THE CPI FOR RENTS: A REVISIONIST HISTORY. Theodore M. Crone Leonard I. Nakamura Federal Reserve Bank of Philadelphia

Transcription

1 THE CPI FOR RENTS: A REVISIONIST HISTORY Theodore M. Crone Leonard I. Nakamura Federal Reserve Bank of Philadelphia Richard Voith Econsult Corporation April 2003 The views expressed here are those of the authors and do not necessarily reflect those of the Federal Reserve Bank of Philadelphia or of the Federal Reserve System. We would like to thank David Genesove, Jack Triplett, and members of the Federal Reserve System Committee on Applied Microeconomics for valuable comments. David Genesove kindly supplied the CPI microdata set we have used extensively.

2 ABSTRACT MEASURING AMERICAN RENTS: A REVISIONIST HISTORY Until the end of 1977, the method used to measure changes in rent of primary residence in the U.S. consumer price index (CPI) tended to omit price changes when units changed tenants or were temporarily vacant. Since such units typically had more rapid increases in rents than average units, omitting them biased inflation estimates downward. Beginning in 1978, the Bureau of Labor Statistics (BLS) implemented a series of methodological changes that reduced this bias. We conclude that from 1940 to 1977 the CPI inflation rate for rent may have been about overstated by 1.5 percent annually in US data. Correspondence to: Theodore M. Crone, Research Department, Federal Reserve Bank of Philadelphia, 10 Independence Mall, Philadelphia, PA 19106, (office), (fax) ted.crone@phil.frb.org ( ). Leonard I. Nakamura, Research Department, Federal Reserve Bank of Philadelphia 10 Independence Mall, Philadelphia, PA 19106, (office), (fax), leonard.nakamura@phil.frb.org ( ). Richard P.Voith, Senior Vice President and Principal, Econsult Corporation, 3600 Market Street, Suite 560, Philadelphia, PA 19104, (office), (fax), voith@econsult.com ( ) 1

3 MEASURING AMERICAN RENTS: A REVISIONIST HISTORY I. Introduction and overview Most studies of price mismeasurement have been devoted to upward bias in inflation measures (Boskin et al, 1996, Price Statistics Review Committee, 1961). This paper discusses a case of downward bias in inflation measurement in an important part of the economy, the case of tenant rents. As this measure for rents is used to deflate nominal housing services, real growth in consumption of housing services during the period from 1940 to 1977 may have been overstated by 1.5 percent annually in U.S. data. Rental units have two characteristics that make them unusual compared to many other goods and services: they are highly differentiated and their prices typically adjust only once a year. Because they are highly differentiated, it is difficult to compare different units; comparing the same unit over time is also problematic because of the effects of age on the unit s quality. Because rents usually don t change more than once a year, how often to sample a given unit each year becomes an important issue. Before 1978 the data used to estimate rental inflation in the U.S. Consumer Price Index (CPI) apparently suffered from two forms of downward bias: aging bias and nonresponse bias. Aging bias occurs when the quality of the average rental unit deteriorates over time because of inadequate maintenance. If the rental price of a unit remains constant and its quality deteriorates, its quality-adjusted rent has risen. Therefore, rental inflation data unadjusted for aging bias may be downwardly biased. Nonresponse bias, the more important of the two biases and the focus of this paper, has its source in rental turnover. When a tenant stops occupying a rental unit being surveyed, its rent may fail to be recorded either because the unit is vacant or because the new tenant is not contacted or does not respond. Since changes in tenancy normally coincide with rental price increases, ignoring nonrespondents may result in a large downward bias. Only the vacancy part of nonresponse bias has been explicitly studied by the U.S. Bureau of Labor Statistics (BLS), and 2

4 the impact of vacancy nonresponse bias and the imputation to correct this bias has not been discussed in either Moulton s review of rental inflation or Stewart and Reed s current methodology research series. 1 We also point out that since 1994, the formula by which rental data are aggregated into the rental index implies a three-month lag in the reporting of rental inflation. Repeated investigations have suggested that prior to 1978, the CPI rental index was downwardly biased (Humes and Schiro, 1948, 1949; Weston, 1972; and Ozanne, 1981). Between 1940 and 1977, a period during which the methodology underlying the index was most vulnerable to nonresponse bias and was uncorrected for aging bias, the CPI for rent rose 2.8 percent annually (Table 1). Bureau of Census measures of rent, reported in the decennial Census of Housing and the biennial American Housing Survey, show that median gross rent rose 5.5 percent annually percentage points faster than the CPI for rent. Taking the CPI data at face value, this implies that the quality of the median rental unit increased 2.7 percent a year during this period. By comparison, from 1930 to 1940 and from 1985 to 1997, median gross rents rose less than half a percentage point faster than the CPI rent index, implying a substantially lower increase in quality. 2 We believe this discrepancy is an anomaly that is explained in part by the downward bias in the CPI rental inflation rate due to nonresponse. Section II of this paper models nonresponse bias in the rental CPI so that we can quantify the impact changes in BLS procedures in the measurement of rent. We then parameterize our model and test it using the BLS microdata for Section III presents two new alternative estimates of the rental price index and some additional data on prices and output to suggest that these new estimates are not unreasonable. 1 Stewart and Reed suggest that the only adjustment needed to pre-1978 data is an adjustment for aging bias. We believe that an adjustment is needed for the nonresponse bias as well. 2 Prior to 1940, the BLS directly interviewed landlords rather than tenants, and it believes the problem of nonresponse bias was not a major one. 3

5 II. The Nature of Nonresponse Bias All sample surveys suffer from nonresponse, i.e., incomplete returns from some part of the targeted sample. Response bias for rents biases rents downward because rents in the U.S. typically are increased once a year, and such increases are often associated with tenants moving, so rent increases and tenant turnover are correlated. Moreover, turnover is a frequent occurrence. When the tenant moves, the contact between price inspector and tenant may be broken, and the rental increase may then go unrecorded. As a consequence, for a substantial period of time, from 1942 to 1977, a large proportion of rental increases on the order of one-third failed to be recorded, and measured rents were biased downwards. Between 1952 and 1994, the BLS largely corrected the biases in the CPI in five steps. However, to our knowledge, the extent of this problem has never been investigated. We estimate the effect on the bias of these changes by the BLS and adjust the historical rental inflation for the change in methodology. The five steps included: (1) a reduction in the frequency of collection of prices from quarterly to semiannually in 1953; (2) a major change in sampling procedures and methodology in January 1978 that resulted in a significant reduction of the number of nonrespondents but introduced a recall bias in the estimate; (3) an adjustment to the rental component of the CPI in January 1985 that corrected vacancy-related nonresponse bias and had the effect of eliminating much of recall bias; (4) an aging-bias adjustment in January 1988, based on Randolph s (1988a and b) estimates; (5) elimination in January 1994 of the recall formula which had introduced recall bias in II.1 Price measurement for annually adjusted prices: theory Rents in the United States are usually, but by no means always, adjusted annually. More frequent adjustment may occur: the lease contract may be for less than a year, there may be no 4

6 lease contract, or the lease contract may provide for rental price changes at an interval less than a year. But, as will appear from our data, most rent increases occur at roughly annual intervals. This fact influences both how BLS measures rents and the biases that appear in rental price collection. How often to obtain observations. First we consider the rate at which a rental price inspector obtains price observations from a given unit. Suppose prices for units to be sampled are known to increase annually, but exactly when any individual rent will increase is unknown. To choose the interval over which to sample units, we face a tradeoff between timeliness and the proportion of observations that yield a price increase. For example, if we price each unit in our sample every month, only one-twelfth of our price observations will show a price increase, but we will know that the price increases we do observe occurred in the past month. On the other hand, if we price each unit every 12 months, all of our price observations will show a price increase, but the price increase for any individual product may have occurred at any time in the past year and the recorded rate of inflation will be the average of increases over the past year. We could ask respondents to tell us exactly when the price increase occurred, and thereby obtain a more accurate picture, except that respondents are generally not very good at doing so, and as we shall see, efforts in this direction have been abandoned in practice with rents. The BLS s current practice for rents is to sample every six months, and to record the monthly rental price increase as the sixth root of the average price increase observed. Response bias due to vacancies. An issue that is special to rents is that rental price increases are often associated with the renter moving from the rental unit. While the unit may be reoccupied immediately, more often a period of vacancy ensues. Vacancy matters in this historical review, because the BLS has been hesitant to rely on rental asking prices, and treats vacant units as lacking a price and therefore requiring an imputation. This is in contrast to the BLS practice for products other than rents, where transactions are frequent enough so that BLS feels confident in relying on the asking price, for example, the marked or posted price of a retail item. 5

7 A simple model of rent collection with vacancies. In order to understand the quantitative impacts of the biases due to turnover and vacancy, and the impact of the frequency of rent sampling, it is useful to set up a simple model that analyzes the tradeoffs between data collection, precision, and bias. Rents are assumed to be determined by an annual lease which specifies fixed level nominal monthly rent payments for the lease duration. Measurement error. We first consider the issue of measurement error associated with the sampling period. The log rental price p it for unit i at month t follows a transition path such that p it = p it-1 with probability 1 - and p it = x it + p it-1 with probability, where x it = * t + e it > 0, θ = 1/12, and e it has zero mean and standard deviation.. Here * t is the underlying annual rate of inflation, so that Ep it = θ* t + p it-1. We sample the log rent p it and wish to form estimates of * t = Ep it - p it-1. If we sample m rental units every month, and there are no missing observations, we obtain x it = p it - p it-1 or 0 according to a binomial distribution with parameters m and. Our monthly inflation measure is m pit pit 1 which has mean θ* t and variance * 2 θ(1- )/m +. 2 /m, so variance is inversely m i= 1 proportional to m. Alternatively, one can sample each unit every n months and obtain p it and p it-n, sampling m units a month. From each unit, we obtain a price increase x it-j (j ε [0.. n-1]) with probability n, for n < 1/, and thus the number of price increases recorded follows a binomial distribution n 1 θπ t j with parameters m and n. Our observations /p it -p it-n /m have mean and variance n 2 n 1 t j j= 0 2 π θ(1 nθ) θσ +, so variance is inversely proportional to mn. By sampling each unit n mn mn less frequently and sampling more units, we reduce the variance of the error term, at the cost of observing values of n 1 j= 0 π t j rather than * t so our inflation measure is, on average, out of date by n/2 periods. This procedure, with n = 6, involves taking the sixth root of the observations taken at six month intervals as the monthly rate of change, is the procedure that BLS has followed j= 0 6

8 since 1994 for rents. Response bias. Now let us turn to the problem of response bias, the possibility of missing observations either because of turnover or vacancy. If rents are rising, as has been the case, this results in downward bias. We continue to assume that the rent increases (x it >0) with probability θ. We add the assumption that with probability ρ the tenant then leaves the unit. For a unit whose tenant has left, in any succeeding month, with probability 1-α a new tenant occupies the unit with a year-long lease at a new fixed price, and with probability α the unit remains vacant. A further issue is that the rate of inflation at rental units from which tenants depart is, on average, higher than the units of continuing tenants where a price increase has occurred. This issue is discussed in Genesove. Let us define the rent increase for continuing tenants as * Ct. Where the tenant moves, the rent increase is larger by some fraction b, (1+b)* Ct. Then the annual rate of increase for complete data would be π t = (1+ρb)π Ct. We shall assume that vacancies and reoccupied units have the same rate of increase. Now response bias is due to the fact that when the tenant moves, the price inspector is less likely to record any price for the unit, either because the unit stands vacant or because of loss of contact with the tenant. Let us call q M the probability that a unit where the tenant has moved will have a price recorded, and q C the probability that a unit where the tenant hasn t moved will be recorded, with q M <q C. Then we can calculate the annualized measured rate of inflation (π m t ) and the complete data rate of inflation (π t ) as follows (see Appendix One for derivation): qm 1 ρ(1 ( (1 + b)) m qc πt = πct qm 1 nθρ(1 ) q qm (1 + ρb)(1 nθρ(1 )) qc π t = (1 + ρb) πct = πt qm 1 ρ(1 (1 + b)) q C C m If q M /q C is equal to 1, then the last fraction simply becomes 1. Response bias is due to 7

9 obtaining observations from units where tenants have moved less often than units tenants continue to occupy. Turnovers omitted: If rental prices of units that are vacated, whether reoccupied or not, are not collected, then we set q M /q C =0 and the mean measured rental inflation ( π ) can be calculated to be: m (1 ρ) π Ct π t =. 1 nρθ If rents are collected annually, nθ =1, the rate of sampling would equal the rate at which prices are changed, and the measured rate of inflation would equal the inflation rate of the continuing tenants. Response bias in that case is reduced to the fact that continuing tenants experience lower rates of inflation than new tenants. But with nθ < 1, as the case is with sampling every six months, measured inflation gives too much weight to tenants who are in the portion of the annual cycle in which the rent does not increase. The complete data rate of inflation (π t ) is equal to: (1 + ρb)(1 nθρ ) m πt = πt. 1 ρ If n=.5,, =.4, and b=.3, (parameters that in round numbers approximate the m parameterization we shall adopt), then π = 0.75π and π = 1.49π m. t Ct t t m t Modelling vacancy nonresponse. For units occupied in a given month, n months later a price increase will have occurred on average at nθunits; at these units nθ(1-ρ) of the old tenants remain, θρ(n-α(1- n )/(1-)) new tenants have moved in, and,θα(1- n )/(1-) units will have become vacant. The ratio of turnovers (vacancies plus new tenants) to units that experience a price change is fixed at,. The ratio of vacant units among those that experienced a price change falls with n:,(1- n )/(1-)n, since more of them are being occupied as time passes. If, for a unit whose original tenant has left, the subsequent rental price is collected when n qm α(1 α ) the apartment is reoccupied but not when the unit remains vacant, then we set = 1 q n(1 α) C 8

10 and the mean measured rental inflation ( π ) can be calculated to be: m t n α(1 α ) (1 + ρb ρ (1 + b)) π Ct m n(1 α) π t = n α(1 α ) 1 ρθ 1 α and the complete data rate of inflation (π t ) is equal to: n α(1 α ) 1 ρθ (1 ) 1 m π t = + ρb πct = α π n t α(1 α ) 1+ b 1 ρ n(1 α) 1+ ρb For collection every six months, the practice for BLS rent collection, we use n=6. The vacancy rate is,θ(1-6 )/(1-). m If n=.5,, =.4, α= 2/3, and b=.3, then π = 1.024π and π = 1.094π m. t Ct t t II.2 Changes in BLS rent collection methodology from 1942 to 1977: overview Nonresponse was less of a problem in the BLS rental survey prior to 1942 as until then price inspectors obtained their data from the files of real estate agents and large-property owners. This system had the advantage of avoiding a dependence on tenant response. The price inspector could directly compare current rents with past rents, even if the tenant had changed. If a unit was vacant, a comparable unit could often be found from the books. Obtaining rents from tenants, Changes in BLS methodology starting in 1942 increased the potential for nonresponse bias in the rental-price series. Price inspectors were instructed to obtain rents directly from tenants rather than from the records of landlords or real estate managers. 3 Following an initial interview to elicit cooperation and gather data about the unit, the tenant was mailed a rent questionnaire quarterly. 3 An important impetus for this change was the implementation of wartime rent controls. It was feared that rental increases that evaded or violated rent control laws might not be accurately reported by real estate agents or landlords. By gathering data on the terms of the rental agreement, the price inspector would be able to detect changes in the terms, such as requiring the tenant to pay for utilities that had previously been included in the rent. 9

11 Semiannual rent collection, In 1953, without any fanfare, we infer that the rate of rental collection was changed from quarterly to semiannually, but we have only indirect evidence of the change. Collection of mortgage rate and other price information on the costs of owneroccupied housing was instituted in the 1953 CPI revision, so we know that this was a period in which major changes did occur to the housing index (Lamale, 1956). And when the 1964 revision was announced, it included information that implied that rent collection had at some previous date become semiannual. Personal visits and telephone surveys, Under the mail method used between 1942 to 1963, one study of the responses from March to September 1947, when collection was still quarterly, found that approximately 50 percent of the initial mail questionnaires were completed without prompting by the tenant, and an additional 20 percent were returned on follow-up, but the nonresponse rate was 30 percent -- 5 percent were returned unable to locate and 25 percent were not returned (Humes and Schiro, 1949). In a mail system, when a tenant moved, the mail questionnaire would have been addressed to a previous occupant, and would be forwarded or returned to the sender. The BLS rental price inspector would have to ascertain who the new occupant was and solicit his or her cooperation with a new interview, and start over again. Such a process would almost invariably miss the rent increases associated with new tenants. The method of survey by mail was deemed unsatisfactory because of the large number of nonrespondents, and in the 1964 revision to the CPI, the BLS instituted a system of using parttime agents to collect rental data by personal visit or telephone. Forty thousand units were surveyed semiannually to obtain a total of 80,000 prices annually, or an average of 6,667 per month. No substitution was permitted for units whose prices were not obtained. Solicitation by telephone would have the same problem of missing new tenants; instead of receiving the mail back, the price inspector would find that the number was no longer in service or had been changed. Again the price inspector would have to begin over with a new solicitation. Personal visits might have a greater likelihood of response from a new tenant, but the new tenant even if successfully contacted would be less likely to cooperate than a tenant who has already agreed to 10

12 participate. The institution of personal visits does not appear to have materially reduced response bias; overlap data showed that the new procedures introduced in 1964 did not raise the measured rate of inflation but actually reduced it. 4 Reducing response bias and introducing recall bias, Beginning in 1978, a new survey method was instituted. The number of rental units surveyed was reduced substantially to 18,000. The intention was to ensure that the sampling of rental units was as thorough as possible and, in particular, to capture rent increases when the tenant moved. Data were also obtained on the length of occupancy of new tenants. Price inspectors could choose to interview the landlord or manager instead of the tenant and typically did so. Price inspectors were to reinterview the tenant, manager, or owner of the unit every six months. In addition, a new method was instituted for using the rental data obtained from the interview. First, respondents were asked the level of last month s rent as well as the current month s rent. Then two comparisons were made: the six-month price increase using the previous interview and the one-month price increase. The rental index was computed using both the onemonth change and the six-month change, weighted so as to minimize fluctuations. Defining I(t) as the level of the index at month t, and Rt,t-k as the change in rent from k months ago, the rental formula was: I(t) =.65 Rt,t-1 I(t-1) +.35 Rt,t-6 I(t-6). (1) This formula, which we call the recall formula, is a nice way to permit the CPI measure to reflect current inflation fully and immediately, while minimizing noise. Unfortunately, use of the formula introduced recall bias, because respondents often failed to remember increases in rent that had occurred in the previous month. The consequence was that the average change from the previous month was less than one-sixth the average change from six months prior. 4 From January to June 1964 the data were collected using both the old and the new survey methods. During this period, there was very little difference between the two series and by the end the revised index for rent was (on a basis of = 100) compared with the unrevised index of 107.9, so the revised index rose more slowly. The June 1963 rent index was 106.8, so the rental CPI at this time was rising at an annual rate of about 1 percent. 11

13 Vacancy bias correction, The BLS implemented a correction for vacancy bias in 1985, which involved the estimation and imputation of expected rents for vacant units. This procedure used rents of newly occupied units in the same location to impute rents for vacant units. Aging bias correction, 1988.The changes to the BLS methodology for collecting rents in 1978 and 1985 did not address the issue of aging bias in the estimate of rental inflation. BLS economists have long worried about aging bias, but it was not until the late 1980s that they were satisfied that they could estimate it accurately. Aging bias refers to the underestimation of rental increases because of the systematic deterioration in the quality of housing services provided by a rental unit as it ages. Historically, the BLS has adjusted the change in rent for observed quality changes, such as the addition of a room. But prior to 1988 the agency did not correct for the systematic deterioration in quality associated with aging. If a unit deteriorates systematically with age, a constant rent over the six-month period implies an increase in rent on a qualityadjusted basis. Recall bias correction, 1994.The recall bias problem introduced in 1978 was solved in 1994 when the BLS discontinued the use of reported one-month rent increases in estimating rental inflation (Armknecht, et al.). At this time, the rent formula was changed so that the monthly rate of rental inflation was calculated as the sixth root of the average 6-month inflation rate. The new formula, while free of downward bias, results in roughly a three-month lag in the reporting of changes in the rental inflation rate. II.3 Estimating the impacts of changes in procedures: parameterization In this section, we argue that the changes in the BLS rent collection methodology have rasied CPI measures of rental inflation substantially. We then discuss how to parameterize the model we have developed to implement a more methodologically consistent rental inflation series. We then discuss the BLS adjustments in 1985 and 1988 in more detail to complete the parameterization to include recall and aging bias. Finally, we use a BLS microdata set from

14 to 1992 as a test bed for our model estimates. The 1978 overlap data. When a major change is instituted in CPI methodology, the BLS sometimes collects data for six months under the old methodology as well as under the revised methodology. In the case of the 1978 revision, in addition to the numerous procedural innovations, BLS introduced a CPI for all urban consumers (the CPI-U) in addition to the revised CPI for urban wage earners and clerical workers (CPI-W). During the overlap period, from January to June 1978, the BLS published statistics for the old CPI as well as the two new ones.. This had the primary benefit of giving contracts that are indexed to the old data more time for changeover. But it also permitted analysis of the direct impacts of the change. The numerical impact of the 1978 revision on the aggregate CPI as revealed in the overlap data was discussed in Layng, 1978, but rental inflation were not commented on specifically. Table 2 presents the 1978 overlap statistics for tenant rents. During the overlap period, the CPI for rents rose roughly ten percent faster under the revised methods than under the old. The old CPI-W for rents did not accelerate, but the new CPI-W for rents did, and the new CPI-U for rents rose even a bit more. BLS estimates of bias. When the Bureau of Labor Statistics introduced corrections for aging bias in 1988 (0.36 percent annually, Lane, Randolph, and Berenson, 1988) and recall bias in 1994 (9 percent of the inflation rate, Armknecht, Moulton, and Stewart, 1995) it discussed the size of the impact of the corrections. It also made an estimate, although somewhat vaguer, when it introduced the vacancy bias correction in 1985 (less than 0.1 percent per month, or as much as percent of the 1983 or 1984 rental inflation rate, CPI Detailed Report, January 1985, p. 171), and the statistics upon which this correction was made had been made available in Rivers and Sommers, If we add the corrections together with the overlap increase, it would appear that the rate of rental inflation was adjusted upwards by over one-third. Parameterizing the model. According to the model we developed, the CPI for rents prior to 1978 suffered from a response bias whenever tenant turnover occurred. The measured rate of inflation followed equation 2.1 plus aging bias. After 1978, the CPI for rents suffered from 13

15 vacancy bias, equation 2.2, plus recall bias and aging bias. In order to examine these relationships quantitatively, we need to use sources that help us to estimate the turnover rate, the vacancy rate, and the higher rate of inflation experienced by units that turnover. The parameters which we need are ρ (the annual turnover rate), 1-α (the reoccupation hazard), b (the higher inflation rate at rental units that turn over), and q M /q C (the relative sampling rate of units where tenants move). We assume that units can turn over only at one-year intervals, and that rent increases only occur at this time. Turnover rate, ρ. The annual turnover rate is the number of persons who move out of rental units in a given year. As far as we know, no published statistics provide this number. Instead, data exists in the American Housing Survey and in Censuses of Housing on recent movers into units. These recent movers into units differ from those who move out of units because they include those who have moved into new and thus previously unoccupied rental housing, and exclude vacancies, where tenants moved out but no tenant has yet moved back in. Annual turnover can be obtained by subtracting new rental units from recent movers, and adding in rental vacancies of less than one year. The 1970 Census of Housing provides data on renters who moved into their units between the beginning of 1969 and March Beginning in 1973, the American Housing Survey (AHS) 6 provides data on renters who moved into their units in the past 12 months. To estimate the number of renters who moved into new units, we use the number of multifamily (2 or more) units constructed during a given year (some new single family units are rented and some multifamily units are sold for owner occupation, but over the period these two have roughly canceled out 7.) We use total rental vacancies to estimate vacancies of 5 The Census period is the previous year and the first three months of the current year. That means that the first quarter is counted twice, a period in which turnover is somewhat lower than during the rest of the year. According to our BLS microdata, 21.6 percent of movers move in during the first quarter of the year; accordingly, we divided this figure by to estimate annual movers. 6 The AHS was known as the Annual Housing Survey from 1973 to 1981, prior to the survey becoming biennial and being renamed the American Housing Survey. We use the new title throughout. 7 According to Components of inventory change, , Pub 9/96, the new rental units added between 1980 and 1993 were.95 times the number of multifamily units completed in the same period. Similar figures apply for 1970 to

16 less than one year. 8 This estimation is carried out in Table 3A. For data available from 1970 to 1993, this turnover rate, ρ, has averaged just under 38.7 percent, varying from 33.6 percent to 41.0 percent. Vacancy rates, α. To infer the reoccupation hazard rate, 1-α, we turn to vacancy statistics. Our aim is to match data for the first six months that a unit is vacant. Statistics on vacancy by length of vacancy can be obtained from either the AHS or the Housing Vacancy Survey (HVS), which is conducted as part of the current population survey (CPS) used in employment estimation. The AHS has the drawback that it is conducted from August to November, while the HVS is conducted year around and is thus unlikely to suffer a strong seasonal bias. The AHS is conducted once every two years; the HVS every month. Samples are roughly the same; the HVS has about 60 thousand units, the AHS about 54 thousand. Thus the effective size of the HVS is much larger than the AHS. The HVS provides information on the proportion of rental vacancies by length of vacancy, units vacant less than 6 months generally account for 80 percent of such units for which the length of vacancy is known. Units vacant less than 6 months were 5 percent from 1970 to 1999, Table 3B. In addition, there are units that are rented but not yet occupied. These appear to be about 1 percent of all units. Assuming that eighty percent of these units have been vacant less than six months, we have total vacancies in a six-month period of 5.8 %. Using the model, the one month vacancy rate is then,/, the total vacancy rate is then, /(1-)) and the 6 month vacancy rate is,(1-6 ) /(1-). If we assume that,=.4, θ = 1/12, and =.646, then the one month vacancy rate is 2.13 %, and the six month vacancy is 5.58 percent. This matches the data for tolerably well (the model predictions are 1 to 2 months is 1.38 %, 2 to 4 is 1.46 % and 4 to 6 is.61 %). However, this model does not match the data well beyond 6 months. The reoccupation rate tends to fall over time; indeed, the vacancy rate in the simple model falls too steeply to match the data from 2 to 4 months to 4 to 6 months, 8 The precise concept we are trying to approximate is rental vacancies of less than one year plus rented units where the tenant has not yet moved in. This is over the long run about equal to total rental vacancies, as rented units where the tenant has not yet moved in and vacancies over one year are both close to one percent of occupied units. 15

17 so it should be kept in mind that α has been calibrated to fit the average three-month and sixmonth vacancy rates. In experiments with the model where n changes, the model will be less accurate when n=12. Faster rental inflation rate for units where tenants move, b. According to the data in Rivers and Sommers (1983) which is described in greater detail below, new tenants during the period October 1979 to March 1981 conditional on their rents increasing -- recorded a 6 month inflation rate averaging percent, compared to continued tenants whose average was 8.94 percent (Table 4). Thus new tenants experienced a 35 percent higher rate of inflation, when their rents increased. The sampling rate of new tenants, q M /q C. The simplest assumption is that no new tenants were sampled, which may have been the case under the early years of the mail survey. However, there may have been a higher rate of sampling of new tenants, certainly when personal visits became more frequent in The model parameters developed thus far would suggest that under the assumption that recall bias was 9 percent, the 1978 overlap data should have shown an increase of 25 percent in the inflation rate if q M /q C = 0. We can use the actual overlap increase of 10 percent to estimate that q M /q C = 0.2. We decided to carry out two estimates, one with q M /q C = 0, which we call the model estimate, and another with q M /q C = 0.2, which we call the conservative estimate. Total impact. Using the parameters b =.35, ρ =.387, α =.646, θ = 1/12, and n = 6, our simple model predicts that the complete data inflation rate is 49 percent greater than the measured inflation rate when only continuing tenants are included in the data (q M /q C =0). Without including the recall formula, the true inflation rate is 8.6 percent greater than the measured inflation rate when vacant units are not included in the data. If q M /q C =.2 instead of 0, then the complete data inflation rate is 33 percent greater than measured. This estimate (using q M /q C =.2) is the parametrization which accords with the ten percent increase in inflation in the overlap data for January to July The impact of different parameters on the base scenario is depicted in Table 5. 16

18 Contractual relationships. The annual lease is the predominant form for rentals. Data from the U.S. Census Bureau s Property Owners and Managers Survey in 1995 showed that 44.4 percent of all units had annual leases, 4.0 percent had leases longer than one year, 36.1 percent had leases less than one year, and 15.5 percent had no leases. 9 These facts suggest that while the annual lease is the modal contract under which rental units are occupied, it is by no means universal. Thus the simple model which underlies our work is a possibly unreliable approximation. II.4 Adjustments to correct for the vacancy-related nonresponse and recall bias (1985) A study of the 1979 to 1981 rental data carried out in 1983 by two BLS economists, Joseph Rivers and John Sommers, revealed that the BLS rental price estimates still suffered from two biases: recall bias and the vacancy component of the nonresponse bias. Recall bias was a systematic tendency for one-month price changes to be less than the sixth root of six-month changes; six-month changes had the advantage of being based on previous records and not the recall of the tenant or landlord. As we discuss below, recall bias downwardly biased rental price increases by roughly 9 percent. Vacancies present a special problem in collecting rental data because a unit that is vacant at the time of a scheduled interview does not have a transacted rent to compare with the previous time or the next time it is collected, although it may have an asking price. Although the 1978 procedures had reduced total nonresponses from possibly 30 percent to 13.6 percent, nonresponses due to vacancies were little changed and now accounted for half of all nonresponses. If rental increases for units that become vacant are higher than the average rental increases, there is a negative nonresponse bias associated with vacancy. To quote Rivers and Sommers: There are more nonreponses in the not at home and vacant categories than reported 9 Single family and multifamily units, excluding data not reported or for rent free units. The survey can be found at 17

19 one month rent changes. This is potentially a serious problem. As was shown in previous parts of this paper, rents tend to be increased with higher probability for units that have a change in occupants as compared to those units with no change in occupants (81 % compared to 46 % for six month changes). This would imply that treating vacancies the same as all other units when applying the noninterview adjustment may result in underestimating the one month relative. That is, imputing data for nonresponse might solve both the vacancy nonresponse bias and the recall bias problem. Vacancy nonresponse bias. Rivers and Sommers use all the CPI rental price microdata observations collected by price inspectors from April 1979 to March 1981 to get a measure of the bias associated with vacancies. 10 In this period, there were 56,510 interview attempts, from which 48,809 good interviews resulted (86.4 percent). Reasons for noninterviews were vacancies (3,833, or 6.8 percent), no one at home (2,619, or 4.6 percent), refusal (745, or 1.3 percent) and other (504, or 0.9 percent). However, only 45,758 six-month changes were recorded for the 48,809 units with good interviews. Presumably a good interview was conducted 3,051 times at a rental, but no six-month price change was recorded because six months previously no price data had been obtained for that particular unit. Rivers and Sommers divided their good interview sample into continuing tenants (those with six or more months of occupancy, 81.2 percent of the sample) and new tenants (18.8 percent). 11 This breakdown is consistent with a turnover rate of about 40 percent annually and, therefore, suggests that the new survey did succeed in capturing almost all new tenants. By separating responses into those of new tenants (less than six months occupancy) and continuing tenants, Rivers and Sommers showed that new tenants had higher rates of price increase than continuing tenants. As shown in Table 5, 46.4 percent of continuing tenants 10 They used additional microdata back to January 1979 for the purpose of calculating six-month changes. 11 The Rivers and Sommers data divides tenants into those with 5 month or less occupancy and 6 months or more. It may thus underestimate the proportion of new tenants included in the data, as tenants with more than 5 months but less than 6 months occupancy may be in the 6 months or more category. 18

20 experienced rent changes in the previous six months, while 80.6 percent of new tenants experienced rent changes. Moreover, those new tenants who experienced rent changes experienced higher rates of rent increase (12.1 percent) than continuing tenants (8.9 percent). This change in methodology was implemented in January 1985 and probably accounted for a 8.6 percent upward adjustment to rental inflation not counting its impact on recall bias, to which we now turn. Recall bias. According to Rivers and Sommers, 24,182 six-month rent changes were reported between April 1979 and March 1981, but only 2,541 one-month rent changes. The number of reported one-month changes is just 63 percent of the 4,030 expected based on the number of six-month changes. This suggests that a large percentage of one-month changes are not being recalled or reported. 12 What is the quantitative impact of a given recall bias on measured rental inflation? Suppose the true monthly inflation rate is *. The six-month rental inflation rate will be (1+*) 6. If the one-month recall bias is e, then the reported one-month change will be (* - e). The formula given in equation (1) to compute the rental index can then be written as the following sixth order difference equation: I(t) =.65(1+*-e) I(t-1) +.35 (1+*) 6 I(t-6). (2) If we assume that measured monthly inflation in the steady state equals 1 + * - de where d = the impact on the measured inflation rate of the recall bias e. Then I(t) =(1+*-de) I(t-1) and I(t) = (1+*-de) t I(0). (3) To compute d we substitute and obtain: 12 Interestingly, managers and landlords provide fewer one-month rent changes than do tenants. One possibility is that managers and landlords sometimes consider a rent to change when the tenant moves out and the asking price is raised, while from the price inspector s standpoint, the rent rises only when the unit is reoccupied. 19

21 (1+*-de) t I(0) =.65(1+*-e)(1+*-de) t-1 I(0) +.35 (1+*) 6 (1+*-de) t-6 I(0) (4) Dividing through by (1+*-de) t-6 I(0) and subtracting the right-hand side, we obtain: (1+*-e)/(1+*-de) -.35 [(1+*)/(1+*-de)] 6 = 0 (5) Now, performing the division indicated by the second term on the left-hand side of equation (5): (1+*-e)/(1+*-de) = 1- e (1-d) + error. (6) The error term is actually (π-de)((e (1- d))/(1+* - de). Both * and e are assumed to be much smaller than one (π is the monthly inflation rate and e is its bias) and d is less than one. Therefore, the error is on the order of * times e. Performing the division indicated by the third term on the left-hand side of equation (5): (1+*)/(1+*-de) = 1 + de + error (7) The error is actually (π-de)de/(1 + * - de), and for the reasons mentioned above, the error is on the order of * times e. Ignoring the error and raising the right-hand side of equation (7) to the sixth power, we obtain (1+de) 6 = 1 + 6de + error (8) where the error represents all the exponentiated values of de and is therefore very small. Ignoring the error terms and substituting the right-hand sides of (6) and (8) into (5), we have approximately (1 - e(1-d)) -.35 (1+ 6 de) = 0.65 e(1-d) -.35(6de)=0 or d = (9) Rivers and Sommers note that their methodology for imputing rent changes to vacant units also implies one-month changes to these vacant units, and that these imputations in their 13 A simulation over a six-year period with a =.005 and e =.001, so that the annual inflation rate is about 6 percent, yields d =

22 simulations substantially eliminate recall bias. In their original calculations with the actual data, the one-month relatives were 62 percent to 67 percent of the change in the six-month relative, or e =.38 π to.33 π. When they used their methodology to impute vacancy data, and imputed data for both six month and one month changes, they found that the one-month relatives reflected 86 percent to 99 percent of the change in the six-month relatives (.14π to.01π). On average, this is a reduction of e of about 0.28 π, which multiplied by.2364 implies a reduction of bias of 6.6 percent. The impact of the vacancy imputation methodology on vacancy response bias and recall bias implies a larger total upward adjustment to measured rental inflation, of 19.4 percent under our model estimates. This figure is within the range of the BLS estimate that the vacancy imputation adjustment would raise the inflation rate for rents by less than 0.1 percent a month. From December 1982 to December 1983, the rental rate rose at an annual rate of 4.8 percent, and from December 1983 to December 1984, at 5.8 percent. Thus 0.1 percent a month could represent 20 to 25 percent of the measured inflation rate, depending on the base against it was calculated. Note that that the announcement, which appeared in the January 1985 CPI detailed report, must have been prepared prior to publication and the writers may not have had complete 1984 data available for their estimates. Table 7 assembles the various impacts on measured rental inflation of the changes in BLS methodology under the assumptions and parameters of our model. II.5 The adjustment for aging bias (1988) There are two potential problems in hedonic regression that estimate the effect of physical deterioration on rents. The first is the so-called vintage effect. This effect arises when there are unmeasured quality characteristics other than physical deterioration associated with age but not other measured characteristics of the residence. For example, the more extensive use of insulation in houses built after the 1970s would raise the unmeasured quality of those units. On the other hand, units built prior to World War II and still occupied may represent the highest quality units built in those years based on the assumption that the lower quality units built at that 21

23 time are no longer in use. These so-called vintage effects make it difficult to get an accurate estimate of the effect of physical deterioration on rent. The second problem in estimating the effect of aging on rent is that units of different types (e.g., apartments versus detached houses) may deteriorate at different rates. In his 1988 article William Randolph (1988b) was satisfied that he had solved both of these problems in estimating the effect of systematic physical deterioration on rents. Randolph argues that including a sufficient number of housing and neighborhood characteristics in a hedonic equation would render the remaining vintage effect minimal. He included housing characteristics like the presence of a dishwasher or washer/dryer and neighborhood characteristics like the percent of the population with a college education. He also estimated different aging effects depending on the number of rooms in the unit, whether the unit was detached, and whether it was rent controlled. His resulting estimate of the average effect of aging on rent was -.36 percentage point a year. The BLS has used this estimate of the effect of aging to adjust the rent component of the CPI since This adjustment increased the rental inflation rate at introduction by 9 percent. II.5 Estimating the impacts: simulation with BLS microdata In this section, we perform various simulations based on a BLS microdata set for rents for the period January 1988 to December In this period, BLS was still collecting data from renters about the previous month s rent and the current month s rent and using the recall formula; it imputed missing data for vacancies and other nonresponding units; and it adjusted the data for aging bias. Moreover, the data set includes information that allows us to tell when data has been imputed, and whether the tenant is continuing from the last rent observation or a new occupant. It is thus a very good data set for verifying whether the data BLS actually used conform to our model behavior, since it provides us the data necessary to compute the impact of changes in BLS practices. Data. The BLS microdata set that we use for our simulations has data on each housing 22

24 unit sampled by the BLS. Our data set has, for each unit and collection period, coded information on the length of occupancy (1 to 6 months and more than 6 months); the type of structure; the completeness of the interview or a reason for failure to obtain information; the current month rent either actual or imputed by BLS; and last month s rent, actual or imputed. The data set does not have the weights BLS used to blow up the sample observations to the universe. We conduct three main exercises using this data. The first is to use the data to replicate the BLS rental CPI, not seasonally adjusted, using the published formulas but equally weighting the data. This test gives us information on whether the unweighted data is a reasonable proxy for the data with weights. The second is to use the data to measure the impact of the vacancy imputations and of response by new tenants. The latter test can be done by using responses only from continuing tenants. The third exercise is to examine the impact of the recall formula on the noisiness and timing of the index. Simulation: Are the missing weights a large problem? In the first exercise, we use all the rental data, imputed and actual. The BLS procedure under the recall formula was to calculate two inflation rates. The first is called the six-month relative, the ratio of the weighted sum of the rents for the current period to the weighted sum for the period six months ago, using all units for which data are available. The second is called the one-month relative, a similar ratio of the current rents to the previous month rents. These two relatives are then combined using the recall formula. We can duplicate this except that we do not have the weights. We then add 0.36 percent annually (0.03 monthly) to the inflation rate. We begin in July 1988 (we can only construct 6 month relatives beginning in July 1988) and continue until December For the period from June 1988 to December 1992, our annualized inflation rate is 3.52 percent while the published measure for the same period is 3.50 percent, a difference of less than 1 percent. The two data series behave somewhat differently, with our data showing a distinct tendency for seasonal variation relative to the published nonseasonally adjusted BLS data, as can be seen on Figure 1. 23