Reliable region predictions for Automated Valuation Models

Transcription

1 Reliable region predictions for Automated Valuation Models Tony Bellotti, Department of Mathematics, Imperial College London Royal Holloway, University of London 29 April 2016

2 Outline Automated valuation models (AVMs) Prediction intervals / region predictions Conformal predictors Empirical studies: London House Prices Conclusions

3 Automated Valuation Models (AVM) Provide valuations for individual properties based on statistical models. Necessary to determine collateral on a mortgage. Traditionally, valuations completed by trained surveyors. So why use AVMs? Cheaper. Faster. Objective. Transparent. Reliable and accurate (we hope). Cheaper and faster means that valuations can be dynamically updated during the lifetime of a mortgage. This gives potential benefits for risk management and also for customers who may like to re-mortgage.

4 How to build an AVM AVM is usually a segmented k-nearest Neighbours (knn) based on a rich data set, including variables such as:- Property characteristics (eg size, number of rooms, garden, view from balcony); Local environment (eg schools, transport); Historic prices and economic conditions.

5 Prediction intervals / region prediction for AVM AVMs will produce point estimate for valuations. However, prediction intervals, or region predictions, are useful since They allow us to determine the overall efficiency of AVM. Allow conservatism in estimating collateral. Indicate stability of prices, at an individual property level. - Immediately, this suggests when to send a human surveyor. Determine property segments for which AVM is not working so well - Longer term, this suggests where to improve the AVM.

6 Reliability of region prediction Important characteristic of prediction interval is Reliability: that the probability that the true value is within the prediction interval is well-calibrated. Typically, set a confidence level 1 δ, for a region prediction R for an observation x, expect P(y R x) 1 δ. Conformal Predictors are an approach that only require data to be identically and independently distributed (IID) and their region predictions are provably reliable. No free lunch: reliability is gained at cost of efficiency of prediction: ie size of region.

7 Conformal predictor: NCM Suppose we have a sequence of observations z 1,, z n where each z i = x i, y i and given new example x n+1, we want to predict y for y i+1. Region prediction: predict a region R which we would like to contain y i+1. A nonconformity n-measure (NCM) is a measurable function A n which assigns a real number to the n + 1 th observations such that for any permutation π of 1,, n, A n z 1,, z n, z n+1 = A n z π 1,, z π n, z n+1 Intuitively, the non-conformity measure should tell us how strange the observation is compared to the others in the sequence. In practice, NCMs are constructed based on an underlying predictive algorithm.

8 Conformal Predictor: Region prediction For a given confidence level 1 δ, construct a region prediction as the set i: α i α n+1 R = y: > δ n + 1 where α i = A n z 1,, z i 1, z i+1,, z n+1, z i, z n+1 = x n+1, y Theorem (Vovk, Gammerman, Saunders 1999): Assuming only that data is exchangeable, P y n+1 R 1 δ Therefore, Reliability is guaranteed. Measure Efficiency as size of region; ie a region R = l, u has size u l.

9 NCM for regression A regression algorithm will output a point estimate y for example x n+1. If y is the true label, then a natural measure of nonconformity is the difference between y and y: A n z 1,, z n, x n+1, y = y y 2 However, it may make sense to scale the strangeness by the variance in the estimate y. Let us suppose this can be estimated by s 2. Then y y 2 A n z 1,, z n, x n+1, y = s 2 + r may be a better proposal for a non-conformity measure. This is possible with knn for regresson, where s 2 is sample variance from the k nearest neighbours.

10 NCM for regression The parameter r controls how much the NCM is affected by the variance in the estimate. Then we can show that the region prediction R = y γ s 2 + r, y + γ s 2 + r where γ is the δ -quantile of the distribution of NCMs based on a calibration data set of observations (based on Papadopoulos, Vovk and Gammerman 2011).

11 Empirical studies Building a good AVM is a complex task. European AVM Alliance is a collection of companies specializing in this area. Focus of this study is to demonstrate the effectiveness of conformal predictors to produce reliable predictions (efficiency will not be so good!). Empirical results based on a large data set of London House Prices.

12 London House Prices London house prices from January 2013 to March Data downloaded from Kaggle, but originally sourced from Land Registry. Link to UK deprivation data. Training and Calibration data: December 2013 (29,855 observations) Test data: month March 2014 (24,150 observations) Delay between train and test to simulate lag in publication of sale prices by UK Land Registry. Estimate price inflation using regression over previous year: 0% to 4.3% increase in 3 months.

13 House Price Data Model outcome variable: log(price). Predictor variables: Location (grid reference). Property type: D for Detached, S for Semi-Detached, T for Terraced, F for Flats/Maisonettes. Freehold / Leasehold. New build? Y/N. Distance from centre of London. Distance to nearest station. Type of station (Tube / Overground). Distance to next nearest station and Type. Deprivation score.

14 Time dependency of house prices Conformal predictors require data to be exchangeable. This is not the case for this problem when prices are changing with time. Overcome this by assuming there is an underlying latent price, adjusted by a systematic factor which is dependent on time. That is, price at time t is y t = y 1 + r(t) where y is latent price such that examples x, y and r(t) is price rise by time t, from time 0 ( r 0 = 0). are exchangeable, This model assumes that price change is only a function of systematic effects such as the economy and market changes. This is a reasonable assumption, at least for short periods of time.

15 Time dependency of house prices Then log y t = log y + log 1 + r(t) Since x, y are exchangeable, the reliability results for CP hold. Hence, we can compute a region R = l, u such that P log y R 1 δ For house price forecasting, r(t) will not be known with certainty, but expert judgement would be available to construct a subjective confidence interval: P r(t) a, b = p Then it follows that P log y t l + log 1 + a, u + log 1 + b P log y R P r(t) a, b 1 δ p

16 Results: Choice of r by minimizing Efficiency This is the result of using different values of r, calibrated on validation data set (using δ = 0.1). Optimal result is achieved when r = For comparison sample variance of y In the validation data set is 0.37.

17 Results: Reliability and Efficiency Accuracy Mean efficiency

18 Results: Distribution of price ranges For δ = 0.1, distribution of size of price regions as ratio of price: exp u exp l /exp(y) Min=0.0981, 1 st Q =1.01, Median=1.22, Mean=1.27, 3 rd Q=1.45, Max=11.5

19 Drivers of Price Uncertainty Linear regression of outcome: region size (u l) Predictor Coefficient Estimate Predictor Coefficient Estimate Log(price) *** Log(distance from centre) *** Property type: Log(distance to station) *** Flat *** Log(distance to next stn) ** Semi-detached *** Nearest station: Terrace *** Underground *** Detached + 0 Overground + 0 Leasehold *** Both New build *** Deprivation score 0 Key: + excluded category; *** ** 0.01 sig.level

20 Conclusions Conformal Predictor can be constructed based on an existing welldesigned AVM. Control for time trend, assuming systematic effect and assuming IID on latent price. Allows for reliable prediction intervals of property prices:- Strict control of predictive accuracy, whilst efficiency is optimized. Efficiency of prediction intervals can be used to determine when to supplement AVM decision with valuation by human surveyor. Association of efficiency with property segments suggests areas of further improvement in AVM development.