Kernel Conditional Quantile Estimation via Reduction Revisited

Transcription

1 Kernel Conditional Quantile Estimation via Reduction Revisited Novi Quadrianto The Australian National University, Australia NICTA, Statistical Machine Learning Program, Australia Joint work with Kristian Kersting, Mark Reid, Tiberio Caetano and Wray Buntine Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page

2 The Problem Normalized Relative Change in Spinal BMD 4 3 mean Normalized Relative Change in Spinal BMD 4 3. quantile. quantile.3 quantile.4 quantile.5 quantile.6 quantile.7 quantile.8 quantile.9 quantile Normalized Age Normalized Age Mean Regression Quantile Regression Mean regression: computing a regression curve corresponding to the mean of a (conditional) distribution. Quantile regression: computing regression curves corresponding to various percentage points of a (conditional) distribution. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page

3 Quantile Regression (Formal) Definition of Quantile Regression Consider a random variable y R and let τ (, ). The conditional quantile q τ (x) for a pair of random variables (x, y) X R is defined as the function q τ : X R for which pointwise q τ (x) is the infimum over q for which Pr(y q τ x) = τ. Applications data mining econometrics social sciences ecology bioinformatics... Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 3

4 Gaussian Processes (Not-so-Formal) Definition of Gaussian Processes It is a generalization of multivariate Gaussian distributions over finite dimensional vectors to infinite dimensionality. Each draw from a Gaussian process is a function. One of Applications: mean regression Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 4

5 Quantile Regression via Reduction Observations : If conditional distribution, p(y x), is known, quantile regression becomes an easy problem. We reduce a hard quantile problem to yet another hard intermediate problem, i.e. distribution modeling of p(y x). However, we can harness Gaussian processes in tackling the distribution modeling. Our Approach: Estimate conditional distribution via Gaussian processes and subsequently slice the distribution at desired quantile levels. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 5

6 Quantile Regression via Reduction Given m observed data points D = {(x i, y i )} m i=, where y i R (the set of outputs) and x i R d (the set of inputs), infer a conditional quantile function q τ (x) from observed data points. The Model: Prior distribution: q GP(m(x), k(x, x )) Likelihood function: y q, x N(q, σn) Predictive distribution (in the form of the standard GP mean regression): for a new input, x q x, X, Y N(µ, σ ) with the moments as follows µ = k T (σ ni + K) Y σ = k(x, x ) k T (σ ni + K) k. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 6

7 Quantile Regression via Reduction The Model: Point Prediction: We need a notion of a loss function, i.e. L τ (ξ) τ Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 7 τ ξ Pinball Loss Function We need to minimize a Bayes risk w.r.t the loss function, i.e. q (opt) τ = argmin E p(y x) [L τ (y q τ )] q τ = argmin{(µ q) [ τ Φ µ,σ (q) ] + σφ µ,σ (q)} q τ

8 Noise Dependent Case (Sad) Reality: in real world problems, the noise rate is dependent on the input variables. Solution: model distribution via heteroscedastic Gaussian processes. Our contribution to heteroscedastic model: joint learning of the free parameters of the latent and observed processes 3. quantile.5 quantile.5 quantile.75 quantile.9 quantile Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 8

9 Performance Guaranteed! Our quantile estimator has bounded regret. Theorem Suppose p = N(µ, σ ) is a predictive distribution at the point x and the true point distribution is p = N(µ, σ ). Then, if KL(p p) ɛ, the regret of the corresponding τ-th quantile estimator q τ satisfies R τ (q) ɛ(τσ + )( Φ (τ) + ). Proof Please refer to the paper. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 9

10 Related Work Linear method (Koenker & Bassett 978): the quantile function is a linear function of inputs, i.e. q τ (x) := x, β(τ), where β(τ) is obtained by minimizing the pinball loss function via linear programming. Kernel method (Takeuchi et al. 6): the dual of regularized pinball loss optimization is minimized via quadratic programming. Reduction method (Langford et al. 6): solving a series of importance weighted binary classification problems. Bayesian method (Yu & Moyeed ): modeling an asymmetric Laplace likelihood and improper uniform prior with MCMC to infer posterior distribution. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page

11 Experiments Toy Data.5. quantile.5 quantile.5 quantile.75 quantile.9 quantile.5. quantile.5 quantile.5 quantile.75 quantile.9 quantile Ground Truth Ours Data: x U(, ) and y = µ(x) + σ(x)ξ with µ(x) = sinc(x), σ(x) =. exp( x), and ξ N(, ). τ QSVM HQGP Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page

12 Experiments Toy Data quantile.5 quantile.5 quantile.75 quantile.9 quantile quantile.5 quantile.5 quantile.75 quantile.9 quantile Ground Truth Ours Data: x U(, ) and y = µ(x) + σ(x)ξ with µ(x) = sin(πx), σ(x) = (. x)/4, and ξ χ (). τ QSVM HQGP Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page

13 Experiments Real Data τ Dataset m Method..5.9 Antigen 97 Linear.97± ±.5.9±.873 QSVM.9± ±.36.8±.8 Reduction.8± ±.3.3±.5 QGP.58±.8.553±.83.68±.48 Weather 38 Linear.9±.34.93±.38.3±.445 QSVM.67±.49.77±.335.8±.3 Reduction.749±.4.757±.75.6±.9 QGP.573±.99.97±.54.68±.68 Motorcycle 33 Linear.3963± ± ±.56 QSVM.9±.5.±.9.85±.76 Reduction.944±.9.93±.49.83±.99 QGP.9± ± ±.99 HQGP.79±.94.87±..697±.6 BMD 485 Linear.383±.8.346± ±.76 QSVM.± ±.387.5±.47 Reduction.5±.97.33± ±.65 QGP.349± ± ±.96 HQGP.8±.7.386± ±.67 California 64 Linear.86±.383.5± ±.678 Housing QSVM Reduction.79±.43.63±.9.673±.6 Sparse QGP.39±.6.7± ±.4 Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 3

14 Experiments Real Data Visualization on Silverman s Motorcyle Benchmark quantile.5 quantile.5 quantile.75 quantile.9 quantile Quantile SVM. quantile.5 quantile.5 quantile.75 quantile.9 quantile 3 Reduction. quantile.5 quantile.5 quantile.75 quantile.9 quantile Ours Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 4 Ours (Heteroscedastic)

15 Summary Take home messages We propose a quantile estimator which is simple to implement; enjoys non-parametric and probabilistic model; principles learning of free parameters; sparse approximation; enforced non-crossing constraint properties; and has performance guaranteed. Novi Quadrianto: Kernel Conditional Quantile Estimation via Reduction Revisited, Page 5