SYSM 6304 Risk and Decision Analysis Lecture 2: Fitting Distributions to Data

Transcription

1 SYSM 6304 Risk and Decision Analysis Lecture 2: Fitting Distributions to Data M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas September 5, 2015

2 Objectives of Lecture Introduce maximum likelihood estimation (MLE) Introduce goodness of fit tests Illustrate the use of empirical and fitted distributions in forecasting percentiles

3 Outline Case Study: Gaussian Fit of Height Data 1 Case Study: Gaussian Fit of Height Data 2 3

4 Outline Case Study: Gaussian Fit of Height Data 1 Case Study: Gaussian Fit of Height Data 2 3

5 Problem Statement Suppose you are given height data for 200 individuals, sampled from a much larger population of millions. You wish to use this data to predict several parameters: What is the average height of the entire population? What is the 99-th percentile and the th percentile of the height in the population? By using this case study, we can illustrate various concepts.

6 Cumulative Distribution Function: Reprise If X is a real-valued random variable, then its cumulative distribution function (CDF) is defined as Φ X (a) := Pr{X a}. In words, Φ X (a) is the probability that X is less than or equal to a. Usually it is unknown and has to be estimated from samples; how can we do this?

7 Empirical Distribution Function: Definition Suppose Y is the random variable of interest, and we have n independent samples of Y, call them y 1,..., y n. For each value of y, define the empirical distribution function ˆΦ Y (y) = 1 n n i=1 I yi y, where I denotes the indicator function it equals one if the statement in the subscript is true, and zero if it is false. So specifically ˆΦ Y (y) is just the fraction of the n samples that are smaller than or equal to y.

8 Empirical Distribution Function (Cont d) To construct the empirical distribution function, first sort all the samples y 1,..., y n in increasing order; call the result z 1,..., z n. Then construct a staircase function that jumps by 1/n at each sample z i. That is the empirical distribution function.

9 Empirical Distribution: Depiction ˆΦ Y (u) z 1 z 2 z 3 z 4 z 5 z 6 u

10 Empirical Distribution to Approximate True CDF The empirical distribution function is not continuous it jumps vertically by 1/n at each sample, where n is the total number of samples. However, as n becomes large, the jumps become smaller.

11 Gaussian Density and Distribution Functions: Reprise The normal distribution function is smooth (i.e., differentiable everywhere). The formula for the normal density function is φ N (x) = 1 2π exp( x 2 /2). It is a special case of the Gaussian density function φ G (x) = 1 2πσ exp( (x µ) 2 /2σ 2 ), where µ and σ are the mean and the standard deviation respectively. The next slide shows the density function with µ = 0 for various values of σ.

12 Gaussian Density Function Gaussian Density Functions for Various Sigma Values sigma = 1 sigma = 0.5 sigma = 2 sigma = phi(x) x

13 Gaussian Density Function in a Logarithmic Scale The tail behavior of the Gaussian density function shows up more clearly on a logarithmic scale. 0 Normal Density Function in Log Scale log phi(x) x/sigma

14 Cumulative Normal Distribution The cdf (cumulative distribution function) of a normal random variable is given as the integral of the density, that is, Φ N (x) = 1 2π x exp( s 2 /2)ds. There is no closed-form expression for the integral, but it has the shape shown on the next slide.

15 Cumulative Normal Distribution (Cont d) 1 Cumumative Normal Distribution Function Phi(x) x

16 Maximum Likelihood Estimation for Gaussian Distributions Suppose we are given random samples y 1,..., y n of a random variable Y. The objective is to find the maximum likelihood estimate for a Gaussian distribution to fit the data. The Gaussian distribution has two adjustable parameters: the mean µ and the standard deviation σ. The question is: What choice of these parameters makes the observations most likely? (Hence the name maximum likelihood estimation, or MLE).

17 MLE for Gaussian Distributions (Cont d) It turns out that the answer is the following: The best choice for µ is the so-called sample mean ˆµ = 1 n n y i. i=1 The best choice for σ is given by [ n ˆσ = i=1 (y i ˆµ) 2 ] 1/2. So ˆµ is just the average value of the samples, and ˆσ is the average deviation of the samples around ˆµ.

18 Heights Data Set We apply MLE to the Hong Kong height data set consisting of 200 samples. The estimates ˆµ and ˆσ can be readily computed using the Matlab commands mean and std. Specifically, if y denotes the vector of the n samples, then the commands ˆµ = mean(y) and ˆσ = std(y) will give the MLE values. In this case it turns out that ˆµ = , ˆσ = , where both numbers are in inches.

19 Comparison of Empirical and Fitted CDFs To see how good the fit is, recall the sorted vector z, which consists of the samples y 1,..., y n arranged in increasing order. By definition, the empirical CDF has value i/n at the sample z i, the staircase function. By computing the fitted Gaussian CDF at each z i, we can get an idea of whether the Gaussian is a good fit or not. The comparison is shown on the next slide.

20 Gaussian Fit to Height Data Gaussian Fit to Hong Kong Heights Data Empirical Gaussian Fit Distribution Function Height

21 Goodness of Fit: The K-S Test The figure shows the best fit to the data from within the family of Gaussian CDFs. But is it a good fit overall? The so-called K-S goodness of fit test allow us to address this question. K-S stands for Kolmogorov-Smirnov, the two Russian mathematicians who invented this test. What we show is the so-called one-sample K-S test, which is all we will use; we will not require the two-sample K-S test. The (one-sample) K-S test is very useful because it is universal it can be used with any candidate distribution, not just the Gaussian.

22 Goodness of Fit: The K-S Test (Cont d) To apply this test, denote ˆΦ(z i ) to be the value of the fitted distribution (Gaussian or otherwise) computed at the i-sorted sample z i. Recall that the empirical distribution equals i/n at the sample z i, the staircase function. Compute the K-S test statistic t KS = max i/n ˆΦ(z i ). i So the test statistic is just the worst-case discrepancy between the empirical and fitted distribution functions across all samples.

23 Goodness of Fit: The K-S Test (Cont d) To apply the K-S test, we need to choose a confidence level δ, usually 0.05 or 0.01 (though any arbitrary δ is permitted). Compute the threshold θ = [ 1 2n log 2 ] 1/2. δ Check to see whether t KS > θ. If so, we can reject the fitted distribution as being bad, with confidence 1 δ. If t KS θ, we accept the fit, or more accurately, we cannot reject the fit.

24 Goodness of Fit Test (Cont d) We aim for a 95% confidence level, so we define a confidence parameter δ = 1.95 = There are 200 samples. Therefore the acceptable threshold, according to the above formula, is θ = The maximum error between the two curves is t KS = Because the maximjum discrepancy between the fitted and the empirical distributions is smaller than the threshold, we accept the Gaussian fit as being a fair description of the data. Later we will see an example where a fitted distribution is rejected because the maximum discrepancy is larger than the permitted threshold.

25 Predicting Percentiles Using Fitted Distributions In risk analysis, we are often interested in percentile values. If X is the random variable of interest, and if ζ is some number between 0 and 1, the ζ (or more accurately 100 ζ) percentile is the value a such that Pr{X a} = ζ, or equivalently Pr{X > a} = 1 ζ. So if ζ = 0.95, then a would be the 95th percentile, and so on. Fitted Distributions can be used to predict the percentile values on the basis of data.

26 Using Empirical vs. Fitted Distributions Suppose we have n samples of the random variable X (e.g., 200 height samples), and we wish to find the ζ percentile value. One way to do this is to sort the samples x 1,..., x n in ascending order, and call the resulting sequence z 1,..., z n. Then define k = ζn (the number ζn rounded up to the nearest integer). We can define the value z k as the empirical ζ percentile value. For example, if there are 200 samples, and ζ = 0.95, then the 190-th largest sample would be the empirically derived 95-th percentile value.

27 Using Empirical vs. Fitted Distributions (Cont d) This approach runs into trouble when we wish to predict very rare events, that is, ζ very close to 1. For instance, if n = 200 and ζ = 0.999, then k = n, and the largest sample is automatically taken as the percentile value, which is clearly not meaningful.

28 Estimating Height Percentiles Take the Hong Kong heights example. Suppose you are required to estimate a threshold such that 95% of the population is below that height. There are two ways to do this. First, you can find the 190-th largest sample, which is invhes. Second, using the Matlab command norminv, you can find a number a = Φ 1 (0.95) using the estimated ˆµ and ˆσ. This gives

29 Estimating Height Percentiles (Cont d) Similarly, the 99-th percentile can either the 198-th largest sample which is , or Φ 1 (0.99) which gives This approach fails to find the 99.9-th percentile. But Φ 1 (0.999) = is a reliable estimate of this value. Note that the last number is larger than the largest number in the sample! Note that we can use the Gaussian fit to estimate percentiles because it passed the K-S test.

30 Outline Case Study: Gaussian Fit of Height Data 1 Case Study: Gaussian Fit of Height Data 2 3

31 Problem Statement You are given the daily prices of Microsoft stock for roughly 5.5 years. You wish to fit an appropriate probability distribution to these prices. Then you wish to use the fitted distribution to predict the so-called Value at Risk (VaR) of the stock price, at variour percentiles.

32 Choice of Appropriate Family of Distributions The stock price is always nonnegative. So, instead of trying to fit the stock price directly, we fit the daily returns of the stock. The daily return is defined as the logarithm of the ratio of today s price and yesterday s price. Clearly it can be both positive or negative. Let us denote the stock price random variable by Y, and the daily return random variable by X. If X has a Gaussian distribution, then Y is said to be log-normal. The more accurate phrase log-gaussian is somehow not used.

33 Gaussian Fit to Daily Returns From the 1342 daily price records, we can compute 1341 daily returns. We already know that the maximum likelihood estimate of a Gaussian distribution to this data is obtained using the sample mean ˆµ and the sample standard deviation ˆσ. In this case ˆµ = So for all practical purposes the sample mean is zero. Also, ˆσ = , which means that on a typical day the stock goes up or down by 2%. Using these parameters, we can fit a Gaussian distribution to the data. The next slide shows the empirical and Gaussian fitted CDFs.

34 Empirical vs. Gaussian Fit Distributions Gaussian Fit of Daily Returns of Microsoft Stock Empirical CDF Gaussian Fit Distribution Function Daily Returns on Microsoft Stock

35 Goodness of Fit Test Next we apply the K-S goodness of fit test. The acceptable threshold at a confidence level δ = 0.05 with n = 1341 samples is given by [ 1 θ = 2n log 2 ] 1/2, δ which in this case turns out to be However the K-S test statistic, defined as t KS = max i/n Φ(z i ) i where z i is the i-th largest sample, turns out to be , which is larger than the acceptable threshold θ. So we can be at least 95% sure that the daily returns do not follow a Gaussian distribution, or in other words, the stock price itself does not follow a log-normal distribution.

36 Goodness of Fit Test (Cont d) In fact we can be a lot more than 95% sure that the data does not fit a Gaussian distribution. Compute δ = 2 exp( 2θnt 2 KS) = So we can be 1 δ, or approximately % sure that the data does not fit a Gaussian distribution. So what can be done? Instead of a Gaussian family, we use the family of stable distributions. Stable distributions contain Gaussian distributions as a special case. So we will get at least as good a fit using stable distributions as with Gaussian distributions, and often a better fit.

37 Parameters of a Stable Distribution A Gaussian r.v. is completely specified by just two parameters, namely its mean µ and its standard deviation σ. A stable distribution is specified by four parameters: An exponent α (0, 2). A skew β [ 1, 1]. A scale γ R +. A location δ R. If α = 2 then we get a Gaussian distribution as a special case.

38 Interpretation of Parameters The four parameters mean what the names suggest: The exponent α controls how slowly the complementary distribution function Φ(u) decays as u. As α gets smaller, the densities get flatter and wider. The skew β is zero if the density function is symmetric, and nonzero otherwise. The scale γ is the spread on the u-axis. As γ is decreased, the density function gets spread out. The location δ centers the location of the distribution. The next several slides illustrate the role of these constants.

39 Varying Alpha phi(x) for various values of alpha Stable Density Functions for Various Values of Alpha alpha = 1.8 alpha = 1.5 alpha = 1.1 alpha = x As α is decreased, the peaks get higher and the tails get flatter.

40 Heavy-Tailed Behavior of Stable Distributions The Gaussian distribution can be thought of as a special case of a stable distribution with α = 2. If α < 2, then the r.v. is heavy-tailed in that its variance is infinite. If α < 1, then even the mean is infinite. Despite this, stable distributions with α < 2 often provide a far better fit to real-world data than Gaussian distributions.

41 Varying Beta Stable Density Functions for Various Values of Beta beta = 0 beta = 1 beta = 1 phi(x) for various values of beta x The brown and red curves are asymmetric, though this is hard to see.

42 Varying Gamma Stable Density Functions for Various Values of Gamma gamma = 1 gamma = 0.5 gamma = 2 phi(x) for various values of gamma x Smaller values of γ spread out the density function.

43 Varying Delta Stable Density Functions for Various Values of Delta delta = 0 delta = 1 delta = 1 phi(x) for various values of delta x Nonzero values of δ shift the curve to the left or the right but do not otherwise change the shape.

44 Fitting a Stable Distribution to Data The theory of maximum likelihood estimation for stable distributions is too advanced to cover in the lectures. However, the Matlab command stblfit can be used to fit a stable distribution to any set of samples. Ensure that the file stblfit.m is in the same directory from which Matlab is being executed.

45 Fitting a Stable Distribution to Microsoft Daily Returns Applying the command stblfit to the 1341 daily return samples of Microsoft results in these parameters α β γ δ = In particular α is far smaller than 2, which corresponds to a Gaussian distribution. This may explain why a Gaussian fit fails the K-S test. The empirical and the stable-fitted distributions are shown on the next slide.

46 Empirical vs. Stable-Fit Distributions Stable Fit to Daily Returns of Microsoft Stock Empirical Stable Fit Distribution Function Daily Returns on Microsoft Stock

47 Empirical vs. Gaussian Fit Distributions: Reprise Gaussian Fit of Daily Returns of Microsoft Stock Empirical CDF Gaussian Fit Distribution Function Daily Returns on Microsoft Stock

48 Goodness of Fit Test for Stable Fit Distribution The K-S test statistic, which is t KS = max i/n Φ(z i ), i in this case turns out to be , which is far smaller than the K-S test statistic for the Gaussian fit, which was On the other hand, the acceptable threshold with δ = 0.05 and n = 1341 samples is given, as before, by θ = [ 1 2n log 2 ] 1/2 = δ Because t KS < θ, we accept the stable fit distribution.

49 Predicting Percentiles Using Stable Fit Distribution Once we have a stable distribution that fits the data (as determined by the K-S test), we can use the Matlab command tt stblinv to determine various percentiles. If we set ζ = 0.95, then we can compute v ζ as stblinv(0.95,alpha,beta,gamma,delta, which equals This means that we are 95% sure that tomorrow s stock price will not increase by more than 2.9% over today s price. If we wish to be 99% sure, we set ζ = 0.99, and compute the threshold stblinv(0.99,alpha,beta,gamma,delta, which equals So we can be 99% sure that tomorrow s stock price won t be more than 6.5% higher.

50 Predicting Confidence Intervals Using Stable Fit Distribution Instead of finding one-sided limits, we can also find so-called confidence intervals using the fitted distribution. If we wish to find a 95% confidence interval for the return, we can compute stblinv(0.975,alpha,beta,gamma,delta) = and stblinv(0.025,alpha,beta,gamma,delta) = = So we can assert with 95% confidence that tomorrow s stock return (daily fluctuation) will lie between and

51 Outline Case Study: Gaussian Fit of Height Data 1 Case Study: Gaussian Fit of Height Data 2 3

52 Conditional PDFs and Densities: Motivation Example: The life expectancy of an American male at birth is years. So if a man is now 65 years old, is his life expectancy years? No! it is a longer! Reason: The life expectancy calculation of years includes those who fail to reach the age of 65. So the expectancy has to be recomputed by taking this into account. Example: The average rainfall in July in a certain city is inches. It has already rained 8 inches by the middle of July. So how much more rain can be expected in the second half of July? Answer is not 2.04 inches!

53 Conditional PDFs and Densities: Definition Suppose X is a random variable with pdf Φ X ( ) and density φ X ( ), and b is some specified number. What are the pdf and density of X once it is specified that X > b? 0, if a b, Φ X>b (a) = Φ X (a) Φ X (b), if a > b, Φ X (b) The formula looks cleaner if expressed in terms of Φ X. 1, if a b, Φ X>b (a) = Φ X (a), if a > b, Φ X (b)

54 Conditional PDFs and Densities: Definition (Cont d) 0, if a b, φ X>b (a) = φ X (a), Φ X (b) if a > b, Using the conditional density, we can compute the conditional mean (usually the quantity of greatest interest), conditional variance etc.

55 Example: Longevity in the USA The US Social Security Administration (SSA) publishes data on the longevity of males and females. Starting with a cohort of 100,000 persons, the data shows how many are alive at the end of year 1, year 2 etc. So this data represents the complementary cumulative distribution function.

56 Longevity Male and Female Longevity in the USA Male Female Complementary cdf Years

57 Longevity 3 The figure below shows the fraction of the original cohort who die each year Male and Female Mortality in the US Male Female Density Years

58 Life Expectancy at Birth and at Age 65 Computing life expectancy at birth gives years for males and years for females. If we focus on those who survive until age 65, the figures are years for males and years for females.