Regression and Simulation

Transcription

1 Regression and Simulation This is an introductory R session, so it may go slowly if you have never used R before. Do not be discouraged. A great way to learn a new language like this is to plunge right in. We will simulate some data suitable for a regression. This will get you started in R and will teach you some useful tricks. I want you to build a script in RStudio (File/New File/R Script) and execute the lines from the script as you proceed. 1. Generate data for simple regression The first task is to generate some data for fitting a simple regression model. By simple, this means a single predictor X 1 and a response Y. We will use a sample size of N=100, and generate N pairs (X 1i, Y i ) from the model Y i = β 0 + X 1i β 1 + e i. In R we will call X 1 simply, since R does not like subscripts in names. Now for some details: The X 1i should come from a Gaussian distribution with mean 20 and standard deviation 3. Take β 0 = 25 and β 1 = 2 to be the true parameters, and assume that e i comes from a normal distribution with mean 0 and standard deviation 15. Generate your data, make a histogram of X 1, and plot the values of Y against those of X 1. Your plot should look something like this: Y Useful functions are rnorm, plot, and hist. If you type?rnorm, for example, you will see a helpfile. In the plot, include the true regression line; abline is useful here. 1

2 Answers First we will generate some data from a Gaussian distribution and call it. N=100?rnorm =rnorm(n,20,3) hist() Histogram of Frequency Now we will make a response Y that depends on plus some error. We will make the error Gaussian as well. Y= 25 + *2 + rnorm(n,0,15) plot(,y) abline(25,2,col="red") 2

3 Y Notice how we included the true regression line in the plot. 2. Fit a linear regression Let s now fit a linear regression of Y on X 1 and look at its properties. For this you will use the function lm in the following manner. Use the summary() command on your fitted object fit to see what is inside. Try coef as well. We can include the fitted regression line in the plot. The abline function knows about fitted lm models, so you can give it as a single argument to abline(). 3

4 Y Once you get this working, see what happens if you execute your entire script (including the data generation) a second time. You should get a somewhat different result, since different data are generated. You can prevent this, if you like, by using the set.seed command. Type?set.seed to learn how. Try this and see that your data generation is now reproducible. Answers Here is our initial regression fit. summary(fit) Call: lm(formula = Y ~ ) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-05 *** --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: on 98 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 98 DF, p-value: 1.036e-05 4

5 coef(fit) (Intercept) plot(,y) abline(25,2,col="red") abline(fit,col="blue",lwd=2) Y If we repeat this process, we get something slightly different =rnorm(n,20,3) Y= 25 + *2 + rnorm(n,0,15) plot(,y) summary(fit) Call: lm(formula = Y ~ ) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) *** ** --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 5

6 Residual standard error: on 98 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 98 DF, p-value: abline(25,2,col="red") abline(fit,col="blue",lwd=2) Y If we want to be able to repeat an analysis, including the random number generation, then we have to set the random number seed. This is easy to do in R. set.seed(101) # any number will do =rnorm(n,20,3) Y= 25 + *2 + rnorm(n,0,15) plot(,y) summary(fit) Call: lm(formula = Y ~ ) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-06 *** --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 6

7 Residual standard error: on 98 degrees of freedom Multiple R-squared: 0.189, Adjusted R-squared: F-statistic: on 1 and 98 DF, p-value: 6.206e-06 coef(fit) (Intercept) abline(25,2,col="red") abline(fit,col="blue",lwd=2) Y Simulation to check aspects of regression We learn from the theory that the variance of the slope estimate in simple regression is given by the expression Var( ˆβ 1 ) = σ 2 N i=1 (X 1i X 1 ) 2, where X 1 is the mean of the X 1i, and σ is the standard deviation of the e i. (Here the hat on top of ˆβ 1 means it is estimated from our data.) What this theory is really telling us is that if we fix our X 1 values and repeatedly generate new e i and hence Y i and refit the regressions each time, the variance of the series of ˆβ 1 s that we get will be given by this expression. Since we typically don t know σ, we need to estimate it as well, using the the expression ˆσ 2 = 1 N 2 N (Y i Ŷi) 2. i=1 7

8 The linear model summary tells us the estimated standard error of the slope is 0.54 for these data (at least mine did). Let s check it out! Run a simulation where you generate a new response vector each time, fit the regression, extract the slope coefficient, and store it. Do this 1000 times using a for loop (?'for'), keeping the same values for X 1. Summarize the results and in particular compute the standard deviation of the 1000 values you get. How closely does it match the theoretical value? Answers We will generate 1000 different response vectors, compute the slope each time, and then claculate the standard deviation of the estimated slopes. set.seed(101) #any number will do beta=double(1000) for(i in 1:1000) { Y = 25 + *2 + rnorm(n,0,15) beta[i]=coef(fit)[2] } Now let s look at these betas (slope estimates). hist(beta) Histogram of beta Frequency beta mean(beta) [1]

9 sd(beta) [1] Pretty close! 4. Slightly different simulation Here we held the original X 1 values fixed. What if we generate new versions of them as well when we do our simulations. Repeat your simulations, adding this little twist. What do you learn about the distribution and standard error of ˆβ 1 in this setting? Any surprises? What would you have expected? Answers set.seed(101) #any number will do beta2=double(1000) for(i in 1:1000){ =rnorm(n,20,3) Y= 25 + *2 + rnorm(n,0,15) beta2[i]=coef(fit)[2] } mean(beta2) [1] sd(beta2) [1] It s smaller! What happened? 5. Multiple Regression We now generate an X 2 and create a multiple regression model Y i = β 0 + X 1i β 1 + X 2i β 2 + e i. Let X 2 have the same distribution as X 1 (but be independent of X 1 ), and let β 2 = 1 in the simulation. Fit the multiple linear regression model the formula is now Y~+X2 and use summary to summarize the results. What do you observe about the standard errors of each coefficient and in particular the coefficient of X 1? For your current version of X 1 and X 2, what is their sample correlation? (cor()). Is your sample correlation zero? Why or why not? Answers We now generate an X 2 and create a multiple regression model. 9

10 set.seed(109) =rnorm(n,20,3) X2=rnorm(N,20,3) Y= 25 + *2 + X2*1+ rnorm(n,0,15) fit2=lm(y~+x2) summary(fit) Call: lm(formula = Y ~ ) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) *** e-07 *** --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: on 98 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 98 DF, p-value: 3.141e-07 summary(fit2) Call: lm(formula = Y ~ + X2) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-07 *** X Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: on 97 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 97 DF, p-value: 4.447e-07 In this case, X 1 and X 2 are realizations of two independent random variables, each Gaussian, so the presence of one does not really effect the other in the regression. Here their sample correlation is 10

11 cor(,x2) [1] Suppose we wanted the distribution of X 2 to be correlated with X 1. How might we do that? Figure out a way to do this, and experiment until your X 2 has sample correlation about 0.6 with X 1. Now generate a Y from your regression model, and use summary to examine the coefficients. What do you observe? Now try to produce X 2 with sample correlation about 0.8 with X 1, and repeat the regression experiment above. What do you observe? Answers How do we generate an X 2 that has correlation with X 1? One simple way is to generate a Z independent of X 1, and then make X 2 = αx 1 + Z for some value of α (below we use α = 0.7). Z=X2 X2=0.7*+Z plot(,x2) X cor(,x2) [1] Now when we fit the linear model, the variance of the coefficients increases in the multiple linear regression versus the simple linear regression, because of the correlation. Y= 25 + *2 + X2*1+ rnorm(n,0,15) fit2=lm(y~+x2) summary(fit) 11

12 Call: lm(formula = Y ~ ) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-06 *** e-09 *** --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: 15.1 on 98 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 98 DF, p-value: 6.404e-09 summary(fit2) Call: lm(formula = Y ~ + X2) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) *** X * --- Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 Residual standard error: on 97 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 97 DF, p-value: 3.799e-09 12