Review: Population, sample, and sampling distributions A population with mean µ and standard deviation σ For instance, µ = 0, σ = 1 0 1 Sample 1, N=30 Sample 2, N=30 Sample 100000000000 InterquartileRange = 1.25 InterquartileRange = 1.7 InterquartileRange = 0.65 The sampling distribution of the interquartile range for samples of size N = 30
How s your IQ? Suppose the population IQ is a normal distribution with mean 100 and standard deviation 20. The mean IQ in this class, 23 students, is 130. Should we reject the null hypothesis that this class is no different in IQ from the population?
The Logic Our result: R = 130 Assume Ho: Π = 100, this class is a random sample drawn from the population of people with mean IQ 100 If the result is very unlikely under Ho, if Pr(R=130 Π = 100) α, then we are inclined to reject Ho. Pick a value of α (say,.01) and calculate the conditional probability p = Pr(R=130 Π = 100) Our residual uncertainty that Ho might be right is less than or equal to α
Calculate p = Pr(R=130 Π = 100) Find the sampling distribution of R for N = 23 Since we know the population parameters (normal, mean = 100, standard deviation = 20) we can get the sampling distribution by Monte Carlo sampling: (defun sampling-distribution (n mean std k) "N is the sample size, MEAN and STD are the parameters of a normal distribution, K is the size (number of samples) of the sampling distribution." (loop repeat k collect (mean (sample-normal-to-list mean std n)))) 30 20 10 90 95 100 105 110
Calculate p = Pr(R=130 Π = 100) Find the sampling distribution of R for N = 23 Since we know the population parameters (normal, mean = 100, standard deviation = 20) we can get the sampling distribution by Monte Carlo sampling: 30 20 10 90 95 100 105 110 130 The probability of getting a sample of size 23 with mean 130 by random sampling from a population with mean 100 and standard deviation 20 is virtually zero.
Another way to write the code: (defun sampling-distribution (n mean std r k) (loop repeat k counting (> (mean (sample-normal-to-list mean std n)) r))) (sampling-distribution 23 100 20 130 1000) => 0
Parametric statistical inference Testing hypotheses by simulating the process of sampling is cool but not always necessary The probability of tossing 15 heads in 20 with a fair coin can be worked out exactly The probability that a sample from a population has a particular mean can be estimated However, theory tells us about the sampling distributions of very few statistics; for the rest, simulation works great
Central Limit Theorem The sampling distribution of the mean of samples of size N drawn from a population with mean µ and standard deviation σ approaches a normal distribution with mean µ and standard deviation σ / N as N becomes large Good news! We know the sampling distribution of the mean and can estimate the probability of sample results!
The Logic Our result: R = 130 Assume Ho: Π = 100, this class is a random sample drawn from the population of people with mean IQ 100 If the result is very unlikely under Ho, if Pr(R=130 Π = 100) α, then we are inclined to reject Ho. Pick a value of α (say,.01) and calculate the conditional probability p = Pr(R=130 Π = 100) The sampling distribution of the mean approaches a normal distribution with mean = 100 and std = 20 / 23 = 4.17 So our sample result is 30 / 4.17 = 7.2 standard deviations above the mean of the sampling distribution!
Standard error: The standard deviation of the sampling distribution Standard Error of the Mean under Ho: Π = 100, the sampling distribution is normal, its mean is 100, its standard deviation is 20 / 23 = 4.17 The standard error is 4.17 The sample result is 4.17 standard error units above the mean under Ho 100 104 130 99% of a normal distribution lies within two standard deviations of the mean. How probable is our sample result?
Try it again with a less extreme result Our result: R = 108 Assume Ho: Π = 100, this class is a random sample drawn from the population of people with mean IQ 100 If the result is very unlikely under Ho, if Pr(R=108 Π = 100) α, then we are inclined to reject Ho. Pick a value of α (say,.01) and calculate the conditional probability p = Pr(R=108 Π = 100) The sampling distribution of the mean approaches a normal distribution with mean = 100 and std = 20 / 23 = 4.17 So our sample result is 8 / 4.17 = 1.92 standard errors above the mean of the sampling distribution.
p values s.e. under Ho: Π = 100, the sampling distribution is normal, its mean is 100, its standard deviation is 20 / 23 = 4.17 The sample result, R=108, is 1.92 standard error units above the mean under Ho. 100 104 108 Now it isn t so obvious that we should reject Ho. How can we find p = Pr(R=108 Π = 100)? State the result in standard error units and look up its probability in a table.
p values s.e. The sample result, R=108, is 1.92 standard error units above the mean under Ho. 100 104 108
Standardizing subtract the mean, divide by the standard error s.e. under Ho: Π = 100, the sampling distribution is normal, its mean is 100, its standard deviation is 20 / 23 = 4.17, and the sample result is 108 100 104 108 s.e. under Ho: Π = 0, the sampling distribution is normal, its mean is 0, its standard deviation is 1.0, the sample result is (108-100) / (20 / 23) = 1.92 0 1 1.92
Z scores or standard scores subtract the mean, divide by the standard error s.e. Z = x µ s.e. = x µ σ / N 100 104 108 s.e. 108-100 4.17 = 1.92 0 1 1.92
The Z test Z = x µ s.e. Z is the number of standard error units the sample mean is from the mean of the sampling distribution under the null hypothesis. 1.92 = 8 100 20 / 23 If Z 1.645 then the sample result has p.05 probability given the null hypothesis If Z 1.96 then the sample result has p.01 probability given the null hypothesis
The Z test Our result: R = 108 Assume Ho: Π = 100, this class is a random sample drawn from the population of people with mean IQ 100 If the result is very unlikely under Ho, if Pr(R=108 Π = 100) α, then we are inclined to reject Ho. Pick a value of α (say,.01) and calculate the conditional probability p = Pr(R=108 Π = 100) The sampling distribution of the mean approaches a normal distribution with mean = 100 and std = 20 / 23 = 4.17 So our sample result is 8 / 4.17 = 1.92 standard errors above the mean of the sampling distribution Equivalently, Z = (108-100) / 4.17 = 1.92 p = Pr(R=108 Π = 100) = Pr(Z).0274, α =.01, do not reject Ho.
You do it: A sample of size 25 has mean 8. Test the hypothesis that the sample is drawn from a population with mean 12, standard deviation 10.
You do it: A sample of size 25 has mean 8. Test the hypothesis that the sample is drawn from a population with mean 12, standard deviation 10. Z = 8-12 10 / 25 = 2
Central limit theorem demo (loop repeat 1000 collect (mean (sample-from-population n)))) Std(population) = 1.11 s.e.(20) = 1.11 / 20 =.248 s.e.(30) = 1.11 / 30 =.203 s.e.(50) = 1.11 / 20 =.157 600 500 400 300 200 100 Population 100 50 N=20 Std =.25 90 80 70 60 50 40 30 20 10 N=30 Std =.21 60 50 40 30 20 10 N=50 Std =.16-10 -9-8 -7-6 -5-4 -3-2 -1 0 1 VAR Histogram OF Var[Dataset-3] -1.5-1 -0.5 0-1.4-1.1
Three components of all test statistics Effect size Z = x x = x N background variance sample size You can make any Z score significant with a big enough sample, but you shouldn t. Always try to control variance before increasing N.
Parametric and computer-intensive hypothesis testing std under Ho: Π = 100, the mean of sampling distribution is 100, the standard deviation is 20 / 23 = 4.17 100 104 130 30 20 Empirically (by simulation) this distribution has a mean of 100.05 and a standard deviation of 4.38 10 90 95 100 105 110 130
We do not know the sampling distribution of most statistics but we can estimate them empirically! (defun sampling-distribution (n mean std k) "N is the sample size, MEAN and STD are the parameters of a normal distribution, K is the size (number of samples) of the sampling distribution." (loop repeat k collect (mean (sample-normal-to-list mean std n)))) median interquartile-range trimmed-mean median-divided-by-mom s-age
Some issues for parametric and computer-intensive tests Z is fine if you know σ, (recall, z = (x - µ ) / (σ / n)) but what if you don t? Estimate σ from s and for smaller samples run t tests. Monte Carlo tests are fine if you know the parameters of the population from which samples are drawn, but what if you don t? Estimate these parameters from the sample and run bootstrap or randomization tests.