Chapter 7: Estimation Sections
|
|
- Ethan Malone
- 5 years ago
- Views:
Transcription
1 1 / 40 Chapter 7: Estimation Sections 7.1 Statistical Inference Bayesian Methods: Chapter Prior and Posterior Distributions 7.3 Conjugate Prior Distributions 7.4 Bayes Estimators Frequentist Methods: 7.5 Maximum Likelihood Estimators 7.6 Properties of Maximum Likelihood Estimators Skip: p (EM algorithm and Sampling Plans) 7.7 Sufficient Statistics Skip: 7.8 Jointly Sufficient Statistics Skip: 7.9 Improving an Estimator
2 2 / 40 Chapter Statistical Inference Statistical Inference We have seen statistical models in the form of probability distributions: f (x θ) In this section the general notation for any parameter will be θ The parameter space will be denoted by Ω For example: Life time of a christmas light series follows the Expo(θ) The average of 63 poured drinks is approximately normal with mean θ The number of people that have a disease out of a group of N people follows the Binomial(N, θ) distribution. In practice the value of the parameter θ is unknown.
3 3 / 40 Chapter Statistical Inference Statistical Inference Statistical Inference: Given the data we have observed what can we say about θ? I.e. we observe random variables X 1,..., X n that we assume follow our statistical model and then we want to draw probabilistic conclusions about the parameter θ. For example: If I tested 5 Christmas light series from the same manufacturer and they lasted for 21, 103, 76, 88 and 96 days. Assuming that the life times are independent and follow Expo(θ), what does this data set tell me about the failure rate θ?
4 4 / 40 Chapter Statistical Inference Statistical Inference Another example Say I take a random sample of 100 people and test them all for a disease. If 3 of them have the disease, what can I say about θ = the prevalence of the disease in the population? Say I estimate θ as ˆθ = 3/100 = 3%. How sure am I about this number? I want uncertainty bounds on my estimate. Can I be confident that the prevalence of the disease is higher than 2%?
5 5 / 40 Chapter Statistical Inference Statistical Inference Examples of different types of inference Prediction Predict random variables that have not yet been observed E.g. If we test 40 more people for the disease, how many people do we predict have the disease? Estimation Estimate (predict) the unknown parameter θ E.g. We estimated the prevalence of the disease as ˆθ = 3%.
6 6 / 40 Chapter Statistical Inference Statistical Inference Examples of different types of inference Making decisions Hypothesis testing, decision theory E.g. If the disease affects 2% or more of the population, the state will launch a costly public health campaign. Can we be confident that θ is higher than 2%? Experimental Design What and how much data should we collect? E.g. How do I select people in my clinical trial? How many do I need to be comfortable making decision based on my analysis? Often limited by time and / or budget constraints
7 7 / 40 Chapter Statistical Inference Bayesian vs. Frequentist Inference Should a parameter θ be treated as a random variable? E.g. consider the prevalence of a disease. Frequentists: No, the proportion q of the population that has the disease, is not a random phenomenon but a fixed number that is simply unknown Example: 95% confidence interval: Wish to find random variables T 1 and T 2 that satisfy the probabilistic statement P(T 1 q T 2 ) 0.9 Interpretation: P(T 1 q T 2 ) is the probability that the random interval [T 1, T 2 ] covers q
8 8 / 40 Chapter Statistical Inference Bayesian vs. Frequentist Inference Should a parameter be treated as a random variable? E.g. consider the prevalence of a disease. Bayesians: Yes, the proportion Q of the population that has the disease is unknown and the distribution of Q is a subjective probability distribution that expresses the experimenters (prior) beliefs about Q Example: 95% credible interval: Wish to find constants t 1 and t 2 that satisfy the probabilistic statement P(t 1 Q t 2 data ) 0.9 Interpretation: P(t 1 Q t 2 ) is the probability that the parameter Q is in the interval [t 1, t 2 ].
9 Chapter Prior and Posterior Distributions Bayesian Inference Prior distribution Prior distribution: The distribution we assign to parameters before observing the random variables. Notation for the prior pdf/pf : We will use p(θ), the book uses ξ(θ) Likelihood When the joint pdf/pf f (x θ) is regarded as a function of θ for given observations x 1,..., x n it is called the likelihood function. Posterior distribution Posterior distribution: The conditional distribution of the parameters θ given the observed random variables X 1,..., X n. Notation for the posterior pdf/pf : We will use p(θ x 1,..., x n ) = p(θ x) 9 / 40
10 Chapter Prior and Posterior Distributions Bayesian Inference Theorem 7.2.1: Calculating the posterior Let X 1,..., X n be a random sample with pdf/pf f (x θ) and let p(θ) be the prior pdf/pf of θ. The the posterior pdf/pf is p(θ x) = f (x 1 θ) f (x n θ)p(θ) g(x) where g(x) = Ω f (x θ)p(θ)dθ is the marginal distribution of X 1,..., X n 10 / 40
11 11 / 40 Chapter Prior and Posterior Distributions Example: Binomial Likelihood and a Beta prior I take a random sample of 100 people and test them all for a disease. Assume that Likelihood: X θ Binomial(100, θ), where X denotes the number of people with the disease Prior: θ Beta(2, 10) I observe X = 3 and I want to find the posterior distribution of θ Generally: Find the posterior distribution of θ when X θ Binomial(n, θ) and θ Beta(α, β) where n, α and β are known.
12 12 / 40 Chapter Prior and Posterior Distributions Example: Binomial Likelihood and a Beta prior Notice how the posterior is more concentrated than the prior. After seeing the data we know more about θ
13 Chapter Prior and Posterior Distributions Bayesian Inference Recall the formula for the posterior distribution: p(θ x) = f (x 1 θ) f (x n θ)p(θ) g n (x) where g(x) = Ω f (x θ)p(θ)dθ is the marginal distribution g(x) does not depend on θ We can therefore write p(θ x) f (x θ)p(θ) In many cases we can recognize the form of the distribution of θ from f (x θ)p(θ), eliminating the need to calculate the marginal distribution Example: The Binomial - Beta case 13 / 40
14 14 / 40 Chapter Prior and Posterior Distributions Sequential Updates If our observations are a random sample, we can do Bayesian Analysis sequentially: Each time we use the posterior from the previous step as a prior: p(θ x 1 ) f (x 1 θ)p(θ) p(θ x 1, x 2 ) f (x 2 θ)p(θ x 1 ) p(θ x 1, x 2, x 3 ) f (x 3 θ)p(θ x 1, x 2 ). p(θ x 1,... x n ) f (x n θ)p(θ x 1,..., x n 1 ) For example: Say I test 40 more people for the disease and 2 tested positive. What is the new posterior?
15 15 / 40 Prior distributions Chapter Prior and Posterior Distributions The prior distribution should reflect what we know a priori about θ For example: Beta(2, 10) puts almost all of the density below 0.5 and has a mean 2/(2 + 10) = 0.167, saying that a prevalence of more then 50% is very unlikely Using Beta(1, 1), i.e. the Uniform(0, 1) indicates that a priori all values between 0 and 1 are equally likely.
16 16 / 40 Choosing a prior Chapter Prior and Posterior Distributions We need to choose prior distributions carefully We need a distribution (e.g. Beta) and its hyperparameters (e.g. α, β) When hyperparameters are difficult to interpret we can sometimes set a mean and a variance and solve for parameters E.g: What Beta prior has mean 0.1 and variance 0.1 2? If more than one option seems sensible, we perform sensitivity analysis: We compare the posteriors we get when using the different priors.
17 Chapter Prior and Posterior Distributions Sensitivity analysis Binomial-Beta example Notice: The posterior mean is always between the prior mean and the observed proportion / 40
18 18 / 40 Chapter Prior and Posterior Distributions Effect of sample size and prior variance The posterior is influenced both by sample size and the prior variance Larger sample size less the prior influences the posterior Larger prior variance the less the prior influences the posterior
19 Chapter Prior and Posterior Distributions Example - Normal distribution Let X 1,..., X n be a random sample from N(θ, σ 2 ) where σ 2 is known Let the prior distribution of θ be N(µ 0, ν 2 0 ) where µ 0 and ν 2 are known. Show that the posterior distribution p(θ x) is N(µ 1, ν 2 1 ) where µ 1 = σ2 µ 0 + nν 2 0 x n σ 2 + nν 2 0 and ν 2 1 = σ2 ν 2 0 σ 2 + nν 2 0 The posterior mean is a linear combination of the prior mean µ 0 and the observed sample mean. What happens when ν 2 0? What happens when ν 2 0 0? What happens when n? 19 / 40
20 20 / 40 Chapter 7 Example - Normal distribution 7.2 Prior and Posterior Distributions
21 21 / 40 Conjugate Priors Chapter Conjugate Prior Distributions Def: Conjugate Priors Let X 1, X 2,... be a random sample from f (x θ). A family Ψ of distributions is called a conjugate family of prior distributions if for any prior distribution p(θ) in Ψ the posterior distribution p(θ x) is also in Ψ Likelihood Bernoulli(θ) Poisson(θ) N(θ, σ 2 ), σ 2 known Exponential(θ) Conjugate Prior for θ The Beta distributions The Gamma distributions The Normal distributions The Gamma distributions Have already see the Bernoulli-Beta and Normal-Normal cases
22 22 / 40 Conjugate prior families Chapter Conjugate Prior Distributions The Gamma distributions are a conjugate family for the Poisson(θ) likelihood: If X 1,..., X n i.i.d. Poisson(θ) and θ Gamma(α, β) then the posterior is ( ) n Gamma α + x i, β + n The Gamma distributions are a conjugate family for the Expo(θ) likelihood: i=1 If X 1,..., X n i.i.d. Expo(θ) and θ Gamma(α, β) then the posterior is ( ) n Gamma α + n, β + x i i=1
23 23 / 40 Chapter Conjugate Prior Distributions Improper priors Improper Prior: A pdf p(θ) where p(θ)dθ = Used to try to put more emphasis on data and down play the prior Used when there is little or no prior information about θ. Caution: We always need to check that the posterior pdf is proper! (Integrates to 1) Example: Let X 1,..., X n be i.i.d. N(θ, σ 2 ) and p(θ) = 1, for θ R. Note: Here the prior variance is Then the posterior is N(x n, σ 2 /n)
24 24 / 40 Chapter 7 continued Chapter 7: Estimation Sections 7.1 Statistical Inference Bayesian Methods: 7.2 Prior and Posterior Distributions 7.3 Conjugate Prior Distributions 7.4 Bayes Estimators Frequentist Methods: 7.5 Maximum Likelihood Estimators 7.6 Properties of Maximum Likelihood Estimators Skip: p (EM algorithm and Sampling Plans) 7.7 Sufficient Statistics Skip: 7.8 Jointly Sufficient Statistics Skip: 7.9 Improving an Estimator
25 Chapter 7 continued 7.4 Bayes Estimators Bayes Estimator In principle, Bayesian inference is the posterior distribution However, often people wish to estimate the unknown parameter θ with a single number A statistic: Any function of observable random variables X 1,..., X n, T = r(x 1, X 2,..., X n ). Example: The sample mean X n is a statistic Def: Estimator / Estimate Suppose our observable data X 1,..., X n is i.i.d. f (x θ), θ Ω R. Estimator of θ: A real valued function δ(x 1,..., X n ) Estimate of θ: δ(x 1,..., x n ), i.e. estimator evaluated at the observed values An estimator is a statistic and a random variable 25 / 40
26 26 / 40 Chapter 7 continued 7.4 Bayes Estimators Bayes Estimator Def: Loss Function Loss function: A real valued function L(θ, a) where θ Ω and a R. L(θ, a) = what we loose by using a as an estimate when θ is the true value of the parameter. Examples: Squared error loss function: L(θ, a) = (θ a) 2 Absolute error loss function: L(θ, a) = θ a
27 27 / 40 Bayes Estimator Chapter 7 continued 7.4 Bayes Estimators Idea: Choose an estimator δ(x) so that we minimize the expected loss Def: Bayes Estimator Minimum expected loss An estimator is called the Bayesian estimator of θ if for all possible observations x of X the expected loss is minimized. For given X = x the expected loss is E (L(θ, a) x) = L(θ, a)p(θ x)dθ Let a (x) be the value of a where the minimum is obtained. Then δ (x) = a (x) is the Bayesian estimate of θ and δ (X) is the Bayesian estimator of θ. Ω
28 28 / 40 Chapter 7 continued 7.4 Bayes Estimators Bayes Estimator For squared error loss: The posterior mean δ (X) = E(θ X) min a E (L(θ, a) x) = min a E ( (θ a) 2 x ). The mean of θ x minimizes this, i.e. the posterior mean. For absolute error loss: The posterior median min a E (L(θ, a) x) = min a E ( θ a x). The median of θ x minimizes this, i.e. the posterior median. The Posterior mean is a more common estimator because it is often difficult to obtain a closed expression of the posterior median.
29 29 / 40 Examples Chapter 7 continued 7.4 Bayes Estimators Normal Bayes Estimator, with respect to squared error loss: If X 1,..., X n are N(θ, σ 2 ) and θ N(µ 0, ν0 2 ) then the Bayesian estimator of θ is δ (X) = σ2 µ 0 + nν 2 0 X n σ 2 + nν 2 0 Binomial Bayes Estimator, with respect to squared error loss: If X Binomial(n, θ) and θ Beta(α, β) then the Bayesian estimator of θ is δ (X) = α + X α + β + n
30 30 / 40 Chapter 7 continued 7.4 Bayes Estimators Bayesian Inference Pros and cons Pros: Cons: Gives a coherent theory for statistical inference such as estimation. Allows for incorporation of prior scientific knowledge about parameters Selecting a scientifically meaningful prior distributions (and loss functions) is often difficult, especially in high dimensions
31 Chapter 7 continued 7.5 Maximum Likelihood Estimators Frequentist Inference Likelihood When the joint pdf/pf f (x θ) is regarded as a function of θ for given observations x 1,..., x n it is called the likelihood function. Maximum Likelihood Estimator Maximum likelihood estimator (MLE): For any given observations x we pick the θ Ω that maximizes f (x θ). Given X = x, the maximum likelihood estimate (MLE) will be a function of x. Notation: ˆθ = δ(x) Potentially confusing notation: Sometimes ˆθ is used for both the estimator and the estimate. Note: The MLE is required to be in the parameter space Ω. Often it is easier to maximize the log-likelihood L(θ) = log (f (x θ) 31 / 40
32 32 / 40 Examples Chapter 7 continued 7.5 Maximum Likelihood Estimators Let X Binomial(n, θ) where n is given. Find the maximum likelihood estimator of θ. Say we observe X = 3, what is the maximum likelihood estimate of θ? Let X 1,..., X n be i.i.d. N(µ, σ 2 ). Find the MLE of µ when σ 2 is known Find the MLE of µ and σ 2 (both unknown) Let X 1,..., X n be i.i.d. Uniform[0, θ], where θ > 0. Find ˆθ Let X 1,..., X n be i.i.d. Uniform[θ, θ + 1]. Find ˆθ
33 33 / 40 Chapter 7 continued 7.5 Maximum Likelihood Estimators MLE Intuition: We pick the parameter that makes the observed data most likely But: The likelihood is not a pdf/pf: If the likelihood of θ 1 is larger than the likelihood of θ 1, i.e. f (x θ 2 ) > f (x θ 1 ) it does NOT mean that θ 2 is more likely Recall: θ is not random here Limitations: Does not always exist Not always appropriate - we cannot incorporate external (prior) knowledge May not be unique
34 34 / 40 Chapter 7 continued Chapter 7: Estimation Sections 7.1 Statistical Inference Bayesian Methods: 7.2 Prior and Posterior Distributions 7.3 Conjugate Prior Distributions 7.4 Bayes Estimators Frequentist Methods: 7.5 Maximum Likelihood Estimators 7.6 Properties of Maximum Likelihood Estimators Skip: p (EM algorithm and Sampling Plans) Skip: 7.7 Sufficient Statistics Skip: 7.8 Jointly Sufficient Statistics Skip: 7.9 Improving an Estimator
35 35 / 40 Properties of MLE s Chapter 7 continued 7.6 Properties of Maximum Likelihood Estimators Theorem 7.6.2: MLE s are invariant If ˆθ is the MLE of θ and g(θ) is a function of θ then g(ˆθ) is the MLE of g(θ) Example: Let ˆp be the MLE of a probability parameter, e.g. the p in Binomial(n, p). Then the MLE of the odds, p 1 p is ˆp 1 ˆp In general this does not hold for Bayes estimators. E.g. for square error loss E(g(θ) x) g(e(θ x))
36 36 / 40 Chapter 7 continued 7.6 Properties of Maximum Likelihood Estimators Computation For MLE s In many practical situations the maximization we need is not available analytically or too cumbersome There exist many numerical optimization methods, Newton s Method (see definition 7.6.2) is one example. For Bayesian estimators In many practical situations the posterior distribution is not available in closed form This happens if we cannot evaluate the integral for the marginal distribution In stead people either approximate the posterior distribution or take random samples from it, e.g. using Markov Chain Monte Carlo (MCMC) methods
37 37 / 40 Chapter 7 continued Method of Moments (MOM) 7.6 Properties of Maximum Likelihood Estimators Let X 1,..., X n be i.i.d. from f (x θ) where θ is k dimensional. The j th sample moment is defined as m j = 1 n n i=1 X j i Method of moments (MOM) estimator: match the theoretical moments and the sample moments and solve for parameters: Example: m 1 = E(X 1 θ), m 2 = E(X 2 1 θ),..., m k = E(X k 1 θ) Let X 1,..., X n be i.i.d. Gamma(α, β). Then E(X) = α β and E(X 2 ) = α(α + 1) β 2 Find the MOM estimator of α and β
38 Chapter 7 continued 7.7 Sufficient Statistics Sufficient Statistics A statistic: T = r(x 1,..., X n ) Def: Sufficient Statistics Let X 1,..., X n be a random sample form f (x θ) and let T be a statistic. If the conditional distribution of X 1,..., X n T = t does not depend on θ then T is called a sufficient statistic The idea: Just as good to have the observed sufficient statistic as it is to have the individual observations of X 1,..., X n Can limit our search for a good estimator to sufficient statistics 38 / 40
39 Chapter 7 continued 7.7 Sufficient Statistics Sufficient Statistics Theorem 7.7.1: Factorization Criterion Let X 1,..., X n be a random sample form f (x θ) where θ Ω is unknown. A statistic T = r(x 1,..., X n ) is a sufficient statistic for θ if and only if for all x R n and all θ Ω, the joint pdf/pf f n (x θ) can be factored as f n (x θ) = u(x)v (r(x), θ) where function u and v are nonnegative. The function u may depend on x but not on θ The function v depends on θ but depends on x only through the value of the statistic r(x) Both MLEs and Bayesian estimators depend on data only through sufficient statistics. 39 / 40
40 40 / 40 Chapter 7 continued END OF CHAPTER 7
Chapter 7: Estimation Sections
1 / 31 : Estimation Sections 7.1 Statistical Inference Bayesian Methods: 7.2 Prior and Posterior Distributions 7.3 Conjugate Prior Distributions 7.4 Bayes Estimators Frequentist Methods: 7.5 Maximum Likelihood
More informationChapter 7: Estimation Sections
Chapter 7: Estimation Sections 7.1 Statistical Inference Bayesian Methods: 7.2 Prior and Posterior Distributions 7.3 Conjugate Prior Distributions Frequentist Methods: 7.5 Maximum Likelihood Estimators
More informationChapter 8: Sampling distributions of estimators Sections
Chapter 8 continued Chapter 8: Sampling distributions of estimators Sections 8.1 Sampling distribution of a statistic 8.2 The Chi-square distributions 8.3 Joint Distribution of the sample mean and sample
More informationST440/550: Applied Bayesian Analysis. (5) Multi-parameter models - Summarizing the posterior
(5) Multi-parameter models - Summarizing the posterior Models with more than one parameter Thus far we have studied single-parameter models, but most analyses have several parameters For example, consider
More informationChapter 8: Sampling distributions of estimators Sections
Chapter 8: Sampling distributions of estimators Sections 8.1 Sampling distribution of a statistic 8.2 The Chi-square distributions 8.3 Joint Distribution of the sample mean and sample variance Skip: p.
More informationPosterior Inference. , where should we start? Consider the following computational procedure: 1. draw samples. 2. convert. 3. compute properties
Posterior Inference Example. Consider a binomial model where we have a posterior distribution for the probability term, θ. Suppose we want to make inferences about the log-odds γ = log ( θ 1 θ), where
More informationDefinition 9.1 A point estimate is any function T (X 1,..., X n ) of a random sample. We often write an estimator of the parameter θ as ˆθ.
9 Point estimation 9.1 Rationale behind point estimation When sampling from a population described by a pdf f(x θ) or probability function P [X = x θ] knowledge of θ gives knowledge of the entire population.
More informationLecture Notes 6. Assume F belongs to a family of distributions, (e.g. F is Normal), indexed by some parameter θ.
Sufficient Statistics Lecture Notes 6 Sufficiency Data reduction in terms of a particular statistic can be thought of as a partition of the sample space X. Definition T is sufficient for θ if the conditional
More informationNon-informative Priors Multiparameter Models
Non-informative Priors Multiparameter Models Statistics 220 Spring 2005 Copyright c 2005 by Mark E. Irwin Prior Types Informative vs Non-informative There has been a desire for a prior distributions that
More informationLecture 17: More on Markov Decision Processes. Reinforcement learning
Lecture 17: More on Markov Decision Processes. Reinforcement learning Learning a model: maximum likelihood Learning a value function directly Monte Carlo Temporal-difference (TD) learning COMP-424, Lecture
More informationCS340 Machine learning Bayesian statistics 3
CS340 Machine learning Bayesian statistics 3 1 Outline Conjugate analysis of µ and σ 2 Bayesian model selection Summarizing the posterior 2 Unknown mean and precision The likelihood function is p(d µ,λ)
More informationBack to estimators...
Back to estimators... So far, we have: Identified estimators for common parameters Discussed the sampling distributions of estimators Introduced ways to judge the goodness of an estimator (bias, MSE, etc.)
More informationTwo hours. To be supplied by the Examinations Office: Mathematical Formula Tables and Statistical Tables THE UNIVERSITY OF MANCHESTER
Two hours MATH20802 To be supplied by the Examinations Office: Mathematical Formula Tables and Statistical Tables THE UNIVERSITY OF MANCHESTER STATISTICAL METHODS Answer any FOUR of the SIX questions.
More informationConjugate priors: Beta and normal Class 15, Jeremy Orloff and Jonathan Bloom
1 Learning Goals Conjugate s: Beta and normal Class 15, 18.05 Jeremy Orloff and Jonathan Bloom 1. Understand the benefits of conjugate s.. Be able to update a beta given a Bernoulli, binomial, or geometric
More informationPoint Estimation. Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage
6 Point Estimation Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage Point Estimation Statistical inference: directed toward conclusions about one or more parameters. We will use the generic
More informationChapter 5. Statistical inference for Parametric Models
Chapter 5. Statistical inference for Parametric Models Outline Overview Parameter estimation Method of moments How good are method of moments estimates? Interval estimation Statistical Inference for Parametric
More informationConfidence Intervals Introduction
Confidence Intervals Introduction A point estimate provides no information about the precision and reliability of estimation. For example, the sample mean X is a point estimate of the population mean μ
More informationCS340 Machine learning Bayesian model selection
CS340 Machine learning Bayesian model selection Bayesian model selection Suppose we have several models, each with potentially different numbers of parameters. Example: M0 = constant, M1 = straight line,
More informationدرس هفتم یادگیري ماشین. (Machine Learning) دانشگاه فردوسی مشهد دانشکده مهندسی رضا منصفی
یادگیري ماشین توزیع هاي نمونه و تخمین نقطه اي پارامترها Sampling Distributions and Point Estimation of Parameter (Machine Learning) دانشگاه فردوسی مشهد دانشکده مهندسی رضا منصفی درس هفتم 1 Outline Introduction
More information6. Genetics examples: Hardy-Weinberg Equilibrium
PBCB 206 (Fall 2006) Instructor: Fei Zou email: fzou@bios.unc.edu office: 3107D McGavran-Greenberg Hall Lecture 4 Topics for Lecture 4 1. Parametric models and estimating parameters from data 2. Method
More informationExam 2 Spring 2015 Statistics for Applications 4/9/2015
18.443 Exam 2 Spring 2015 Statistics for Applications 4/9/2015 1. True or False (and state why). (a). The significance level of a statistical test is not equal to the probability that the null hypothesis
More information(5) Multi-parameter models - Summarizing the posterior
(5) Multi-parameter models - Summarizing the posterior Spring, 2017 Models with more than one parameter Thus far we have studied single-parameter models, but most analyses have several parameters For example,
More informationLecture 10: Point Estimation
Lecture 10: Point Estimation MSU-STT-351-Sum-17B (P. Vellaisamy: MSU-STT-351-Sum-17B) Probability & Statistics for Engineers 1 / 31 Basic Concepts of Point Estimation A point estimate of a parameter θ,
More informationCS 361: Probability & Statistics
March 12, 2018 CS 361: Probability & Statistics Inference Binomial likelihood: Example Suppose we have a coin with an unknown probability of heads. We flip the coin 10 times and observe 2 heads. What can
More informationBayesian course - problem set 3 (lecture 4)
Bayesian course - problem set 3 (lecture 4) Ben Lambert November 14, 2016 1 Ticked off Imagine once again that you are investigating the occurrence of Lyme disease in the UK. This is a vector-borne disease
More informationPractice Exercises for Midterm Exam ST Statistical Theory - II The ACTUAL exam will consists of less number of problems.
Practice Exercises for Midterm Exam ST 522 - Statistical Theory - II The ACTUAL exam will consists of less number of problems. 1. Suppose X i F ( ) for i = 1,..., n, where F ( ) is a strictly increasing
More informationReview for Final Exam Spring 2014 Jeremy Orloff and Jonathan Bloom
Review for Final Exam 18.05 Spring 2014 Jeremy Orloff and Jonathan Bloom THANK YOU!!!! JON!! PETER!! RUTHI!! ERIKA!! ALL OF YOU!!!! Probability Counting Sets Inclusion-exclusion principle Rule of product
More informationCommon one-parameter models
Common one-parameter models In this section we will explore common one-parameter models, including: 1. Binomial data with beta prior on the probability 2. Poisson data with gamma prior on the rate 3. Gaussian
More informationLikelihood Methods of Inference. Toss coin 6 times and get Heads twice.
Methods of Inference Toss coin 6 times and get Heads twice. p is probability of getting H. Probability of getting exactly 2 heads is 15p 2 (1 p) 4 This function of p, is likelihood function. Definition:
More informationConjugate Models. Patrick Lam
Conjugate Models Patrick Lam Outline Conjugate Models What is Conjugacy? The Beta-Binomial Model The Normal Model Normal Model with Unknown Mean, Known Variance Normal Model with Known Mean, Unknown Variance
More informationMulti-armed bandit problems
Multi-armed bandit problems Stochastic Decision Theory (2WB12) Arnoud den Boer 13 March 2013 Set-up 13 and 14 March: Lectures. 20 and 21 March: Paper presentations (Four groups, 45 min per group). Before
More informationGenerating Random Numbers
Generating Random Numbers Aim: produce random variables for given distribution Inverse Method Let F be the distribution function of an univariate distribution and let F 1 (y) = inf{x F (x) y} (generalized
More informationPoint Estimation. Some General Concepts of Point Estimation. Example. Estimator quality
Point Estimation Some General Concepts of Point Estimation Statistical inference = conclusions about parameters Parameters == population characteristics A point estimate of a parameter is a value (based
More informationMetropolis-Hastings algorithm
Metropolis-Hastings algorithm Dr. Jarad Niemi STAT 544 - Iowa State University March 27, 2018 Jarad Niemi (STAT544@ISU) Metropolis-Hastings March 27, 2018 1 / 32 Outline Metropolis-Hastings algorithm Independence
More informationPoint Estimation. Copyright Cengage Learning. All rights reserved.
6 Point Estimation Copyright Cengage Learning. All rights reserved. 6.2 Methods of Point Estimation Copyright Cengage Learning. All rights reserved. Methods of Point Estimation The definition of unbiasedness
More informationExtracting Information from the Markets: A Bayesian Approach
Extracting Information from the Markets: A Bayesian Approach Daniel Waggoner The Federal Reserve Bank of Atlanta Florida State University, February 29, 2008 Disclaimer: The views expressed are the author
More informationA New Hybrid Estimation Method for the Generalized Pareto Distribution
A New Hybrid Estimation Method for the Generalized Pareto Distribution Chunlin Wang Department of Mathematics and Statistics University of Calgary May 18, 2011 A New Hybrid Estimation Method for the GPD
More informationEE641 Digital Image Processing II: Purdue University VISE - October 29,
EE64 Digital Image Processing II: Purdue University VISE - October 9, 004 The EM Algorithm. Suffient Statistics and Exponential Distributions Let p(y θ) be a family of density functions parameterized by
More informationChapter 5. Sampling Distributions
Lecture notes, Lang Wu, UBC 1 Chapter 5. Sampling Distributions 5.1. Introduction In statistical inference, we attempt to estimate an unknown population characteristic, such as the population mean, µ,
More informationPoint Estimators. STATISTICS Lecture no. 10. Department of Econometrics FEM UO Brno office 69a, tel
STATISTICS Lecture no. 10 Department of Econometrics FEM UO Brno office 69a, tel. 973 442029 email:jiri.neubauer@unob.cz 8. 12. 2009 Introduction Suppose that we manufacture lightbulbs and we want to state
More informationProbability Theory. Probability and Statistics for Data Science CSE594 - Spring 2016
Probability Theory Probability and Statistics for Data Science CSE594 - Spring 2016 What is Probability? 2 What is Probability? Examples outcome of flipping a coin (seminal example) amount of snowfall
More informationINSTITUTE AND FACULTY OF ACTUARIES. Curriculum 2019 SPECIMEN EXAMINATION
INSTITUTE AND FACULTY OF ACTUARIES Curriculum 2019 SPECIMEN EXAMINATION Subject CS1A Actuarial Statistics Time allowed: Three hours and fifteen minutes INSTRUCTIONS TO THE CANDIDATE 1. Enter all the candidate
More informationStatistical estimation
Statistical estimation Statistical modelling: theory and practice Gilles Guillot gigu@dtu.dk September 3, 2013 Gilles Guillot (gigu@dtu.dk) Estimation September 3, 2013 1 / 27 1 Introductory example 2
More informationSTA 532: Theory of Statistical Inference
STA 532: Theory of Statistical Inference Robert L. Wolpert Department of Statistical Science Duke University, Durham, NC, USA 2 Estimating CDFs and Statistical Functionals Empirical CDFs Let {X i : i n}
More informationChapter 8. Introduction to Statistical Inference
Chapter 8. Introduction to Statistical Inference Point Estimation Statistical inference is to draw some type of conclusion about one or more parameters(population characteristics). Now you know that a
More informationChapter 4: Asymptotic Properties of MLE (Part 3)
Chapter 4: Asymptotic Properties of MLE (Part 3) Daniel O. Scharfstein 09/30/13 1 / 1 Breakdown of Assumptions Non-Existence of the MLE Multiple Solutions to Maximization Problem Multiple Solutions to
More informationIEOR E4602: Quantitative Risk Management
IEOR E4602: Quantitative Risk Management Basic Concepts and Techniques of Risk Management Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com
More informationBayesian Normal Stuff
Bayesian Normal Stuff - Set-up of the basic model of a normally distributed random variable with unknown mean and variance (a two-parameter model). - Discuss philosophies of prior selection - Implementation
More informationIntro to Decision Theory
Intro to Decision Theory Rebecca C. Steorts Bayesian Methods and Modern Statistics: STA 360/601 Lecture 3 1 Please be patient with the Windows machine... 2 Topics Loss function Risk Posterior Risk Bayes
More informationA potentially useful approach to model nonlinearities in time series is to assume different behavior (structural break) in different subsamples
1.3 Regime switching models A potentially useful approach to model nonlinearities in time series is to assume different behavior (structural break) in different subsamples (or regimes). If the dates, the
More informationSection 0: Introduction and Review of Basic Concepts
Section 0: Introduction and Review of Basic Concepts Carlos M. Carvalho The University of Texas McCombs School of Business mccombs.utexas.edu/faculty/carlos.carvalho/teaching 1 Getting Started Syllabus
More informationGOV 2001/ 1002/ E-200 Section 3 Inference and Likelihood
GOV 2001/ 1002/ E-200 Section 3 Inference and Likelihood Anton Strezhnev Harvard University February 10, 2016 1 / 44 LOGISTICS Reading Assignment- Unifying Political Methodology ch 4 and Eschewing Obfuscation
More informationMVE051/MSG Lecture 7
MVE051/MSG810 2017 Lecture 7 Petter Mostad Chalmers November 20, 2017 The purpose of collecting and analyzing data Purpose: To build and select models for parts of the real world (which can be used for
More informationCSE 312 Winter Learning From Data: Maximum Likelihood Estimators (MLE)
CSE 312 Winter 2017 Learning From Data: Maximum Likelihood Estimators (MLE) 1 Parameter Estimation Given: independent samples x1, x2,..., xn from a parametric distribution f(x θ) Goal: estimate θ. Not
More informationChapter 6: Point Estimation
Chapter 6: Point Estimation Professor Sharabati Purdue University March 10, 2014 Professor Sharabati (Purdue University) Point Estimation Spring 2014 1 / 37 Chapter Overview Point estimator and point estimate
More informationSTA 114: Statistics. Notes 10. Conjugate Priors
STA 114: Statistics Notes 10. Conjugate Priors Conjugate family Once we get a /pmf ξ(θ x) by combining a model X f(x θ) with a /pmf ξ(θ) on θ Θ, a report can be made by summarizing the. It helps to have
More informationAdaptive Experiments for Policy Choice. March 8, 2019
Adaptive Experiments for Policy Choice Maximilian Kasy Anja Sautmann March 8, 2019 Introduction The goal of many experiments is to inform policy choices: 1. Job search assistance for refugees: Treatments:
More informationSTAT 111 Recitation 3
STAT 111 Recitation 3 Linjun Zhang stat.wharton.upenn.edu/~linjunz/ September 23, 2017 Misc. The unpicked-up homeworks will be put in the STAT 111 box in the Stats Department lobby (It s on the 4th floor
More information1 Bayesian Bias Correction Model
1 Bayesian Bias Correction Model Assuming that n iid samples {X 1,...,X n }, were collected from a normal population with mean µ and variance σ 2. The model likelihood has the form, P( X µ, σ 2, T n >
More informationVersion A. Problem 1. Let X be the continuous random variable defined by the following pdf: 1 x/2 when 0 x 2, f(x) = 0 otherwise.
Math 224 Q Exam 3A Fall 217 Tues Dec 12 Version A Problem 1. Let X be the continuous random variable defined by the following pdf: { 1 x/2 when x 2, f(x) otherwise. (a) Compute the mean µ E[X]. E[X] x
More informationChapter 7: Point Estimation and Sampling Distributions
Chapter 7: Point Estimation and Sampling Distributions Seungchul Baek Department of Statistics, University of South Carolina STAT 509: Statistics for Engineers 1 / 20 Motivation In chapter 3, we learned
More informationInterval estimation. September 29, Outline Basic ideas Sampling variation and CLT Interval estimation using X More general problems
Interval estimation September 29, 2017 STAT 151 Class 7 Slide 1 Outline of Topics 1 Basic ideas 2 Sampling variation and CLT 3 Interval estimation using X 4 More general problems STAT 151 Class 7 Slide
More informationMachine Learning for Quantitative Finance
Machine Learning for Quantitative Finance Fast derivative pricing Sofie Reyners Joint work with Jan De Spiegeleer, Dilip Madan and Wim Schoutens Derivative pricing is time-consuming... Vanilla option pricing
More information3 ˆθ B = X 1 + X 2 + X 3. 7 a) Find the Bias, Variance and MSE of each estimator. Which estimator is the best according
STAT 345 Spring 2018 Homework 9 - Point Estimation Name: Please adhere to the homework rules as given in the Syllabus. 1. Mean Squared Error. Suppose that X 1, X 2 and X 3 are independent random variables
More informationSimulation Wrap-up, Statistics COS 323
Simulation Wrap-up, Statistics COS 323 Today Simulation Re-cap Statistics Variance and confidence intervals for simulations Simulation wrap-up FYI: No class or office hours Thursday Simulation wrap-up
More informationIEOR E4703: Monte-Carlo Simulation
IEOR E4703: Monte-Carlo Simulation Simulation Efficiency and an Introduction to Variance Reduction Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University
More informationStochastic Claims Reserving _ Methods in Insurance
Stochastic Claims Reserving _ Methods in Insurance and John Wiley & Sons, Ltd ! Contents Preface Acknowledgement, xiii r xi» J.. '..- 1 Introduction and Notation : :.... 1 1.1 Claims process.:.-.. : 1
More informationSTAT 509: Statistics for Engineers Dr. Dewei Wang. Copyright 2014 John Wiley & Sons, Inc. All rights reserved.
STAT 509: Statistics for Engineers Dr. Dewei Wang Applied Statistics and Probability for Engineers Sixth Edition Douglas C. Montgomery George C. Runger 7 Point CHAPTER OUTLINE 7-1 Point Estimation 7-2
More informationUsing Monte Carlo Integration and Control Variates to Estimate π
Using Monte Carlo Integration and Control Variates to Estimate π N. Cannady, P. Faciane, D. Miksa LSU July 9, 2009 Abstract We will demonstrate the utility of Monte Carlo integration by using this algorithm
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Monte Carlo Methods Mark Schmidt University of British Columbia Winter 2018 Last Time: Markov Chains We can use Markov chains for density estimation, p(x) = p(x 1 ) }{{} d p(x
More informationThe Bernoulli distribution
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this
More informationIntroduction to Sequential Monte Carlo Methods
Introduction to Sequential Monte Carlo Methods Arnaud Doucet NCSU, October 2008 Arnaud Doucet () Introduction to SMC NCSU, October 2008 1 / 36 Preliminary Remarks Sequential Monte Carlo (SMC) are a set
More informationCSC 411: Lecture 08: Generative Models for Classification
CSC 411: Lecture 08: Generative Models for Classification Richard Zemel, Raquel Urtasun and Sanja Fidler University of Toronto Zemel, Urtasun, Fidler (UofT) CSC 411: 08-Generative Models 1 / 23 Today Classification
More informationPoint Estimation. Principle of Unbiased Estimation. When choosing among several different estimators of θ, select one that is unbiased.
Point Estimation Point Estimation Definition A point estimate of a parameter θ is a single number that can be regarded as a sensible value for θ. A point estimate is obtained by selecting a suitable statistic
More informationEstimation after Model Selection
Estimation after Model Selection Vanja M. Dukić Department of Health Studies University of Chicago E-Mail: vanja@uchicago.edu Edsel A. Peña* Department of Statistics University of South Carolina E-Mail:
More informationECE 295: Lecture 03 Estimation and Confidence Interval
ECE 295: Lecture 03 Estimation and Confidence Interval Spring 2018 Prof Stanley Chan School of Electrical and Computer Engineering Purdue University 1 / 23 Theme of this Lecture What is Estimation? You
More informationPart II: Computation for Bayesian Analyses
Part II: Computation for Bayesian Analyses 62 BIO 233, HSPH Spring 2015 Conjugacy In both birth weight eamples the posterior distribution is from the same family as the prior: Prior Likelihood Posterior
More informationSTAT 425: Introduction to Bayesian Analysis
STAT 45: Introduction to Bayesian Analysis Marina Vannucci Rice University, USA Fall 018 Marina Vannucci (Rice University, USA) Bayesian Analysis (Part 1) Fall 018 1 / 37 Lectures 9-11: Multi-parameter
More informationSTART HERE: Instructions. 1 Exponential Family [Zhou, Manzil]
START HERE: Instructions Thanks a lot to John A.W.B. Constanzo and Shi Zong for providing and allowing to use the latex source files for quick preparation of the HW solution. The homework was due at 9:00am
More informationM.Sc. ACTUARIAL SCIENCE. Term-End Examination
No. of Printed Pages : 15 LMJA-010 (F2F) M.Sc. ACTUARIAL SCIENCE Term-End Examination O CD December, 2011 MIA-010 (F2F) : STATISTICAL METHOD Time : 3 hours Maximum Marks : 100 SECTION - A Attempt any five
More informationChapter 4: Commonly Used Distributions. Statistics for Engineers and Scientists Fourth Edition William Navidi
Chapter 4: Commonly Used Distributions Statistics for Engineers and Scientists Fourth Edition William Navidi 2014 by Education. This is proprietary material solely for authorized instructor use. Not authorized
More informationUQ, STAT2201, 2017, Lectures 3 and 4 Unit 3 Probability Distributions.
UQ, STAT2201, 2017, Lectures 3 and 4 Unit 3 Probability Distributions. Random Variables 2 A random variable X is a numerical (integer, real, complex, vector etc.) summary of the outcome of the random experiment.
More informationChapter 7 presents the beginning of inferential statistics. The two major activities of inferential statistics are
Chapter 7 presents the beginning of inferential statistics. Concept: Inferential Statistics The two major activities of inferential statistics are 1 to use sample data to estimate values of population
More informationBIO5312 Biostatistics Lecture 5: Estimations
BIO5312 Biostatistics Lecture 5: Estimations Yujin Chung September 27th, 2016 Fall 2016 Yujin Chung Lec5: Estimations Fall 2016 1/34 Recap Yujin Chung Lec5: Estimations Fall 2016 2/34 Today s lecture and
More informationIntroduction to the Maximum Likelihood Estimation Technique. September 24, 2015
Introduction to the Maximum Likelihood Estimation Technique September 24, 2015 So far our Dependent Variable is Continuous That is, our outcome variable Y is assumed to follow a normal distribution having
More informationChapter 7 - Lecture 1 General concepts and criteria
Chapter 7 - Lecture 1 General concepts and criteria January 29th, 2010 Best estimator Mean Square error Unbiased estimators Example Unbiased estimators not unique Special case MVUE Bootstrap General Question
More informationAn Introduction to Bayesian Inference and MCMC Methods for Capture-Recapture
An Introduction to Bayesian Inference and MCMC Methods for Capture-Recapture Trinity River Restoration Program Workshop on Outmigration: Population Estimation October 6 8, 2009 An Introduction to Bayesian
More informationUnit 5: Sampling Distributions of Statistics
Unit 5: Sampling Distributions of Statistics Statistics 571: Statistical Methods Ramón V. León 6/12/2004 Unit 5 - Stat 571 - Ramon V. Leon 1 Definitions and Key Concepts A sample statistic used to estimate
More informationUnit 5: Sampling Distributions of Statistics
Unit 5: Sampling Distributions of Statistics Statistics 571: Statistical Methods Ramón V. León 6/12/2004 Unit 5 - Stat 571 - Ramon V. Leon 1 Definitions and Key Concepts A sample statistic used to estimate
More informationLearning From Data: MLE. Maximum Likelihood Estimators
Learning From Data: MLE Maximum Likelihood Estimators 1 Parameter Estimation Assuming sample x1, x2,..., xn is from a parametric distribution f(x θ), estimate θ. E.g.: Given sample HHTTTTTHTHTTTHH of (possibly
More information15 : Approximate Inference: Monte Carlo Methods
10-708: Probabilistic Graphical Models 10-708, Spring 2016 15 : Approximate Inference: Monte Carlo Methods Lecturer: Eric P. Xing Scribes: Binxuan Huang, Yotam Hechtlinger, Fuchen Liu 1 Introduction to
More informationSTAT 111 Recitation 4
STAT 111 Recitation 4 Linjun Zhang http://stat.wharton.upenn.edu/~linjunz/ September 29, 2017 Misc. Mid-term exam time: 6-8 pm, Wednesday, Oct. 11 The mid-term break is Oct. 5-8 The next recitation class
More informationProbability Theory and Simulation Methods. April 9th, Lecture 20: Special distributions
April 9th, 2018 Lecture 20: Special distributions Week 1 Chapter 1: Axioms of probability Week 2 Chapter 3: Conditional probability and independence Week 4 Chapters 4, 6: Random variables Week 9 Chapter
More informationTwo hours UNIVERSITY OF MANCHESTER. 23 May :00 16:00. Answer ALL SIX questions The total number of marks in the paper is 90.
Two hours MATH39542 UNIVERSITY OF MANCHESTER RISK THEORY 23 May 2016 14:00 16:00 Answer ALL SIX questions The total number of marks in the paper is 90. University approved calculators may be used 1 of
More informationThis is a open-book exam. Assigned: Friday November 27th 2009 at 16:00. Due: Monday November 30th 2009 before 10:00.
University of Iceland School of Engineering and Sciences Department of Industrial Engineering, Mechanical Engineering and Computer Science IÐN106F Industrial Statistics II - Bayesian Data Analysis Fall
More informationChapter 5. Continuous Random Variables and Probability Distributions. 5.1 Continuous Random Variables
Chapter 5 Continuous Random Variables and Probability Distributions 5.1 Continuous Random Variables 1 2CHAPTER 5. CONTINUOUS RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS Probability Distributions Probability
More informationApplied Statistics I
Applied Statistics I Liang Zhang Department of Mathematics, University of Utah July 14, 2008 Liang Zhang (UofU) Applied Statistics I July 14, 2008 1 / 18 Point Estimation Liang Zhang (UofU) Applied Statistics
More informationExercise. Show the corrected sample variance is an unbiased estimator of population variance. S 2 = n i=1 (X i X ) 2 n 1. Exercise Estimation
Exercise Show the corrected sample variance is an unbiased estimator of population variance. S 2 = n i=1 (X i X ) 2 n 1 Exercise S 2 = = = = n i=1 (X i x) 2 n i=1 = (X i µ + µ X ) 2 = n 1 n 1 n i=1 ((X
More informationPractice Exam 1. Loss Amount Number of Losses
Practice Exam 1 1. You are given the following data on loss sizes: An ogive is used as a model for loss sizes. Determine the fitted median. Loss Amount Number of Losses 0 1000 5 1000 5000 4 5000 10000
More informationMonte-Carlo Planning: Introduction and Bandit Basics. Alan Fern
Monte-Carlo Planning: Introduction and Bandit Basics Alan Fern 1 Large Worlds We have considered basic model-based planning algorithms Model-based planning: assumes MDP model is available Methods we learned
More information