Modern Methods of Data Analysis - SS 2009

Transcription

1 Modern Methods of Data Analysis Lecture II ( ) Contents: Characterize data samples Characterize distributions Correlations, covariance

2 Reminder: Average of a Sample arithmetic mean of data set: weighted mean of data set: mode most prob. value (peak in distribution, not unique) median smallest value which is 50% of events` better use median than mean, more robust against outliers! similar defined Quantile: Median = 50% Quantil truncated mean: useful if the underlying distribution is expected to be asymmetric

3 Measure the Spread of a Sample How to characterize width/spread? First thought... mean deviation from the mean: Could consider average absolute deviation: However hard to handle mathematically.

4 Sample Variance Way better quantity: mean square deviation called sample variance s² or V For any random variable :

5 Sample Variance For data analysis, preferably loop only once over data: mean square square of the mean

6 Sample Variance For large numbers, safer to shift distribution by estimated mean :

7 Standard Deviation (RMS), FWHM standard deviation σ or RMS: root mean squared [ standard is a joke, there are several standards in literature...] FWHM: full width at half maximum more robust against outliers, fluctuations harder at low statistics; for Gaussian distributed events: FWHM = 2.35σ

8 Example: Give sample variance, RMS and FWHM:

9 Expectation Values So far characterized given set realization of an experiment (sum over N) by sample mean, sample spread... Now talk about mean, spread of a distribution: Note However for N->, Law of large numbers

10 Variance of a Distribution: V[x] = E[(x-μ)²] = V[x] = V[x] = E[x²] µ² f(x): PDF V[x] is the measure of the spread of the distribution, not how well the mean is measured!

11 Example: N = 100 N = N = 1000 µ=5 σ=1

12 How to determine uncertainty on the mean? E[ x ] =??? V[ x ] =???

13 Expectation Value of sample mean

14 Variance of the Sample Mean

15 m(b0) = ± 0.53 (stat) ± 0.33 (sys) CDF has a mass resolution of 16 MeV: the reconstructed mass of a single B meson is spread around the true B mass with σ=16 MeV The B mass can be measured with way better precision

16 Unbiased Estimators: Unbiased Estimator erwartungstreuer Schätzer unbiased estimator for true mean µ is : for n data points, we estimate the true variance V(x) by the sample variance s² - if true mean µ is known! - If the true mean is unknown, then an unbiased estimator for the variance σ² is the sample variance s² : beware of N-1! One single value is not enough to determine mean and spread.

17 Solution: Unbiased Estimator for V(x)

18 Solution: Unbiased Estimators for V(x)

19 Efficiency of Estimators Optimal Estimator: optimal smallest variance (Likelihood maximization gives optimal estimator, will be proven in later lecture) Efficiency of Estimator: variance of optimal estimator/variance of estimator For Gaussian distribution non optimal estimators are called not robust E.g. Median of Gauss distribution has 64% efficiency is optimal estimator

20 Symmetric truncated Mean truncated mean ( getrimmter Mittelwert ): e.g. r = 40% truncated mean: 10% lowest and 10% highest values ignored, calculate mean of 80% central values r = 50% truncated mean -> arithmetic mean r -> 0% -> median

21 Laplace or double exponential efficiency Cauchy r = 0.23 truncated mean best estimator for unkown sym. distribution r

22 Moments r-th algebraic moment r-th central moment Expectation value: 1. algebraic moment Variance: 2. central moment Schiefe /skewness - pos. for right winged distributions Wölbung /kurtosis - measure for ratio of core relative to tails - pos. kurtosis: longer tails than Gaussian

23 Skewness & Kurtosis kurtosis < 0 kurtosis > 0 Gaussian distribution have kurtosis = 0

24 Which fraction of events is within 1,2,3 σ 1σ 2σ 3σ 4σ This is only true for Gaussian distributions!

25 Biennaymé-Tchebycheff-Inequality For every distribution the following inequality is valid: k Gauss Tchebycheff

26 Solution: Biennaymé-Tchebycheff-Inequality Given a PDF f(x) and a function positive w(x) 0: with :

27 Two Dimensional Distributions Multiple ways to visualize 2-dim distributions box plot lego plot surface plot numbers scatter plot color map contour plot...

28 Two dimensional Distributions straight generalization of 1-dim PDFs A 2-dim PDF is a function f(x,y) 0 with

29 Marginal Distributions Marginal distributions: projection on the axis Randverteilungen

30 Conditional Probability

31 Exercise: Compute Compute