Descriptive Statistics Bios 662

Transcription

1 Descriptive Statistics Bios 662 Michael G. Hudgens, Ph.D. mhudgens :51 BIOS Descriptive Statistics

2 Descriptive Statistics Types of variables Measures of location Measures of spread, shape Data displays BIOS Descriptive Statistics

3 Types of Variables A variable is a quantity that may vary from object to object A sample or data set is a collection of values of one or more variables. Types of variables Quantitative variable intrinsically numerical e.g. age, height, counts Qualitative (categorical) - intrinsically nonnumerical e.g. gender, province, country BIOS Descriptive Statistics

4 Types of Variables Qualitative (categorical) - intrinsically nonnumerical Binary, dichotomous e.g., alive/dead, female/male Ordinal - natural ordering e.g., diagnosis (certain, probable, unlikely,...) e.g., attitude (strongly agree, agree, neutral,...) Nominal - no natural ordering e.g., religion, race In recording qualitative data, numerical values may be assigned BIOS Descriptive Statistics

5 Descriptive Statistics Types of variables Measures of location Measures of spread, shape Data displays BIOS Descriptive Statistics

6 Measures of Location (Arithmetic) Mean Percentiles Median Mode Geometric mean BIOS Descriptive Statistics

7 Arithmetic mean Data: x 1, x 2,..., x n Mean: x = x 1 + x x n n = 1 n n i=1 x i BIOS Descriptive Statistics

8 Example Duration of hospital stay in days: x 1 = 5, x 2 = 10, x 3 = 6, x 4 = 11 Mean: x = 1 32 ( ) = 4 4 = 8 BIOS Descriptive Statistics

9 Reporting of decimals Report mean with one more significant digit than the observations Example: If x is measured in whole numbers and x = 6.345, report x = 6.3 BIOS Descriptive Statistics

10 Let c be any constant Properties of Mean If then If then y i = x i + c for i = 1, 2, 3,..., n, ȳ = x + c y i = cx i for i = 1, 2, 3,..., n, ȳ = c x BIOS Descriptive Statistics

11 Properties of Mean - Example A sample of birth weights in a hospital found 1 oz = g ȳ = grams Therefore the mean in ozs. is x = ȳ = BIOS Descriptive Statistics

12 Order statistics Data: x 1, x 2,..., x n Order data from smallest to largest x (1) x (2) x (n) x (1), x (2),..., x (n) are order statisitics Note x (1) = min{x 1, x 2,..., x n } x (n) = max{x 1, x 2,..., x n } BIOS Descriptive Statistics

13 Example Duration of hospital stay in days: x 1 = 5, x 2 = 10, x 3 = 6, x 4 = 11 Order statistics: x (1) = 5, x (2) = 6, x (3) = 10, x (4) = 11 BIOS Descriptive Statistics

14 Percentiles Intuitive definition: the x percentile is such that x% of the observations are less than that value Also known as sample quantile BIOS Descriptive Statistics

15 Percentiles: Text definition The (p 100) th percentile of a sample y (np+p) if np + p is an integer ˆζ p = {y ( np+p ) + y ( np+p ) }/2 otherwise for 0 < p < 1 Note: y is the greatest integer y; i.e., the floor function y is the smallest integer y; i.e., the ceiling function Cf Def 3.11 of text BIOS Descriptive Statistics

16 Percentiles: General form General form (Hyndman and Fan, Am Stat 1996) ˆζ p = (1 γ)y (j) + γy (j+1) where j = pn + m for some m R and 0 γ 1. Let g = pn + m j If m = p and γ = 0 if g = 0 1/2 if g > 0 then j = pn + p and we recover text definition BIOS Descriptive Statistics

17 Percentiles: Software SAS Proc Univariate: 5 definitions of percentile R: 9 definitions Claim: none of these match the book definition BIOS Descriptive Statistics

18 R quantile() function >?quantile quantile package:stats R Documentation Sample Quantiles Description: The generic function quantile produces sample quantiles corresponding to the given probabilities. The smallest observation corresponds to a probability of 0 and the largest to a probability of 1. Usage: quantile(x,...) ## Default S3 method: quantile(x, probs = seq(0, 1, 0.25), na.rm = FALSE, names = TRUE, type = 7,...) Arguments: BIOS Descriptive Statistics

19 x: numeric vectors whose sample quantiles are wanted. probs: numeric vector of probabilities with values in [0,1]. na.rm: logical; if true, any NA and NaN s are removed from x before the quantiles are computed. names: logical; if true, the result has a names attribute. FALSE for speedup with many probs. Set to type: an integer between 1 and 9 selecting one of the nine quantile algorithms detailed below to be used....: further arguments passed to or from other methods. Types: quantile returns estimates of underlying distribution quantiles based on one or two order statistics from the supplied elements in x at probabilities in probs. One of the nine quantile algorithms discussed in Hyndman and Fan (1996), selected by type, is employed. BIOS Descriptive Statistics

20 Percentiles: Class Definition The (p 100) th percentile of a sample: y ( np +1) if np is not an integer ˆζ p = {y (np) + y (np+1) }/2 if np is an integer for 0 < p < 1 Defintion 2 of R/Hyndman and Fan: m = 0 and 1 if g > 0 γ = 1/2 if g = 0 Defintion 5 of SAS BIOS Descriptive Statistics

21 Example Suppose n = 278 and we want the 75th percentile R such that > x <- 1:278 > quantile(x,.75,type=2) 75% 209 np = = ˆζ.75 = x (209) BIOS Descriptive Statistics

22 Example: SAS data; infile "H:/WWW/bios/662/2007fall/percentile.txt"; input x; proc univariate; var x; run; The UNIVARIATE Procedure Variable: x Quantiles (Definition 5) Quantile Estimate 75% Q % Median % Q % % % 3.0 0% Min 1.0 BIOS Descriptive Statistics

23 Median The sample median is the 50th percentile y ( n+1 2 ˆζ ) if n is odd.5 = {y (n/2) + y (n/2+1) }/2 if n is even for 0 < p < 1 BIOS Descriptive Statistics

24 Example Duration of hospital stay in days: x 1 = 5, x 2 = 10, x 3 = 6, x 4 = 11 Median: ˆζ.5 = {x (2) + x (3) }/2 = (6 + 10)/2 = 8 BIOS Descriptive Statistics

25 Mode The mode is the most frequently occurring value in the data set E.g., if then mode is 11 x 1 = 5, x 2 = 11, x 3 = 6, x 4 = 11 BIOS Descriptive Statistics

26 Geometric Mean Data: x 1, x 2,..., x n The geometric mean of x is x g = (x 1 x 2 x n ) 1/n Let y i = log(x i ) for i = 1, 2,..., n. Then x g = exp(ȳ) x g is used when data are of the form c k Eg, suppose x 1 = 10 and x 2 = 0.1. Then x g = 1 BIOS Descriptive Statistics

27 Comments Mean is most often used measure Median is better if there are influential observations (more robust to extreme values) Mode rarely used (exception: nominal data) BIOS Descriptive Statistics

28 Example Duration of hospital stay in days: x 1 = 5, x 2 = 10, x 3 = 6, x 4 = 11 ˆζ.5 = x = 8, x g = 7.6 Alter last observation: x 1 = 5, x 2 = 10, x 3 = 6, x 4 = 50 ˆζ.5 = 8, x = 17.7, x g = 11.1 BIOS Descriptive Statistics

29 Descriptive Statistics Types of variables Measures of location Measures of spread, shape Data displays BIOS Descriptive Statistics

30 Measures of Spread, Shape Range Variance and standard deviation Interquartile range Skewness, Kurtosis BIOS Descriptive Statistics

31 Range Range: r a = x (n) x (1) Easy to calculate Sensitive to unusual observations (outliers) Usually, the larger n is, the larger r a BIOS Descriptive Statistics

32 Sample Variance and Standard Deviation Want to measure deviation from mean Sample variance s 2 = 1 n 1 n i=1 (x i x) 2 = 1 n 1 n i=1 x 2 i n x2 Sample standard deviation s = s 2 BIOS Descriptive Statistics

33 Sample Variance and Standard Deviation An alternative form of the sample variance is s 2 1 = 1 n (x n i x) 2 i=1 Can show s 2 is unbiased for population variance σ 2, however E(s 2 1 ) = σ2 σ2 n van Belle et al. argue for s 2 based on d.f. (Note 3.5) BIOS Descriptive Statistics

34 Sample Standard Deviation The units of s are the same as the units of x i If s is large, the data are spread over a wide range Report the standard deviation with two more significant digits than the original observations BIOS Descriptive Statistics

35 Properties of the Standard Deviation If c is a constant and y i = x i + c, then s y = s x If then y i = cx i s y = cs x BIOS Descriptive Statistics

36 Some approximations The interval x ± s will contain approx 68% of the observations The interval x ± 2s will contain approx 95% of the observations Approx s by Note s ˆζ.75 ˆζ ˆζ.75 ˆζ.25 is called interquartile range BIOS Descriptive Statistics

37 Symmetry and Skewness Informally, define symmetry to indicate having a uniform or even distribution about the mean If a distribution is symmetric, mean=median Data sets that are not symmetric are said to be skewed Skewness is a measurement of the degree to which a data set is skewed BIOS Descriptive Statistics

38 Skewness Define rth sample moment about the mean m r = i (y i ȳ) r for r = 1, 2, 3,... n Text definition of sample skewness: a 3 = m 3 (m 2 ) 3/2 = i (y i ȳ) 3 /n { i (y i ȳ) 2 /n} 3/2 = n i (y i ȳ) 3 { i (y i ȳ) 2 } 3/2 Typo in text page 51 SAS Proc Univariate VARDEF=N BIOS Descriptive Statistics

39 Interpretation? Text: skewed to the right if mean is greater than mode Values of a 3 > 0 indicate... skewness to the right However, for {0, 2, 2, 3, 4} x = 2.2, mode equals 2, and skewness equals BIOS Descriptive Statistics

40 Alternative Definitions Another definition of skewness: b 3 = n n 1 i (y i ȳ) 3 n 2 { i (y i ȳ) 2 } 3/2 Default in SAS Many more definitions; cf Joanes and Gill (JRSS D 1998) BIOS Descriptive Statistics

41 Kurtosis Kurtosis is a measure of the flatness or peakedness of a distribution; degree of archedness; thickness of tails Text definition of sample kurtosis: a 4 = m 4 (m 2 ) 2 = i (y i ȳ) 4 /n i { i (y i ȳ) 2 /n} 2 = n (y i ȳ) 4 { i (y i ȳ) 2 } 2 Typo in text page 51 BIOS Descriptive Statistics

42 Kurtosis: SAS Proc Univariate VARDEF=N a 4 = 1 ( ) y i ȳ 4 3 n s i.e., i.e., Why minus 3? a 4 = (yi ȳ) 4 /n s 4 3 a 4 = m 4 (m 2 ) 2 3 BIOS Descriptive Statistics

43 Descriptive Statistics Types of variables Measures of location Measures of spread, shape Data displays BIOS Descriptive Statistics

44 Data display Simplest form is a line listing A frequency table gives the frequency of observations within a set of ordered intervals Intervals should be mutually exclusive and exhaustive 8 to 10 intervals is usually sufficient With the exception of the end intervals, the length of the intervals should be constant BIOS Descriptive Statistics

45 Frequency Table - Example: Table 3.6 Blood Pressure Native 1st 2nd < > Total BIOS Descriptive Statistics

46 Frequency Tables Table on previous slide example of empirical frequency distribution Difficult to compare blood pressure distributions due to different sample sizes Divide by sample size to get empirical relative frequency distribution BIOS Descriptive Statistics

47 ERFD - Example: Table 3.7 Blood Pressure Native 1st 2nd < > Total BIOS Descriptive Statistics

48 Empirical distribution function Def 3.9 The empirical cumulative distribution of a variable is a listing of the variable with the proportion of observations less than or equal to that value (cumulative proportion) Aka empirical distribution function (EDF) Does not necessarily entail binning BIOS Descriptive Statistics

49 ECD - Example Blood Pressure Native 1st 2nd < < Total BIOS Descriptive Statistics

50 ECD - Example ECD Native First Second ECD BP BP ECD ECD BP BP BIOS Descriptive Statistics

51 Graphs ECD/EDF Histogram Stem and leaf plot Box plot Trellis/conditional plots BIOS Descriptive Statistics

52 Histogram Data are divided into intervals as in a frequency table A histogram is a bar graph with the area of each bar equal to the relative frequency in the interval. Can compare histograms from samples of different size Intervals need not be the same width Beware effect of choice of interval width (Fig 3.1 text) BIOS Descriptive Statistics

53 > par(mfcol=c(1,2)) Histogram: Example (Fig 3.1 text) > hist(liver$albumin,col="gray",xlab="albumin (mg/dl)",breaks=7,freq=f,main="") > hist(liver$albumin,col="gray",xlab="albumin (mg/dl)",breaks=30,freq=f,main="") Density Density Albumin (mg/dl) Albumin (mg/dl) BIOS Descriptive Statistics

54 Sample Kurtosis Kurtosis= 1.79 Kurtosis= 2.98 Density Density x x Kurtosis= 4.09 Kurtosis= 7.96 Density Density x x BIOS Descriptive Statistics

55 Stem and Leaf Plot Stem consists of leading digits Leaves consist of last digit Example: x=496, stem=49, leaf=6 Make a column of stems from smallest to largest To the right of each stem, list in a row the leaves, in ascending order. Note: there will be one leaf for each observation BIOS Descriptive Statistics

56 Stem and Leaf Plot: Example > stem(liver$albumin) The decimal point is 1 digit(s) to the left of the BIOS Descriptive Statistics

57 > stem(liver$albumin,width=100) Stem and Leaf Plot: Example The decimal point is 1 digit(s) to the left of the BIOS Descriptive Statistics

58 > stem(liver$albumin,scale=2) The decimal point is 1 digit(s) to the left of the BIOS Descriptive Statistics

59 Stem and Leaf Advantage: visualize all (or almost all) of the data Disadvantage: loss of ordering of data set BIOS Descriptive Statistics

60 Box plot The top of the box is the 75th percentile (ˆζ.75 ); the bottom is the 25th percentile (ˆζ.25 ) A line through the box is drawn at the median BIOS Descriptive Statistics

61 Box plot The lines extending out of the box (whiskers) may extend to the 90th and 10th percentiles the largest and smallest values largest observation ˆζ x IQR; smallest observation ˆζ x IQR (text is wrong! cf Tukey 1977, Chambers et al 1983) Data beyond whiskers may be plotted individually BIOS Descriptive Statistics

62 Box plot: Example > boxplot(liver$albumin) BIOS Descriptive Statistics

63 Box plot What proportion of the data should we expect to be between the whiskers? If data normal, 95-98% for 6 n 20, 99% for n > 20 Ref: Hoaglin et al. (JASA 1986) Note so whiskers cover 1.5 IQR 1.5(1.35)s 2s ˆζ.5 ± 2.68s BIOS Descriptive Statistics

64 Box plot and Histogram Example Density Density x x BIOS Descriptive Statistics

65 Multivariate plots Describe relationships/associations between more than one variable Scatterplots Simple for two variables Add color, symbols for > 2 variables Trellis/conditional plots BIOS Descriptive Statistics

66 Scatterplot Example I y x BIOS Descriptive Statistics

67 Scatterplot Example II y z=0 z= x BIOS Descriptive Statistics

68 Trellis plots Solar.R Ozone Temperature Temperature Temperature Temperature BIOS Descriptive Statistics

69 Table or graph? Tables best suited for looking up specific information Graphs better for perceiving trends, making comparisons and predictions Ref Gelman et al (Amer Stat 2002) BIOS Descriptive Statistics