Frequency Distribution and Summary Statistics

Transcription

1 Frequency Distribution and Summary Statistics Dongmei Li Department of Public Health Sciences Office of Public Health Studies University of Hawai i at Mānoa

2 Outline 1. Stemplot 2. Frequency table 3. Summary statistics 2

3 1. Stem-and-leaf plots (stemplots) Always start by looking at the data with graphs and plots Our favorite technique for looking at a single variable is the stemplot A stemplot is a graphical technique that organizes data into a histogram-like display You can observe a lot by looking Yogi Berra 3

4 Stemplot Illustrative Example Select an SRS of 10 ages List data as an ordered array Divide each data point into a stem-value and leaf-value In this example the tens place will be the stem-value and the ones place will be the leaf value, e.g., 21 has a stem value of 2 and leaf value of 1 4

5 Stemplot illustration (cont.) Draw an axis for the stem-values: axis multiplier (important!) Place leaves next to their stem value 21 plotted (animation) 5

6 Stemplot illustration continued Plot all data points and rearrange in rank order: Here is the plot horizontally: (for demonstration purposes) Rotated stemplot 6

7 Interpreting Stemplots Shape Symmetry Modality (number of peaks) Kurtosis (width of tails) Departures (outliers) Location Gravitational center mean Middle value median Spread Range and inter-quartile range Standard deviation and variance 7

8 Shape Shape refers to the pattern when plotted Here s the silhouette of our data X X X X X X X X X X Consider: symmetry, modality, kurtosis 8

9 Shape: Idealized Density Curve A large dataset is introduced An density curve is superimposed to better discuss shape 9

10 Symmetrical Shapes 10

11 Asymmetrical shapes 11

12 Modality (no. of peaks) 12

13 Kurtosis (width of tails) Mesokurtic (medium) Platykurtic (flat) fat tails Leptokurtic (steep) skinny tails Kurtosis is not be easily judged by eye 13

14 Stemplot Second Example Data: 1.47, 2.06, 2.36, 3.43, 3.74, 3.78, 3.94, 4.42 Stem = ones-place Leaves = tenths-place Round to keep one digit after decimal point (e.g., ) Do not plot decimal ( 1) Shape: asymmetric, skewed to the left, unimodal, no outliers 14

15 Draw a stemplot using JMP Analyze---Distribution---Data---Stem and Leaf Open the JMP data set named Stem_and_leaf_plot.jmp 15

16 Third Illustrative Example (n = 26) Age data set from 26 subjects {14, 17, 18, 19, 22, 22, 23, 24, 24, 26, 26, 27, 28, 29, 30, 30, 30, 31, 32, 33, 34, 34, 35, 36, 37, 38} Data set: Stem_and_leaf_plot_example2.jmp Distribution of the age variable? 16

17 2. Frequency Table Frequency = count Relative frequency = proportion or % Cumulative frequency % less than or equal to level AGE Freq Rel.Freq Cum.Freq % 0.3% % 1.7% % 6.0% % 11.6% % 19.9% % 32.9% % 47.2% % 59.6% % 73.4% % 82.1% % 88.7% % 92.5% % 95.4% % 97.4% % 98.6% % 99.5% % 100.0% Total % 17

18 Frequency Table with Class Intervals When data are sparse, group data into class intervals Create 4 to 12 class intervals Classes can be uniform or non-uniform End-point convention: e.g., first class interval of 0 to 10 will include 0 but exclude 10 (0 to 9.99) Talley frequencies Calculate relative frequency Calculate cumulative frequency 18

19 Class Intervals Uniform class intervals table (width 10) for data: Class Freq Relative Freq. (%) Cumulative Freq (%) Total

20 Histogram A histogram is a frequency chart for a quantitative measurement. Notice how the bars touch _ Age Class 20

21 Bar Chart A bar chart with non-touching bars is reserved for categorical measurements and non-uniform class intervals Pre- Elem. Middle High School-level 21

22 3. Summary Statistics Central location Mean Median Mode Spread Range and interquartile range (IQR) Variance and standard deviation 22

23 Location: Mean Eye-ball method visualize where plot would balance Arithmetic method = sum values and divide by n ^ Grav.Center Eye-ball method around 25 to 30 (takes practice) Arithmetic method mean = 290 / 10 = 29 23

24 Notation n sample size X the variable (e.g., ages of subjects) x i the value of individual i for variable X sum all values (capital sigma) Illustrative data (ages of participants): n = 10 X = AGE variable x 1 = 21, x 2 = 42,, x 10 = 52 x i = x 1 + x x 10 = =

25 Central Location: Sample Mean Arithmetic average Traditional measure of central location Sum the values and divide by n xbar refers to the sample mean x 1 n x x 1 2 n 1 xn xi n i 1 25

26 Example: Sample Mean Ten individuals selected at random have the following ages: Note that n = 10, x i = = 290, and x 1 n x i 1 10 (290 ) 29.0 The sample mean is the gravitational center of a distribution Mean = 29 26

27 Uses of the Sample Mean The sample mean can be used to predict: The value of an observation drawn at random from the sample The value of an observation drawn at random from the population The population mean 27

28 Population Mean xi 1 N N x i Same operation as sample mean except based on entire population (N population size) Conceptually important Usually not available in practice Sometimes referred to as the expected value 28

29 Central Location: Median Ordered array: When n is even, the median is the average of the (n 2)th data and the (n 2+1)th data. When n is odd, the median is the ((n+1) 2)th data. For illustrative data: n = 10 the median falls between 27 and 28=(27+28) 2 = median Average the adjacent values: M =

30 More Examples of Medians Example A: Median = 4 Example B: Median = 5 (average of 4 and 6) Example C: Median 2 (Values must be ordered first) 30

31 The Median is Robust The median is more resistant to skews and outliers than the mean; it is more robust. This data set has a mean of 1636: Here s the same data set with a data entry error outlier (highlighted). This data set has a mean of 2743: The median is 1614 in both instances, demonstrating its robustness in the face of outliers. 31

32 Mode The mode is the most commonly encountered value in the dataset This data set has a mode of 7 {4, 7, 7, 7, 8, 8, 9} This data set has no mode {4, 6, 7, 8} (each point appears only once) The mode is useful only in large data sets with repeating values 32

33 Comparison of Mean, Median, Mode Note how the mean gets pulled toward the longer tail more than the median mean = median symmetrical distrib mean > median positive skew mean < median negative skew 33

34 Spread: Quartiles Two distributions can be quite different yet can have the same mean This data compares particulate matter in air samples (μg/m 3 ) at two sites. Both sites have a mean of 36, but Site 1 exhibits much greater variability. We would miss the high pollution days if we relied solely on the mean. Site 1 Site

35 Spread: Range Range = maximum minimum Illustrative example: Site 1 range = = 46 Site 2 range = = 8 Beware: the sample range will tend to underestimate the population range. Always supplement the range with at least one addition measure of spread Site 1 Site

36 Spread: Quartiles Quartile 1 (Q1): cuts off bottom quarter of data = median of the lower half of the data set Quartile 3 (Q3): cuts off top quarter of data = median of the upper half of the data set Interquartile Range (IQR) = Q3 Q1 covers the middle 50% of the distribution Q1 median Q3 Q1 = 21, Q3 = 42, and IQR = = 21 36

37 Quartiles (Tukey s Hinges) Example 2 Data are metabolic rates (cal/day), n = median When n is odd, include the median in both halves of the data set. Bottom half: which has a median of (Q1) Top half: which has a median of 1729 (Q3) 37

38 Five-Point Summary Q0 (the minimum) Q1 (25 th percentile) Q2 (median) Q3 (75 th percentile) Q4 (the maximum) 38

39 Boxplots 1. Calculate 5-point summary. Draw box from Q1 to Q3 w/ line at median 2. Calculate IQR and fences as follows: Fence Lower = Q1 1.5(IQR) Fence Upper = Q (IQR) Do not draw fences 3. Determine if any values lie outside the fences (outside values). If so, plot these separately. 4. Determine values inside the fences (inside values) Draw whisker from Q3 to upper inside value. Draw whisker from Q1 to lower inside value 39

40 Illustrative Example: Boxplot Data: pt summary: {5, 21, 27.5, 42, 52}; box from 21 to 42 with IQR = = 21. F U = Q (IQR) = 42 + (1.5)(21) = 73.5 F L = Q1 1.5(IQR) = 21 (1.5)(21) = None values above upper fence None values below lower fence 4. Upper inside value = 52 Lower inside value = 5 Draws whiskers Upper inside = 52 Q3 = 42 Q2 = 27.5 Q1 = 21 Lower inside = 5 40

41 Illustrative Example: Boxplot 2 Data: point summary: 3, 22, 25.5, 29, 51: draw box 2. IQR = = 7 F U = Q (IQR) = 29 + (1.5)(7) = 39.5 F L = Q1 1.5(IQR) = 22 (1.5)(7) = One above top fence (51) One below bottom fence (3) O utside v alue (5 1) In side valu e (31) Up per hin ge (29) Me dian (25.5 ) Lo wer h ing e (2 2) In side valu e (21) O utside v alue (3 ) 4. Upper inside value is 31 0 Lower inside value is 21 Draw whiskers 41

42 Illustrative Example: Boxplot 3 Seven metabolic rates: point summary: 1362, , 1614, 1729, IQR = = F U = Q (IQR) = (1.5)(279.5) = F L = Q1 1.5(IQR) = (1.5)(279.5) = None outside 4. Whiskers 1867 and 1362 N = 7 Data source: Moore, 42

43 Boxplots: Interpretation Location Position of median Position of box Spread Hinge-spread (IQR) Whisker-to-whisker spread Range Shape Symmetry or direction of skew Long whiskers (tails) indicate leptokurtosis (Long tails?) 43

44 Side-by-side boxplots Boxplots are especially useful when comparing groups 44

45 Spread: Standard Deviation Most common descriptive measures of spread Based on deviations around the mean. This figure demonstrates the deviations of two of its values This data set has a mean of 36. The data point 33 has a deviation of = 3. The data point 40 has a deviation of = 4. 45

46 Variance and Standard Deviation Deviation = x i x Sum of squared deviations = SS x x 2 i Sample variance = s 2 n SS 1 Sample standard deviation = s s 2 46

47 Standard deviation (formula) Sum of Squares s n 1 1 ( x i x ) 2 Sample standard deviation s is the estimator of population standard deviation. 47

48 Illustrative Example: Standard Deviation Observation Deviations Squared deviations x i x i x x 2 i x = = = = = = = = = = = 2 ( 2) 2 = = 3 ( 3) 2 = = 4 ( 4) 2 = 16 SUMS 0* SS = 58 * Sum of deviations always equals zero 48

49 Illustrative Example (cont.) Sample variance (s 2 ) s 2 SS ( n g/m 3 ) 2 Standard deviation (s) s s g/m 3 49

50 Interpretation of Standard Deviation Measure spread (e.g., if group was s 1 = 15 and group 2 s 2 = 10, group 1 has more spread, i.e., variability) rule (next slide) Chebychev s rule (two slides hence) 50

51 Rule Normal Distributions Only! 68% of data in the range μ ± σ 95% of data in the range μ ± 2σ 99.7% of data in the range μ ± 3σ Example. Suppose a variable has a Normal distribution with μ = 30 and σ = 10. Then: 68% of values are between 30 ± 10 = 20 to 40 95% are between 30 ± (2)(10) = 30 ± 20 = 10 to % are between 30 ± (3)(10) = 30 ± 30 = 0 to 60 51

52 Chebychev s Rule All Distributions Chebychev s rule says that at least 75% of the values will fall in the range μ ± 2σ (for any shaped distribution) Example: A distribution with μ = 30 and σ = 10 has at least 75% of the values in the range 30 ± (2)(10) = 10 to 50 52

53 Rules for Rounding Carry at least four significant digits during calculations. Round at last step of operation Always report units Always use common sense and good judgment. 53

54 Choosing Summary Statistics Always report a measure of central location, a measure of spread, and the sample size Symmetrical mound-shaped distributions report mean and standard deviation Odd shaped distributions report 5- point summaries (or median and IQR) 54

55 Software and Calculators Use software and calculators to check work. 55

56 Excel Data Analysis ToolPak Data set: Boxplot.xlsx Summary statistics using Excel 56

57 Excel Data Analysis ToolPak Get summary statistics using Excel 57

58 Boxplot using Excel Youtube links to get boxplot in Excel 4PVarwE&feature=related Female age Male age 58

59 Boxplot and summary statistics by JMP Data set: Boxplot.jmp What do you say about the comparison of distributions of ages for females and males? 59

60 60

61 Exercise Surgical times. Durations of surgeries (hours) for 15 patients receiving artificial hearts are shown here. Create a stem plot of these data. Describe the distribution. Are there any outliers? What is the standard deviation of this data set? Draw a boxplot based on this data set Data set: Presentation2_Exercise.xlsx Presentation2_exercise.jmp 61