Statistics I Chapter 2: Analysis of univariate data

Transcription

1 Statistics I Chapter 2: Analysis of univariate data

2 Chapter 2: Analysis of univariate data Contents 1. Representations and graphs Frequency tables. Bar and pie charts, pictograms, histograms, frequency polygons. Other graphs. Lying with graphs. 2. Numerical summary: Central tendency (mean, median, mode) Location (quartiles and percentiles). Box plots. Spread (variance, standard deviation, range, IQR, coefficient of variation) Shape (coefficients of skewness and kurtosis)

3 Chapter 2: Analysis of univariate data Recommended reading Peña, D., Romo, J. Introducción a la Estadística para las Ciencias Sociales (1997). Chapters 2, 3, 4 y 5. Newbold, P. Statistics for Business and Economics (2008). Chapters 1 y 2

4 Description of qualitative variables Sample: 46 professionals of a computer company in the United States. Variable: EDUC: education level (1=High School; 2=College; 3=Advanced Degree) Variable: MGT: position of responsibility (1=yes; 0=no) In order to obtain information: How to summarize primary data in a more useful way that allows a quick visual interpretation?

5 Description of qualitative variables: frequency tables and bar charts Education level Number of employees Proportion of employees High School College Advanced Degree Total 46 1

6 Description of qualitative variables: general outline of a frequency table Note: Freq. Freq. Class, c i Absolute, n i Relative, f i c 1 n 1 f 1 = n1 n c 2 n 2 f 2 = n2 n... c k n k f k = n k n Total n 1 n i = number of c i in the sample, f i = n i n 0 f i 1

7 Description of qualitative variables: the bar chart Bars are of the same width and equally-spaced, with heights corresponding to frequencies There are gaps between bars Bars are labeled with class names

8 Other graphics: the pie chart Each slice is a fraction of the total size of the pie Many software programs rank slices alphabetically Although pretty harder to interpret than barcharts Avoid 3D piecharts, for those the area in the background seems to be smaller than the area in the foreground

9 The pie chart: example Tabla dinámica (Pivot table) Sample: The Simpsons 568 first episodes Variable: character performing the leading role (the one whospeaksmore) in anepisode Note: Youcan obtainthesamechart formtheraw data (without using a Pivot table). Check the Supplementary materials about Excel usage.

10 Other graphics: the Pareto chart Bar chart in which the categories of the variable are ranked in decreasing order of frequency. Applies only to nominal qualitative variables. Useful in the detection of the more significant reasons (a few options account for almost all the purchasing frequency) Pareto Principle (80-20 rule) Based on empirical knowledge Pareto stated in 1896 that society was divided into two proportional groups 80-20, the few of many and the many of few : A minority group made up of 20 % of the population who owned 80 % of something. A majority group made up of 80 % of the population who owned the remaining 20 %.

11 The Pareto chart: example Visitar la colección del Museo 16,6 Visitar o estar en la cafetería del Museo 7,7 Sample: Among the 1100 visitors of the art exhibition Turner and Visitar la tienda del Museo 28,1 the Masters (Prado Museum, June 22 to September ), Estar o visitar otros espacios del Museo que no tienen 33,0 thosecolección who bought their tickets online (a 20.3 %) Source: Institute for Tourism Esperar en Studies el exterior del Museo 27,5 Variable: Main reason for buying the ticket online % Tabla 9. Visitantes por la razón principal para adquirir la entrada por vía telemática Filtro: Adquiere la entrada por vía telemática % Por comodidad 60,5 Rapidez 10,1 Puedo elegir el día y la hora de la visita 14,0 No tengo que esperar en taquilla 9,5 Porque la entrada es más barata 4,3 Por el horario 24 horas 1,2 Había oído hablar bien del servicio 0,4 Total 100,0

12 The Pareto chart: example

13 Other graphics: pictograms Sample: 70 university students from Madrid Variable: Preferred political party Preferred political party Students numb. Students prop. PSOE PP Unidos Podemos Ciudadanos Otros Total 70 1 The area of the graph is proportional to the frequency.

14 Exercise Results from a survey conducted among year-olds about their favorite leisure activity What is the variable and who are the individuals? For what percentage of young people is reading the preferred leisure activity?

15 Exercise From a test taken by a group of students, graded between 1 and 8, the following table was obtained: Grade, c i n i f i How many students took the test? What percentage of students obtained a grade greater than or equal to 6?

16 Exercise In a survey about health habits, 30 randomly chosen students were asked about the sport they usually practice. The results are shown in the following table: Sport, c i n i f i Basket Swimming Football None Total 30 1 Which of the following charts corresponds to the data above?

17 Exercise Estadística Aplicada a) c) Deporte Deporte Baloncesto Natación Fútbol Ningún deporte 0 Baloncesto Natación Fútbol Ningún deporte b) d) Deporte Deporte Baloncesto Natación Fútbol Ningún deporte 2 0 Baloncesto Natación Fútbol Ningún deporte

18 Description of discrete quantitative variables: table of frequencies Sample: 100 shopping malls in which a promotion of a certain service was launched last November. Variable: number of new customers gained due to the promotion. Absolute Relative Absolute Relative Cumulative Cumulative c i Frequency n i Frequency f i Frequency N i Frequency F i 0 1 0,01 1 0, ,04 5 0, , , , , , , , , , , , , ,1 86 0, , , , Total 100 1

19 Description of discrete quantitative variables: table of frequencies What percentage of the sampled malls gained only 5 new customers? How many malls attracted at least 3 new customers? How many malls attracted less than 6 new customers? What percentage of the sampled malls gained between 4 and 8 new customers? What percentage of malls gained at most 7 new customers?

20 Description of discrete quantitative variables: the bar chart Bar charts can also be created for discrete data if there are not too many different values.

21 Description of discrete quantitative variables: general format of the table Note: Cumulative Cumulative Absolute Relative Absolute Relative Class, c i Freq., n i Freq., f i Freq., N i Freq., F i c 1 n 1 f 1 = n1 n N 1 = n 1 F 1 = f 1 c 2 n 2 f 2 = n2 n N 2 = N 1 + n 2 F 2 = F 1 + f c k n k f k = n k n N k = n F k = 1 Total n 1 c 1 < c 2 <... < c k n i = number of individuals in the sample in class c i,f i = n i n N i = N i 1 + n i, F i = F i 1 + f i 0 f i, F i 1 F i and N i also make sense for qualitative ordinal variables

22 Qualitative ordinal variables: cumulative frequencies We can also include cumulative frequencies in the table. Sample: 901 employees. Variable: levels of satisfaction (S=satisfied, V=very, U=unsatisfied) Cumulative Cumulative Absolute Relative Absolute Relative Class Frequency Frequency Frequency Frequency VU U S VS Total 901 1

23 Qualitative ordinal variables: bar charts with cumulative frequencies Beware! Many software programs rank the classes in alphabetical order when the variable is qualitative. If it is an ordinal variable, it must be ranked in ascending order.

24 Bar charts for discrete data Sample: 46 professionals of a computer company in the United States. variable: EXPRNC: number of years working in the company Experience, c i Absolute freq., n i Relative freq., f i 1 5 0, , , , , , , , , , , , , , , , ,022 Total 46 1

25 Description of discrete quantitative variables: the bar chart Too many different values.

26 Description of continuous quantitative variables Sample: 46 professionals of a computer company in the United States. Variable: EXPRNC: years of experience Variable: SALARY: anual gross income (in US dollars)

27 Grouping by class intervals: continuous (or discrete) data Note: Class Interval Midpoint n i f i N i F i [l 0, l 1 ] c 1 = l0+l1 2 n 1 f 1 N 1 F 1 (l 1, l 2 ] c 2 = l1+l2 2 n 2 f 2 N 2 F (l k 1, l k ] c k = l k 1+l k 2 n k f k n 1 Total n 1 Left end-point is excluded, but right end-point is included in Excel (it is a convention) Reverse end-point convention can be applied - check your software for definition Useful for tabulating discrete data if X takes many values

28 Grouping by class intervals Very often class intervals have the same width Determine the width w of each interval by w = largest number - smallest number number of desired intervals How many intervals? Roughly between 5 and 20. Practice and experience provide the best guidelines (From Newbold): Intervals never overlap Sample size Number of classes Fewer than to to to to More than Round up the interval width to get desirable interval endpoints

29 Grouping by class intervals: histogram and frequency polygon Find range: 20 1 = 19 Select number of classes: say k = 46 = Compute interval width: 19/7 = Determine the end-points (beginning before the first one and ending after the last one): [0, 3], (3, 6],..., (19, 21]

30 Description of quantitative variables: histogram and frequency polygon There are no gaps between the bars/bins Bin widths = widths of class intervals (identical), class boundaries are marked on the horizontal axis Bin heights = frequencies (here, absolute) Bin areas are proportional to the frequencies

31 Quantitative variables: the histogram

32 Description of quantitative variables: histogram and frequency polygon

33 Other graphics: cartograms (INE, Encuesta de Turismo de residentes) Average trips expenditure per person during the third term of 2016 Average excursions expenditure per person during the third term of 2016

34 Other graphics: pictograms

35 Other graphics: time series INE, Encuesta de Población Activa

36 How to lie with pictograms Published in La Voz de Galicia, on October 24, Letting height proportional to frequency gives a false impression. Is there anything else you don t like?

37 Lying with graphs Improper use of scales: the coordinate origin is not 0

38 Lying with graphs

39 Lying with graphs The vertical axes scale is upside down

40 Lying with Statistics A classic book: How to Lie with Statistics, by Darrell Huff, Available online:

41 Numerical summary Central tendency Location Spread Shape mean quartiles range coeff. skewness median percentiles interquartile range coeff. kurtosis mode variance standard deviation coeff. of variation

42 Descriptive statistics Why are they useful? Can we calculate them for all types of variables? Which are the most useful in each case? How can we compute them with a calculator or Excel?

43 Measures of central tendency The mean The median The mode

44 Central tendency: the (arithmetic) mean The (arithmetic) mean The mean is the average of all the data n i=1 x = x i n = x x n n It is the most common measure of location It is the center of gravity of the data It can be calculated only for quantitative variables

45 The mean: example For the experience of the 46 professionals of a computer company, What is the mean? x = With Excel: function PROMEDIO(número1; [número 2];...) = 7.5 años

46 The mean: example How to calculate the mean from the absolute frequency table? And from the relative frequency table?

47 The mean with grouped data It is the same formula but using the center of each interval. For the salary of the 46 professionals of a computer company, What is the mean? Note: the mean salary using the raw data equals

48 The mean: properties Linearity: If Y = a + bx ȳ = a + b x If Z = X + Y z = x + ȳ If the 46 professionals salaries increase by 2 %, How does the mean salary change? If the salary is reduced in 100 dollars, What is then the new mean salary? If the salary is increased with a productivity bonus that is recorded in variable Y, with mean ȳ, What is the new mean salary? Disadvantages: Affected by extreme values (outliers) Example: X : 3, 1, 5, 4, 2, Y : 3, 1, 5, 4, 200 x = = 3 ȳ = = 42.6! When the data is skewed, an alternative robust measure of central tendency is more appropriate

49 Central tendency: the median... it is the most central datum Order the data from smallest to largest 2. Include repetitions 3. The median is the physical centre M = Median Ordered list from smallest to largest: x (1), x (2),..., x (n) if n odd M = x ((n+1)/2) x (n/2) +x (n/2+1) 2 if n even With Excel: function MEDIANA(número1; [número2];...) = 4

50 Finding the median from a frequency table Experience, x i n i f i N i F i 1 5 0, , , , , , , , , , 435 < 0.5 M=6 4 0, , 522 > , , , , , , , , , , , , , , , , , , , , , , , , ,000

51 The median: properties Linearity: If Y = a + bx with b > 0 M y = a + bm x If the 46 professionals salaries are increased by 2 %, How does the median salary change? Afterwards the salary is reduced in 100 dollars. What is the final median salary? Can we calculate the median with the education level data? Can we calculate the median with the 0-1 position of responsibility variable? Advantage: Not affected by outliers Example: X : 3, 1, 5, 4, 2, Y : 3, 1, 5, 4, 200 M x = 3 M y = 4 When the data is skewed it is a better measure of central tendency than the mean.

52 The median and the mean for asymmetric (skewed) data Annual gross salary in 2014, Encuesta de Estructura Salarial 2014, INE La diferencia entre el salario medio y el mediano se explica porque en el cálculo del valor medio influyen notablemente los salarios muy altos aunque se refieran a pocos trabajadores. (Nota de Prensa del INE de 28 de octubre de 2016)

53 Central tendency: the mode... it is the most frequent value The mode of the variable experience in the 46 professionals example is 1 year, with an absolute frequency of 5 employees. The values 2,3,4,8 and 10 have an absolute frequency of 4 employees.

54 Central tendency: the mode Does this definition make sense with the education level data? Does this definition make sense with the 0-1 position of responsability variable?

55 Central tendency: the mode Does this definition make sense with continuous data? modal interval

56 The mode: properties It can be calculated for both qualitative and quantitative variables. Indeed, it is the only descriptive measure (mean, median, mode) that makes sense for nominal qualitative variables. Not affected by outliers There can be no mode. There can be more than one mode: bimodal trimodal plurimodal What does it indicate?

57 Bimodal distribution Time (in minutes) to complete a marathon. Data fromanopen marathon(everybody can participate). 160 Tiempo en correr un maratón: histograma Whatdo youthinkishappening? Can youguesswhichtypesof runners makeup theblue and the greengroups? Wouldyouexpectto observe thesamehistogramshapeifthedata came instead from a marathon at the Olympic Games?

58 Location measures Quartiles Percentiles

59 Location measures: quartiles and percentiles Quartiles split the ranked data into four segments with an equal number of values per segment. Percentiles split the ranked data into a hundred segments with an equal number of values per segment. 1. Order the data from smallest to largest 2. Include repetitions 3. Select each quartile (percentile) according to: The first quartil Q1 has position 1 (n + 1). 4 The second quartil Q2 (= median) has position 1 (n + 1). 2 The third quartil Q3 has position 3 (n + 1). 4 The k-th percentile Pk, has position k(n + 1)/100, k = 1,..., 99.

60 Quartiles and percentiles with Excel Note: In most cases the fractions 1 4 (n + 1), 3 4 (n + 1) and k 100 (n + 1) are not integer to get the (integer) position of the given quartile (or percentile) a rounding criterion must be used. With Excel, the functions are: CUARTIL.INC(matriz;cuartil), with: 1=first quartil, 2=median, 3=third quartil PERCENTIL.INC(matriz;p), with: p = k 100 (0, 1), k-th percentile

61 Masures of spread The range and the interquartile range The variance and the standard deviation The coefficient of variation

62 Variation: range and interquartile range (IQR) The Range is the simplest measure of variation R = x máx x mín Ignores the way the data is distributed Sensitive to outliers Example: Given observations 3, 1, 5, 4, 2, R = 5 1 = 4 Example: Given observations 3, 1, 5, 4, 100, R = = 99 The Interquartile range (IQR) can eliminate some outlier problems. Eliminate high and low observations and calculate the range of the middle 50 % of the data RIC = 3rd cuartil 1st cuartil = Q 3 Q 1

63 Variation: Interquartile range and boxplot Outliers are observations that fall below the value of Q1 1.5 IQR above the value of Q IQR For extreme outliers, replace 1.5 by 3 in the above definition MEDIANA x min Q 1 (Q 2) Q 3 x max 25% 25% 25% 25% RI=18

64 Box-Plot It shows five central location measures. It shows a robust dispersion measure. It allows the study of the symmetry of the data. It gives a criterion to detect outliers. It is very useful to compare different datasets. Variation: when several box-plots are depicted in the same chart, the boxes widths can be proportional to the sample sizes.

65 Homer and his antagonists Homer Simpson has two main antagonists: Flanders and Mr. Burns: In thoseepisodesin whichat leastoneof themappears, Howis Homer s protagonism distributed?

66 Homer and his antagonists Youmustuse thefiltervariable definedin Exercise5 (Problemset 1) 1) Create 4 variables with the values of columm(variable) Homer for each of the following cases: Homer&Burns, Homer&Flanders, Homer&Both, Homer&None 2) Selectallcases and inserta Diagrama de Cajas y Bigotes

67 Measure of variation: variance Average of squared deviations of values from the mean Sample variance n ˆσ 2 i=1 = (x i x) 2 n faster to calculate { }}{ n i=1 = x i 2 n( x) 2 n divided by n Sample quasi-variance (corrected sample variance) n s 2 i=1 = (x i x) 2 n 1 They are related via = n i=1 x 2 i n( x) 2 n 1 ˆσ 2 = n 1 n s2 divided by n 1 If a, b are real numbers and y = a + bx, then s 2 y = b 2 s 2 x

68 Measure of variation: standard deviation (SD) The most-commonly used measure of spread Population standard deviation, sample standard deviation and sample quasi-standard deviation are respectively ˆσ = ˆσ 2 s = s 2 Measures variation about the mean Has the same units as the original data, whilst variance is in units 2 Variance and SD are both affected by outliers

69 Calculating variance and standard deviation Example: X : 11, 12, 13, 16, 16, 17, 18, 21, Y : 14, 15, 15, 15, 16, 16, 16, 17, Z : 11, 11, 11, 12, 19, 20, 20, 20 x = = 15.5 ȳ = = 15.5 z = = 15.5 n i=1 n i=1 n i=1 x 2 i = = 2000 y 2 i = = 1928 z 2 i = = 2068 n sx 2 i=1 = x i 2 n( x) (15.5)2 = = 78 = sx = n sy (15.5)2 = = 6 = sy = sz (15.5)2 = = 146 = sz =

70 Calculating variance and standard deviation with Excel

71 Comparing standard deviations Example cont.: X : 11, 12, 13, 16, 16, 17, 18, 21, Y : 14, 15, 15, 15, 16, 16, 16, 17, Z : 11, 11, 11, 12, 19, 20, 20, 20 x = 15.5 s x = y = 15.5 s y = z = 15.5 s z =

72 Measure of variation: coefficient of variation (CV) Measures relative variation and is defined as CV = s x Is a unitless number (sometimes given in % s) Shows variation relative to the mean Example: Stock A: Average price last year = 50, Standard deviation = 5 Stock B: Average price last year = 100, Standard deviation = 5 CV A = 5 50 = 0.10 CV B = = 0.05 Both stocks have the same SDs, but stock B is less variable relative to its mean price

73 Z-scores. In which ODS (SDG) is Spain performing better? ODS 4: Quality education, Spain: 88,9 ODS 5: Gender equality and women s empowerment, Spain: 80,6 ODS8: Decent work and economic growth, Spain: 80,9 ODS 12: Responsible consumption and production, Spain: 60,8 ODS16: Peace, justice and strong institutions, Spain: 69,5 Medidas Resumen SDG 4 SDG5 SDG8 SDG12 SDG16 Media 72, , , , , Error típico 1, , , , , Mediana 80, , , , , Moda #N/A #N/A #N/A #N/A #N/A Desviación estándar 22, , , , , Varianza de la muestra 517, , , , , Curtosis 0, , , , , Coeficiente de asimetría -1, , , , , Rango 95, , , , , Mínimo 3, , , , , Máximo 99, , , , , Suma 11357, , , , ,21239 Cuenta

74 Numerical summaries and frequency tables. Standardization. To standardize variable x means to calculate x x s If you apply this formula to all observations x 1,..., x n and call the transformed ones z 1,..., z n, then the mean of the z s is zero with standard deviation of one Standardization = finding z-score

75 Z-scores. In which ODS is Spain performing better? Medidas Resumen SDG 4 SDG5 SDG8 SDG12 SDG16 Media 72, , , , , Error típico 1, , , , , Mediana 80, , , , , Moda #N/A #N/A #N/A #N/A #N/A Desviación estándar 22, , , , , Varianza de la muestra 517, , , , , Curtosis 0, , , , , Coeficiente de asimetría -1, , , , , Rango 95, , , , , Mínimo 3, , , , , Máximo 99, , , , , Suma 11357, , , , ,21239 Cuenta Spain 88,9 80,6 80,9 60,8 69,5 Con respecto a la media 16, , , , , Incorporando variabilidad 0, , , , ,

76 Measures of shape Fisher Pearson coefficient of skewness Fisher coefficient of kurtosis Empirical rule

77 Measures of shape: Skewness Be AWARE of not making a decision about the shape just by means of a comparison between the Mean, the Median and the Mode. Fisher Pearson coefficient of skewness γ 1 = 1 n ( ) 3 xi x n s i=1 With Excel: COEFICIENTE.ASIMETRIA(número1; número2;...) n n ( ) 3 xi x (n 1)(n 2) s i=1

78 Measures of shape: kurtosis Fisher s coefficient of kurtosis n ( ) 4 xi x 3 γ 2 = 1 n With Excel: CURTOSIS(número1; número2;...) n(n + 1) n ( ) 4 xi x (n 1) 2 3 (n 1)(n 2)(n 3) s (n 2)(n 3) i=1 i=1 s

79 Measures of shape: skewness and kurtosis Excel function

80 Análisis de Datos in Excel: Estadística descriptiva [OECD-only] Average PISA score across Maths/Reading/Science(0-600) Media 491, Error típico 4, Mediana 496, Moda #N/A Desviación estándar 26, Varianza de la muestra 679, Curtosis 1, Coeficiente de asimetría -1, Rango 113, Mínimo 415, Máximo 528, Suma 17219,46943 Cuenta 35 FRECUENCIA [OECD-only] Average PISA score across Maths/Reading/Science(0-600) 406,00 425,00 444,00 463,00 482,00 501,00 520,00 539,00 Spain: 491,4 MARCAS DE CLASE Data source: SDG Index& Dashboards Report 2017,

81 Empirical rule If the data is bell-shaped (normal), that is, symmetric with light tails, the following rule holds: 68 % of the data are in ( x 1s, x + 1s) 95 % of the data are in ( x 2s, x + 2s) 99.7 % of the data are in ( x 3s, x + 3s) Note: This rule is also known as rule Example: We know that for a sample of 100 observations, the mean is 40 and the quasi-standard deviation is 5. Assuming that the data is bell-shaped, give the limits of an interval that captures 95 % of the observations. 95 % of x i s are in: ( x ± 2s) = (40 ± 2(5)) = (30, 50)