Lecture 2 Describing Data

Transcription

1 Lecture 2 Describing Data Thais Paiva STA Summer 2013 Term II July 2, 2013

2 Lecture Plan 1 Types of data 2 Describing the data with plots 3 Summary statistics for central tendency and spread 4 Histograms and boxplots

3 Data Recall that A population is composed of units. Units can represent individuals, financial stocks, nations, trees, etc... A sample is a collection of units drawn from the population For each unit we can measure several different characteristics (variables) To present a dataset, we typically use rows to represent units and columns to represent variables.

4 An Example Dataset Salary # of Name Gender ( 1,000) Education Family members AA F 50 B 2 BB M 43 B 3 CC F 65 M 1 DD M 200 B 2 EE M 60 M 4 FF M 25 S 2 GG F 15 S 0 HH F 80 D 3 II M 22 S 1 JJ F 69 B 4 KK F 70 M 2 M = male; F = female B = bachelor; S = senior high; M = masters, D = doctoral

5 Types of Variables Qualitative Variables These values can fall into one of several groups or categories Represent characteristics which cannot be naturally associated to a number Ordinal: although not naturally associated to a number, can be somehow ordered (i.e., education) Nominal (Categorical): there is no natural order (i.e., gender) Quantitative Variables Take numerical values such that using numerical operations makes sense Discrete: only take integer values (i.e., # of family members) Continuous: can take any value (sometimes within a specified range) (i.e., height) Sometimes a variable can be treated as quantitative or qualitative.

6 Sample versus Population Recall that a parameter is a descriptive measure of a population a statistic is a descriptive measure of a sample In many cases, for statistical inference we want the statistics to approximate the unknown parameters. Some approximations will be better than others We will see which are good approximations This lecture, we will focus on some statistics calculated using our sample.

7 Gender According to the data, there are 5 males and 6 females in the sample. A very simple graph is the bar chart representation F M Here the y-axis represents frequency. An alternative is to use percentage. Frequency is simply the number of data values that falls into each category.

8 Education B D M S Bar chart allows us to quickly see the distribution of the variable in our sample. The distribution of a single variable describes the overall pattern of the data. Example: which value is most common? Here we treat education as nominal. We can also re-arrange the x-axis as S B M D and treat education as ordinal.

9 Salary: Frequencies Using bar chart to describe salary is more complicated because it is continuous. One way is to divide data into categories. For example smaller or bigger than 100 (thousands). Frequency Salary This representation that creates categories (bins) for continuous data is called a histogram.

10 Salary: Histograms There are many ways to define the salary categories! By 50K: By 20K: Frequency Salary Frequency Salary Categories can be chosen based on scientific reason or to display certain features of the distribution.

11 Salary: Distribution Density Salary Instead of plotting the number of data point that falls into each bin, we can convert these into relative frequencies. This graph is related to the density of the distribution. More on this later. We simply changed the scale but the general shape of the distribution remains the same.

12 Salary: Distribution The construction of the distribution requires a little bit of attention since it is based on the relative frequencies. Relative frequency = frequency sample size f <100 = # salaries<100 # salaries = f 100 = # salaries 100 # salaries = 1 11 Then the relative frequencies must be distributed evenly along the cells. height <100 = f<100 base <100 = = height 100 = f 100 base 100 = = 0.001

13 Salary: Distribution There is an important property for the histogram built in terms of relative frequencies: The total area of all the rectangles is equal to 1 height <100 base <100 + height 100 base 100 = (100 0) ( ) = 1 This property is worth remembering! It will will be very important later in the course.

14 Describing Distributions It is important to convey certain characteristics of a distribution, for example to compare different distributions. We will focus on three specific areas: 1. Shape 2. Center 3. Spread Do the extreme values tend to be small or large? What values are most common? Where is the middle of the distribution? How much variability is present?

15 Shape: Mode The mode of a variable is the most frequent observation occurring in the data set. Mode(gender)=FEMALE [6(F) and 5(M)] Mode(education)=BACHELOR [4(B), 3(S), 3(M), 1(D)] If you look at the graphs you will realize that the mode coincides with the peaks of a histogram However for salary, because the variable is continuous, we need to group the data first. Again, the mode will depend on the choice of the groups.

16 Modes While the mode is a useful statistic, often a distribution exhibits several peaks. Ex. Unimodal Ex. Bimodal Frequency Frequency

17 Skewness Sometimes a variable is more extreme in one direction, e.g. medical cost, housing price,... Left Skewed Right Skewed Frequency Frequency A symmetric distribution does not exhibit skewness.

18 Measuring the Center: Mean Given that y 1, y 2,..., y n are the n observations within a sample, then the sample mean is defined as Same as the average ȳ = n y i i=1 We will use the bar notation to denote a sample mean Note how each observed value contributes equally to the mean n

19 Example: mean salary If the salary is NOT grouped then the mean is equal to = If salary is grouped (e.g. by 20K), we can use the midpoint of each group: n i n i n = 62.73

20 Measuring the Center: Median The median is a statistic based on the order of the observations. It is a particular case of percentile: the 50th-percentile. Example Median(education)=Bachelor since [SSSBB(B)BMMMD] Median(# family members)=2 since [0112(2)2233] Median(gender)=? It cannot be evaluated because the variable is not ordinal

21 Salary Median If the data are NOT grouped, then the median is 60 Since [ (60) ] If the data are grouped [40-60] n i n i n kp n i 1 3 (6) i=1 kp i=1 n i n (0.54) The median is where the corresponding cumulative frequency is (or the minimum value above) 0.50.

22 Mean versus Median For measuring central tendency: Mean: quantitative data with symmetric distribution Median: quantitative data with skewed distribution Because the median is solely based on the rank of the data, for a unimodal distribution: Shape Left skewed Symmetric Right skewed mean < median mean median mean > median

23 Spread of a Distribution When describing a distribution, knowing of the center may be not sufficient. For example: financial analysts are usually interested in the expected returns (mean) and the risk (spread) In this sense, the spread indicates how much uncertainty (variability) is associated with the expected return Consider two hypothetical stock returns at time 1, 2, 3, 4, 5: 1 2, 1, 0, 1, 2 with mean , 100, 0, 100, 200 with mean 0

24 Measuring Spread: Range The range is simply defined as range = max min Examples: range(salary) = = 185 range(# family members) = 4 0 = 4 range(gender) = not possible range(education) = not possible

25 Measuring Spread: Percentile The p-percentile is the observation within the sample such that p% of the remaining observations are equal or lower than the p-percentile. p The position of the p percentile is 100 (n + 1) the 25% percentile is called 1 st quartile (q 1 ) the 50% percentile is called Median the 75% percentile is called 3 rd quartile (q 3 ) Note the data must be qualitative ordinal or quantitative. Also, the quartiles are also based on the order of the data.

26 Measuring Spread: Percentile Salary Example For salary The original data are: ( ) The sorted data are: ( ) salary [0.25 (11+1)] = salary [3] = 25 > 25th percentile salary [0.5 (11+1)] = salary [6] = 60 > median salary [0.75 (11+1)] = salary [9] = 70 > 75th percentile What if only 10 data were available? A little bit more difficult!

27 Measuring Spread: Percentile Example Suppose you have the following data: y = 1, 3, 6, 8. What are the q 1, median, q 3? According to the percentile formula, the locations are Therefore = = = 3.75 q 1 = 1.5 (or between 1 and 3) median = 4.5 (or between 3 and 6) q 3 = 7.5 (or between 6 and 8)

28 Boxplot The box plot is a useful graphical representation which combines the five number summary: minimum, q 1, median, q 3, maximum Salary # of Family Members

29 Measure Spread: Interquartile Range Compared to a histogram, the boxplot shows a lot less detail. Useful to compare the distribution across different groups The box between the first and third quartile gives the interquartile range (IQR) For salary, IQR = = 45 IQR = q 3 q 1 IQR measures the central spread and is not influenced by the tails - very useful for describing the spread of a skewed distribution! The location of the thick median line in the box also shows the skewness around the center of the distribution

30 Outliers Sometimes very extreme (strange) observations (outliers) can occur and it is important to identify them. Outliers can: Represent recording errors Represent interesting observations Influence summary statistics (e.g. the mean) A popular rule is to suspect an observation if it falls more than 1.5 IQR below the first quartile or above the third quartile.

31 Modified Boxplot Salary Using the 1.5 IQR rule, a modified boxplot shows the potential outliers. The upper and lower limits are (70 25) = (70 25) = Here the 200K value is shown as a point. Excluding the outlier, our new sample mean is 49.9 instead of 63.5K!

32 Measuring Spread: Variance The sample variance, s 2, is computed as the sum of the squared deviations about the mean, x, divided by (n-1). s 2 = n (x i x) 2 i=1 n 1 variance(gender) = not possible!! variance(# family members) = [(2 2.18) 2 + (3 2.18) 2 + (1 2.18) 2 + (2 2.18) 2 + (4 2.18) 2 + (2 2.18) 2 + (0 2.18) 2 +(3 2.18) 2 +(1 2.18) 2 +(4 2.18) 2 +(2 2.18) 2 ]/(11 1) = 1.56

33 Measuring Spread: Standard Deviation The standard deviation, s, is defined as the square root of the variance: Example: sd(gender) = not possible!! sd(# family members) = (1.56) = 1.25

34 Degrees of Freedom The degrees of freedom represent the effective number of values free to vary in the computation a statistic. In computing the variance, we need to calculate the difference d i = (x i x) for each observation. However, x itself is calculated from the n observations!!! In fact, if you give me the difference for any (n 1)-many observations, I can calculate the remaining one! This is because Remember this property! n (x i x) = 0 i=1

35 Putting It All Together A Frequency Frequency B A B C Mean Median Variance S.D C Frequency

36 Putting It All Together a b c

37 Summary Measures of Central Tendency: Mean Median Measures of Spread: Percentiles and quartiles Interquartile range Variance and standard deviation Graphical Representations of Distribution: Histogram Boxplot and modified boxplot Mode and skewness