Most of the transformations we will deal with will be in the families of powers and roots: p X -> (X -1)/-1.

Transcription

1 Powers and Roots Quite often when we re dealing with quantitative data, it turns out that for the purposes of analysis, it is useful to carry out a transformation of one of the variables of interest. This could occur for any one of a number of reasons: C C C C A distribution may be skewed. This can be problematic for any one of a number of reasons. 1) Skewed distributions are difficult to eamine because many observations may be piled up in one place. 2) Unusual observations in the direction opposite from that of the skew may be hidden. 3) Summary measures like the mean and median don t necessarily make much sense for a skewed distribution. The relationship between two variables many be nonlinear. Again, this can be a problem for any one of a number of reasons. 1) Linear relationships are easier to interpret than nonlinear ones. With a linear relationship, after all, a one unit change in the independent variable leads to some unit change in the dependent variable. 2) The statistical theory for linear models is simple and well-developed. This is what we will be learning for the rest of the quarter. 3) Nonparametric regression is not feasible when there is more than independent variable. Spread may be nonconstant. There are cases where the spread of a variable will depend on its level. In that situation, comparing two distributions of the same variable from populations are at different levels may be difficult. One or both of the variables may be a proportion. Most of the transformations we will deal with will be in the families of powers and roots: p X -> (X -1)/-1. In the case that p is zero, we will substitute the log transformation, X -> log. The relationships of these transformed variables to the original are summarized in the figure below: Note that you will not always be using the specific formula listed above. You may need to fiddle a bit so that all of the independent variables are positive. Also, as a general rule, carrying out a transformation only makes sense when the ratio of the largest to the smallest value is quite large.

2 Creating transformed variables in STATA with the generate command. An important command you will need to use in STATA to create transformed variables is the generate command. The format is generate new variable = epression involving eisting variables for eample, the curves in the previous transparency were generated from an original variable by the following commands: generate y0 = ln() generate y1 = - 1 generate y2 = ^2-1 generate y3 = ^3-1 generate ym1 = (^-1-1)/-1 The original variable had been created to consist of 1000 observations ranging from 0 to 4. Note that ln() is the natural logarithm of, log() is the log base 10 of, and ^3 is cubed. The range of functions you can include in your epression is quite wide. help functions will summarize them for you. For eample, normd() is the height of the normal distribution at, normprob() is the value of the cumulative normal at, and so on y = normd() y = normprob() y = sin(1/) y = sin 2-4 4

3

4 Dealing with skewness If you have a variable that is skewed, you may use a power/root or log transformation to try to make it look more normal, or at least more symmetric. In general, if a distribution is skewed to the right, you may need to eperiment with a log or a root transformation, while if it is skewed to the left, you may need to square or cube it. Here are some eamples: First, a situation where a variable is skewed to the left. On the left is a smoothed univariate distribution (using STATA s kdensity) histogram of countries according to the percentage of males with a primary school education in 1993 (prim93m). Note that it is skewed to the left, indicating that while most countries have a fairly high proportion of males with some primary school education, some have very little. On the right is a smoothed distribution of a variable prim93m2 generated by squaring prim93m. Note that it is much more symmetric, and even sort of normal Primary 93 M (World Bank) prim93m2 Here s a situation where transformation of a left-skewed distribution improves symmetry, but not normality. On the left is a distribution of life epectancies in 1995, on the right is a distribution of th life epectancy to the 4 power: e0 in 1995 (World Bank) e+06 e0_ e+07 The smoothed distribution is the univariate equivalent of the bivariate smoothing we saw earlier.

5 Just as we can raise to a power to adjust for a left skew, we can log or take a root to correct for a right skew. On the left is a smoothed distribution of per capita GDP, on the right is the equivalent for logged per capita GDP United Nations per capita GDP lpcgdp95 Below is an etreme eample. On the left is a distribution of surface areas of countries. It is right skewed, indicating that while most countries have a small surface area, some have very large ones. On the right is a distribution of the tenth-root of surface areas, the result of some eperimentation Surface Area in sq. km. World B AreaRt2

6 Transforming to deal with nonlinearity Often we will want to transform a variable to linearize a relationship. An important rule of thumb is that if we plot the dependent variable on the vertical versus the independent variable on the horizontal, if the plot ehibits a bulge up and to the left, we will probably need to either transform the independent variable downward or the dependent variable upward. If the bulge is down and to the left, we may need to shift transform the independent variable upward or the dependent variable downward. Here s an eample where on the left, male life epectancy is plotted against per capita GDP, and on the right, it is plotted against logged per capita GDP Male e from U.N. Male e from U.N United Nations per capita GDP lpcgdp95 Here s an eample where the dependent variable is TFR United Nations per capita GDP lpcgdp95

7 Nonconstant spread Sometimes the spread of a variable depends on the level of another variable. For eample, according to the figure on the left below, how spread out the distribution of the infant mortality rate is varies according to the level of fertility, even though the relationship between the two is linear. That can sometimes be a problem if you want in some way to compare the distribution of the variable at different levels. Outliers may be harder to locate at one of the levels, for eample. When spread increases with level, working down the ladder of powers can stabilize it. The right on the right below plots the cube root of the infant mortality rate as a function of the TFR IMR IMR

8 Proportions An important transformation when one of the variables represents a proportion is the logit transformation, ln(p/(1-p)). This is the log of the odds of P. Proportions are unusual beasts because they are bounded, by 0 and 1, and often they behave differently when you look at distributions of them. Often you will see eamples of nonconstant spread when you are looking at proportions as a dependent variable, and the logit transformation is another way of dealing with this. Here is the IMR/TFR relationship from the previous page, followed by one where the IMR is transformed into the logit: IMR logitimr

9 Here s an eample where we re looking at a distribution of countries by the percentage of dwellings with running water. It s right skewed, but squaring the percentage doesn t do much that s useful, indeed it produces a bimodal distribution that still looks a bit skewed Fraction Fraction Water (World Bank) mwater95 Here s what happens when we look at a logistic transformation: Fraction lwater95 Notice the distribution is much better-behaved.