A Monte Carlo Measure to Improve Fairness in Equity Analyst Evaluation

Transcription

1 A Monte Carlo Measure to Improve Fairness in Equity Analyst Evaluation John Robert Yaros and Tomasz Imieliński Abstract The Wall Street Journal s Best on the Street, StarMine and many other systems measure analyst stock-rating performance using variations on a method we term the portfolio method, whereby a synthetic portfolio is formed to track the analyst s ratings. At the end of the evaluation period, analysts are compared by their respective portfolio returns. Of the pitfalls to this method, one most troubling is that the analysts are generally covering different sets of stocks over different time periods. Thus, each analyst has access to different opportunities and just comparing portfolio values is unfair. In response, we present a Monte Carlo method where, for each analyst, we generate numerous pseudo-analysts with the same coverage over the same time periods as the real analyst. Using this method, we are better able to compare analysts, adjusted for their individual opportunities. We draw comparisons between our results and the results from the existing systems, showing that those systems are less precise in reflecting analyst performance. 1 Introduction Numerous systems for evaluating stock analysts have emerged over the years. This reflects the investor s desire to know which analysts are the best predictors of future stock behavior. Good predictions can mean the investor can achieve higher returns, so s/he is willing to pay substantially for such advice as long as s/he perceives it to be the most accurate. At the same time, measuring analyst performance is not straightforward. Each analyst likely covers a subset of stocks, such as major pharmaceutical companies, and the subsets of stocks are nearly always different across John Robert Yaros Rutgers University, New Brunswick, NJ, USA, yaros@cs.rutgers.edu Tomasz Imieliński Rutgers University, New Brunswick, NJ, USA, imielins@cs.rutgers.edu 1

2 2 John Robert Yaros and Tomasz Imieliński research firms employing different analysts. For example, one retail analyst may cover Walmart, Target and Costco, while a retail analyst at another firm covers Walmart, Best Buy and RadioShack. Moreover, the stocks covered by each analyst may vary in time, so making comparisons for a specified interval can be difficult since the composition of stocks covered can change frequently throughout the interval. The de facto approach 1 to handling these challenges has been the portfolio method, wherein a synthetic portfolio is created to track ratings made by the analyst. For example, whenever the analyst gives a positive rating, the portfolio goes long one unit of that stock. For negative ratings, a short unit is added to the portfolio. These positions are exited when the rating ends, such as when the analyst stops coverage. The intent is that the portfolio value will reflect the accuracy of the analyst s decisions. At the end of a given time period, analysts can be ranked with the belief that the most accurate analyst will have highest portfolio value. We find three inter-related shortcomings. First, each analyst covers different stocks and, thus, has access to different opportunities. So, the portfolio return for one analyst may be higher than another analyst simply because his/her stocks have greater price changes. Second, while positive and negative ratings have counterpart actions of buying and selling in the portfolio, neutral ratings do not have a clear action. A frequent approach is to simply ignore them. Another approach is to invest in a benchmark asset such that returns of the overall portfolio are diluted. It is true that returns may be lowered by missed opportunity, but again considering that not all analysts cover the same stocks, the missed opportunity may not be reflected when comparing to other analysts if few or no other analysts covered that stock. Third, to interpret portfolio return, one must have reference to the movement of the underlying stocks. For example, suppose an analyst covering only one stock has a portfolio return of 5%. This might be excellent if perfect predictions would lead to a 6% return overall during the period, but would be much weaker if 60% was possible. In recognition of these issues, we present a Monte Carlo (MC) approach that harks back to a 1933 study by Cowles, who wanted to measure the accuracy of the stock market predictors of his time. To do so, he generated several time series of predictions by simply drawing from a stack of cards labeled positive, negative, etc. Using these, he could determine if analysts were truly making predictions better than chance. Similarly, we judge analysts in reference to pseudo-analysts, which we generate such that they cover the same stock at the same time. So, against the pseudo-analysts, the real analyst has access to the same opportunities. Based on stock returns during the period, we compute the analyst s percentile score against the pseudo-analysts. These percentile scores are much fairer means of comparison than simply comparing raw portfolio returns. Moreover, our result allows for a more interpretable statement like the analyst beat 60% of pseudo-analysts, rather than a statement from the portfolio method like the analyst generated a +5% return where interpretation is difficult without a great deal of context. 1 Another approach is to use surveys, such as Institutional Investor s annual awards, where experts, such as brokerage clients, are asked to rank analysts. They have been called beauty contests [2] since they can lack objectivity. Surveys can also be expensive and require expert participation.

3 A Monte Carlo Measure to Improve Fairness in Equity Analyst Evaluation 3 2 Background Since 1993, the Wall Street Journal (WSJ) has published its Best on the Street ranking of analysts. The methodology states: For a stock rated a buy, a positive total return yielded a positive score on that stock, but a negative return produced a negative score. Similarly, for a stock rated sell, a negative total return yielded a positive score while a positive return resulted in a negative score. Hold recommendations did not affect the score [1]. Consider the 2007 Pharmaceuticals Sector [6]: BEING ON THE RIGHT side of huge swings in small companies helped propel Jonathan Aschoff into the No.1 spot among pharmaceutical analysts... Mr. Aschoff... upgraded shares of Adolor(c) Corp. to buy from hold in early February, the day its shares surged 41%... Mr. Aschoff downgraded the stock to sell in early September, the day the shares plunged 45%... He benefited from the methodology of this survey, which calculates returns from the closing the day before the recommendation change, scoring for both the 58% return while he rated the stock a buy and a nearly 73% decline during his sell recommendation. Two shortcomings are evident. First, the previous day s close is considered the starting point of the rating. In two instances, the largest portion of return came from stock movement occurring before the rating and, thus, does not reflect predictive ability. Second, unless the other analysts had access to stocks with similar huge swings, they would be unable to have a high rank, even if they were highly accurate on their lower-opportunity stocks. As discussed in Section 1, analysts typically have limited control of their coverage, so the rankings can involve luck as much as skill. StarMine has recognized some of these issues. The 2010 U.S. award methodology states The portfolio return is opportunity adjusted to facilitate a fair comparison of analyst performance regardless of their coverage universe. The adjustment method is not specified, but there is strong evidence [8, 7] that they normalize by the volatility of the covered stocks. This can help, but volatility really measures noise rather than opportunity. Consider Fig. 1. Stocks A and B have standard deviations 5.2% and 5.7%, respectively. StarMine s methodology suggests stock B has greater opportunity. Yet, stock B essentially has noise around an upward path, while stock A has a sequence of returns that we might reasonably expect a good analyst could label so that an investor would know when to be long or short. StarMine s methodology also states Holds invest one unit in the benchmark (i.e., for an excess return of zero). Consider Fig. 2. Suppose an analyst covers one of the stocks and issues a hold. Regardless of whether the stock is C or D, his/her portfolio return would be identical under the StarMine methodology. Yet, the analyst would clearly be less correct about C than D (assuming a benchmark return of 0%) Time Fig. 1 Stock A Stock B 40% 20% 0% Cumulative Return Time Fig. 2 Stock C Stock D 20% 10% 0% Cumulative Return

4 4 John Robert Yaros and Tomasz Imieliński 3 Data and Method We use the Center for Research in Security Prices (CRSP) s daily total return for each stock which includes not only price changes, but all payouts (e.g. cash dividends). Values after delisting are also used, which prevents upward biases. We use the Capital Asset Pricing Model (CAPM) to calculate abnormal return, which is the difference between a stock s actual and expected return. Abnormal return reflects the job of the analyst, which is usually not to predict if a stock will go up or down in absolute, but to predict its performance relative to the market or peers [4]. For a stock s over time period T, we calculate abnormal return ˆR s,t as ˆR s,t = R s,t R f,t + β s,t (R m,t R f,t ) (1) where R s,t, R m,t and R f,t are the returns over time period T of stock s, the U.S. market and a risk-free instrument, respectively. Market and risk-free returns are obtained from [3]. The sensitivity of stock s to the market is denoted by β s,t, which is calculated immediately prior to time period T using regression r s,t r f,t = α + β s,t (r m,t r f,t ) + ε t (2) where r s,t, r m,t and r f,t are 20-trading-day returns for the stock, market and a riskfree instrument at time t, respectively, and ε t is the error at time t. The interval of twenty trading days is approximately one calender month and we regress over a 500-day period, so the regression is over 25 points. For ratings, we use the I/B/E/S U.S. dataset, which assigns each analyst a unique identifier that remains constant even if s/he switches firms. I/B/E/S also standardizes each firm s ratings into a five level system of 1-Strong Buy, 2-Buy, 3-Hold, 4-Underperform and 5-Sell. For simplicity, we use 1 & 2 as Buy, 3 as Hold and 4 & 5 as Sell. Once an analyst makes a rating, we consider it active until 1) the analyst issues a new rating for the stock, 2) a different analyst at the same firm issues a new rating (i.e. stock was reassigned), 3) a stop coverage is issued by the firm, 4) the stock is delisted, or 5) 250 trading days (approx. 1 calendar year) elapses. WSJ and StarMine awards are annual. Correspondingly, we break our data into years. For a single stock s in a single year, suppose an analyst has ratings with time periods T 1,T 2,...,T n. We compute the cumulative abnormal return as R s = n k=1 d k ˆR s,tk (3) where d k corresponds to the rating at T k, where Buy is +1, Hold is 0 and Sell is 1. As Fig. 3 exemplifies, returns are only counted within the measurement year. Let F y denote the set of all ratings of all analysts on all stocks where the ratings overlapped the measurement year, y. To evaluate a single analyst on a single stock over year y, we generate multiple pseudo-analysts where each analyst begins on the start date of the analyst s earliest rating that overlaps year y (t 0 in Fig. 3). We

5 A Monte Carlo Measure to Improve Fairness in Equity Analyst Evaluation 5 Fig. 3 An example timeline of analyst ratings for a single stock are shown on top. The lengths considered for the year s returns calculations are shown on bottom. t BUY HOLD BUY SELL then randomly sample from F y to generate a sequence of rating lengths until the end of the measurement year is reached or exceeded. Buy, Hold and Sell levels are subsequently applied by again sampling from F y. Thus, both the length and level distributions come directly from the real analyst population, matching our desire that the pseudo-analysts replicate the real analysts. This helps avoid biases (dis)favoring the real analysts, although it replicates their behavior, even if irrational (e.g. analysts tend to issue more buys than sells [5]). For each pseudo-analyst, abnormal return is calculated in the same manner as the real analyst. We then compute a percentile value p s, which is the fraction of pseudoanalysts that had lower abnormal return R s than the real analyst. We compute a composite score over all stocks S in the analyst s coverage as p s = s S l s p s s S l s (4) where l s is the number of days stock s was covered during the year. Analysts can be ranked and compared by p s. As stated in Section 1, p s is easily interpretable since it indicates how many random analysts the real analyst outperformed. We recognize contention may exist over some aspects of our approach (e.g., returns should use a sector benchmark rather than a market benchmark, Strong Buy should be differentiated from Buy, etc.). These aspects can be altered for particular situations and tastes, yet, the shift to a MC approach is a significant structural improvement. For example, in the case of holds, lost opportunity is truly captured in the MC method because pseudo-analysts will have higher returns if a buy or sell rating was more appropriate. In the portfolio method, it is unclear if other analysts will have higher portfolio return since few others may be covering the same stock. 4 Experimental Results WSJ and StarMine award analysts were collected from their respective websites for years 2001 to 2009 (award years 2002 to 2010). Analysts were manually linked to I/B/E/S using their names, firm at award time and textual descriptions. For each stock covered by each analyst in a given year, we generate 10,000 pseudo-analysts. Average percentile values for all real analysts and for WSJ and Starmine analysts are shown in Table 1. As can be seen in the All Analysts column, the median percentile value is near 0.5, indicating that the average analyst tends to do no better than an average pseudo-analyst. With statistical significance at the 5% level for all years using a Mann-Whitney U test, WSJ award analysts tend to do better than analysts

6 6 John Robert Yaros and Tomasz Imieliński Table 1 Median Monte Carlo Percentiles Year All Analysts WSJ StarMine Frequency All Analysts WSJ StarMine Analyst Percentile Value Fig. 4 Percentile Distributions (2007 shown) without a WSJ award. The same is true of StarMine analysts. This is expected since an analyst with higher score under the MC approach would tend to have higher value under the portfolio method. Yet, as seen in Fig. 4, the WSJ and StarMine awards do not capture many of the best analysts identified by the MC method. In analysing the results, we find this may occur in some instances because an analyst with high MC score did not meet certain WSJ or StarMine requirements, such as covering a sufficient number of stocks in a particular industry. However, we find that it occurs frequently when an analyst has less opportunity to capture large returns. It also occurs when an analyst is outranked by other analysts with erroneous hold ratings, but were not penalized for those rating under the WSJ and StarMine methodologies. These results support our claim that the popular portfolio method does not properly capture analyst performance. We suggest the presented MC method alleviates the identified issues and offers a fairer representation of analyst accuracy. References 1. Best on the street (a special report): 2007 analysts survey - how the survey was conducted. Wall Street Journal p. R4 (2007). May Emery, D.R., Li, X.: Are the wall street analyst rankings popularity contests? Journal of Financial and Quantitative Analysis 44, (2009) 3. French, K.: Kenneth R. French-Data Library. french/data library.html (2012) 4. Kadan, O., Madureira, L., Wang, R., Zach, T.: Industry recommendations: Characteristics, investment value, and relation to firm recommendations (2009) 5. McNichols, M., O Brien, P.C.: Self-selection and analyst coverage. Journal of Accounting Research 35, (1997) 6. Rubenstein, S.: Best on the street (a special report): 2007 analysts survey - pharmaceuticals. Wall Street Journal p. R7 (2007). May StarMine: 2003 North America industry analyst awards. award/starmine (2003) 8. StarMine: Coverage-relative rating. analysts.phtml?page set=coverage relative rating (2013)