A Unified New Method for the Evaluation and Monitoring of Investment Managers

Transcription

1 A Unified New Method for the Evaluation and Monitoring of Investment Managers Dan dibartolomeo and Sandy Warrick Northfield Information Services, Inc. August-September 2005

2 Topics for Today Investors are constantly looking to invest with superior active managers, but have a hard time finding the managers that will be superior in the future Active managers are typically evaluated by looking at simple performance measures over fixed past time periods We will present a more sophisticated method of manager monitoring and evaluation that involves a three part statistical process We will present an empirical study of a large sample of US equity mutual funds over eight years that indicates that our method produces highly statistically and economically significant results

3 Motivation To hire active managers we must believe at least one of three things: The average professional investment manager outperforms passive index funds because individual investors have below index performance Manager active returns persist. We can predict with reasonable reliability which managers are going to outperform in the future, even if the average manager is just average We are doing a societal good because if all investors were passive, there would be no functional mechanism to ration capital in the economy. Our economy would break down over time

4 Problems in the Typical Current Process Much manager evaluation occurs relative to benchmarks that are often not suitable for the manager s investment approach Evaluation of past performance is based on standardized periods (i.e. 5 years) rather than periods that are relevant to the manager in question Many evaluation measures such as Sharpe ratio or information ratio correspond to meaningful investor utility for only a small fraction of investors The statistical significance of ex-post performance is measured in a simple time series fashion. It does not include the context of whether the manager exists among a tightly bunched set of peers or a widely dispersed set This is critical in examining the luck versus skill issue

5 A Simple Prescription for Success Classification Make sure each fund is being measured against the right benchmark and the right peers We use an augmented method of returns-based style analysis Process Control Evaluate each manager over the evaluation period that is the best for that particular manager We use a Sequential Probability Ratio Test called CUSUM to find the optimal evaluation period Evaluate Past Performance Use a return measure such as alpha, not the Sharpe Ratio or Information Ratio as your measure Use a Bayesian framework adjustment to ex-post alpha to reflect contemporaneous dispersion across managers Does the CUSUM analysis show improving or declining efficiency?

6 The Persistence Literature If markets are very efficient, there should be no persistence patterns in active management returns. While there are there are innumerable studies showing markets are relatively efficient, many fund studies show that some persistence does exist Hendricks, Patel and Zeckahuaser (1993) Find positive persistence only over time horizons less than a year Stronger persistence among worst managers who stay worst Elton, Gruber, and Blake (1996) Persistence of risk adjusted returns over one to three year time horizons Appears to be correlated with investor capital flows Goetzmann and Ibbotson (1994) Persistence over one to two year horizon Effect is stronger for more volatile funds

7 The Persistence Literature is Persistent Carhart (1997) Some persistence over a one year horizon but not longer Investment style and expense ratios explain most peristence effects. No evidence of stockpicking stockpicking skill Stewart (1998) Funds that have consistently outperformed the &P 500 over a screening period also outperform during subsequent periods Consistent performers hold more diversified portfolios Brown & Goetzmann (1995) Find performance persistence in mutual funds using several Superior performance is correlated across managers (style herding) Detzel and Weigand (1998) Some persistence in mutual fund returns, but after adjusting for manager investment style, all persistence in returns is explained.

8 Classification Issues The best way to win a contest for the largest tomato is to paint a cantaloupe red and hope the judges don t notice dibartolomeo and Witkowski (1997) Imagine you operate two funds One aggressive and one conservative To outperform your peer groups you actually have to be more skillful than competitors. Its not easy. You take the easy way out Mischaracterize both funds: market the aggressive fund as conservative, market the conservative one as aggressive Depending on market conditions, one of the two will ALWAYS compare well to the intentionally wrong peer group Any observed persistence in active returns could be an artifact of misclassification

9 Classification Empirical Studies dibartolomeo and Witkowski (1997) Studied 748 mutual funds from 1990 through 1995 About 40 percent of funds appear to be misclassified in terms of o objective Much of the misclassification is intentional, not random Investors ability to diversify fund types is diminished, with an a annual associated cost in the billions of dollars Similar results on different data, using different methods are obtained by Brown and Goetzmann (1997) Kim, Shukla and Thomas (2000) Problems with institutional money managers are less severe but still present

10 Our Approach to Classification Forming normal portfolios by comparing actual portfolio holdings to benchmarks across time is the best method Kritzman (1987) Very labor intensive if the data is available Data isn t available for broad universes of institutional managers We use an augmented form of returns-based style analysis to match funds to a broad range of benchmarks Sharpe (1992) Form a portfolio of indices that mimics fund behavior over time The key improvement is the calculation of confidence intervals on the Sharpe style weights Saying a fund is 10% small cap value is meaningless if it turns out to be 10% plus or minus 30% dibartolomeo and Lobosco (1997) Most commercial vendors don t include this calculation because it will scare away users

11 Time Horizons for Evaluating Investment Track Records Practitioner tradition in the investment industry is to evaluate active manager track records over a long period At least 3 to 5 years Some will argue a full market cycle is needed As we ve seen, all the academic studies refute this No evidence that long-term past performance is predictive of future performance If there is any meaning to past performance at all, its short-lived, perhaps the last year

12 The Key Question What time portion of a track record do we really need to evaluate as part of our monitoring of manager quality control What we need is a procedure to draw the line between getting enough meaningful data within a manager s record and older, stale data that should be ignored Enter CUSUM

13 Statistical Process Control Developed at Bell Labs in the 1930 s by Walter Shewart, whose key insight was to focus on results. The product is what counts If it is good, the the process is good If it us bad, then the process is bad Similar in spirit to performance monitoring Originally used to monitor Western Electric s telephone production lines Separate signal from noise

14 The CUSUM Technique Backward looking sequential probability ratio test Created by E.S. Page in 1954 Reliably detects small process shifts Insensitive to probability distribution Provably optimal: detects process shifts faster than any other method. Robust, good under almost any definition of optimality Much better than exponentially weighted moving average. Mathematically very tractable: its literally adding up a series of numbers Easily analyzed algebraically or graphically

15 A Robust Method of Monitoring Manager Returns: CUSUM CUSUM analysis defines key turning points in the active return time series, and defines statistical significance of results subsequent to the key turning point Use of CUSUM to monitor active managers started with the IBM pension fund Philips, Stein and Yashchin (2003) Used to monitor over $500 Billion in externally managed funds The PSY method involves using CUSUM to classify active managers into three categories: Good, We Don t Know, and Bad Managers are reviewed whenever a class boundary is crossed Not an automatic hire/fire signal

16 CUSUM (Green) Plot Shows Regimes of Over and Under Performance CUSUM Manager E / / / / / / / / / / / / / / / / / / / / / / / / /01 Ac tive R e turn CUS UM Upper CUSUM Low e r CU S U M

17 Our Implementation of CUSUM is Different from PSY In our approach we use CUSUM to evaluate whether the effectiveness of a manager s process is improving or declining This is not the same as being good or bad A manager that has been really outstanding in the past could decline in effectiveness but still be very good A manager that has been really awful in the past could improve in effectiveness but still be very bad We use CUSUM to find the key turning point in the past where the effectiveness of the manager seems to have changed most profoundly

18 The Math to Get CUSUM is Easy Calculate excess returns for a manager over a carefully selected benchmark Hold out a short sample period at the beginning to get an initial estimate of the mean and standard deviation of excess return Standardize each excess return by subtracting prior mean and dividing by prior standard deviation We re measuring change in the information ratio Calculate the cumulative sum of the standardized excess returns over time M i = (E( i Avg(E 1 :E i-1 )) / Stdev(E 1 :E i-1 ) E i = excess return for period I M t = Sum (M 1 :M t-1 )

19 Finding the Critical Time Point To find the critical time point, we look back through time from the current moment We re looking for the past moment in time for which the interim change in the CUSUM is least likely to have occurred by accident. If its not an accident, its probably meaningful Mathematically this is the point that maximizes the absolute value of Z t Z t = (M( n M t ) / (n-t).5 M t is the value of CUSUM at time t n is the time index of the last data point (i.e. now)

20 What does CUSUM Tell Us? Something happened at time t to change the effectiveness of the manager Statisticians call this a regime change The time interval between time n and time t is the best interval over which to evaluate this manager s performance record If the manager s effectiveness is constant over the time interval, the CUSUM will be constant at zero If the effectiveness is improving, the CUSUM will plot as an upward sloping line If the effectiveness is declining, the CUSUM will plot as a downward sloping line

21 Now That We Know the When, Lets Deal with the What Many performance measures are not congruent to adding value for investors degroot and Plantinga (2001) Consider a manager that adds exactly one basis point of return in every time period. The information ratio is infinite, but very little investor wealth is added We chose to measure excess return above a carefully chosen benchmark that should reflect both risk and investing style This directly measures added value for investors Our CUSUM analysis is already a variation on information ratios

22 Separating Luck from Skill To maximally exploit our information about manager performance we need to separate skillful managers from the merely lucky We need to adjust for the fact that if manager returns are widely dispersed within a peer group, its easier to have a high excess return. If the dispersion of returns is low, its harder We adopt a method a Bayesian framework of a precision weighted estimate that incorporates information about the dispersion of peer fund returns during the evaluation period for each fund Similar to Shanken and Jones (2004) without the elaborate Monte-Carlo simulations

23 The Precision Weighted Excess Return Estimate: An Example Lets assume Manager X has an excess return (A) of 4% per year with a standard deviation (S) of 4% Over the same time period, the average peer manager had an annual excess return of.25% (Mean), and the dispersion (CSD) of the excess returns across the peer group is 1.5% E = (A/S 2 + Mean/CSD 2 ) / (1/S 2 + 1/CSD 2 ) A = 4, S = 4, MEAN =.25, CSD = 1.5 E (precision weighted) = (0.361) / (0.5069) = We assume the manager has skill sufficient to add 71 basis points per year over the benchmark

24 Our Study: Fund Data We start with eleven years of monthly returns on all US equity mutual funds in existence at April 30, 2005 This dataset is survivorship biased but frequency of fund extinction below levels that could sufficiently bias results Brown, Goetzmann,, Ibbotson and Ross (1992) Elton, Gruber and Blake (1996) Carpenter and Lynch (1999) Carhart,, Carpenter, Lynch and Musto (2002) We eliminate multiple classes of shares in the same fund We eliminate about 200 specialized funds: index funds, tax-efficient, long-short, etc. The final sample is about 1200 funds

25 Benchmarks We use 15 benchmark indices Russell 1000, 1000 Growth, 1000 Value Russell 2000, 2000 Growth, 2000 Value Russell 3000, 3000 Growth, 3000 Value S&P 500, S&P 500 Growth, S&P 500 Value S&P MidCap, MidCap Growth, MidCap Value The Russell and S&P definitions of Growth and Value are demonstrably different Do a style analysis of each index against the others We tested both returns based style analysis and simple highest correlation to assign funds to benchmarks Not much difference in the resulting assignments Next version of the study will allow for assignment changes during the sample periods

26 Experimental Design Hypothesis: Excess returns measured over CUSUM defined past sample periods will be predictive of future excess returns Assign each fund to a benchmark At each calendar year end, compute the CUSUM best look-back interval for each manager Compute excess return for each manager from the CUSUM critical point to the test date Compute the precision weighted excess return from the CUSUM critical point to test date Use OLS cross-sectional sectional regression analysis to relate excess returns during the subsequent year to the excess returns during the CUSUM look-back period ending at the test date

27 Regression Equation If this all works b will be positive and statistically significant R it = a + b t-1 (Y it-1 ) + e it Rit = excess return on fund i during year t Yit = annualized excess return excess return (or precision weighted return) during the CUSUM look- back period ending at the end of year t-1t b t-1 = regression coefficient that relates observed past and future excess returns at the end of year t-1t a = intercept it = error term for fund i during year t e it

28 Benchmark Assignment Benchmark Assignment Percentages Percentages Total Total Value Value Growth Growth Blend Blend Total Total Russell Russell Russell Russell S&P S&P MidCap MidCap Russell Russell S&P S&P Style/ Style/ Size Size

29 Distribution of Fund Returns (all funds with complete data for last 72 months) Excess Return Tracking Error Information Ratio Average Median Standard Deviation

30 Regression Results: Domestic funds Pooled regression of excess returns 6303 fund years from 1998 through 2004 B coefficient is.126 with a T stat of 12 Correlation of past to future is.15 Individual years using excess returns Average annual coefficient is.152, average T 4.9 Best year is 1999, coefficient.40, T = 10.6 One Really Bad Year 2003, coefficient -.14, T = -5.6 We also investigated regressions using the precision weighted excess returns as the independent variable. In most cases, cross sectional results were equally good or better using precision weighted returns, and these results showed less year-to to-year variation.

31 Empirical Conclusions: Domestic Mutual Funds Our hypothesis that past returns can be used to predict future returns is supported to a degree of virtual statistical certainty Using annual cross sectional regression of raw excess returns, the expected excess returns are about 15% of the observed past returns. The average t-statistic t for the regression above is almost 5. Using precision weighted excess returns as the independent variable, the expected values are over 68% of the past values (t-stat = 3.6) Given the observed dispersion among manager returns, large and economically significant excess returns should be available to investors

32 Empirical Conclusions: International Mutual Funds For international funds, the benchmarks used were either EAFE or S&P/IFCI, depending on which returns were more closely correlated to the fund returns. Using annual cross sectional regression of raw excess returns, the expected excess returns are about 12% of the observed past returns. The average t-statistic t for the regression above is almost 3. Using precision weighted excess returns as the independent variable, the expected values are about 130% of the past values (t-stat = 3.0).

33 Empirical Conclusions: Hedge Funds For hedge funds, the benchmarks used were either: Cash, for those hedge funds whose returns most closely correlated d with t-bills or domestic bonds The MSCI World Equity index for those funds most closely correlated with MSCIW. The MSCI World Equity index for those funds most closely correlated with EAFE or IFCI and showed only slightly less correlation with the MSCIW The Russell 3000 for returns most closely correlated to domestic equity indices. The returns were adjusted for their beta to the benchmark. Using annual cross sectional regression of raw excess returns, the t expected excess returns are about 7% of the observed past returns. (t-stat=1.7) Using precision weighted excess returns as the independent variable, the expected values are 60% of the past values (t-stat = 3.0)

34 Final Comments We have demonstrated a methodology for the monitoring and evaluation of active managers Use carefully selected benchmarks to capture manager and style effectse Use CUSUM analysis to determine the most effective period over which w to evaluate a particular manager s performance record Use a performance measure that is congruent with investor utility y and is refined by considering cross-sectional sectional dispersion of peer manager returns An empirical study of 1200 domestic equity mutual funds strongly supports the efficacy of the approach Additional empirical study of more than 200 international equity mutual funds and 500 hedge funds shows that this approach is also applicable to other funds for which it can be difficult to separate ate luck from skill.

35 References Hendricks, Darryll, Jayendu Patel and Richard Zeckhauser.. "Hot Hands In Mutual Funds: Short-Run Persistence Of Relative Performance, ," 1988," Journal of Finance, 1993, v48(1), Elton, Edwin J., Martin J. Gruber and Christopher R. Blake. "The Persistence Of Risk-Adjusted Mutual Fund Performance," Journal of Business, 1996, v69(2,apr), Goetzmann,, William N. and Roger G. Ibbotson. "Do Winners Repeat?," Journal of Portfolio Management, 1994, v20(2), Carhart,, Mark M. "On Persistence In Mutual Fund Performance," Journal of Finance, 1997, v52(1,mar), Stewart, Scott D. "Is Consistency Of Performance A Good Measure Of Manager Skill?," Journal of Portfolio Management, 1998, v24(3,spring), Brown, Stephen J. and William N. Goetzmann.. "Performance Persistence," Journal of Finance, 1995, v50(2), Detzel,, Larry F. and Robert Weigand,, Explaining Persistence in Mutual Fund Performance, Financial Services Review, 1998

36 References dibartolomeo, Dan and Erik Witkowski. "Mutual Fund Misclassification: Evidence Based On Style Analysis," Financial Analyst Journal, 1997, v53(5,sep/oct), Brown, Stephen J. and William N. Goetzmann.. "Mutual Fund Styles," Journal of Financial Economics, 1997, v43(3,mar), Kim, Moon, Ravi Shukla and Michael Thomas. "Mutual Fund Objective Misclassification," Journal of Economics and Business, 2000, v52(4,jul/aug), Kritzman, Mark. "How To Build A Normal Portfolio In Three Easy Steps," Journal of Portfolio Management, 1987, v13(4), Sharpe, William F. "Asset Allocation: Management Style And Performance Measurement," Journal of Portfolio Management, 1992, v18(2), Lobosco,, Angelo and Dan DiBartolomeo. "Approximately The Confidence Intervals For Sharpe Style Weights," Financial Analyst Journal, 1997, v53(4,jul/aug), Page, 85.Page, E.S. Continuous Inspection Schemes, Biometrika,, 1954 Philips, Thomas K., David Stein and Emmanuel Yashchin,, Using Statistical Process Control to Monitor Active Managers, Journal of Portfolio Management, 2003

37 References degroot,, Sebastien and Auke Plantinga,, Risk Adjusted Performance Measures and Implied Risk Attitudes, Journal of Performance Measurement, Winter 2001/2002 Jones, Christopher and Jay Shanken,, Mutual Fund Performance with Learning Across Funds, Working Paper FMA Annual Meeting Proceedings, 2003 Brown, Stephen J., William Goetzmann,, Roger G. Ibbotson and Stephen A. Ross. "Survivorship Bias In Performance Studies," Review of Financial Studies, 1992, v5(4), Elton, Edwin J., Martin J. Gruber and Christopher R. Blake. "Survivorship Bias And Mutual Fund Performance," Review of Financial Studies, 1996, v9(4,winter), Carpenter, Jennifer N. and Anthony W. Lynch. "Survivorship Bias And Attrition Effects In Measures Of Performance Persistence," Journal of Financial Economics, 1999, v54(3,dec), Carhart,, M. M., J. N. Carpenter, A. W. Lynch and D. K. Musto. "Mutual Fund Survivorship," Review of Financial Studies, 2002, v15(5),