On the Robustness of Smart Beta. Felix Goltz, PhD Head of Applied Research, EDHEC-Risk Institute Research Director, ERI Scientific Beta

Transcription

1 1

2 On the Robustness of Smart Beta Felix Goltz, PhD Head of Applied Research, EDHEC-Risk Institute Research Director, ERI Scientific Beta 2

3 Is smart beta smart enough to last? Questioning Robustness Is there real investment merit in these new indices? Or are they simply the product of data mining? (Morningstar) The historical tests that don t predict the future for smart beta strategies (Financial News) Market conditions may present a headwind or tailwind for certain strategies.. (Towers Watson) 3

4 What is Robustness of Smart Beta Performance? Relative and Absolute Robustness In general, robustness refers to the capacity of a system to perform effectively in face of change. In statistics, models are said to be robust if they are not unduly affected by outliers or by small departures from model assumptions. In the context of smart beta strategies, two kinds of robustness need to be taken into account relative robustness and absolute robustness. Relative robustness A strategy is assumed to be robust if it is able to deliver similar outperformance in similar market conditions. e.g. A value factor index would be deemed robust if it aligns well with the value factor performance and does not suffer idiosyncratic losses due to any other causes including but not limited to stock specific and sector specific events. Absolute robustness A strategy shown to outperform irrespective of prevailing market conditions, the absence of pronounced state and/or time dependencies, can be termed as robust in absolute terms. Absolute robustness can be achieved by allocating across different rewarded sources of risks rather than concentrating in a single one. 4

5 Outline Factor fishing versus consensual factors Model mining versus diversifying strategy-specific risk Single factor tilts versus multi factor allocation Performance analysis 5

6 Factor fishing versus consensual factors Model mining versus diversifying strategy-specific risk Single factor tilts versus multi factor allocation Performance analysis 6

7 Factor Fishing Risks The latest fashion: Enhanced factor indices Enhanced or prime factor indices of providers introduce substantial originality on the factor definitions This also creates a severe mismatch with consensual definitions used in the academic foundations of factor investing Provider Value indicator Momentum indicator Fama-French (2012, 2015) Goldman Sachs Equity Factor Index World MSCI Multi Factor Indices FTSE Global Factor Index Series Price to Book Value score from proprietary model (Axioma), relative to stock s regional industry group Sector-relative composite based on Enterprise Value/Operating CF, Forward P/E, Price to Book Composite based on CF to price, net income to price, and country-relative sales to price Past 12 Months return (omitting last month) Residuals from cross sectional regression of past 12 month return (omitting last month) on stock volatility Composite based on excess return divided by annualized volatility over past 12 months and past six months Mean./Std. dev. of avg. residual return from 11 rolling window regressions of past 36 months returns on country and industry index 7

8 Factor Fishing Risks The latest fashion: Enhanced factor indices Such enhanced factor definitions surely lead to enhanced results in the backtests But the risk is that enhancements may be simply due to factor fishing and thus not robust Harvey et al [2016] document a total of 314 of factors with positive historical risk premium showing that the discovery of the premium could be a result of data mining ( selection bias ). The practice of identifying merely empirical factors is known as factor fishing (see Ang [2013]). Below, we provide a discussion on the risks associated with such factor fishing 1. Humorous illustration 2. Empirical illustration: A Datamining experiment 3. Practical Examples: Smart Beta product design in practice 8

9 Factor Fishing Risks Dangers of Multiple Testing : Illustration Source: xkcd.com 9

10 Source: xkcd.com 10

11 Source: xkcd.com 11

12 Source: xkcd.com 12

13 Variable Selection for Value A mostly harmless data-mining experiment We conduct an in-sample variable picking exercise Can we beat the simple and consensual book-to-market ratio? We test 10 definitions: Sales-to-Price, Earnings-to-Price, Cash Flow-to- Price, Dividend Yield, Payout Yield (unadjusted & sector-relative) The value portfolios select the top 50% of stocks by value score and cap-weights them We assess in-sample and out-of-sample performance We look at the trailing ten year performance across different variables We also look at out-of-sample returns to strategeis which select the best in-sample variable Running such backtests leads to strong biases, even though our setup understates the possible harm We use only ten single variables that are highly correlated and share the same economic intuition Data-mining biases would be greater with more variables, less correlated variables, or economically non-sensical variables. 13

14 Variable Selection for Value In-Sample Return Differences over Rolling Ten-Year Periods Ann. excess return over broad CW Return Difference Winning variable: 10.00% StP SN-StP CFtP SN-StP EtP StP 8.00% 6.00% 4.00% 2.00% 0.00% -2.00% -4.00% 5.00% 4.00% Maximum Excess Return Minimum Excess Return Max-Min Spread 3.00% 2.00% 1.00% 0.00% Extremes of annualized excess returns of ten cap weighted value strategies for ten year lookback period This chart plots the maximum and minimum of annualized excess returns with respect to the broad US market cap weighted benchmark of 500 stocks of annually rebalanced cap weighted value tilted strategies 14 with 50% stock selection out of the universe of 500 US stocks based on 10 value variable combinations - Earnings-to-Price, Cashflow-to-Price, Sales-to-Price, Dividend-to-Price and Payout-to-Price, both plain-vanilla and sector neutral versions for each. Ten year trailing returns are obtained with annual step size.

15 Variable Selection for Value Out-of-Sample Degradation High in-sample dispersion suggests ample room for enhancing backtests We found annualised return differences of about 1% to 4% over 10 year backtest periods across the ten variables Capturing the winning variable requires frequent changes in variable definitions We can enhance backtests but should we? We compare the performance of in-sample picking with using the simple Book-to- Market ratio Event-study approach: Each year, we select the prior 5 year best performing variable (in-sample) and hold it for five years (out of sample) Can we do better than with Book-to-Market? In-Sample Out-of-Sample 15

16 - 5 Years - 4 Years - 3 Years - 2 Years - 1 Year Formation + 1 Year + 2 Years + 3 Years + 4 Years + 5 Years Cumulative excess return over Book-to-Market Variable Selection for Value Event Study Picking the past winner yields cumulative outperformance over bookto-market of +1.79% in sample Pre-Formation 2.00% 1.00% 0.00% -1.00% -2.00% -3.00% Comparison of cumulative relative returns of the average best in-sample alternative Value strategy with respect to portfolio based on Book-to-Market This chart plots the cumulative excess returns of ten annually rebalanced cap weighted value tilted strategies with 50% stock selection out of the universe of 500 US stocks based on five variable definitions, with respect to a similarly constructed portfolio based on Book-to-Market. Between 1984 and 2009, five year formation period is used to pick the best portfolio based on alternative value definitions and this portfolio is held for another five years. This is done every year for a total of 26 event studies. The chart plots the average outperformance pre- and post-formation with respect to the Book-to-Market portfolio. The alternative value definitions are Earnings-to-Price, Cashflow-to-Price, Sales-to-Price, 16 Dividend-to-Price and Payout-to-Price, both plain-vanilla and sector neutral versions for each. The graph is smoothed by using yearly values.

17 - 5 Years - 4 Years - 3 Years - 2 Years - 1 Year Formation + 1 Year + 2 Years + 3 Years + 4 Years + 5 Years Cumulative excess return over Book-to-Market Variable Selection for Value Event Study Over the following five years, having picked the in-sample winner, leads to cumulative underperformance of -2.72% out-of-sample Pre-Formation Post-Formation 2.00% 1.00% 0.00% -1.00% -2.00% -3.00% Comparison of cumulative relative returns of the average best in-sample alternative Value strategy with respect to portfolio based on Book-to-Market This chart plots the cumulative excess returns of ten annually rebalanced cap weighted value tilted strategies with 50% stock selection out of the universe of 500 US stocks based on five variable definitions, with respect to a similarly constructed portfolio based on Book-to-Market. Between 1984 and 2009, five year formation period is used to pick the best portfolio based on alternative value definitions and this portfolio is held for another five years. This is done every year for a total of 26 event studies. The chart plots the average outperformance pre- and post-formation with respect to the Book-to-Market portfolio. The alternative value definitions are Earnings-to-Price, Cashflow-to-Price, Sales-to-Price, 17 Dividend-to-Price and Payout-to-Price, both plain-vanilla and sector neutral versions for each. The graph is smoothed by using yearly values.

18 Variable Selection for Value Testing Different Calibration and Holding Periods We assess different strategies that pick a variable based on a calibration period and stick to it for a holding periods Such variable picking strategies have a high risk of degradation in live results compared to backtested results Bookto-Mkt In-sample results 10 Years 5 years Average Out-of-sample results of variable picking strategies Calibration Period = 10 years Calibration Period = 5 years HP=5 HP=4 HP=3 HP=2 HP=5 HP=4 HP=3 HP=2 Average Returns 13.1% 13.7% 13.9% 13.8% 12.7% 12.5% 12.7% 12.6% 12.4% 12.3% 12.3% 12.4% 12.5% Volatility 19.0% 18.4% 17.9% 18.1% 18.6% 18.6% 18.6% 18.5% 18.1% 18.6% 18.4% 18.3% 18.5% Sharpe Ratio Ret. Diff. with % 0.78% 0.67% -0.37% -0.60% -0.43% -0.46% -0.69% -0.80% -0.77% -0.75% -0.61% Book-to-Mkt Ret. Diff with in-sample % -1.17% -1.00% -1.03% -1.47% -1.58% -1.55% -1.53% -1.28% Difference with 10Y In-sample Difference with 5Y In-sample Performance of Variable Picking Strategies for Value Tilted Portfolios - This table shows the performance of 8 active strategies formed on the basis of calibration period and held for holding period. At formation, the best performing strategy based on the calibration period of 10 or 5 years is selected and held for a holding period of 2 to 5 years. Book-to-Market portfolio is formed annually by cap-weighting the 50% selection of the stocks with the highest Book-to-Market ratio. The In-sample results select the ex-post best 18 returns of the alternative value strategies. All based on US 500 stocks.

19 Factor Fishing Risks Composite Scores and Overfitting Above, we considered selecting a single variable Novy-Marx (2015) argues that composite scores entail a much higher risk of biased backtests than single variables. Selecting the best performing composite ( best k of n variables) creates a bias similar to selecting the best single variable among n to the power of k candidate variables. Using a composite score of 5 variables out of 10 candidate variables is as biased as selecting with hindsight a single variable from 100,000 (10 to the power of 5) candidate variables. Combining signals that backtest positively can yield impressive backtested results, even when none of the signals employed to construct the composite signal has real power (Novy Marx 2015) 19

20 Factor Fishing Risks Composite Scores and Overlaps Multiple variables for single factor indices lead to implicit exposures: Example: popular quality definitions combine Return on Equity, Debt-to-Equity and earnings variability. Such a definition introduces low volatility exposure into the index; since there is a positive link between vol and debt-to-equity (Choi and Richardson 2012) respectively vol and earnings variability (Chen, Huang and Jha 2012) Stocks that rank well on composite score may be grey stocks If the composite score combines uncorrelated variables composite scoring may end up selecting stocks that do not correspond to large exposure to either of the criteria taken individually. Composite scores may introduce implicit risks that did not exist in the seminal academic research. Risks include overconcentration or sector exclusion. Such risks are expected to be unrewarded in the long term Example: overweighting of financial stocks in fundamentally weighted indices, which helped during the tech bubble (backtest period) but hurt during the financial crisis (live period) 20

21 Factor Fishing Risks Multiple Testing in Practice (I) An Example: Index Design e.g. for Value (data for Developed markets stocks, March 2000 to January 2013) "For each composite value index, factors are selected on the basis of the most significant t-stat values" We base the choice on the following considerations: Book to Price and Dividend Yield historically have a detrimental effect on performance e.g. for Momentum (Developed markets stocks, January 2001 to December 2013) Our preferred measure of momentum is the Residual Sharpe Ratio, which displays relatively high risk-adjusted performance outcomes, and relatively low levels of volatility See: and 21

22 Factor Fishing Risks Multiple Testing in Practice (II) Another Example: A Factor Library 500+ Stock Selection Signals encompassing millions of backtests spanning seminal academic literature and the latest practitioner expertise updated daily to determine what is working currently and historically allows users to upload their own proprietary factors See: 22

23 Inconsistency Increases Factor Fishing Risks Two types of inconsisteny Index providers argue that cap-weighted indices should have a consistent set of rules across regions to avoid unintended investment outcomes. But consistency is often forgotten for factor indices. Inconsistency across factors: One can use the resulting freedom to optimise in-sample results with little replicability out of sample (during live period) Inconsistency across time: A newly re-engineered version of a given factor tilt can provide backtests that look better than the live performance of older indices for the same factor. This seems to be similar to the practice of fund management houses that launch new funds to hide bad performance of older funds. 23

24 Inconsistency Increases Factor Fishing Risks Inconsistency across Factors: Illustration Factor Index Stock selection Weighting Risk controls Size Value Equal-Weight Index Value-weighted index Index methodologies (MSCI factor indices) All stocks in CW parent index universe All stocks in CW parent index universe Equal-weighted Score adjusted by investability factor None None Mom. Momentum Index Selection by momentum score (fixed number of constituents to target 30% market cap coverage) Market cap * momentum score Cap on weight of individual security Low Vol. Minimum Volatility Index All stocks in CW parent index universe Optimisation to minimise portfolio risk Sector and country weight constrarints Cap on multiple of market cap of individual security Yield High Dividend Yield Index Select stocks with dividend yield > 1.3x parent index dividend yield Market cap weighted Cap on weight of individual security 24 See Deploying Multi-Factor Index Allocations in Institutional Portfolios, Research Insight, MSCI, December 2013

25 MSCI indices Value Quality Size Mom. Inconsistency Increases Factor Fishing Risks Inconsistency over Time: Illustration Many index providers have replaced existing indices with new indices tracking the same factor The table below shows that, the factor definitions of a index provider MSCI in 2015 versus 2013 have changed in a pronounced manner. Scoring Adjustments Sales, book value, earnings and cash earnings. Past 3 year average values. Simple average across variables. Return on Equity, Debt to Equity and Earnings Variability. Average of z-score for each variable. None: equal-weighting of large/mid cap stocks is prescribed as a way to capture the size premium. 12-month and 6-month local price performance. Simple average of z-scores. Price-to-Book Value, Price-to-Forward Earnings and Enterprise Value-to-Cash flow from Operations. Current values. Average of z-score for each variable. Return on Equity, Debt to Equity and Earnings Variability. Average of z-score for each variable. No sector control. No sector control. Sector-relative scoring. Sector-relative scoring. No the proprietary Negative of the exposure from the Proprietary Equity country or model's descriptor is Model: model uses a z-score based on the logarithm of sector on a country-relative the market cap of the relevant firm. control. basis Exposure from the Proprietary Equity Model based on 12-month relative strength (25% weight), 6-month relative strength (37.5% weight), historical alpha (37.5% weight). Weighted sum of z-scores. Mom. score is riskadjusted. No explicit risk adjustment (use of proprietary model exposure). 25 See Deploying Multi-Factor Index Allocations in Institutional Portfolios, Research Insight, MSCI, December 2013 and The MSCI Diversified Multi-Factor Indexes - Maximizing Factor Exposure While Controlling Volatility, Research Insight, MSCI, May 2015.

26 Avoiding Factor Fishing Risks Common Sense Recommendations Use a consistent strategy design framework Lo (1994): we need some kind of framework to limit the number of possibilities that we search over. Require a clear economic rationale Why should the exposure to this factor constitute a systematic risk that requires a reward and is likely to continue producing a positive risk premium (Kogan and Tian [2013]). Stay with standard factor definitions Very long term backtests are available 25 years of post publication evidence extensive scrutiny by the academic community (free due diligence) 26

27 Avoiding Factor Fishing Risks The Value of Parsimony Over-elaboration and overparameterization is often the mark of mediocrity. George E. P. Box (1976) 27

28 Factor fishing versus robust factors Model mining versus diversifying strategy-specific risk Single factor tilts versus multi factor allocation Performance analysis 28

29 Diversifying strategy-specific risk Unrewarded Risks of Weighting Schemes Firm-Specific Risk Unrewarded Risks Risks that are specific to the company itself (its management, the risk of the poor quality of its products, the relevance of its R&D and innovation, etc.). Portfolio theory considers it to be neither predictable nor rewarded, so it is better to avoid it by investing in a well-diversified portfolio. Financial Risk Factors which do not carry a premium The academic literature considers that commodity, currency, and sector risks do not have a positive long-term premium. These risks can have a strong influence on the volatility, tracking error, or max relative drawdown over a particular periods. Model-Specific Risk of Weighting Schemes All weighting schemes have specific operational risk which depends on the diversification model used. For e.g., robustness of Maximum Sharpe Ratio scheme depends on a good estimation of covariance matrix and expected returns. 29

30 Diversifying strategy-specific risk Unrewarded Risks of Weighting Schemes Diversification-based weighting schemes have different conditions of optimality and different parameters to be estimated. This difference leads to varying degrees of model specific risks which are unrewarded. 30

31 Diversifying strategy-specific risk Combining Multiple Models Single Diversification Strategies Combination of weighting schemes 1 Diversified Multi-Strategy Maximum Deconcentration Diversified Risk Weighted Maximum Decorrelation Diversify model-specific risk 2 Exploit low correlation of parameter estimation errors Diversify across different optimality conditions 3 Efficient Minimum Volatility Efficient Max. Sharpe Ratio Diversify stock-specific risk 1. Diversified Multi-Strategy indices equal-weight each of the five diversification strategies (Amenc, Goltz, Lodh and Martellini (2014)). 2. See Timmermann (2006), Kan and Zhou (2007), Tu and Zhou (2010), Amenc, Goltz, Lodh, Martellini (2012) on benefits of combining portfolio strategies. 3. Martellini, Milhau and Tarelli (2013) provide a quantitative analysis of the trade-off between optimality risk and estimation risk. They look at optimality risk in isolation by considering a large number of possible equity universes, defined in terms of many different possible reasonable true population values for risk and return parameters, and measuring the difference for these parameter values (in terms of ex-ante Sharpe ratios, i.e. based on true parameter values) between the true MSR portfolios and various heuristic portfolios. In a second step of their analysis, estimation risk is introduced so as to help measure the distance of various heuristic benchmarks using imperfect estimates with respect to the true MSR portfolio. This analysis allows to analyse the interaction between estimation risk and optimality risk. Their results show that under the assumption of true parameter knowledge, the MSR portfolio exhibits a Sharpe ratio far superior than that of other strategies, thus underlining the opportunity costs involved in estimation risk for such portfolios. On the other hand, when a realistic estimate of estimation error is introduced for covariance and expected return parameters, the average Sharpe ratio 31 of the scientifically-diversified portfolios is substantially reduced. Interestingly, GMV dominates the MSR portfolio after estimation risk is taken into account, and also that a mixture of GMV and EW portfolios generates the highest average Sharpe ratio, with the lowest standard deviation.

32 Diversifying strategy-specific risk Correlation across Weighting Schemes (same tilt) The weighting schemes have imperfect correlation between the relative returns of the strategies for a given choice of factor exposure. By combining the different strategies we can take advantage of the imperfect correlation and reduce the model specific risks. Pair-wise Correlations of Excess Returns across Five Weighting Schemes US Long Term (Dec 1970 Dec 2015) Average Correlation across Five Weighting Schemes Minimum Correlation across Five Weighting Schemes Mid Cap High Momentum Low Volatility Value Low Investment High Profitability The analysis is based on daily total returns in USD of US Long-Term Track Records from 31/12/1970 to 31/12/2015 (45 years). The average and minimum pair-wise correlations across the five weighting schemes Max Deconcentration, Max Decorrelation, Max Sharpe Ratio, Min Volatility and Diversified Risk Weighted for the six factors Momentum, Low Volatility, Value, Size, Low Investment and Low Profitability are provided. Source: 32

33 Using diversified multi-strategy weigting The case of value 31/12/ /12/2015 Absolute Performance Annualized Returns SciBeta Developed Value Diversified Multi-Strategy FTSE RAFI Developed 1000 FTSE Developed Value Factor MSCI World Enhanced Value MSCI World Value Weighted Russell Developed Value S&P Enhanced Value Developed LMC S&P Intrinsic Value Weighted Developed S&P Developed BMI Value MSCI World (Benchmark) 6.37% 5.57% 4.86% 5.87% 4.63% 4.58% 3.89% 5.51% 5.15% 5.54% Volatility 17.43% 19.44% 18.96% 19.83% 18.88% 18.49% 21.90% 17.94% 18.17% 17.80% Sharpe Ratio Maximum Drawdown Relative Performance Annualized Excess Returns 57.32% 61.00% 60.47% 61.67% 61.55% 61.02% 68.99% 59.08% 61.16% 57.46% 0.83% 0.02% -0.68% 0.33% -0.91% -0.97% -1.65% -0.04% -0.39% - Tracking Error 2.41% 3.98% 3.54% 5.22% 2.72% 2.27% 9.54% 2.17% 1.88% - Information Ratio Maximum Relative Drawdown The Diversified Multi-Strategy weighted value index follows the methodology of Amenc et al (2014) and uses a consensual value proxy variable (book-to-market) and the Diversified Multi- Strategy weighting scheme. Below, we compare the performance of this value smart factor index to other indices that ignore diversification of model specific risk and use proprietary variable definitions and Risk-adjusted performance of such an approach aimed at robustness exceeds that of many indices which aim at more concentrated approaches using proprietary value definitions 5.68% 13.87% 12.35% 18.09% 11.54% 12.93% 28.98% 7.53% 10.20% - The tables show the performance and risks of the Scientific Beta Value Smart Factor Index and its competitors in the Developed Universe. The analysis is based on daily USD total return data from 31 December 2005 to 31 December The MSCI World index is used as the cap-weighted reference. Yield on Secondary US Treasury Bills (3M) is used as a proxy for the risk-free rate. Data for external indexes come from Bloomberg and DataStream or provider s official websites. Data for Scientific Beta indexes come from 33

34 Diversifying strategy-specific risk Benefits of Model Combinations Essentially, all models are wrong, but some are useful. George E. P. Box (1987) 34

35 Factor fishing versus robust factors Model mining versus diversifying strategy-specific risk Single factor tilts versus multi factor allocation Performance analysis 35

36 Single (or Few) Factor Tilts Strong Dependency on Individual Factor Exposures Popular smart beta strategies have embedded exposures to risk factors. Performance may be heavily dependent on one or two particular factors FTSE RAFI USA 1000 Index (31-Dec-1970 to 31-Dec-2015) Monthly. Rel. Returns (over C.W.) Negative Size Returns Positive Size Returns Negative Value Returns Positive Value Returns -0.92% 0.43% -0.37% 1.04% MSCI USA Minimum Volatility Index (31-May-1988 to 31-Dec-2015) Monthly. Rel. Returns (over C.W.) Negative Market Returns Positive Market Returns Negative BAB Returns Positive BAB Returns 0.05% 0.80% -0.90% -0.01% Factor returns in each calendar month are used to classify the time period into different market conditions (positive/negative). All reported excess returns are monthly in nature. Daily total returns of the FTSE RAFI USA 1000 Index and MSCI USA Minimum Volatility index are obtained from Bloomberg. Due to data unavailability, the time period of MSCI Minimum Volatility Index is restricted to the longest possible period 31-May-1988 to 31-Dec Analysis on FTSE RAFI USA index is based on a 45 year time period from 31- Dec-1970 to 31-Dec The cap-weighted reference is the daily total returns of cap-weighted index of the 500 largest stocks in United States. The Market (MKT-Rf), 36 Small size (SMB) and Value (HML) factors are obtained from Kenneth French data library. Betting-Against-Beta (BAB) factor is obtained from Andrea Frazzini data library. The missing values in BAB factor time series for the period 01-Dec-2015 to 31-Dec-2015 are assumed to be zero.

37 Single Factor Tilts Strong Dependency on Individual Factor Exposures Exposure to a single factor is risky in absolute terms. Periods of poor factor returns are common but occur at different times Such losses are consistent with the economic explanation of a risk premium Calendar Year Returns of Risk Factors Factors are obtained from the Scientific Beta US Long-Term Track Records. The analysis is based on daily total returns in USD from 31/12/1970 to 31/12/2015 (45 years). Small Size/Value/Momentum factors are long short cap-weighted portfolios long in small cap stocks (in broad market) /30% highest book-to-market/30% past 12M-1M high returns stocks and short in 30% largest cap stocks/30% lowest book-to-market/30% past 12M-1M low returns stocks. Low Vol/High Profitability/Low Investment factors are long short cap-weighted portfolios long in 30% lowest past 2Y volatility/30% highest gross profit-to-total asset ratio/30% lowest 2Y total asset growth rate stocks and short in 30% highest past 2Y volatility/30% lowest gross profit-to-total asset ratio/30% highest 2Y total asset growth rate stocks. Average Across 6 factors are the mean annual returns in each year. 37

38 Diversified MultiStrategy Multi Factor Allocation Avoiding Concentration in a Single Factor The reward for exposure to these factors has been shown to vary over time (see e.g. Harvey [1989]; Asness [1992]; Cohen, Polk and Vuolteenaho [2003]). The indices are not perfectly correlated with each other which shows a potential for diversification across factors. There is strong intuition suggesting that multi-factor allocations will tend to result in improved risk-adjusted performance. US Long Term (Dec 1970 Dec 2015) Mid Cap Diversified MultiStrategy High Momentum Low Volatility Value Low Investment High Profitability Mid Cap High Momentum Low Volatility Value Low Investment High Profitability 1.00 Daily total returns in USD from 31/12/1970 to 31/12/2015 (45 years) are used for the US Long Term universe. The universe contains 500 stocks. The full names of the indices used are: SciBeta United States LTTR Mid-Cap Diversified Multi-Strategy, SciBeta United States LTTR High-Momentum Diversified Multi-Strategy, SciBeta United States LTTR Low- Volatility Diversified Multi-Strategy, SciBeta United States LTTR Value Diversified Multi-Strategy, SciBeta United States LTTR Low Investment Diversified Multi-Strategy and SciBeta United States LTTR High Profitability Diversified Multi-Strategy. Source: 38

39 Factor Fishing risks versus robust factors Model mining risks versus diverifying strategy specific risk Single factor tilts versus multiple factor tilts Performance analysis 39

40 Performance Analysis Long Term and Live Performance Smart beta providers often emphasize back-test results over a limited time period, usually over 10 to 15 years. Importance of Long Term Assessment (~40 years) Producing long-term track records prevents providers from favoring strategies that are based on short term trends and not based on consensual academic research. Such short periods do not allow one to assess the performance across different market conditions. Importance of Live Performance The persistence of outperformance beyond the back-test period points to absolute robustness of the strategies. The persistence conditional performance patterns points to relative robustness of the strategies. Below, we illustrate an assessment of long term and live performance for a set of multi factor indices published by Scientific Beta 40

41 USA LTTR 31-Dec-1972 to 31-Dec-2015 Annual Performance Analysis Long Term Performance of (Multi) Smart Factor Indices Smart factor indices generate attractive risk-adjusted performance. Combining them improves relative risk-adjusted return. Broad Cap- Weighted Scientific Beta USA LTTR Diversified Multi-Strategy Indexes High Low MBMS MBMS Mid Cap Value Momentum Volatility EW ERC 10.16% 14.30% 13.32% 12.94% 14.47% 13.83% 13.64% Returns Volatility 17.15% 16.03% 16.27% 14.08% 15.90% 15.34% 15.35% Sharpe Ratio Relative Returns % 3.16% 2.77% 4.31% 3.67% 3.48% Tracking Error % 4.90% 6.08% 5.53% 5.10% 4.78% Information Ratio Performance and Risk (Long Term Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The Multi-Beta Multi-Strategy EW (ERC) index is the equal weighted (equal relative risk contribution) combination of the four Diversified Multi-Strategy indices with stock selection based on Mid Cap, Momentum, Low Volatility, and Value respectively. All statistics are annualised and daily total returns from 31/12/1972 to 31/12/2015 are used for the analysis. The SciBeta CW US-500 index is used as the cap-weighted benchmark. The yield on Secondary US Treasury Bills (3M) is used as a proxy for the risk-free rate. The full names of the US indices used are: SciBeta Long Term United States Mid-Cap Diversified Multi-Strategy, SciBeta Long Term United States High-Momentum Diversified Multi-Strategy, SciBeta Long Term United States Low-Volatility Diversified Multi-Strategy, SciBeta Long Term United States Value Diversified Multi-Strategy, SciBeta Long Term United States Multi-Beta Multi-Strategy EW, SciBeta Long Term United States Multi-Beta Multi-Strategy ERC. Source: Scientific Beta US Long Term Smart Factor Indices have a 45-year track record, of which 2 years are used for calibration of parameters of MBMS ERC index. Consequently performance is reported only for a 43 year period. 41

42 USA LTTR 31-Dec-1972 to 31-Dec-2015 Mid Cap Scientific Beta USA LTTR Diversified Multi-Strategy Indexes High Low Value Momentum Volatility MBMS EW MBMS ERC Bull Markets Relative Returns 4.15% 3.11% -0.77% 2.94% 2.41% 2.38% Tracking Error 5.53% 4.07% 5.09% 4.76% 4.29% 4.05% Information Ratio Bear Markets Relative Returns 3.82% 2.97% 7.52% 5.86% 5.10% 4.71% Tracking Error 8.30% 6.42% 7.89% 6.98% 6.59% 6.13% Information Ratio Performance Analysis Long Term Conditional Performance Combining factor tilts leads to smoother outperformance across market regimes compared to the average component index to be improved Conditional Performance (Long Term Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The Multi-Beta Multi-Strategy EW (ERC) index is the equal weighted (equal relative risk contribution) combination of the four Diversified Multi-Strategy indices with stock selection based on Mid Cap, Momentum, Low Volatility, and Value respectively. All statistics are annualised and daily total returns from 31/12/1972 to 31/12/2015 are used for the analysis. The SciBeta CW US-500 index is used as the cap-weighted benchmark. The yield on Secondary US Treasury Bills (3M) is used as a proxy for the risk-free rate. Calendar quarters with positive benchmark returns comprise bull markets and the rest constitute bear markets. The full names of the US indices used are: SciBeta Long Term United States Mid-Cap Diversified Multi-Strategy, SciBeta Long Term United States High-Momentum Diversified Multi-Strategy, SciBeta Long Term United States Low-Volatility Diversified Multi-Strategy, SciBeta Long Term United States Value Diversified Multi-Strategy, SciBeta Long Term United States Multi-Beta Multi-Strategy EW, SciBeta Long Term United States Multi-Beta Multi-Strategy ERC. Source: Scientific Beta US Long Term Smart Factor Indices have a 45-year track record, of which 2 years are used for calibration of parameters of MBMS ERC index. Consequently performance is reported only for a 43 year period. 42

43 Performance Analysis Live Performance of Smart Factor Indices Single factor indices have live outperformance and also show pronounced differences: relative returns range from 2.74% to 0.13% Average live performance across the four indices smoothes such differences Multi-factor allocation helps to avoid being concentrated in the worst performing factor in a given period. Scientific Beta Developed World Diversified Multi- Average of Developed World Strategy Indexes the 4 smart 21-Dec-2012 to MSCI World High Low factor 31-Dec-2015 Mid Cap Value Momentum Volatility indexes Annual Returns 10.06% 11.93% 12.80% 12.56% 10.18% 11.87% Volatility 10.86% 10.04% 10.37% 8.96% 10.42% 9.95% Sharpe Ratio Relative Returns % 2.74% 2.50% 0.13% 1.81% Tracking Error % 2.51% 3.15% 1.96% 2.52% Information Ratio Performance and Risk (Live Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The analysis period is from 21-Dec-2012 to 31- Dec Daily total return series in USD are used. The risk-free rate is the return on the 3-month US Treasury Bill and the benchmark is the MSCI World Index. The live date of the four SciBeta single factor multi-strategy indexes is 21-Dec The full names of the indices used are: SciBeta Developed Mid-Cap Diversified Multi-Strategy, SciBeta Developed High-Momentum Diversified Multi-Strategy, SciBeta Developed Low-Volatility Diversified Multi-Strategy, SciBeta Developed Value Diversified Multi-Strategy. Source: 43

44 Performance Analysis Conditional Live Performance of SciBeta Indices In line with the long term performance, individual factor indices perform differently in different market regimes over the live period. Average conditional performance across the four indices smoothes such differences This suggests value-added of combining the single factor tilts during this period Developed World 21-Dec-2012 to 31-Dec-2015 Scientific Beta Developed World Diversified Multi- Strategy Indexes High Low Mid Cap Value Momentum Volatility Average of the 4 smart factor indexes Bull Markets Relative Returns 2.02% 2.13% 1.33% 0.70% 1.55% Tracking Error 2.47% 2.55% 3.07% 1.88% 2.49% Information Ratio Bear Markets Relative Returns 1.47% 4.95% 7.09% -1.73% 2.94% Tracking Error 2.56% 2.31% 3.53% 2.34% 2.68% Information Ratio Conditional Performance(Live Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The analysis period is from 21-Dec-2012 to 31-Dec Daily total return series in USD are used. The risk-free rate is the return on the 3-month US Treasury Bill and the benchmark is the MSCI World Index. Calendar quarters with positive benchmark returns comprise bull markets and the rest constitute bear markets. The live date of the four SciBeta single factor multi-strategy indexes is 21-Dec The full names of the indices used are: SciBeta Developed Mid-Cap Diversified Multi-Strategy, SciBeta Developed High-Momentum Diversified Multi-Strategy, SciBeta Developed Low-Volatility Diversified Multi- Strategy, SciBeta Developed Value Diversified Multi-Strategy. Source: 44

45 Conclusions Best Practices to Improve Robustness Factor Definitions Best Practice: Requirements for Robustness Consistent framework and deliberately conservative factors Common practice: Risk of Lack of Robustness Inconsistent methodologies and overfitted proprietary factors Weighting scheme Diversification (asking which strategies complement each other) Model Picking (asking which strategy has performed best) Factor selection Performance Analysis Multi factor allocation Long term backtests (~40 years), conditional performance analysis and live performance analysis Picking the winning factor Reliance on short term backtests (10 to 15 years) 45

46 References Amenc, N., F. Goltz, A. Lodh and L. Martellini Diversifying the Diversifiers and Tracking the Tracking Error: Outperforming Cap-Weighted Indices with Limited Risk of Underperformance. Journal of Portfolio Management 38(3) Amenc, N., F. Goltz, A. Lodh and L. Martellini Investing in Multi Smart Beta Portfolios: Reconciling Risk Factor Allocation and Smart Beta Allocation. EDHEC-Risk Institute Working Paper. Ang, A Factor Investing. Working Paper, Columbia University. Asness, C Changing Equity Risk Premia and Changing Betas over the Business Cycle and January. University of Chicago Working Paper. Chen, C., A. Huang and R. Jha Idiosyncratic Return Volatility and the Information Quality Underlying Managerial Discretion. Journal of Financial Quantitative Analysis 47(4): Choi, J. and M. Richardson The Volatility of Firm s Assets and the Leverage Effect. Working Paper. Cohen, R. B., C. Polk and T. Vuolteenaho The Value Spread. Journal of Finance 58(2): Fama, E., and K. French Size, Value, and Momentum in International Stock Returns. Journal of Financial Economics 105(3): Fama, E., and K. French A Five-Factor Asset Pricing Model. Journal of Financial Economics 116(1): Harvey, C. R Time-Varying Conditional Covariances in Tests of Asset Pricing Models. Journal of Financial Economics 24: Harvey, C., Y. Liu, and H. Zhu and the Cross-Section of Expected Returns. Review of Financial Studies 29(1): Kan, R. and G. Zhou Optimal Portfolio Choice with Parameter Uncertainty. Journal of Financial and Quantitative Analysis 42(3): Kogan, L., and M. Tian Firm Characteristics and Empirical Factor Models: A Data-Mining Experiment. Working Paper. Lo, A.W Data-Snooping Biases in Financial Analysis. In: H. Russell Fogler (ed.), Blending Quantitative and Traditional Equity Analysis AIMR. Martellini, L., V. Milhau and A. Tarelli The Trade-Off between Estimation Risk and Ignorance Risk in Portfolio Construction. EDHEC-Risk Institute Working Paper. Novy-Marx, R Backtesting Strategies Based on Multiple Signals. Working Paper. Timmermann, A Forecast Combinations. In: Elliott, G., C.W.J. Granger and A. Timmermann (Eds.), Handbook of Economic Forecasting Amsterdam: North Holland. Tu, J., and G. Zhou Incorporating Economic Objectives into Bayesian Priors: Portfolio Choice under Parameter Uncertainty. Journal of Financial and Quantitative Analysis 45:

47 Appendix Performance Analysis of MBMS Quality Indices 47

48 Performance Analysis Long Term Performance of SciBeta Quality Indices Investment and Profitability factor tilts generates attractive risk-adjusted performance. Combining them allows the relative risk-adjusted return to be improved. Scientific Beta USA LTTR USA LTTR Broad Cap- Diversified Multi-Strategy Indexes 31-Dec-1972 to 31-Dec- Weighted Low High MBMS 2015 Investment Profitability Quality Annual Returns 10.16% 14.08% 12.79% 13.47% Volatility 17.15% 15.15% 15.98% 15.44% Sharpe Ratio Relative Returns % 2.63% 3.31% Tracking Error % 4.40% 4.57% Information Ratio Performance and Risk (Long Term Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The Multi-Beta Multi-Strategy Quality index is the equal weighted combination of the two Diversified Multi-Strategy indices with stock selection based on Low Investment and High Profitability. All statistics are annualised and daily total returns from 31/12/1972 to 31/12/2015 are used for the analysis. The SciBeta CW US-500 index is used as the cap-weighted benchmark. The yield on Secondary US Treasury Bills (3M) is used as a proxy for the risk-free rate. The full names of the US indices used are: SciBeta Long Term United States Low Investment Diversified Multi-Strategy, SciBeta Long Term United States High Profitability Diversified Multi-Strategy, SciBeta Long Term United States Multi-Beta Multi-Strategy Quality. Source: Scientific Beta US Long Term Smart Factor Indices have a 45-year track record, of which 2 years are used for calibration of parameters of MBMS ERC index. Consequently performance is reported only for a 43 year period. 48

49 Performance Analysis Long Term Conditional Performance of SciBeta Quality Indices Combining factor tilts leads to smoother outperformance across market regimes compared to the average component index. USA LTTR 31-Dec-1972 to 31-Dec Scientific Beta USA LTTR Diversified Multi- Strategy Indexes Low Investment High Profitability MBMS Quality Bull Markets Relative Returns 2.50% 3.50% 3.03% Tracking Error 4.66% 3.85% 3.87% Information Ratio Bear Markets Relative Returns 5.56% 1.28% 3.43% Tracking Error 7.06% 5.46% 5.86% Information Ratio Conditional Performance (Long Term Track Record) The table compares the performance and risk of the SciBeta Diversified Multi-Strategy indices. The Multi-Beta Multi-Strategy Quality index is the equal weighted combination of the two Diversified Multi-Strategy indices with stock selection based on Low Investment and High Profitability. All statistics are annualised and daily total returns from 31/12/1972 to 31/12/2015 are used for the analysis. The SciBeta CW US-500 index is used as the cap-weighted benchmark. The yield on Secondary US Treasury Bills (3M) is used as a proxy for the risk-free rate. Calendar quarters with positive benchmark returns comprise bull markets and the rest constitute bear markets. The full names of the US indices used are: SciBeta Long Term United States Low Investment Diversified Multi-Strategy, SciBeta Long Term United States High Profitability Diversified Multi-Strategy, SciBeta Long Term United States Multi-Beta Multi-Strategy Quality. Source: Scientific Beta US Long Term Smart Factor Indices have a 45-year track record, of which 2 years are used for calibration of parameters of MBMS ERC index. Consequently performance is reported only for a 43 year period. 49