POWER AND LEVEL VALIDATION OF MOODY S KMV EDF CREDIT MEASURES IN NORTH AMERICA, EUROPE, AND ASIA

Transcription

1 SEPTEMBER 10, 2007 POWER AND LEVEL VALIDATION OF MOODY S KMV EDF CREDIT MEASURES IN NORTH AMERICA, EUROPE, AND ASIA MODELINGMETHODOLOGY AUTHORS Irina Korablev Douglas Dwyer ABSTRACT In this paper, we validate the performance of Moody s KMV EDF credit measures in its timeliness of default prediction, ability to discriminate good firms from bad firms, and accuracy of levels in three regions: North America, Europe, and Asia. We focus on the period for most of our tests. Wherever possible, we compare the performance to that of other popular alternatives, such as agency ratings, Moody s KMV RiskCalc EDF credit measures, Altman s Z-Scores, and a simpler version of the Merton model. We find that EDF credit measures perform consistently well across different time horizons, and different subsamples based on firm size and credit quality. Our tests indicate that EDF credit measures provide a very useful measure of credit risk that can be applied throughout the world.

2 Copyright 2007, Moody s KMV Company. All rights reserved. Credit Monitor, CreditEdge, CreditEdge Plus, CreditMark, DealAnalyzer, EDFCalc, Private Firm Model, Portfolio Preprocessor, GCorr, the Moody s KMV logo, Moody s KMV Financial Analyst, Moody s KMV LossCalc, Moody s KMV Portfolio Manager, Moody s KMV Risk Advisor, Moody s KMV RiskCalc, RiskAnalyst, Expected Default Frequency, and EDF are trademarks owned by of MIS Quality Management Corp. and used under license by Moody s KMV Company. Published by: Moody s KMV Company To Learn More Please contact your Moody s KMV client representative, visit us online at contact Moody s KMV via at info@mkmv.com, or call us at: NORTH AND SOUTH AMERICA, NEW ZEALAND AND AUSTRALIA, CALL: MKMV (6568) or EUROPE, THE MIDDLE EAST, AFRICA AND INDIA, CALL: FROM ASIA CALL:

3 TABLE OF CONTENTS 1 INTRODUCTION CREDIT RISK ASSESSMENT APPROACHES Moody s KMV EDF Credit Measures Agency Ratings Moody s KMV RiskCalc EDF Credit Measures Merton s Structural Model Altman s Z-Score EMPIRICAL METHODOLOGY Timely Default Prediction Default Predictive Power Level Validation with Default Data Interpreting the Analytical Outputs for Level Validation Level Validation with CDS Data Median EDF by Rating Category across Regions EMPIRICAL RESULTS North America Data Timely Default Prediction U.S Default Predictive Power U.S Accuracy of Levels U.S Timely Default Prediction Outside the U.S Default Predictive Power Outside the U.S Accuracy of Levels Outside the U.S Conclusion Europe Diversity in Bankruptcy Mechanisms and Creditor Protection Data Timely Default Prediction Default Predictive Power Level validation with default data Level Validation with CDS Data Conclusion

4 4.3 Asia Data Timely Default Prediction Default Predictive Power Level Validation Conclusion Median EDF by Rating Category across Regions CONCLUSION APPENDIX B: SUMMARY OF ACCURACY RATIOS FOR EDF CREDIT MEASURES AND AGENCY RATINGS BY YEAR

5 1 INTRODUCTION The new Basel Capital Accord states: The methodology for assigning credit assessments must be rigorous, systematic, and subject to some form of validation based on historical experience. There are two important components to this validation process: the ability to predict defaults and the accuracy of the default predictive measure. The first criterion implies that a credit measure should be dynamic enough to be a meaningful and timely signal of deteriorating credit quality or an impending credit event. In this regard, the Basel Accord states: Assessments must be subject to ongoing review and responsive to changes in financial condition. Before being recognized by supervisors, an assessment methodology for each market segment, including rigorous back-testing, must have been established for at least one year. This also means that the credit assessment technology should have the ability to distinguish between defaulters and non-defaulters. It should not allow defaulters to enter the sample while trying to create a sample of good quality firms (Type I Error). Conversely, it should not exclude good quality firms from the sample while trying to exclude potential defaulters (Type II Error). The second criterion is focused on the accuracy of the credit assessment measure so that it can be useful to banks and other financial institutions in their efforts toward risk measurement, valuation, and capital allocation. The Basel Accord states: Banks must have a robust system in place to validate the accuracy and consistency of rating systems processes, and the estimation of PDs (Probabilities of Default). The objective of this document is to compare the performance, based on the above validation criteria, of EDF credit measures with some of the other popular credit assessment approaches. The popular approaches that we consider are the following: Agency ratings RiskCalc U.S. v3.1 private firm model A Simple Merton structural model Altman s Z-Score In this paper we present our test results for three regions: North America, Europe and Asia. The rest of the paper is organized as follows: Section 2 discusses briefly the credit assessment approaches that we consider in our paper. Section 3 highlights the empirical methodology we follow to compare the approaches. Section 4 presents the results of our tests by region and interprets the economic meaningfulness of these results. 1 Section 5 concludes the paper. 2 CREDIT RISK ASSESSMENT APPROACHES The credit risk assessment approaches considered in this paper are: Moody s KMV EDF credit measures Agency ratings Moody s KMV RiskCalc private firm model Merton s structural model Altman s Z-Scores 2 In the following section we briefly discuss each of the approaches. 1 Section 4.1 presents the results for North America, section 4.2 presents the results for Europe, and section 4.3 presents the results for Asia. 2 For reasons explained in the next two sections, not all the approaches can be subjected to tests on all the criteria. We try to include as many of these approaches as possible in our test of each criterion. 5

6 2.1 Moody s KMV EDF Credit Measures The structural view on credit risk was first made commercially viable with the introduction of the Vasicek-Kealhofer (VK) model. This model offers a rich framework that treats equity as a perpetual down-and-out option on the underlying assets of the firm. This framework incorporates five different classes of liabilities: short-term liabilities, longterm liabilities, convertible debt, preferred shares, and common shares. To overcome the regular problems encountered by structural models due to the assumption of normality, the VK model uses an empirical mapping based on actual default data to get the default probabilities, known as EDF credit measures and offered by Moody s KMV. 3 Volatility is estimated through a Bayesian approach that combines a comparables analysis with an iterative approach. EDF credit measures are the outputs of Moody s KMV Credit Monitor and CreditEdge applications. An EDF credit measure is a quantitative measure of credit quality. More specifically, an EDF credit measure is an estimate of the physical probability of default for a given firm. For an overview of the EDF credit measure, see Crosbie and Bohn (2003). In 2007, Moody s KMV released EDF 8.0, which refines the mapping of the Distance-to-Default to the EDF credit measure using a much larger default database observed over a longer time period. Details of the new model enhancement can be found in Dwyer and Qu (2007). The EDF estimates are now bounded between 0.01% (for an EDF value of 0.01) and 35% (for an EDF value of 35). Moody s KMV offers a term-structure of EDF credit measures for 1 to 10 years and an extrapolation scheme to get shorter-term EDF credit measures. The risk free rate used in the calculation of EDF credit measures is now updated monthly. 2.2 Agency Ratings Moody s Investors Service, Standard and Poor s Corporation, and other well-known rating agencies around the world have been assigning credit ratings to major borrowers for decades. These are ordinal measures of credit measures (i.e., they help rank firms by their quality of credit). These ratings have established international credibility because of the long history of rating agencies, and the extensive testing of their relative performance. 2.3 Moody s KMV RiskCalc EDF Credit Measures Moody s KMV RiskCalc is designed to calculate EDF credit measures for private companies. Private companies are typically smaller than public companies and are not required to file financial statements with SEC. The RiskCalc model incorporates aspects of both the structural, market-based approach in the form of industry-level distance-to-default measures, and the localized financial statement-based approach. While it incorporates equity market information at the aggregate level, RiskCalc does not take advantage of the equity information of the specific company. We used the RiskCalc v3.1 U.S. model to obtain RiskCalc EDF credit measures for the set of publicly traded companies. Comparing public firm EDF credit measures to RiskCalc EDF credit measures computed on public firms represents an out-of-universe test of RiskCalc. 2.4 Merton s Structural Model The Merton model of risky debt is the original structural model of credit risk, and perhaps the most significant contribution to the area of quantitative credit risk research. This model assumes that equity is a call option on the value of assets of the firm. From this insight, the value of debt can be derived based on the observed equity value. The default event is modeled as the firm s asset value falling below a threshold level (i.e., default barrier). Given the default barrier, and the asset value parameters, the probability of default can be estimated for various horizons. A detailed description of this model can be found in most standard finance textbooks. 4 3 See Eom, Helwege, and Huang (2003) for details of the discussion. 4 See, for example, Hull (1999). 6

7 For our specific tests, the model has been implemented as: Default Point i,merton = Short Term Liabilities Long Term Liabilities The default probability for a firm i for a time horizon t is computed as: PD i =Φ AVL + Default Pointi,Merton σ t 2 ( μi σi ) i ln 0.5 t equity EVL i EVL i σi = σi AVLi AVLi i 1 (1) (2) rt ( ) ( ) EVL = AVL Φ d Default Point e Φ d d = i i 1 i,merton 2 AVL + + Default Pointi,Merton σ t 2 ( σ i ) i ln r 0.5 t d = d σ t i i (3) σ equity σ i, i, AVL i, and EVL i are the asset volatility, equity volatility, asset value and equity value of firm i, respectively. Φ (x) is the cumulative normal distribution function. μ i is the drift rate for the asset returns of firm i while r is the equity riskless rate of return. σ i is computed as the standard deviation of three years of weekly equity returns for company i. Asset value AVL i is computed by solving equations (2) and (3) simultaneously Altman s Z-Score Altman s Z-Score came as a response to the need for identifying the financial health of any business based on observable accounting and market ratios. This original measure was developed in 1968 by Edward Altman, whose Z-Score is available in various forms. We chose the public firm form, which includes market capitalization in the leverage ratio, and calculated Z-Scores as follows: 5 In contrast to the two equations and two unknowns, we use an iterative approach to solve for empirical volatility which is combined with modeled volatility in a Bayesian fashion. 7

8 Where Z = ( X + X + X + X + X ) CurrentLiabilities X 1 = 1.2 BookAssetValue (4) is the ratio of Current Liabilities to Total Assets; Retained Earnings X 2 = 1.4 Book Asset Value is the Profitability Ratio; Operating Income before Depreciation X 3 = 3.3 Book Asset Value is the ratio of EBIDTA to Total Assets; X = 4 Market Capitalization 0.6 Book Value of Liabilities is the ratio of Market Value of Equity to Book Value of Liabilities; and X = 5 Sales Book Asset Value is the ratio of Sales to Total Assets. The calculation typically produces a Z-Score between 5 and 10, with a high Z-Score implying a better credit quality and lower chance of bankruptcy. Z-Scores are not interpreted directly as default probabilities and therefore work as ordinal measures of financial health. Therefore, they cannot be used directly for valuation, quantitative risk assessment, and capital allocation purposes. 3 EMPIRICAL METHODOLOGY In this section, we describe the methodology we chose for tests of each criterion. 3.1 Timely Default Prediction Timeliness measures how many months before impending credit event EDF credit measures give signal of deteriorating credit quality. To test timeliness, we create a sample of defaulted firms, retaining monthly observations from 24 months prior to default up to12 months after default. We compute the median EDF credit measure and the median Moody s rating by months to default. We overlay and compare the median EDF credit measure and the median Moody s rating. For testing timeliness against rating, we use the Moody s rating. To ensure that the measure has stood the test of time and the rating grades and size, we also provide the analysis, wherever possible, for the subsets of data based on time period: and beyond 8

9 3.2 Default Predictive Power While a default predictive measure can be timely for warning of impending defaults, it may not be so effective in distinguishing a good firm from a bad firm. The calibration of the model may be on the conservative side inflating the default probability of all suspect names, of which some names might not be genuinely distressed. In this case, even though one could claim that the model performed well in predicting impending defaults, it would be fairly mediocre in its ability to distinguish good firms from bad firms. One of the essential features of a good model is that it should be sophisticated enough to differentiate bad (genuinely distressed) firms from good (false alarms) firms. There are two wellknown approaches to testing a model for its power: Cumulative Accuracy Profile (CAP) with its output known as Accuracy Ratio (AR). Receiver Operating Characteristic (ROC) with its output known as Area Under Curve (AUC). Typically, the larger the Accuracy Ratio or Area Under Curve, the better the model. In extreme cases, a totally random model that bears no information on impending defaults has AR = 0, and AUC = 0.5. For a perfect model, AR = AUC = 1. The two approaches are equivalent with AR = 2AUC-1. A more detailed discussion can be found in Appendix A. In this article, we use the Cumulative Accuracy Profile approach, and provide AR as our output. We compared EDF credit measures to: Ratings RiskCalc EDF credit measure Simple Merton model Altman s equity-based Z-Score. 3.3 Level Validation with Default Data The level validation of EDF credit measures verifies how well the model s predicted default rates track realized default rates. We employ the same methodology described in Bohn, Arora and Korablev (2005) which was first developed in Kurbat and Korablev (2002). The procedure is summarized into the following four steps: 1. Using Monte Carlo technique, we simulate asset value movements based on a single factor Gaussian model to capture correlated defaults. 2. We determine default/non-default state based on the level of each firm s EDF credit measure and each simulation outcome. 3. We compare the actual default rate to the median, 10 th percentile and 90 th percentile of the simulated distribution. 4. We compute the probability of observing a default rate less than or equal to the realized default rate given the model and the correlation coefficient. We extend this methodology by using Bayesian methods to compute the posterior distribution of the aggregate shock given the realized default rate, the model, and the correlation coefficient. The extension to the original methodology is developed in Dwyer (2007) Interpreting the Analytical Outputs for Level Validation We create two graphs as an output to the level validation test. Figure 1 is the illustrative example of the output, and is the comparison of the median predicted (by simulation) default rate and realized default rate. The median predicted default rate is the black line. Red line represents the actual default rate. Fifty percent of the time the actual default rate should be above (or below) the median. We also show the mean of predicted default rate, which is the blue line. Most of the time the actual default rate should be below the average predicted default rate. The two gray lines correspond to the prediction interval which represents the range of variability that is expected in the realized default rates given the EDF 9

10 values and the assumed correlation model. This prediction interval implies that eighty percent of the time the realized default rate should lie within the 10 th and the 90 th percentiles. 6 The actual default rate should lie within the 10 th and 90 th percentile 80% of the time. The actual default rate. The average predicted default rate. Most of the time the actual default rate should be below this average. The median predicted default rate. Fifty percent of the time the actual default rate should be above (or below) the median. FIGURE 1 Illustrative example of the level validation output. Comparison of median predicted default rate and realized default rate. 6 This prediction interval differs from the concept of a confidence interval. An x% confidence interval is random interval for which the probability of it holding the true value of a parameter is x%. In our context here, an x% prediction interval has the interpretation that x% of the time the realized default rate will be within this range given the EDFs levels and the correlation model. 10

11 P-value measures the probability of observing a default rate at or lower than the actual default rate Median value of the aggregate shock given the actual default rate FIGURE 2 Illustrative example of the level validation output. Posterior distribution of the aggregate shock and P-value of the actual default rate The figure depicts the posterior distribution for the aggregate shock that was derived given the realized default rate, the model and the correlation coefficient. We also computed the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. This P-value is shown as a blue line. 3.4 Level Validation with CDS Data This test analyzes the level bias in European EDF credit measures relative to that of U.S. EDF credit measures. The rationale for the test is based on the assumption that similar risks should offer similar premium in the U.S. and Europe. We compare the median as well as 25 th and 75 th percentile CDS levels of two regions: U.S. and Europe across EDF-implied rating groups. The same EDF categories should have same aggregate median spreads in CDS market across two regions. We used Mark-It composite CDS data from January 2003 to December The Europe region is based on the following currency information: Euro, Austrian Schilling, Belgian Franc, Swiss Franc, Czech Republic Koruna, Deutsche Mark, Danish Kroner, Spanish Peseta, Finnish Markka, French Franc, Greek Drachmae, Hungarian Forint, and British Pound. The U.S. region is based on the U.S. dollar. 3.5 Median EDF by Rating Category across Regions We calculate and compare median EDF credit measures for North American non-financial companies, Asian-Pacific non-financial companies, European non-financial companies and global financial companies by several rating categories. In the absence of other measures of credit risk, e.g., spreads or defaults, a comparison with rating provides a sanity check on the rank ordering of risk produced by the EDF credit measure and the comparableness of level of the EDF credit measure across geographies. 11

12 4 EMPIRICAL RESULTS In this section, we describe empirical results. 4.1 North America In this section, we describe empirical results obtained in North America. Results are separated into U.S. and North American companies that are headquartered outside of the U.S. These companies are predominantly headquartered in Canada, Bermuda and the Cayman Islands Data We start with all U.S. firms that have publicly traded equity from , unless otherwise specified. We restrict the sample to non-financial firms with more than $30 million in size. 7 For level validation we impose further restriction of $300 million in size. We also present results for comparable North American firms that are outside of the U.S. (Canada, Bermuda, Cayman Islands, Bahamas, Belize, Panama, Virgin Islands, and Netherlands Antilles). Table 1 shows the countries and the number of firm-months in each country that constitute North American module in Credit Monitor and CreditEdge. Outside of the U.S., the largest countries are Canada, Bermuda and the Cayman Islands. TABLE 1 Countries in the North American Database Country Number of Observations (firm-month) Netherlands Antilles 776 Bahamas 440 Belize 85 Bermuda 3,552 Canada 153,971 Cayman Islands 975 Panama 245 USA 1,127,452 Virgin Islands 491 For all comparison against ratings, we used Moody s ratings. Defaults are based on the Moody s KMV Default database and include missed payments, distressed exchanges, and insolvency proceedings. The defaults have been collected on a daily basis for more than ten years using a variety of printed and on-line sources. 8 By the end of 2006, we had about 7,900 public defaults worldwide. About 5,600 defaults were from North America. 7 Size is measured by the sales of the firm for non-financial firms. Wherever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using a deflation adjustor calculated internally at Moody s KMV. 8 To collect defaults, we use numerous printed and online sources from around the world on a daily basis. We use government fillings, government agency sources, company announcements, news services, specialized default news sources and even sources within financial institutions to ensure to the greatest extent possible that we find all defaults. We also keep evidences in electronic format so that content can be easily verified. As a result, Moody s KMV has the most extensive default database for public firms. 12

13 4.1.2 Timely Default Prediction U.S. In this section, we compare the performance of EDF credit measures against agency ratings in their ability to predict timely defaults. Figure 3 demonstrates how the median EDF credit measure (represented by the solid black line) starts rising 24 months before the actual default, while the median Moody s rating stays flat until 13 months before default, and then shows a steep rise about 5 months before default. In that sense, the EDF credit measures seem to lead the ratings. This is also helped further by the fact that the EDF credit measure is more continuous, and therefore one can see a steady and continuous rise in the aggregate. Ratings, on the other hand, are discrete, and therefore one sees a step-like function with flat stretches implying that this measure does not instantaneously pick up the most currently available information. To test for the robustness of the results, we further divided our data into the subperiods: The period is shown on the left panel of Figure 4, and the period is shown on the right panel of Figure 4. Both EDF credit measures and ratings start at a higher level 24 months prior to default in the latter half of the sample. EDF credit measures continued to lead the agency rating in each subperiod, indicating that EDF credit measures indeed provide a more timely warning of impending defaults. EDF measure is leading rating by 11 months FIGURE 3 Comparison of median agency ratings with Moody s KMV EDF values for rated defaulted firms in the U.S. from 2 years before default to 1 year after default between 1996 and

14 FIGURE 4 Comparison of median agency ratings with Moody s KMV EDF values for rated defaulted firms in the U.S. from 2 years before default to 1 year after default for subsamples: (left panel) and (right panel) Default Predictive Power U.S. In this section, we compare the performance of EDF credit measures against agency ratings, Z-Scores, and a simple Merton model in its ability to discriminate between good and bad firms. Our test statistic is the Accuracy Ratio as defined earlier. We also show the plots of Cumulative Accuracy Profiles of these measures for various subsamples selected using different horizons and size filters. EDF Credit Measure vs. Agency Rating Figure 5 shows the performance of EDF credit measures against ratings on the entire sample period of By design, this test is restricted to the sample of rated firms only. It is clear that the EDF credit measure performs better than ratings on the entire sample period with their Accuracy Ratios at 0.88 and 0.75, respectively. To ensure that the measure is robust in its performance across various time horizons, we divide our sample into two subsets of data based on time periods: We provide the analysis by three different size categories: Size is greater than $30 million Size is between $30 and $300 million Size is greater than $300 million 14

15 FIGURE 5 Cumulative Accuracy Performance (CAP) curves comparing Moody s KMV EDF credit measures and agency ratings for U.S. non-financial companies between 1996 and The Accuracy Ratios for EDF measure and agency rating are 0.88 and 0.75, respectively. Table 2 illustrates the results for the subsamples. We find that the EDF credit measure substantially outperforms ratings, in all categories by at least 12%. TABLE 2 Accuracy Ratios by category for EDF Credit Measures and agency ratings for U.S. non-financial companies Date EDF Credit Measure Ratings , Size > $30 Million , Size $30-$300 Million , Size> $300 Million

16 We also calculated Accuracy Ratios at the horizons longer than one year. The results are presented in Table 3. EDF credit measures have more discriminatory power than agency ratings at all horizons, but the difference is smaller at longer horizons. TABLE 3 Accuracy Ratios of one- to five-year EDF credit measures and agency ratings for U.S. non-financial companies between 1991 and 2006 One-year EDF credit measure Two-year EDF credit measure Three-year EDF credit measure Four-year EDF credit measure Five-year EDF credit measure EDF Credit Measure Ratings Number of Observations Number of Defaults The Accuracy Ratios (AR) for both the EDF credit measure and agency rating decreases with horizon. The difference between ARs becomes more compressed at longer horizons. Figure 6 and Figure 7 present the Accuracy Ratios for the EDF credit measure and agency rating by year at one- and five-year horizons respectively. 9 For each year, we used the EDF credit measure as of the last market day of the prior year to predict default during the next one or five years. At a one-year horizon, the EDF credit measure has better discriminatory power than agency rating in all years, except 1996, which had the least number of defaults. At a five-year horizon, the EDF credit measure also outperforms agency rating in all years except The numbers underlying Figures 6 and 7 are summarized in Tables 15 th and 16 th of Appendix B. 16

17 Year EDF Credit Measure Agency Rating FIGURE 6 Accuracy Ratios for EDF credit measures and agency ratings for U.S. non-financial companies by year at the one-year horizon Year EDF Credit Measure Agency Rating FIGURE 7 Accuracy Ratios for EDF Credit Measures and agency ratings for U.S. non-financial companies by year at the five-year horizon EDF Credit Measure vs. Merton Default Probability and Z-Score In this section we compare the performance of EDF credit measures to the Merton model s implied default probabilities and Z-Scores as described in Section 2. The sample period used is between 1996 and Unlike the rated firms, which are usually larger and higher profile, some of the unrated firms can be very small and their defaults can go unnoticed. In some cases, there can be some informal negotiations or bailouts, avoiding the default. These cases are likely 17

18 to contaminate our results. Therefore we filtered out very small firms (size < 30 million dollars) from our sample. 10 For the entire period , the results are shown in Figure 8. The results are presented on a joined sample of Z-Scores, Merton default probabilities, and EDF credit measures, which require each of these values to be non-missing. We find that the EDF credit measure substantially outperforms Merton default probability and Z-Score in terms of their ability to discriminate good firms from bad firms with their Accuracy Ratios at 0.82, 0.72, and 0.66 respectively. We further divide the sample into subsets of sizes 30 million dollars to 300 million dollars, and 300 million dollars and above. In both cases, the EDF credit measure outperforms the Merton model and Z-Score, as shown in Table 4. Once again, as a robustness check, we compared the performance of the two measures across the time horizons , and The results are shown in Table 4. As expected, our results are fairly robust with EDF credit measures outperforming Merton default probabilities and Z-Scores across both horizons. FIGURE 8 Cumulative Accuracy Performance (CAP) curves comparing Moody s KMV EDF credit measures, Merton default probability and Z-Scores for U.S. non-financial companies between 1996 and 2006.The Accuracy Ratios for EDF measure, Merton Default Probability and Z-Score are 0.82, 0.72 and 0.66 respectively. 10 Size is measured by the sales of the firm for non-financial firms. Whenever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using a deflation adjustor calculated internally at Moody s KMV. 18

19 TABLE 4 Summary of Accuracy Ratios across various size buckets and time horizons for EDF credit measure, Merton default probability, and Z-Score for U.S. non-financial companies Date/Size EDF Credit Measure Z-Score Merton Default Probability , Size >$30Mln , Size >$30Mln , Size >$30Mln , Size $30-$300 Million , Size> $300 Million EDF Credit Measure vs. RiskCalc EDF Credit Measure In this section we compare the performance of EDF credit measures to RiskCalc EDF credit measures calculated for Public firms as described in Section 2. The sample period used was As before, we filtered out very small firms (size < 30 million dollars) from our sample. 11 For the entire period , the results are shown in Figure 9. We find that EDF credit measures have more discriminatory power than RiskCalc EDF credit measures, which we expected because RiskCalc does not incorporate firm-specific equity market information. Their Accuracy Ratios are at 0.82 and 0.68 respectively. We further divided the sample into subsets of sizes of 30 million dollars to 300 million dollars, and 300 million dollars and above. In both cases, EDF credit measures outperform RiskCalc EDF credit measures, as shown in Table 5. Both measures perform better for larger firms. Once again, as a robustness check, we compared the performance of the two measures across the time horizons , and The results are presented in Table 5. As expected, our results are fairly robust with the EDF credit measures outperforming the RiskCalc EDF credit measures across both horizons. The Accuracy Ratio of the EDF credit measure is higher in the second period while Accuracy Ratio of the RiskCalc EDF stays the same. 11 Size is measured by the sales of the firm for non-financial firms. Wherever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using a deflation adjustor calculated internally at Moody s KMV. 19

20 20 FIGURE 9 Cumulative Accuracy Performance (CAP) curves comparing Moody s KMV EDF credit measures and RiskCalc EDF credit measures between 1996 and 2006 for U.S. non-financial companies. The Accuracy Ratios for EDF measure and RiskCalc EDF measure are 0.82 and 0.68 respectively.

21 TABLE 5 Summary of Accuracy Ratios for EDF Credit Measures and RiskCalc EDF Credit Measures for U.S. non-financial companies by different size buckets and time periods Date / Size , Size >$30 Million , Size >$30 Million , Size >$30 Million , Size $ Million , Size>$300 Million EDF Credit Measure RiskCalc EDF Credit Measure The EDF credit measure effectively discriminates between good and bad credits. It performed better than Z-Score, RiskCalc for private firms applied for publics, and simple implementation of a Merton model. It leads rating changes in predicting defaults and it performs well across multiple cuts of the data and multiple horizons Accuracy of Levels U.S. The test for this criterion draws from the methodology used by Korablev and Kurbat (2002), and Bohn, Arora and Korablev (2005), which is described in Section 3. We also extended this methodology by using Bayesian methods to compute the posterior distribution of the aggregate shock given the realized default rate, the model and the correlation coefficient as described in Dwyer (2007). The other alternatives of credit risk measurement cannot be directly interpreted as physical default probabilities, or provide a framework that can account for the underlying correlations between assets. Therefore they cannot be compared against EDF credit measures for the level test. 12 Secondly, we have issues of hidden defaults or missing defaults for smaller firms, as explained in Kurbat and Korablev (2002). Therefore, consistent with that paper, we restrict this test to firms of size 300 million and above. We first present results broken down by coarser levels of the EDF credit measure, then repeat the analysis for narrower ranges of the measure. Results for Firms with EDF Values Below 35% In the previous validation studies (Kurbat and Korablev (2002), Bohn, Arora and Korablev (2004)), the test was performed on the EDF 7.1 model, which was capped at 20%. In that case the predicted number of defaults was likely to underestimate the realized number of defaults due to the truncation effect. Therefore we divided our sample into two: EDF credit measures less than 20% and EDF credit measures equal to 20%. One of the main features of the EDF 8.0 model is the new cap of 35%. Now we can expect that the truncation effect would lessen or even disappear. Nevertheless, to be consistent with the previous studies we decided to split the sample into two: firms with EDF values less than 35% (3500 bps) and firms with EDF values equal to 35%. The comparison for the sample of firms with EDF values less than 35% is shown in Figure 10. The left panel of Figure 10 displays mean, median predicted (by simulation) and actual default rate for EDF values below 35% along with 80% confidence set for the predicted default rate. We used an asset correlation of 0.19 to simulate defaults in each year. The right panel of the 12 The exception to this is the Merton model but the default probabilities are too low as implied by the Merton model, and therefore it would usually underestimate the predicted number of defaults. 21

22 Figure 10 presents the posterior distribution for the aggregate shock given the actual default rate and P-values of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. The predicted default rate clearly tracks the realized default rate very well. All predicted default rates fall within the confidence set. The exception is year 2003, which was an uncharacteristically good year for the economy leading to a substantially lower number of defaults. In year 2003, to explain the low default rate, we estimate that the U.S. economy received a positive 0.84 standard deviation shock relative to market expectations. Such a positive shock is consistent with the high returns on the S&P 500 observed during that year. The P-values of the realized default rate range from 21% to 75%, which is within the sampling variability that would be expected. FIGURE 10 Comparison of median predicted default rate with the realized default rate, The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure less than 35%. We used an asset correlation of 0.19 to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. We summarize the numbers that underlie Figure 10 in Tables 6 and 7. Table 6 contains the number of firms, number of defaults, median and mean predicted default rate per year as well as the 10 th and 90 th percentiles for predicted default rate. It is clear from this table that the correlation effect skews the distribution of default rates to the left. If we ignored this effect and had simply taken the mean default rate of the sample, we would have grossly over-predicted the realized default rate. Table 7 contains the median aggregate shock, the 10 th and 90 th percentiles of the aggregate shock, and the p-value by year. 22

23 TABLE 6 Comparison of mean and median predicted default rate with the realized default rate between 1991 and 2006 Year Mean Predicted Default Rate Median Predicted Default Rate Realized Default Rate 10th percentile 90th percentile Firms Defaults % 1.7% 2.5% 0.5% 4.9% % 1.0% 1.0% 0.2% 3.2% % 0.9% 0.9% 0.2% 2.9% % 0.7% 0.6% 0.1% 2.5% % 0.7% 0.9% 0.1% 2.6% % 0.8% 0.9% 0.2% 2.8% % 0.8% 0.8% 0.2% 2.6% % 0.7% 0.9% 0.1% 2.4% % 1.2% 1.0% 0.3% 3.9% % 1.9% 1.9% 0.5% 5.5% % 2.8% 2.7% 0.8% 7.3% % 1.9% 1.8% 0.5% 5.4% % 2.3% 1.0% 0.6% 6.2% % 0.8% 0.7% 0.2% 2.8% % 0.5% 1.0% 0.1% 1.9% % 0.4% 0.2% 0.1% 1.5% The sample was restricted to U.S. firms larger than 300 million dollars with EDF credit measures less than 35%. 23

24 TABLE 7 Summary table of aggregate shock and year-wise probability of realizing the actual number of defaults between 1991 and 2006 Year 10th Percentile Median Aggregate Shock 90th Percentile Probability of having actual defaults or even lower % % % % % % % % % % % % % % % % The sample was restricted to U.S. firms larger than 300 million dollars with EDF credit measures less than 35%. Results for Firms with EDF Values Equal to 35% Figure 11 shows the median predicted and actual number of defaults for EDF credit measures of 35%. We used as an asset correlation for pairs of firms in each year to simulate defaults. The companies in this sample are, on average, somewhat less correlated with each other than the set of firms with EDF credit measures of less than 35%. We find that the realized default rate ranges from 11% to 67%. The high default rate in 1998 is indicative of a large negative shock which is shown in Figure 11 along with the P-values of the realized default rate. The P-values range from 8% to 93%, which is within the sampling variability that would be expected over a 15-year period. 24

25 FIGURE 11 Comparison of median predicted default rate with the realized default rate, The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure equal to 35%. We used an asset correlation of to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. We summarize the numbers that underlie Figure 11 in Tables 8 and 9. Table 8 contains the number of firms, the number of defaults, the median and mean predicted default rate per year, as well as the 10 th and 90 th percentiles for the predicted default rate. It is clear from this table that the correlation effect skews the distribution of default rates to the left. Table 9 contains the median aggregate shock, the 10 th and 90 th percentiles of the aggregate shock, and the P-value by year. 25

26 TABLE 8 Comparison of mean and median predicted default rate with the realized default rate between 1991 and 2006 Year Mean Predicted Default Rate Median Predicted Default Rate Realized Default Rate 10th Percentile 90th Percentile Firms Defaults % 33.4% 40.0% 12.7% 59.7% % 33.4% 24.0% 12.3% 60.3% % 33.5% 33.3% 11.8% 60.9% % 33.5% 11.8% 11.2% 61.9% % 33.7% 15.4% 10.4% 63.5% % 33.6% 12.5% 11.0% 62.2% % 34.2% 11.1% 9.2% 67.1% % 33.6% 66.7% 10.8% 62.6% % 33.4% 35.1% 13.1% 59.2% % 33.5% 38.3% 13.6% 58.8% % 33.5% 40.2% 14.5% 57.8% % 33.5% 41.0% 13.9% 58.4% % 33.5% 38.0% 14.1% 58.2% % 33.4% 23.1% 12.4% 60.1% % 33.5% 10.0% 11.7% 61.1% % 33.5% 16.7% 11.4% 61.6% 18 3 The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure equal to 35%. 26

27 TABLE 9 Summary table of aggregate shock and year-wise probability of realizing the actual number of defaults between 1991 and 2006 Year 10th Percentile Median Aggregate Shock 90th Percentile The probability of having actual defaults or even lower % % % % % % % % % % % % % % % % The sample was restricted to U.S firms larger than 300 million dollars with EDF credit measures equal to 35%. Results by EDF Subgroups To test the robustness of our results, we further divide the sample of firms with EDF values less than 35% into smaller groups. EDF buckets that we used along with correlation for default simulation in each bucket are presented in Table 10. TABLE 10 EDF buckets Stratum EDF Range Correlation Number of Firms Figures 12, 13, and 14 show the median, mean, and the prediction interval for the realized default rate and actual default rate for EDF values in the range [0.02, 5), [5,12), and [12,35), respectively. It is clear from these figures that while the predicted and realized default rates can deviate from each other in certain years, there is no substantial bias in their levels over the long run. In general, the two levels track each other very well. All predicted default rates fall within the 27

28 prediction interval. Year 2003 was an uncharacteristically good year for the economy leading to a substantially lower number of defaults in two of the three subgroups. FIGURE 12 Comparison of median predicted default rate with the realized default rate, The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure between 0.01% and 5%. We used an asset correlation of to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. FIGURE 13 Comparison of median predicted default rate with the realized default rate, The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure between 5% and 12%. We used an asset correlation of to simulate defaults in each year. On the left panel, the gray lines represent 80% prediction interval for predicted default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the posterior distribution of the aggregate shock and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th 28

29 percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. FIGURE 14 Comparison of median predicted default rate with the realized default rate, The sample was restricted to U.S. firms larger than 300 million dollars and EDF credit measure between 12% and 34.99%. We used an asset correlation of to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate Timely Default Prediction Outside the U.S. The Timeliness test outside the U.S. produces very similar results to those in the U.S. The median EDF credit measure starts rising 24 months before the actual default, while the median rating rises 18 months before default from B2 to B3, then stays flat until six months before default at which point it rises sharply. EDF credit measures clearly lead ratings. 29

30 FIGURE 15 Comparison of median agency ratings with Moody s KMV EDF values for defaulted firms from two years before default to one year after default for North American companies outside the U.S. and sample period between 1996 and Default Predictive Power Outside the U.S. In this section, we compare the performance of Moody s KMV EDF credit measures Z-Scores and a simple Merton model in its ability to discriminate between good and bad firms. We do not perform a power test against Agency Rating because of the small number of rated defaults outside the U.S. EDF Credit Measure vs. Merton Default Probability and Z-Score In this section we compare the performance of EDF credit measures to the Merton model and Z-Scores as described in Section 2. The sample period used was We filtered out very small firms (size < 30 million dollars) from our sample as we did in the case of U.S. companies. 13 Results for the entire period , are shown in Figure 16. The results are presented as a sample of Z-Scores, Merton Default Probabilities, and EDF credit measures. All three values should be non-missing to be included in the sample. We find that the EDF credit measure outperforms Merton Default Probability and Z-Score as a more effective statistic to discriminate good firms from bad firms with their Accuracy Ratios at 0.78, 0.70 and 0.65, respectively. Because of the sample size, we do not divide the sample into two subsamples as we did in the U.S. 13 Size is measured by the sales of the firm for non-financial firms. Whenever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using a deflation adjustor calculated internally at Moody s KMV. 30

31 FIGURE 16 Cumulative Accuracy Performance (CAP) curves comparing EDF credit measures, Merton Default Probability and Z-Scores between 1996 and 2006 for North American companies outside the U.S. The Accuracy Ratios for the EDF credit measure, Merton Default Probability and Z-Score are 0.78, 0.70 and 0.65 respectively Accuracy of Levels Outside the U.S. Figure 17 presents the level validation results for the sample of firms with EDF credit measures below 35%. The left panel of the Figure 17 displays mean, median predicted, and actual default rate as well as 80% confidence set for predicted defaults. We used an asset correlation of 0.19 to simulate defaults in each year. The right panel of Figure 17 displays the posterior distribution for the aggregate shock given the actual default rate and P-values of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. Predicted default rate tracks the realized default rate very well. Realized default rate fluctuates around median predicted default rate. In all years, except 1991, predicted default rates fall within the confidence set. Year 1991 was a good year, leading to a lower number of defaults. In year 1991, to explain the low default rate, we estimate that the U.S. economy received a positive 0.82 standard deviation shock relative to market expectations. P-values are between 8% and 75% which is in the range we expect over 15-year period. 31

32 FIGURE 17 Comparison of median predicted default rate with the realized default rate, The sample was restricted to North American firms outside the U.S. larger than 300 million dollars and EDF credit measure less than 35%. We used an asset correlation of 0.19 to simulate defaults in each year. On the left panel the gray lines represent 80% prediction interval for the realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate and red dotted line is the realized default rate. The right panel shows the posterior distribution of the aggregate shock and P-values. The dark black line, rm50 is the median of the aggregate shock; and the grey lines, rm10 and rm90, are the 10 th and 90 th percentiles for the aggregate shock; the blue line is the P-vale of the actual default rate, which is the probability of observing a default at or lower than the actual default rate Conclusion Results obtained for the North American sample show that the EDF credit measure leads the agency rating in timely default prediction. The EDF credit measure leads other alternative measures in its ability to discriminate good firms from bad firms over time and across various subsections of the data. We also showed that the model predicted default rates track realized default rates well and the model works well not only in the U.S., but also in North America excluding the U.S. 4.2 Europe In this section, we describe the results obtained in Europe Diversity in Bankruptcy Mechanisms and Creditor Protection Bankruptcy mechanisms can differ between regions. For example, Davydenko and Franks (2005) found that while the British bankruptcy mechanism is designed to be extremely creditor friendly, the French system is geared toward protecting a business as a going concern even at the expense of its creditors. 14 While interpreting the validation results, it is important to understand the impact of these mechanisms on the outcome of the model. For example, if a system is too creditor-friendly, the creditors can pressure the firm at the slightest hint of distress. This action may cause a firm to file for bankruptcy sooner, although the recovery for creditors may be higher. On the other hand, if the system is too geared toward protecting a firm, the creditors may not be allowed to take a firm to court even if it is in severe distress. 14 A brief description of the similarities and differences among British, French, and German bankruptcy mechanisms is provided in Korablev (2005). 32

33 A second characteristic is the nature of debt in an economy. A creditor-debtor relationship might be close (as in Japan), or at arm s length (as in the U.S.). If the creditors are few and have a close relationship with the debtor, they are more likely to evaluate the long-term potential of the debtor before taking it into bankruptcy. If the creditors are scattered, there is a higher likelihood of a free-rider problem, leading to a forced bankruptcy even if the debtor may have some long-term positive potential. In general, we see an equal contribution of non-bankruptcy defaults and bankruptcies in North America, while the European cases of distress are dominated by bankruptcies as shown in Figure This may be influenced by two factors. First, in many economies within Europe, the debt is held more closely relative to that in the U.S., making it more likely to enter private renegotiations of debt and avoid default during times of a liquidity crunch. Second, many cases of defaults may not be covered by the media, and are in that sense hidden. These two factors should not be applicable to larger firms because their debt is usually widely held, and they are followed more closely by media. Figure 19 compares the percentage representation of default cases in Europe and North America by size over the period of Defaults as a fraction of total distress cases are substantially smaller in Europe for small and mid-sized firms. Larger firms, however, have more comparable default behavior across Europe and North America. This shows that the model validation is more reliable on the sample of large companies because of the quality of data on actual defaults. North America Europe 100% 100% 80% 80% 60% 60% 40% 40% 20% 20% 0% % Bankruptcy Defaults Bankruptcy Defaults FIGURE 18 Percentage representation of defaults and bankruptcies in North American and European Markets between 1996 and 2006 North America Europe 80% 70% 70% 60% 50% 40% 30% 20% 10% 60% 50% 40% 30% 20% 10% 0% % Size < 30 Million 30 Million <= Size <= 300 Million Size > 300 Million Size < 30 Million 30 Million <= Size <= 300 Million Size > 300 Million FIGURE 19 Default events as a percentage of all distress cases across three size buckets between 1996 and The following events constitute non-bankruptcy defaults: missed interest or principle payment, distressed extension of a loan, distressed exchange offer, delay in paying substantial portion of trade debt, and government takeover of financial institution to prevent market collapse. 33

34 The success of a model relies on the ability of the inputs to take regional nuances into account. A model whose inputs are not universal in concept may have more difficulty capturing the differences in characteristics of the system in which it is being implemented. As long as the economic fundamentals of a model are universal in nature, it is not necessary to interpret its output differently across different regions. For the Moody s KMV EDF model, one of the main drivers is asset value, which is inferred from the equity value and an underlying structural framework. The model should work well for data from individual regions and for data pooled across them because the equity markets take into account the regional differences. The extent to which different equity markets accurately reflect firm value and volatility has implications for the power and the level performances of the model. In fact, even if a model is powerful in discriminating defaulters from non-defaulters in different regions, but is off in its level performance, the aggregation of data across regions will make the model seem less powerful. For example, if a distance-to-default (DD) of 2 corresponds to an EDF credit measure of 5% in the U.K., but 2% in France, then an aggregation of data would incorrectly suggest that both a U.K. firm and a French firm with a DD of 2 correspond to the same rank in our test. In that sense, a default predictive power test on a dataset aggregated across different regions essentially tests a joint hypothesis that the model is powerful and that the DD-to-EDF mapping is similar across different regions. It could be the case that the model might be powerful in two regions separately, but may appear less powerful if the data are aggregated. Similarly, while testing for levels, one could imagine that the model had specified levels in two regions incorrectly, overestimating the default rate in one region and underestimating it in the other. However, it may work well on the aggregated dataset. Therefore, a reasonable level performance on aggregated data is a necessary, but not a sufficient, test for the level performance of the model in each region. Unfortunately, there is an insufficient number of defaults available to perform a reliable level test in each subregion of Europe Data We start with all European firms that have publicly traded equity between 1996 and The sample was then restricted to non-financial firms with more than $30 million in size to avoid missing and hidden default problem. 16 For level validation we imposed a further restriction of $300 million in size. 16 Following our practice in North America, size is measured by the sales of the firm for non-financial firms. Whenever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using the appropriate consumer price index and exchange rate. 34

35 TABLE 11 Number of companies by country in the European Module of Credit Monitor and CreditEdge Country Country Code Size >= $30 Million Size >= $300 Million Austria AUT Belgium BEL Switzerland CHE Czech Republic CZE Germany DEU Denmark DNK Spain ESP Finland FIN France FRA Great Britain GBR Greece GRC Hungary HUN Ireland IRL Iceland ISL 7 5 Israel ISR Italy ITA Luxemburg LUX Netherlands NLD Norway NOR Poland POL Portuguese PRT Russia RUS Slovakia SVK 16 4 Slovenia SVN 8 7 Sweden SWE Turkey TUR We also present the results for level validation for subsample of countries that have more than 100 companies of size $300 million. These countries include Switzerland, Germany, Spain, France, Great Britain, Italy, Netherlands, and Sweden. The number of firms by country and size is shown in Table

36 Defaults are based on the Moody s KMV Default database and include missed payments, distressed exchanges and insolvency proceedings. 17 For all comparisons against agency ratings we used Moody s ratings Timely Default Prediction In this section, we compare the performance of EDF credit measures against agency ratings in their ability to predict timely defaults according with methodology described in section 3.1. We create a sample of defaulted firms retaining monthly observations from 24 months prior to default until 10 months after default. Only those observations were included in the sample that had non-missing history of EDF credit measures and ratings 24 months prior to default. We compute the median of the EDF credit measure and the median rating by months to default and overlay the median EDF and the median rating. Figure 20 demonstrates that in the event of default, EDF credit measures become elevated 11 months before ratings. Ratings move later and more abruptly, giving the most signal in the last nine months. FIGURE 20 Median agency ratings and Moody s KMV EDF values for rated defaulted firms in Europe from 24 months before default to 10 months after default between 1996 and Default Predictive Power EDF credit measures outperform simple Merton model implied default probabilities and Z-Scores in its ability to discriminate between defaulters and non-defaulters, which can be seen from Figure 21. The Accuracy Ratios for the EDF credit measure, Merton default probability, and Z-Score are 0.79, 0.70 and 0.61, respectively. 17 To collect defaults, we use numerous printed and online sources from around the world on a daily basis. We use government fillings, government agency sources, company announcements, news services, specialized default news sources and even sources within financial institutions to ensure, to the greatest extent possible that we find all defaults. We also keep evidences in electronic format so that content can be easily verified. As a result, Moody s KMV has the most extensive default database for public firms. 36

37 We divide the sample into subsets of sizes $30 million to $300 million, and $300 million and above. In both cases the EDF credit measure outperforms the Merton model implied default probability and Z-Score, as shown in Table 10. All the measures improve for larger firms. As a robustness check, we compared the performance of the three measures across time horizons and The results, presented in Table 10, illustrate that the EDF credit measure outperforms the Merton model and Z-Score with EDF credit measure and Merton default probability performing better in period while Z-Score has higher Accuracy Ratio in the second period. FIGURE 21 Cumulative Accuracy Profile curves (CAP) comparing Moody s KMV EDF credit measures, Z-Scores and Merton default probabilities for European non-financial firms between 1996 and The Accuracy Ratios for EDF measure, Z-Score and Merton default probability are 0.79, 0.61 and 0.70, respectively. We summarize our findings in this section in Table 12. The results clearly show that the EDF credit measure in Europe outperforms the other popular alternative in its ability to discriminate good firms from bad firms at a 1-year horizon. 37

38 TABLE 12 Summary of Accuracy Ratios, across various size buckets and time periods for European non-financial firms Date , Size >$30 Million Size >$30 Million Size >$30 Million , Size between $30 $300 Million , Size>$300 Million EDF Credit Measure Z-Score Simple Merton Model Level validation with default data To validate the accuracy of levels we followed the methodology described in Section 3.3. Results for the Whole Sample Figure 22 shows the level validation results for the sample of European firms with size greater than $300 million. The left panel of the Figure 22 displays the mean, median predicted and actual default rate along with 80% prediction interval for the default rate. We used an asset correlation of 0.25 to simulate defaults in each year. The right panel of the Figure 22 presents the posterior distribution for the aggregate shock given the actual default rate and P-values of the actual default rate, which is the probability of observing default at or lower than the actual default rate. The predicted default rate tracks the realized default rate well. There are exceptions, however, during times of systematic shock. For example, 2002 was a year when the markets crashed and there were an unexpectedly high number of defaults compared to what was predicted by the model. We estimated that the shock was negative 0.29 standard deviations. Similarly, year 2003 was an uncharacteristically good year for the economy leading to a substantially lower number of defaults. The graph of the aggregate shocks shows that in year 2003 the economy experienced a positive shock of 1.02 standard deviations that led to that small default rate. The results show that all realized default rates fall within the prediction interval. The P-values of the realized default rate range from 17% to 65%, which is within the sampling variability that would be expected. 38

39 FIGURE 22 Comparison of median predicted default rate with the realized default rate, The sample was restricted to European non-financial firms larger than 300 million dollars. We used an asset correlation of 0.25 to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. We summarize the numbers that underlie Figure 22 in Tables 12 and 13. Table 12 contains number of firms, number of defaults, median and mean predicted default rate per year, as well as 10 th and 90 th percentiles for predicted default rate. We find that the mean predicted default rates are much larger than the median default rates indicating that the correlation effect skews the distribution of default rates to the left. If we ignored this effect and had simply taken the mean default rate of the sample, we would have falsely concluded that the model over predicts defaults. Table 13 contains the median aggregate shock, 10 th and 90 th percentiles of the aggregate shock and the P-value by year. 39

40 TABLE 13 Comparison of mean and median predicted number of defaults with the realized number of defaults between 1996 and 2006 Year Mean Predicted Default Rate Median Predicted Default Rate Realized Default Rate 10th Percentile 90th Percentile Firms Defaults % 0.40% 0.44% 0.00% 2.00% % 0.50% 0.37% 0.00% 2.10% % 0.30% 0.38% 0.00% 1.40% % 0.50% 0.35% 0.00% 2.20% % 0.50% 0.38% 0.00% 2.10% % 0.80% 0.88% 0.10% 3.10% % 1.20% 1.70% 0.20% 4.40% % 2.20% 0.66% 0.50% 6.80% % 1.00% 0.77% 0.20% 3.80% % 0.70% 0.31% 0.10% 2.60% % 0.20% 0.06% 0.00% 0.80% The sample was restricted to European firms larger than 300 million dollars. TABLE 14 Summary of aggregate shock and year-wise probability of realizing the actual number of defaults between 1996 and 2006 Year 10th Percentile Median Aggregate Shock 90th Percentile Probability of having actual defaults or even lower % % % % % % % % % % % The sample was restricted to the European firms larger than 300 million dollars. 40

41 Results for Countries having at Least 100 Companies with Size Greater than $300 Million We restricted the sample to the countries that have at least 100 companies with size greater than $300 million. These countries tend to have larger equity markets. For these companies, the predicted default rate tracks the realized default rate very well as shown in Figure 23. The relatively low default rate in year 2003 is indicative of a large positive shock. The P-values of the realized default rate range from 20% to 69%, which is within the sampling variability that would be expected. FIGURE 23 Comparison of median predicted default rate with the realized default rate, The sample was restricted to European non-financial firms larger than 300 million dollars from the following countries: Switzerland, Germany, Spain, France, Great Britain, Italy, Netherlands, and Sweden. We used an asset correlation of 0.25 to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate Level Validation with CDS Data The number of defaults observed for larger firms in Europe was less than in the North America, making the power of the test somewhat weaker compared to that in North America. Therefore, we present another indirect validation of EDF credit measures in Europe. This test analyzes the level bias in European EDF credit measure relative to that of the U.S. EDF credit measure. The rationale for the test is based on the assumption that similar risks offer similar premia in the U.S. and Europe. So, if we subdivide the firms based on EDF categories, then the same EDF categories should have same aggregate median spreads in CDS markets across the two regions. For example, if EDF levels in Europe substantially overstated the level of default risk in Europe relative to North America, then if we were to compare a European firm to a North American firm with a comparable EDF, the European firm on average would have a substantially lower CDS spread. Conversely, if there were no such systematic bias between EDF credit measures in North America versus Europe, then the median spread should be approximately the same. In Figure 24, we compare the median, 25 th and 75 th percentile CDS spreads for Aa and above and A EDF implied rating categories. The median spreads as well as 25 th and 75 th percentiles over time are comparable in the U.S. and Europe, thereby indicating no relative bias in EDF levels of Europe over that in the U.S.. We also tried this for Baa, Ba, B and 41

42 Caa EDF implied rating categories and found comparable results. 12 The results are shown in Figure 25, and 26 respectively. There was some overlap in the underlying names in the two currencies. However, our findings are robust to using a completely non-overlapping sample as well. The subinvestment names can be impacted by liquidity risk that can be different in different regions, thereby making the test less reliable. Aa and above A FIGURE 24 Comparison of CDS spreads in the U.S. and Europe for Aa and above and A EDF-implied rating categories Blue lines represent 25th, median and 75th percentile of the CDS spread in Europe and red lines are similar data for the U.S. 12 The category Aaa is not shown because there were very few observations for CDS contracts in this category. 42

43 Baa Ba FIGURE 25 Comparison of CDS spreads in the U.S. and Europe for Baa and B EDF-implied rating categories Blue lines represent 25th, median and 75th percentile of the CDS spread in Europe and red lines are similar data for the U.S. B Caa FIGURE 26 Comparison of CDS spreads in the U.S. and Europe for B and Caa EDF implied rating categories Blue lines represent 25 th, median and 75 th percentile of the CDS spread in Europe and red lines are similar data for the U.S Conclusion We showed that in Europe, EDF credit Measures lead Agency Ratings in timely default prediction. EDF credit measures lead other alternative measures in their ability to discriminate good firms from bad firms over time and across various 43

44 subsections of the data. Model-predicted default rates track realized default rates well and CDS spreads are similar to those in the U.S. for the same EDF-implied rating categories. 4.3 Asia In this section, we describe the results obtained in Asia Data We start with all Asian firms that have publicly traded equity from 1996 to We restrict the sample to nonfinancial firms with more than $30 million in size (unless otherwise specified) to account for hidden or missing defaults. 18 Defaults are based on the Moody s KMV Default database and include missed payments, distressed exchanges, and insolvency proceedings. Table 14 shows the number of companies by country for two size categories: above $30 million and above $300 million that are in Asian module of Credit Monitor and CreditEdge. We decided to exclude some countries from level validation: China, because the government intervention default definition is not clear Australia and New Zealand, because they belong to the Pacific region Japan, because it has a different economic structure and a hidden default problem Pakistan and Sri Lanka, because they have a small number of companies The remaining countries have the most comprehensive default coverage. These countries are Hong Kong, India, Indonesia, Korea, Malaysia Philippines, Singapore, Thailand, and Taiwan. We ran power and level validation tests separately for Japan. 18 Size is measured by the sales of the firm for non-financial firms. Whenever the firm s total sales number was not available, we used the book asset value of the firm. This number was further adjusted for inflation effect across years by adjusting the numbers to a common denomination by using a deflation adjustor calculated internally at Moody s KMV. 44

45 TABLE 15 Number of companies in Asian Module of Credit Monitor and CreditEdge by country and size Country Country Code Size >= $30 Million Size >= $300 Million Australia AUS China CHN Hong Kong HKG Indonesia IDN India IND Japan JPN Korea KOR Sri Lanka LKA 19 1 Malaysia MYS New Zealand NZL Pakistan PAK Philippines PHL Singapore SGP Thailand THA Taiwan TWN Timely Default Prediction In this section, we compare the performance of EDF credit measures against agency ratings in their ability to predict timely defaults according to methodology described in section 3.1. We create a sample of defaulted firms retaining monthly observations from 24 months prior to default until 10 months after default. Only those observations were included in the sample that had non-missing history of EDF values and ratings 24 months prior to default. We compute the median of the EDF credit measure and the median rating by months to default and overlay the median EDF and the median rating. Figure 27 demonstrates that in the event of default, EDF credit measures become elevated 10 months before ratings. 45

46 FIGURE 27 Median agency ratings and Moody s KMV EDF values for all rated defaulted firms in Asia from 24 months before default to 10 months after default between 1996 and EDF values are displayed on log scale Default Predictive Power The EDF credit measure has more discriminatory power than Z-Score and Merton Default Probability in Hong Kong, India, Indonesia, Korea, Malaysia Philippines, Singapore, Thailand and Taiwan as can be seen in Figure 28. The Accuracy Ratio for the EDF credit measure is Contrary to the power tests performed in North America and Europe, Z-Score outperforms simple Merton model implied default probability in its ability to discriminate between bad and good firms with Accuracy Ratios being 0.57 and 0.56, respectively. 46

47 FIGURE 28 Cumulative Accuracy Profile (CAP) curves comparing Moody s KMV EDF credit measures and Z-Scores for Asian non-financial companies between 10/2001 and 12/2006. The Accuracy Ratios for EDF measure, Z-Score and Merton Default Probabilities are 0.67, 0.57 and 0.56, respectively. The EDF credit measure has more discriminatory power than Z-Score and Merton Default Probability in Japan. Consistent with the results in other nine countries, Z-Score has higher Accuracy ratio than Merton default probability. CAP curves are presented in Figure 29. Accuracy ratios of EDF credit measure, Merton default probability and Z-Score are 0.89, 0.79 and 0.77, respectively. 47

48 FIGURE 29 CAP curves comparing Moody s KMV EDF credit measures and Z-Scores for Japanese nonfinancial companies between 10/2001 and 12/2006. The Accuracy Ratios for the EDF credit measure, Z-Score, and Merton Default Probabilities are 0.89, 0.79 and 0.77, respectively Level Validation Figure 30 shows the level validation results for the sample of Asian firms (Hong Kong, India, Indonesia, Korea, Malaysia Philippines, Singapore, Thailand and Taiwan) with size greater than $300 million. The left panel of Figure 30 displays the mean, median predicted, and actual default rate along with 80% prediction interval for predicted default rate. We used an asset correlation of 0.25 to simulate defaults in each year. The right panel of Figure 30 presents the posterior distribution for the aggregate shock given the actual default rate and P-values of the actual default rate, which is the probability of observing default at or lower than the actual default rate. Collection of default data in Asia is more difficult than in the U.S. and Europe because of language barriers, poor reporting of default events, and government intervention to prevent company collapse, which often goes unreported. We could expect the under prediction of defaults in 1996, 1997, and 1998 because of the severe Asian financial crisis. The over-prediction of defaults in 2001 and 2002 may reflect market uncertainties regarding the Asian recovery while Europe and North America were in recessions. The P-values of the realized default rate range from 11% to 87%, which is within the sampling variability that would be expected. 48

49 FIGURE 30 Comparison of median predicted default rate with the realized default rate, The sample was restricted to Asian non-financial firms larger than 300 million dollars from the following countries: Hong Kong, India, Indonesia, Korea, Malaysia Philippines, Singapore, Thailand, and Taiwan. We used an asset correlation of 0.25 to simulate defaults in each year. On the left panel, the gray lines represent 80% prediction interval for predicted default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the posterior distribution of the aggregate shock distribution and P-values. The dark black line is the median aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles for the aggregate shock; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. Accuracy of Levels in Japan Figure 31 presents the level validation results for the sample of Japanese firms with size greater than $300 million. As above, the left panel of the Figure 31 displays the mean, median predicted, and actual default rate along with 80% prediction interval for predicted defaults. We used an asset correlation of 0.25 to simulate defaults in each year. The right panel of the Figure 31 presents the posterior distribution for the aggregate shock given the actual default rate and P-values of the actual default rate, which is the probability of observing a default at or lower than the actual default rate. As expected, the EDF credit measure is higher than observed default rate in Japan, due to the practice of banks and parent companies extending credit to companies that otherwise would default. 49

50 FIGURE 31 Comparison of median predicted default rate with the realized default rate, The sample was restricted to Japanese non-financial firms larger than 300 million dollars. We used an asset correlation of 0.25 to simulate defaults in each year. On the left panel, the gray lines represent the 80% prediction interval for realized default rate, the black line is the median predicted default rate, the blue line is the mean predicted default rate, and the red dotted line is the realized default rate. The right panel shows the aggregate shock distribution and P-values. The dark black line, rm50 is the median for the posterior distribution of the aggregate shock; the grey lines, rm10 and rm90 are the 10 th and 90 th percentiles; the blue line is the P-value of the actual default rate, which is the probability of observing a default at or lower than the actual default rate Conclusion We showed that in Asia, the EDF credit measures lead agency ratings in timely default prediction. For countries where we have best default coverage, EDF credit measures lead other alternative measures in their ability to discriminate good firms from bad firms over time and across various subsections of the data. Realized default rate for countries with better default coverage lies within the prediction interval. In Japan, the EDF model discriminates distressed firms from healthy firms very well. 4.4 Median EDF by Rating Category across Regions Four panel graphs in Figure 32 display the median EDF credit measure by rating categories across different regions. Levels of EDF credit measure for North American non-financial companies, Asia-Pacific non-financial companies, and global financial companies are comparable for all rating categories. Levels in Europe are a bit lower for better quality firms in rating categories of A and Baa. 50

51 100% 10.0% B Ba Baa A 100% 10.0% B Ba Baa A EDF8 1.00% EDF8 1.00% 0.10% 0.10% 0.01% 01M90 01M95 01M00 01M05 01M % 01M90 01M95 01M00 01M05 01M10 North American non-financial companies 100% 10.0% B Ba Baa A 100% 10.0% Asian-Pacific non-financial companies B Ba Baa A EDF8 1.00% EDF8 1.00% 0.10% 0.10% 0.01% 01M90 01M95 01M00 01M05 01M % 01M90 01M95 01M00 01M05 01M10 European non-financial companies Global financial companies FIGURE 32 Comparison of median EDF across different regions by Moody s rating categories 5 CONCLUSION In this document, we tested the performance of the Moody s KMV EDF credit measure in its timeliness of default prediction, ability to discriminate good firms from bad firms, and accuracy of levels. Whenever possible, we compared the performance to other popular alternatives available to the market. We find that the EDF credit measure performed well on all counts over time and across various subsections of the data. We also showed that the Moody s KMV model works well not only in North America, but also in Europe and Asia. In Europe, our findings are especially significant because our European sample consisted of various subregions with substantially different debt-holding structures and bankruptcy mechanisms which could have adversely impacted the results if the model was not universal in concept. While research at Moody s KMV continues to make efforts to make this measure superior and take into account all the nuances of the data as the markets evolve and become more complex, we feel that as of now, this measure sets a standard in the industry for a transparent and predictive absolute measure of the probability of default. 51

52 APPENDIX A: CAP VS. ROC The most popular validation techniques available today are Cumulative Accuracy Profile (CAP) and Receiver Operating Characteristic (ROC). CAP has its summary statistic known as the Accuracy Ratio, while ROC has its summary statistic as the area under the ROC curve. As a specific case, let us think of a sample that has N defaulters 19 and M non-defaulters. The i th firm in the sample is assigned a default probability p i. Without loss of generality, let us assume the order to be p 1 p 2 p 3. p M+N. Therefore, for each p i, one can assign a set (m(p i ),n(p i )), where m(p i ) represents the number of non-defaulters that have probability of default greater than or equal to p i, and n(p i ) represents the number of defaulters that have probability of default greater than or equal to p i. Obviously m(p M+N ) = M and n(p M+N ) = N. One can translate these numbers to fraction of defaulters and non-defaulters as f m (p i ) and f n (p i ) where f m (p i ) = m( p i ) represents the fraction of non-defaulters that have M a default probability greater than or equal to p i, and f n (p i ) = n( p i ) represents the fraction of non-defaulters that have a N default probability greater than or equal to p i. Similarly, one can also create an overall fraction f(p i ) = m( pi ) + n( pi ) that represents the fraction of firms in the sample that have a default probability greater than or equal M + N to p i. CAP is now defined as the graph of f n (p i ) against f(p i ) for all values of p i. ROC is the graph of f n (p i ) against f m (p i ) for all values of p i. Alternatively, ROC can also be interpreted as the curve that plots the hit rate against the false alarm rate for any cut-off C, across all values of C. Receiving Operating Characteristics is a popular approach borrowed from medical science. ROC curves, also known as power curves, are well-known ways of establishing the ability of a model to distinguish signals from noise, or in our case, defaulters from non-defaulters. The basic approach takes a sample of M+N firms, of which M are good firms (nondefaulters) and N are defaulted firms. If we rank the firms in their likelihood of defaulting from the highest potential defaults to lowest potential defaults, and exclude z% of the riskiest firms from the sample, then we will end up excluding some actual defaults and some non-defaults. In this way, we end up excluding z/100*(m+n) of the firms. A perfect model would have excluded all defaults, as long as z/100 is less than N/(M+N). A random model with no information would exclude zm/100, i.e., z% of non-defaults and zn/100 (i.e., x% of defaults). Let us assume that we exclude x(z)% of non-defaults, and y(z)% of defaults by excluding the riskiest z% of the sample. By varying z from 0 to 100, we can get various pairs of (x,y). By plotting y against x on an X-Y plot, we should be able to construct a graph that indicates the Accuracy Ratio of the model. For example, a model with no predictive power should have its (x,y) plot as a 45 degree line. A perfect model should be a flat horizontal line at 100%. Note that both x and y will vary from 0 to 100%. Also, this test needs only ordinal ranking and therefore can be used to test all the various approaches of credit risk on the same plane. The area of the CAP curve above the 45-degree line as a fraction of the area of the perfect model s CAP curve above the 45-degree line is called the Accuracy Ratio (AR). The area under the ROC curve is called the AUC (Area Under Curve). This is illustrated in figures 33 and 34. Both of these figures are based on the same population of non-defaulters and defaulters. Figure 33 shows the Cumulative Accuracy Profile Curve that is the curve outside area A. A perfect model would have the Cumulative Accuracy Curve represented by the straight line outside area B. The Accuracy Ratio is A/(A+B). Figure 34 shows the ROC curve outside the shaded area. The shaded area represents the AUC. It can be shown that 2AUC-1 = AR. For more details and proof of this relationship, refer to Engelmann, Hayden, and Tasche (2003). 19 Defaulters are usually counted over a certain horizon. Therefore these tests are horizon specific. In this document, all tests are for 1- year horizon. 52

53 FIGURE 33 FIGURE 34 53