SM JULY 2016 A Decade of Validation Demonstrates Superior Performance Contents Highlights 2013-15 VantageScore Performance Compared to CRC In-House Models 2013-15 Consumer Score Consistency 2013-15 Universe Expansion Recession Performance the Value of Developing on a Blended Timeframe A Decade of Performance Validation & Risk Conclusion
A Decade of Validation Demonstrates Superior Performance VantageScore Solutions, as part of its mission, annually validates VantageScore credit score models at each of the credit reporting companies (CRCs) - Experian, Equifax and TransUnion. To promote transparency and aid in model governance, VantageScore Solutions publishes the results of these validations annually, along with updated odds/performance charts. In recognition of our ten-year anniversary, this paper highlights our most recent validation results and also presents insights gleaned from a decade of credit score model performance validations. As we have done since 2005, we began the annual validation process by selecting credit files for 15 million consumers (at random) from CRC databases as a representation of the U.S. population. Each model is validated by comparing predictive performance on both originations and existing-account management applications against the CRCs in-house models for the bankcard, auto and real estate industries. We also determine performance and consumer score consistency across the industry. In addition, underlying drivers of performance due to population stability, characteristic trends, and risk and population volumes are reviewed for performance context. Upon completion of our validation, we confirmed that our VantageScore models continue to perform at extremely high levels. To understand these results in greater detail or for further questions, lenders should contact VantageScore Solutions or your CRC partner(s). HIGHLIGHTS: The VantageScore 3.0 model continues to outperform the CRC in-house scores for all major industries and business applications. Additionally, VantageScore 3.0 delivers superior performance stability during both peak and nonpeak recession timeframes. Over the last ten years, the VantageScore models have consistently outperformed all CRC in-house models. VantageScore 3.0 continues to score 30 to 35 million additional consumers more than conventional credit scoring models. In part as a result of the highly conservative postrecession lending and risk environment, all VantageScore credit score models discussesd in this paper are currently performing very strongly. 2013-15 VANTAGESCORE PERFORMANCE COMPARED WITH CRC IN-HOUSE MODELS (Figure 1: Comparing predictive performance between VantageScore 3.0 and in-house CRC models within key industries) VantageScore 3.0 delivers consistently superior performance compared with the CRCs in-house credit score models. While all scores are performing at extremely high levels, VantageScore 3.0 outperforms the CRC models by an average of 1.7%-3.4% in key industries. In addition, VantageScore 3.0 s predictive performance is highly consistent across all three CRCs, varying by an average of only 0.34 Gini points 1 between each CRC. 1 The Gini coefficient of a credit score compares the distribution of defaulting consumers with the distribution of non-defaulting consumers across the credit score range. The coefficient has a value of 0 to 100. A value of 0 indicates that defaulting consumers are equally distributed across the entire credit score range. In other words, the credit score fails to assign more defaulting consumers to lower credit scores. A coefficient value of 100 indicates that the credit score has successfully assigned all defaulting consumers to the lowest score possible. A Gini coefficient of 45 or greater is considered a good result by industry standards. 1 - VantageScore: A Decade of Validation Demonstrates Superior Performance
Figure 1: Comparing predictive performance between VantageScore 3.0 and in-house CRC models within key industries Existing Account Management Gini statistic 82 78 76 74 72 70 68 66 64.9% 81.1% 81.1% % 79.7% 79.2% 75.4% 75.6% 75.6% 75.7% 74.6% 71.4% CRC1 VS3 CRC1 CRC2 VS3 CRC2 CRC3 VS3 CRC3 77.9% 78% 78.1% 77.4% 81.1% 81% 81.1%.2%.1% Figure 2: Consumer Score Consistency using VantageScore 3.0 91.1% 91.5% 93.4% 93.2% 77.4% 81.3% 76.8%.8% 74.3% 78.6% Overall Auto Bankcard Real Estate Figure 3: Scoring the Conventionally Unscoreable 73.8% 72.8% 71.6% Bankcard Installment Auto Real Estate Student Loan 69.9% < 20 pts < 40 pts 2013-15 CONSUMER SCORE CONSISTENCY (Figure 2: Consumer Score Consistency using VantageScore 3.0) VantageScore 3.0 was used to score a sample of one million consumers at the three CRCs. For consumers with real estate trades,.8 percent of the sample had scores that fell within 20 points of each other across the three CRCs, and 93.2 percent of the sample had scores within 40 points across the CRCs. Unlike competitors, who build credit score models using algorithms that differ between the CRCs, all VantageScore models use one algorithm that is deployed identically at all three CRCs. Patented characteristic-leveling processes built into VantageScore models ensure that any variation in a consumer s score between CRCs is attributable principally to differences in credit-file data, and not to variations in model design (as is true of other credit scoring models). VantageScore models deliver much greater score consistency across CRCs, and provide tighter risk assessment to lenders who use multiple scores for decision making. 2013-15 UNIVERSE EXPANSION (Figure 3: Scoring the Conventionally Unscoreable) VantageScore 3.0 continues to score approximately 30-35 million consumers who cannot be scored by conventional credit scoring models. Traditional models ignore members of this sizable unscoreable population, which includes about 9.5 million Hispanic- and Conventional, 184 Expanded, 35 VantageScore 3.0 continues to score approximately 30-35 million consumers who cannot be scored by conventional credit scoring models VantageScore: A Decade of Validation Demonstrates Superior Performance - 2
African-American consumers. These consumers are not scored by conventional models because their credit files do not contain enough data to satisfy those models inclusion requirements. Advanced model design, including multiple scorecards tailored to consumers with short or sporadic credit histories, enables VantageScore 3.0 to score these once-unscoreable consumers accurately. RECESSION PERFORMANCE THE VALUE OF DEVELOPING ON A BLENDED TIMEFRAME (Figure 4: Recession Performance the Value of Developing on a Blended Timeframe) A key question for lenders as they consider whether to upgrade to newer credit scoring models is: How will a recently developed model perform in more volatile risk periods?. To answer this question, VantageScore 1.0, which was developed pre-recession (2003-05); VantageScore 2.0, which was developed mid-recession (2006-09); and VantageScore 3.0, which was developed post-recession (2009-12) were all validated on consumer originations performance during a period of economic volatility, from 2007 to 2009. As expected, VantageScore 2.0 performed optimally because it was developed on the same recession timeframe. VantageScore 3.0, however, substantially outperformed VantageScore 1.0. In fact, VantageScore 3.0 s performance was within a half Gini point of VantageScore 2.0 for the bankcard, installment and auto industries. For real estate, where score performance was most challenged during the recession, VantageScore 3.0 was within three Gini points of VantageScore 2.0, while VantageScore 1.0 was nearly nine points below. The strength and stability of VantageScore 3.0 s performance is attributable to a methodology that blends two development windows, 2009-11 and 2010-12, and uses 45 million consumer credit files (nearly two times the sample size used on earlier models), to uncover a broader and deeper array of behaviors for consideration in model design and calibration. A DECADE OF PERFORMANCE VALIDATION & RISK (Figure 5: Model Performance through the Recession Existing Account Management) Predictive performance for VantageScore models and a representative CRC in-house model (Figure 5) demonstrate the adverse impact periods of greater economic volatility have on all models. Performance eroded, as to be expected, as economic and product volatility increased, with all models showing a loss in performance of approximately five percent in the 2007-11 timeframe. Despite this marginal loss in performance, all models continued to rank order very effectively. With the onset of more conservative lending behavior over the last several years, model performance has improved again, returning to pre-recession levels. Note that VantageScore 3.0 and VantageScore 2.0 are currently performing better than they did during their original development timeframes. That improvement is largely attributable to the use of a blended timeframe development window. (Figure 6: The Meaning of 660) Finally, risk levels associated with a specific score cut-off also have returned to near pre-recession levels, as is demonstrated in Figure 6 by shifts in probability of default (PD). Perhaps the most relevant insight we learned from sharing these validations is that many industry participants were unaware that risk associated with a specific score varied quite significantly over the last decade. The critical learning for lenders is to incorporate into their lending strategies both a robust score cut-off review and update an mechanism to adjust for this variance in risk exposure. CONCLUSION A decade of credit score model validations has shown that modern credit scoring models are remarkably robust. Even in the midst of recession, these models effectively continued to rank order. VantageScore models, in particular, have continued to outperform competitor models on all key dimensions. Given this degree of robustness, perhaps the more relevant question becomes which credit scoring model will deliver both superior and stable performance regardless of the level of economic volatility. VantageScore 3.0 answers in the affirmative on both measures. VantageScore delivered superior and stable performance during both non-recession and recession timeframes. This is primarily due to the unique and unusually large development sample data and the associated methodology that blends two lending cycles as described above. An old cliché in consumer lending is that bad loans are made in good times, and good loans are made in bad times. Our ten-year examination of the VantageScore model indicates that although that may be true, modernized credit scoring models remain effective predictors of default despite changes in the economy. 3 - VantageScore: A Decade of Validation Demonstrates Superior Performance
Figure 4: Recession Performance the Value of Developing on a Blended Timeframe Recession Performance Analysis VS 1.0 (2003-05) VS 2.0 (2006-08, 2007-09) VS 3.0 (2009-11, 2010-12) Gini statistic 75 70 65 60 55 50 45 40 35 30 74.1 74 75.2 68.1 69.4 69.3 62.5 63.9 63.8 61.9 63.6 63.1 61.9 59.3 53.2 Bankcard Installment Auto Real Estate Student Loan Figure 5: Model Performance through the Recession (Existing Account Management) VantageScore Model Performance, 2005-2015 Gini Performance (U.S. Population) 81 79 78 77 76 75 74 73 72 71.3 78.5 79.6 77.9 78.9 77.4 77.2 76.2 77.4 77.5 77.5 77.6 76.4 75.9 76.0 74.5 74.4 CRC Score VS 1.0 VS 2.0 VS 3.0 PD.4.2 79.4 79.6.3 79.2 79.2 79.3 12 79.0 79.0 79.4 78.8 11 2003-05 2005-07 2006-08 2007-09 2008-10 2009-11 2010-12 2011-13 2012-14 2013-15 10 9 8 7 6 Interval default rate: 90+ days past due The VantageScore credit score models are sold and marketed only through individual licensing arrangements with the three major credit reporting companies (CRCs): Equifax, Experian and TransUnion. Lenders and other commercial entities interested in learning more about the VantageScore credit score models, including the VantageScore 3.0 credit score model, may contact one of the following CRCs listed for additional assistance: Call 1-888-202-4025 http://vantagescore.com/equifax Figure 6: The meaning of 660 Score-to-Risk Relationship, 2005-2015 PD at 720 PD at 660 PD at 620 Call 1-888-414-4025 5.1% 4.3% 4.4% 4.5% 2.2% 2.2% 2.3% 2.7% 0.5% 0.5% 0.6% 0.7% 7.1% 4% 1.1% 8.5% 5.2% 1.6% 6.9% 4% 1.2% 5.8% 5.8% 5.2% 5.2% 3.2% 3.1% 2.7% 2.7% 0.9% 0.9% 0.8% 0.8% http://vantagescore.com/experian Call 1-866-922-2100 http://vantagescore.com/transunion 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 VantageScore July 2016 Copyright VantageScore www.vantagescore.com VantageScore: A Decade of Validation Demonstrates Superior Performance - 4