Credit Performance Scorecard White Paper (2016 Scorecard Updates, version 4.1) November 2015 2011-2015 Fannie Mae
Table of Contents About This Document... 3 STAR Introduction... 4 General Servicing Metric Definitions... 7 Solution Delivery Metric Definitions... 9 Timeline Management Metric Definitions... 11 Comparable Pool Construct Methodology... 12 Comp Evaluation and Scoring Framework... 18 Resources... 24 Appendix... 25 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 2 of 25
About This Document This document provides a brief overview of the STAR TM Program, the STAR Performance Scorecard, and provides context for servicers as they evaluate their results as represented in the STAR Performance Scorecard. The STAR Credit Performance Scorecard White Paper examines the following topics: STAR Introduction General Servicing Metric Definitions Solution Delivery Metric Definitions Timeline Management Metric Definitions Comparable Pool Construct Comp Evaluation and Scoring Framework Resources This document supplements the STAR Reference Guide and will be updated as the program continues to evolve. STAR TM and Servicer Total Achievement and Rewards TM are trademarks of Fannie Mae. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 3 of 25
STAR Introduction What is STAR? The Servicer Total Achievement and Rewards (STAR ) Program is Fannie Mae s servicer performance management system and incentive framework. STAR is designed to: Provide consistent, specific, and measurable feedback to our servicers Align servicer activities to Fannie Mae's business goals Focus on success drivers to help prioritize efforts Measure and communicate servicer performance Provide opportunities to recognize top performers Servicers included in the program represent the majority of Fannie Mae s total credit exposure. Participants are evaluated based on their performance in the Servicer Capability Model (SCM) metrics, the STAR Performance Scorecard, and the People, Process and Technology reviews conducted through the Operational Assessments. Servicer Capability Model (SCM) The SCM establishes a single and consistent reference point to evaluate and measure servicer operations across People, Process, and Technology. For more information, please see the STAR Data Dictionary. Performance Scorecard The 2016 STAR Performance Scorecard evaluates servicer performance in three key process areas defined in the STAR Reference Guide: General Servicing, Solution Delivery, and Timeline Management. A servicer s eligibility for evaluation in each category of the scorecard will be determined based on the individual servicer s profile and the volume of Fannie Mae loans it services relevant to that process area. A matrix of servicer eligibility is provided in Table 1. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 4 of 25
Table 1: STAR Performance Scorecard - Relevant Profiles General Servicing (Current to 30 Days) Fannie Mae Volume Solution Delivery (60+ to Time Frame) Timeline Management (Beyond Time Frame) Profile A Profile B Profile C High Low Yes No Volume: Evaluation: General Servicing Servicers are measured on the basis of their performance in the following categories: roll rate management (1 metric), investor reporting (3 metrics), and customer service (2 metrics). Servicer performance in the roll rate metric is evaluated based on the servicer s relative performance to a similar portfolio created from the Fannie Mae book of business, and the investor reporting and customer service metrics are measured against established thresholds. Solution Delivery Servicers are measured on the basis of their performance in the following categories: roll rate management (1 metric), solution delivery (2 metrics), workout effectiveness (1 metric), and customer service (2 metrics). Servicer performance in the roll rate, solution delivery and workout effectiveness metrics are evaluated based on the servicer s relative performance to a similar portfolio created from the Fannie Mae book of business, and the customer service metrics are measured against established thresholds. Timeline Management Servicers are evaluated based on their performance in the following categories: delinquent loan resolution (2 metrics), bankruptcy management (1 metric), and REO reporting (2 metrics). Servicer performance in the delinquent loan resolution metrics are evaluated based on the servicer s relative performance to a similar portfolio created from the Fannie Mae book of business, and the bankruptcy management and REO metrics are measured against established thresholds. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 5 of 25
Summary The STAR Performance Scorecard compares a servicer s performance relative to other servicers and against established thresholds in three process areas related to loan servicing. To be evaluated in an individual process area within the scorecard, a servicer must service a significant volume of Fannie Mae loans relevant to those metrics. The metrics measured in the Performance Scorecard are deemed of strategic importance in achieving Fannie Mae s business objectives. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 6 of 25
General Servicing Metric Definitions Roll Rates 1 Transition to 60+ Measures the number of loans that roll from a less than 60 day delinquent status to a 60+ day delinquent status or a loss liquidation 2 status over a 3-month reporting period. Active trial modifications less than 4 months old are excluded from the metric. For example, if a servicer had 1,005 loans that were less than 60 days delinquent in January, and as of April, 20 of those loans were 60 or more days delinquent, but 5 had active trial modifications less than 4 months old; the 60+ transition rate for April would be 1.5%. Investor Reporting Multi-Occurrence Hard Reject Rate Measures the percentage of loans with multiple hard reject occurrences within the report period; for a full list of reject codes, please refer to the Help Center section in the Servicer s Reconciliation Facility (SURF TM ). The numerator for the metric is based on the count of unique loans which had a payment hard reject in the report period and at least one or more times in the previous 5 months. The denominator is based on the total number of loans at the beginning of the processing period. If the loan is a bi-weekly loan and had a hard reject, it is included if the payment due date falls within the selected activity month, but it is only included once even if there are multiple payment due dates or hard reject occurrences within the selected activity month. Multi-Occurrence Soft Reject Rate Measures the percentage of loans with multiple soft reject occurrences within the report period; for a full list of reject codes, please refer to the Help Center section in the Servicer s Reconciliation Facility (SURF TM ). The numerator for the metric is based on the count of unique loans which had a payment soft reject in the report period and at least one or more times in the previous 5 months. The denominator is based on the total number of loans at the beginning of the processing period. If the loan is a bi-weekly loan and had a soft reject, it is included if the payment due date falls within the selected activity month, but it is only included once even if there are multiple payment due dates or soft reject occurrences within the selected activity month. 1 Roll rate metrics utilize the MBA methodology for determining delinquency. 2 Loss liquidation status includes short sales, Mortgage Release, foreclosure sales and third-party sales. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 7 of 25
Shortage Percent Measures the rate at which a servicer s monthly remittance is less than the total amount expected by Fannie Mae s Investor Reporting System (SIR). The shortage percentage is calculated as the total shortage amount divided by the total due, where total due is equal to the total monthly remittance plus the total shortage amount less the total surplus amount. The total shortage amount is the shortage balance at the close of the cash reconciliation activity period for A/A, S/A & S/S cash remittance types. Customer Service Average Speed to Answer Average number of seconds calls wait in the Customer Service queue to be answered by an agent. Abandonment Rate Percentage of calls that enter the Customer Service agent queue that are not answered by a live agent before the caller disconnects. This percentage represents all calls that enter the queue, with no exclusions for disconnects within a minimum threshold, blocked calls, or requested call backs. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 8 of 25
Solution Delivery Metric Definitions Roll Rates 3 60+ to Cure Measures the number of loans between 60 days delinquent and the allowable foreclosure time frame 4 at the beginning of the month that are brought current, paid in full, or repurchased over a 3-month reporting period. Active trial modifications less than 4 months old that are not converted during the reporting period are excluded from the metric. For example, if a servicer had 120 loans between 60 days and the maximum allowable foreclosure time frame at the end of January, and as of April month end, 25 of those loans were current or paid in full and 20 had active trial modifications less than 4 months old (and not yet current), the 60+ to Cure rate would be 25% for April. Solution Delivery Retention Efficiency Measures the number of modifications initiated during the month as a percentage of loans between 60 days delinquent 5 and the allowable foreclosure time frame at the beginning of the month. Active trial modifications less than 4 months old are excluded from the metric. The metric ratio reported on the Performance Scorecard represents a 3-month total of retention solutions in the numerator, divided by the 3-month total of the monthly denominator population. Liquidation Efficiency Measures the number completed Mortgage Releases and short sales during the month as a percentage of loans between 60 days delinquent and the allowable foreclosure time frame at the beginning of the month. Active trial modifications less than 4 months old are excluded from the metric. The metric ratio reported on the Performance Scorecard represents a 3-month total of liquidation solutions in the numerator, divided by the 3-month total of the monthly denominator population. 3 Roll rate metrics utilize the MBA methodology for determining delinquency. 4 Refer to the Fannie Mae Servicing Guide for details on allowable foreclosure time frames by state. 5 Retention and liquidation solutions delivered prior to the 60 th day of delinquency are included in the metric numerator and denominator and are treated as 60 days delinquent in the Comp. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 9 of 25
Workout Effectiveness 12-Month Modification Performance Measures the number of loan modifications completed twelve months prior that are now less than 30 days delinquent, paid in full, or repurchased, divided by the total number of loan modifications completed and not cancelled in the same time period. This metric uses a rolling 3-month total in both the numerator and denominator. For example, the March metric denominator would be all modifications completed in January, February and March of 2015, and the metric numerator would be the sum of January 2015 modifications current as of January 2016, February 2015 modifications current as of February 2016, and March 2015 modifications current as of March 2016. Customer Service Average Speed to Answer Average number of seconds calls wait in the Collections/Loss Mitigation queues to be answered by an agent. Abandonment Rate Percentage of calls that enter the Collections/Loss Mitigation agent queue that are not answered by a live agent before the caller disconnects. This percentage represents all calls that enter the queue, with no exclusions for disconnects within a minimum threshold, blocked calls, or requested call backs. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 10 of 25
Timeline Management Metric Definitions Delinquent Loan Resolution Average Age of Loans Beyond Allowable Foreclosure Time Frame Measures the average number of days beyond the allowable foreclosure time frame for all loans that exceed the published time frame. For example, if a servicer had 1,000 loans beyond the allowable foreclosure time frame with a total of 15,000 days beyond, then the metric would be calculated as 15,000 / 1,000 for a metric result of 15 days beyond time frame on average. Beyond Time Frame Resolution Measures loan dispositions 6 and delinquency cures as a percentage of loans that began the month beyond the allowable foreclosure time frame. This metric uses a rolling 3-month total in both the numerator and denominator. For example, the March metric denominator would be all loans beyond the foreclosure time frame at the beginning of January, February and March, and the metric numerator would be the number of liquidations and cures in January, February and March that began the respective month beyond the allowable foreclosure time frame. Bankruptcy Management Motions for Relief Referred Timely Measures the percentage of Motion for Relief referrals submitted to the bankruptcy attorney within Fannie Mae guidelines. REO Fulfillment Percentage of REOgrams Completed Timely Measures the percentage of REOgrams submitted within Fannie Mae timeline requirements. Percentage of Title Issues Resolved within 45 Days Measures the percentage of vesting-related title issues (HOA, deeds, and taxes) resolved within 45 days of the initial discovery or notification date. 6 Dispositions include completed foreclosure sales, third-party sales, short sales, Mortgage Release TM, repurchases, and full payoffs. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 11 of 25
Comparable Pool Construct Methodology Key Points Establish Control Variables Segment Loan Populations Measure Differences 1 2 3 Establish set of key loan, property and geographic characteristics to control for portfolio differences Segment all like loans together by bucketing loans with like characteristics based on the control variables Calculate performance differences for servicer s actual book of business vs. the benchmark at the bucket level, then aggregate results Background As discussed in the STAR Introduction section, metrics in the roll rate, solution delivery, workout effectiveness, and timeline management categories of the STAR Performance Scorecard are evaluated based on the servicer s relative performance compared to like loans from the Fannie Mae book of business. To evaluate a servicer s performance in this manner, it is first necessary to set up a mechanism to account for the different credit risk profiles among servicers portfolios. This allows the relative performance of one servicer versus another servicer to be compared on an "apples to apples" basis. By setting up such a mechanism, the unique credit characteristics of a servicer's portfolio will be taken into account so that each servicer is not penalized or given an unfair advantage based on the credit characteristics of its portfolio. In doing this, Fannie Mae seeks to base performance measures on the things servicers do to impact credit performance, not simply how their individual loans perform. This section describes how a servicer's portfolio is segmented based on key loan attributes or control variables, and how those segmented populations are compared against loans with similar characteristics. The process of comparing groups of loans with like characteristics is referred to as the Comp throughout this White Paper. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 12 of 25
Key Loan Attributes To compare performance across servicers for a given metric, a set of key loan attributes is first selected. The selection of these attributes is based on statistical research performed by the analytics team at Fannie Mae using the historical loan performance of Fannie Mae's entire book of business. To ensure that loans being compared are similar, the portfolio is segmented or bucketed into many distinct groupings based on credit characteristics. Each of the STAR credit performance metrics has control variables that are pertinent to that measure. For example, the amount of payment reduction received via a loan modification is a relevant variable when benchmarking 12-Month Modification Performance; however, it is not a relevant variable when benchmarking performance for Liquidation Efficiency. Appendix 1 at the end of this White Paper lists the control variables applied to construct the Comp for each of the STAR Performance Scorecard metrics. To further illustrate this concept, in the Transition to 60+ metric, there can be one bucket ( bucket #1 ) for all current loans originated after 2012 with an ACI 7 score between 720 and 760, a mark-to-market LTV between 80% and 100%, and that have not been modified or 60+ days delinquent in the last 24 months. To determine the relative performance of a particular servicer, Servicer A, the performance of Servicer A s loans in bucket #1 would be compared to the performance of all other loans in Fannie Mae s portfolio (excluding loans serviced by Servicer A) that match the criteria for bucket #1. A simplified example of the Comp construction is included in Table 2 in the following section. While the metric variables are designed to differentiate the credit characteristics of loans in the portfolio, the analytics team is careful not to create so many stratifications that there are too few loans to make valid comparisons. For example, if there are five control variables for a metric, and each control variable is segmented into three distinct values or ranges, there would be a total of 243 possible buckets created for the metric. Though efforts are made to ensure each Comp bucket has sufficient data for each servicer, when portfolios are segmented to the extent illustrated above, there are some instances where there will not be enough data to assert that a servicer performed better or worse than Comp. In those cases, the servicer s performance will be set at Comp for the reporting period. Additional information regarding how this treatment is applied is discussed in the Scoring Framework section of the White Paper. 7 ACI (Acquisition Credit Index) is a proprietary score used by Fannie Mae to evaluate the credit worthiness of a loan at origination. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 13 of 25
Exceptions and Special Handling Loan products with unique characteristics or limited distribution among servicers are excluded from the relative performance metrics in Performance Scorecard as are loans in certain geographic regions. As such, second liens, government insured loans, and loans securing properties in insular areas (e.g., Puerto Rico, Guam, and the U.S. Virgin Islands) are excluded from the metric and Comp calculations. Loans that transfer from one servicer to another are also treated differently based on the metric being measured and the status of the loan at transfer. The following table represents how servicing transfers are treated for each of the metrics that are measured relative to Comp. Roll Rates Loans transferred from one servicer to another during the reporting period, but before reaching the delinquency transition measured by the metric, will be excluded from metric and Comp for 2 months post transfer. Solution Delivery Loans transferred from one servicer to another will be excluded from the metric and Comp for 1 month post transfer for the retention efficiency metric and 2 months post transfer for the liquidation efficiency metric. Workout Effectiveness Loans transferred after the modification closing date are removed from the metric and Comp. Timeline Management Loans greater than 120 days delinquent at the time of transfer are excluded from the transferee servicer s metric calculation for 24 months post transfer in states where the allowable foreclosure time frame is greater than 365 days and 12 months in all other states. Summary The attributes used to establish comparable portfolio characteristics for each metric are chosen based on analysis performed by Fannie Mae. The variables are then adjusted to balance the need to differentiate portfolio makeup while still producing enough observations to make reasonable inferences of servicer performance. Exceptions to the process are made in instances where there is not enough data to make conclusions and to account for servicing transfers that might impact a servicer s ability to influence loan performance. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 14 of 25
Calculating Relative Performance As described previously, once the various buckets have been created for each of the metrics that are measured relative to Comp, a servicer's performance in each bucket is compared to the performance of all the other loans in the bucket. The tables below will illustrate the method by which servicer performance is calculated. Here, the performance of Servicer A for the Transition to 60+ roll rate metric is compared to the performance of the Comp, which consists of all the loans in the current and 30 day delinquent buckets, excluding Servicer A's loans, and against Servicer B who has a similar population of loans but with slightly different attributes. In this simplified example only two loan attributes are used, LTV and origination year. LTV is segmented into low and high LTV variables. Origination year is segmented into an old vintage population (pre-2010 originations) and a new vintage population (2010 and newer originations). This results in four buckets (two attributes multiplied by two segments each) where we can compare performance for this metric. Table 2: Comparable Pool Construction Servicer Bucket LTV Vintage Fannie Mae Numerator Fannie Mae Denominator Servicer Numerator Servicer Denominator Comp Numerator Comp Denominator Servicer A 1 High Old 36,350 1,500,000 200 8,500 36,150 1,491,500 2 High New 5,250 1,650,000 29 9,500 5,221 1,640,500 3 Low Old 57,500 6,500,000 660 74,250 56,840 6,425,750 4 Low New 8,500 7,000,000 59 50,500 8,441 6,949,500 107,600 16,650,000 948 142,750 106,652 16,507,250 Servicer B 1 High Old 36,350 1,500,000 420 17,500 35,930 1,482,500 2 High New 5,250 1,650,000 65 20,000 5,185 1,630,000 3 Low Old 57,500 6,500,000 435 48,000 57,065 6,452,000 4 Low New 8,500 7,000,000 75 65,200 8,425 6,934,800 107,600 16,650,000 995 150,700 106,605 16,499,300 In reviewing the Fannie Mae portfolio populations, notice that 16,650,000 loans started the month as less than 60 days delinquent. Servicer A has 142,750 loans that started out the month less than 60 days delinquent and Servicer B has 150,700. By the end of the third month, 948 of Servicer A s loans rolled to a worse status versus 995 for Servicer B (excluding active trials). Servicer A's Comp is the other 16,507,250 Fannie Mae loans that started the month as less than 60 days delinquent. Servicer B s comp, while a similar number of loans is slightly different from servicer A s based on the different credit profile of their portfolio. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 15 of 25
Table 3: Understanding Bucket Weighting Based on Servicer s Portfolio Servicer Bucket LTV Vintage Servicer Numerator Servicer Denominator Servicer Metric Denominator Weight Contribution to Metric Ratio Servicer A 1 High Old 200 8,500 2.35% 5.95% 0.14% 2 High New 29 9,500 0.31% 6.65% 0.02% 3 Low Old 660 74,250 0.89% 52.01% 0.46% 4 Low New 59 50,500 0.12% 35.38% 0.04% 948 142,750 0.66% 100.00% 0.66% Servicer B 1 High Old 420 17,500 2.40% 11.61% 0.28% 2 High New 65 20,000 0.33% 13.27% 0.04% 3 Low Old 435 48,000 0.91% 31.85% 0.29% 4 Low New 75 65,200 0.12% 43.26% 0.05% 995 150,700 0.66% 100.00% 0.66% Table 3 shows how a servicer s metric value is influenced by its portfolio makeup. Here, the denominator weight represents the portion of the servicer s less than 60 day delinquent population represented by that bucket. The product of the metric and the denominator weight yields that bucket s contribution to the servicer s overall metric value. You can see that the buckets with the larger number of loans have a greater impact on the metric value than those with fewer loans. This helps to illustrate the importance of segmenting the portfolio based on key loan attributes to account for the servicer s unique credit characteristics. Now we will take into account the key loan attributes and look at the performance in each of the four segments or buckets. Table 4: Calculating Servicer and Comp Performance Servicer Bucket LTV Vintage Servicer Numerator Servicer Denominator Servicer Metric Comp Numerator Comp Denominator Comp Ratio Servicer A 1 High Old 200 8,500 2.35% 36,150 1,491,500 2.42% 2 High New 29 9,500 0.31% 5,221 1,640,500 0.32% 3 Low Old 660 74,250 0.89% 56,840 6,425,750 0.88% 4 Low New 59 50,500 0.12% 8,441 6,949,500 0.12% 948 142,750 0.66% Servicer B 1 High Old 420 17,500 2.40% 35,930 1,482,500 2.42% 2 High New 65 20,000 0.33% 5,185 1,630,000 0.32% 3 Low Old 435 48,000 0.91% 57,065 6,452,000 0.88% 4 Low New 75 65,200 0.12% 8,425 6,934,800 0.12% 995 150,700 0.66% Once the Comp population is established, the next step in calculating performance is determining the metric value by dividing the numerator by the denominator for each bucket where performance will be measured. Notice in Table 4 while the total number of less than 60 day delinquent loans (servicer denominator) for each 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 16 of 25
servicer is similar, their delinquent loan populations are stratified in different buckets based on their portfolio composition. While both servicers have relatively the same mix of old and new loans in these buckets, Servicer B s portfolio contains a greater percentage of high LTV loans (25%) compared to Servicer A (13%). Once the servicer s performance in each bucket is established, the same calculation is made against the servicer s Comp pool of loans to establish the performance of all loans in the Fannie Mae portfolio with those characteristics. Once the Comp ratio is calculated (Table 4), you can determine how many loans would have rolled 60 or more days delinquent if the servicer performed at Comp by multiplying the Comp ratio by the number of delinquent loans (servicer denominator) for each bucket as in Table 5 below. The servicer s Comp for the metric is then calculated as the sum of the individual bucket Comp values. Table 5: Determining Metric Comp Values Servicer Bucket LTV Vintage Comp Ratio Servicer Denominator Comp Value Servicer A 1 High Old 2.42% 8,500 206.02 2 High New 0.32% 9,500 30.23 3 Low Old 0.88% 74,250 656.79 4 Low New 0.12% 50,500 61.34 142,750 954.38 Servicer B 1 High Old 2.42% 17,500 424.13 2 High New 0.32% 20,000 63.62 3 Low Old 0.88% 48,000 424.54 4 Low New 0.12% 65,200 79.21 150,700 991.50 Looking back at Table 4, you can see that each servicer has buckets where they achieve lower roll rates than the Comp and others where they exhibit similar or higher roll percentages. To see to what degree the rates for each servicer are higher or lower than the competition, we will find the percentage their performance is different. That value will become the basis of how the STAR Performance Scorecard measures servicer s relative performance. Summary For STAR metrics that measure relative performance, we segment the loan population for each metric based on key loan attributes that impact performance for that metric to ensure a proper comparison across portfolios. These attributes are further segmented into control variables to create multiple performance buckets for each metric. The servicer's performance in each bucket is then compared to the performance of its Comp for that bucket. Please refer the Appendix for additional details regarding the control variables applied for each metric. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 17 of 25
Comp Evaluation and Scoring Framework Key Points Calculate Relative Performance Evaluate Significance Rank and Score 1 2 3 Servicer s performance versus Comp is calculated for each relative performance metric Confirm there is sufficient volume to measure performance and verify the evaluation of Servicer s performance versus Comp Servicers are ranked based on their relative performance, then a score is assigned based on the distribution Background This section describes how STAR interprets performance relative to Comp and how servicers metric scores are calculated. To help servicers track progress, scores are provided on a monthly basis. Step 1: Calculation of the Relative Performance of the Servicer for an Individual Metric Metric scores are determined based on the servicer s variance to Comp for each individual metric. To determine this value, the servicer's actual performance is compared to that of its comparable pool, as outlined in the previous section. The following table reflects how the Comp values we calculated in the previous section are used to calculate the percent difference to Comp for each bucket and servicer. To do this, start by subtracting the Comp value from the servicer s numerator value to establish the difference between those numbers. Then divide that total by the Comp value to establish the percent at which it varies. For the Transition to 60+ and Average Age of Loans Beyond Foreclosure Time Frame metrics where a lower value is more desirable, the result is adjusted by a factor of -1 so that larger values are always better. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 18 of 25
Table 6: Calculating Variance to Comp Servicer Bucket LTV Vintage Servicer Numerator Comp Value Variance to Comp Pct. Variance to Comp Adjusted Variance to Comp Servicer A 1 High Old 200 206.02-6.02-2.92% 2.92% 2 High New 29 30.23-1.23-4.08% 4.08% 3 Low Old 660 656.79 3.21 0.49% -0.49% 4 Low New 59 61.34-2.34-3.81% 3.81% 948 954.38-6.38-0.67% 0.67% Servicer B 1 High Old 420 424.13-4.13-0.97% 0.97% 2 High New 65 63.62 1.38 2.17% -2.17% 3 Low Old 435 424.54 10.46 2.46% -2.46% 4 Low New 75 79.21-4.21-5.32% 5.32% 995 991.50 3.50 0.35% -0.35% Notice again that each servicer performs well in certain buckets, but below the Comp in others. When those Comp values are aggregated, the cumulative result shows that Servicer A performed better than Comp while Servicer B performed just below it, despite having nearly identical percentage of loans roll 60+ days delinquent than Servicer A. The servicer's relative performance as measured by the adjusted variance to Comp is calculated on a monthly basis. Step 2: Evaluate Comp for Minimum Thresholds Once the servicer s performance versus Comp is calculated, STAR then evaluates the Comp to determine whether or not there are enough observations to confidently conclude whether the servicer performed better or worse than comp. As mentioned previously in the Comparable Pool Construct section of the White Paper, while efforts are made to ensure each Comp bucket has sufficient data to make reasonable inferences into servicer performance, from time to time there are limited occurrences within the Comp pool for an individual servicer. To account for these limited observations, in cases where there are fewer than 5 observations in the servicer s Comp, the servicer s performance is considered undeterminable; however, if the servicer s metric numerator is greater than 10, and there are at least 2 Comp observations, the servicer s performance will be measured as favorable to the Comp. This rule is applied to all metrics for all servicers, but metrics derived from activity based functions, such as Liquidation Efficiency, Retention Efficiency, and 12-Month Modification Performance are the metrics most likely to be impacted by limited Comp data. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 19 of 25
Step 3: Determine Inference Based on Variance to Comp Once it is determined that there are sufficient observations to evaluate the servicer s performance, an inference is made to determine, within a reasonable degree of certainty, the servicer performed above Comp, at Comp or below Comp. To do this, STAR evaluates the variance to comp for each servicer to first determine whether the servicer s measured variance is statistically different from zero at a 99% confidence level. If the servicer s variance to comp does not meet the threshold of being statistically different, then the servicer s performance is deemed to be at Comp for the evaluation period 8. If the servicer s variance is statistically different from zero with a 99% degree of certainty, then the inference will be determined as above Comp if the variance is positive or below Comp if the variance is negative. At Comp Below Comp Above Comp -1 0 1 Variance to Comp Step 4: Calculation of the Relative Performance of the Servicer for an Individual Metric for a Quarter or Year STAR measures servicer performance for multiple time periods including monthly, quarterly, and year-todate. The relative performance of a servicer for a quarter or year is calculated in much the same way as the monthly performance; however, instead of comparing monthly numbers we compare the total performance of the servicer for the quarter or year to the total performance of the Comp. Thus, the formula for measuring quarterly performance would be as follows: QQ = (AA 1 + AA 2 + AA 3 ) (CC 1 + CC 2 + CC 3 ) (CC 1 + CC 2 + CC 3 ) Where A is the servicer's performance, C is the performance of the Comp, and the 1, 2, and 3 represent each month of the quarter. 8 While an inference for an individual month or consecutive months may deem a servicer s performance at Comp, the servicer s variance over a longer period (quarter or year-to-date) may be deemed above or below Comp if the variance is determined to be statistically different from zero for the longer observation period. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 20 of 25
By adding the three months together, we take into account changes in the size of the portfolio over time. In the following table, you can see that the servicer performed 160 basis points better in February than January and 126 basis points worse in March than January. A pure average would result in an average performance of 0.78%. However, there were more loans in January than February and March, and thus the adjusted variance to Comp weighted average of 0.82% is calculated, reflecting the larger volume in January when performance was best compared to Comp. Table 7: Calculation of Individual Metric Variance to Comp for a quarter Description January February March Q1 Actual 948 905 850 2703 Comp 954.38 926 845 2725.4 Actual - Comp -6.4-21 5-22.38 (Actual - Comp) / Comp -0.67% -2.27% 0.59% -0.82% Adjusted Difference to Comp 0.67% 2.27% -0.59% 0.82% To calculate the performance of the metric for the entire year, one would extend the table to show the additional months. Step 5: Convert Adjusted Variance to Comp to a Numerical Score for an Individual Metric The next step is to change the adjusted variance to Comp into a numerical score. This is a multi-step calculation that takes into account the range of outcomes for the servicers peer group and then re-scales the range to a numerical score from 5 to 95 in the following manner: 1. Determine the maximum and minimum variance to Comp for the peer group for the month (Note: the same general method would apply for quarterly or yearly outcomes). 2. Calculate the range between the maximum and minimum variances to the Comp for the peer group. 3. Locate where in this range the particular servicer performed by subtracting the minimum variance to Comp from the servicer's actual variance to Comp and dividing the result by the range of the peer group s variance to Comp. The best servicer in the peer group will have a relative position of 100%, the worst servicer will have a relative position of 0, and all other servicers will fall somewhere in-between based on their actual variance to Comp. 4. Convert the servicer s relative position in the range of the peer group s variance to Comp to a score ranging from 5 to 95 in the following manner: Multiply the servicer s relative position in the range by the difference of the maximum score (95) and the minimum possible score (5) and then add the 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 21 of 25
minimum possible score (5) to that result. The purpose behind doing this is to provide a possible range of scores that is not automatically set at 0 to 100. Table 8 illustrates the calculations used to translate the servicer s performance relative to the Comp into a simple numerical score from 5 to 95 based on how the servicer performed relative to others in their peer group for that metric. In this example, the servicer s relative performance translates to a score of 57.2, indicating that the servicer achieved above-median performance within the peer group. Table 8: Calculate Servicer Score for an Individual Metric Step Description Formula Example Result 1 Calculate variance to Comp for servicer (Actual - Comp) / Comp (948-954.38) / 954.38-0.67% 2 Calculate adjusted variance to Comp Variance to Comp * -1 (only for metrics where lower score is better) -0.67% x -1 0.67% 3 Determine the highest variance to Comp for the month Maximum variance to the Comp for the servicer's peer group for the month 12.18% 12.18% 4 Determine the lowest variance to Comp for the month Minimum variance to the Comp for the servicer's peer group for the month -15.20% -15.20% 5 Calculate the range of differences Maximum - Minimum 12.18% - (-15.20%) 27.38% 6 Calculate the relative position for the servicer within the range (Adjusted Variance to Comp Minimum of Range) / Range (0.67% - (-15.20%)) / 27.38% 57.96% 7 Convert relative position to score (Relative Position of Servicer * (95-5)) + 5 (0.5796 x (95-5)) + 5 57.2 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 22 of 25
Summary A servicer s variance to Comp is adjusted so that higher values are always better. The scoring system takes into account the range of performance for all the servicers used in the comparison as measured by the variance to Comp and assigns a numerical score of 5 to 95. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 23 of 25
Resources Servicer Support For servicer support or additional information related to the STAR Program: Contact your Fannie Mae Portfolio Manager Email us at STAR_Mailbox@fanniemae.com Call our Servicer Support Center at 1-888-FANNIE5 Online Resources Online resources can be accessed at https://www.fanniemae.com/singlefamily/star. The site contains a Program Overview, the STAR Reference Guide, Frequently Asked Questions (FAQs), the STAR Program Data Dictionary, and other training resources. For additional details regarding Investor Reporting and Investor Reporting Scorecard metrics, please visit https://www.fanniemae.com/singlefamily/servicing. 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 24 of 25
Appendix Appendix 1: Control Variables for STAR Scorecard Metrics (Number of Variables for Each Metric) Attribute Transition to 60+ 60+ to Cure Retention Efficiency Liquidation Efficiency 12 Month Performance Loans Beyond Time Frame (Avg Age) Beyond Time Frame Resolution Delinquency Status Loan delinquency status 2 4 4 3 3 Mark-to-Market LTV The current LTV based on Fannie Mae s home price model 5 5 4 4 4 Collectibility Status Loan status that could impact ability to collect past due payments 2 2 2 2 2 Modification Indicator Whether the loan has had a prior modification in past 24 months 2 1 2 2 2 ACI Score Fannie Mae s internal score for acquisition credit quality 5 3 3 Foreclosure Jurisdiction Detail Allowable time frame for jurisdiction 8 2 8 2 8 2 Payment Change Change in the loan principal and interest payment amount 2 3 3 4 Origination Year Year the loan was originated 5 3 Geography Property location (state or census region) 11 5 Modification Documentation Level of documentation required to underwrite the modification 2 Value Change (12 Months) Twelve-month change in Housing Price Index 3 Cash Flow Number of payments made in past 3 months 2 Days Beyond Time Frame Number of days beyond state time frame 4 1 Or ever 60+ days delinquent in the past 24 months 2 Range of allowable time frames, plus separate variables for California, Florida, Illinois, New York and New Jersey 3 Payment increase of 5% or more 4 The amount of payment relief post-modification 5 Census region plus California and Florida 2011-2015 Fannie Mae STAR Credit Performance Scorecard White Paper Page 25 of 25