Scorecard Monitoring

-- By Kari Michel

This blog completes my discussion on monitoring new account decisions with a final focus: scorecard monitoring and performance.  It is imperative to validate acquisitions scorecards regularly to measure how well a model is able to distinguish good accounts from bad accounts. With a sufficient number of aged accounts, performance charts can be used to:

• Validate the predictive power of a credit scoring model;
• Determine if the model effectively ranks risk; and
• Identify the delinquency rate of recently booked accounts at various intervals above and below the primary cutoff score.

To summarize, successful lenders maximize their scoring investment by incorporating a number of best practices into their account acquisitions processes:

1. They keep a close watch on their scores, policies, and strategies to improve portfolio strength.
2. They create monthly reports to look at population stability, decision management, scoring models and scorecard performance.
3. They update their strategies to meet their organization’s profitability goals through sound acquisition strategies, scorecard monitoring and scorecard management.
 

--  Kari Michel

This blog is a continuation of my previous discussion about monitoring your new account acquisition decisions with a focus on decision management. 

Decision management reports provide the insight to make more targeted decisions that are sound and profitable. These reports are used to identify: which lending decisions are consistent with scorecard recommendations; the effectiveness of overrides; and/or whether cutoffs should be adjusted.

Decision management reports include:

• Accept versus decline score distributions
• Override rates
• Override reason report
• Override by loan officer
• Decision by loan officer

Successful lending organizations review this type of information regularly to make better lending policy decisions.  Proactive monitoring provides feedback on existing strategies and helps evaluate if you are making the most effective use of your score(s). It helps to identify areas of opportunity to improve portfolio profitability. 

In my next blog, I will discuss the last set of monitoring reports, scorecard performance.

-- By Tracy Bremmer

In our last blog (July 30), we covered the first three stages of model development which are necessary whether developing a custom or generic model.  We will now discuss the next three stages, beginning with the “baking” stage:  scorecard development.
 
Scorecard development begins as segmentation analysis is taking place and any reject inference (if needed) is put into place. Considerations for scorecard development are whether the model will be binned (divides predictive attributes into intervals) or continuous (variable is modeled in its entirety), how to account for missing values (or “false zeros”), how to evaluate the validation sample (hold-out sample vs. an out-of-time sample), avoidance of over-fitting the model, and finally what statistics will be used to measure scorecard performance (KS, Gini coefficient, divergence, etc.).

Many times lenders assume that once the scorecard is developed, the work is done.   However, the remaining two steps are critical to development and application of a predictive model:  implementation/documentation and scorecard monitoring.   Neglecting these two steps is like baking a cake but never taking a bite to make sure it tastes good. 

Implementation and documentation is the last stage in developing a model that can be put to use for enhanced decisioning. Where the model will be implemented will determine the timeliness and complexity for when the models can be put into practice. Models can be developed in an in-house system, a third-party processor, a credit reporting agency, etc. Accurate documentation outlining the specifications of the model will be critical for successful implementation and model audits.

Scorecard monitoring will need to be put into place once the model is developed, implemented and put into use. Scorecard monitoring evaluates population stability, scorecard performance, and decision management to ensure that the model is performing as expected over the course of time. If at any time there are variations based on initial expectations, then scorecard monitoring allows for immediate modifications to strategies.

With all the right ingredients, the right approach, and the checks and balances in place, your model development process has the potential to come out “just right!”

-- By Wendy Greenawalt

When consulting with lenders, we are frequently asked what credit attributes are most predictive and valuable when developing models and scorecards. Because we receive this request often, we recently decided to perform the arduous analysis required to determine if there are material differences in the attribute make up of a credit risk model based on the portfolio on which it is applied.

The process we used to identify the most predictive attributes was a combination of art and sciences -- for which our data experts drew upon their extensive data bureau experience and knowledge obtained through engagements with clients from all types of industries. In addition, they applied an empirical process which provided statistical analysis and validation of the credit attributes included. Next, we built credit risk models for a variety of portfolios including bankcard, mortgage and auto and compared the credit attribute included in each.

What we found is that there are some attributes that are inherently predictive regardless for which portfolio the model was being developed. However, when we took the analysis one step further, we identified that there can be significant differences in the account-level data when comparing different portfolio models.

This discovery pointed to differences, not just in the behavior captured with the attributes, but in the mix of account designations included in the model. For example, in an auto risk model, we might see a mix of attributes from all trades, auto, installment and personal finance…as compared to a bankcard risk model which may be mainly comprised of bankcard, mortgage, student loan and all trades.  Additionally, the attribute granularity included in the models may be quite different, from specific derogatory and public record data to high level account balance or utilization characteristics.

What we concluded is that it is a valuable exercise to carefully analyze available data and consider all the possible credit attribute options in the model-building process – since substantial incremental lift in model performance can be gained from accounts and behavior that may not have been previously considered when assessing credit risk.

-- By Tracy Bremmer

Preheat the oven to 350 degrees. Grease the bottom of your pan. Mix all of your ingredients until combined. Pour mixture into pan and bake for 35 minutes. Cool before serving.

Model development, whether it is a custom or generic model, is much like baking. You need to conduct your preparatory stages (project design), collect all of your ingredients (data), mix appropriately (analysis), bake (development), prepare for consumption (implementation and documentation) and enjoy (monitor)!  

This blog will cover the first three steps in creating your model! 

Project design involves meetings with the business users and model developers to thoroughly investigate what kind of scoring system is needed for enhanced decision strategies. Is it a credit risk score, bankruptcy score, response score, etc.? Will the model be used for front-end acquisition, account management, collections or fraud?

Data collection and preparation evaluates what data sources are available and how best to incorporate these data elements within the model build process. Dependent variables (what you are trying to predict) and the type of independent variables (predictive attributes) to incorporate must be defined. Attribute standardization (leveling) and attribute auditing occur at this point. The final step before a model can be built is to define your sample selection.

Segmentation analysis provides the analytical basis to determine the optimal population splits for a suite of models to maximize the predictive power of the overall scoring system. Segmentation helps determine the degree to which multiple scores built on an individual population can provide lift over building just one single score.

Join us for our next blog where we will cover the next three stages of model development:  scorecard development; implementation/documentation; and scorecard monitoring. 
 

-- By Kari Michel

In my last blog I gave an overview of monitoring reports for new account acquisition decisions listing three main categories that reports typically fall into:  (1) population stability; (2) decision management; (3) scorecard performance.

Today, I want to focus on population stability.   Applicant pools may change over time as a result of new marketing strategies, changes in product mix, pricing updates, competition, economic changes or a combination of these. Population stability reports identify acquisition trends and the degree to which the applicant pool has shifted over time, including the scorecard components driving the shift in custom credit scoring models. 

Population stability reports include:

• Actual versus expected score distribution
• Actual versus expected scorecard characteristics distributions (available with custom models)
• Mean applicant scores
• Volumes, approval and booking rates

These types of reports provide information to help monitor trends over time, rather than spikes from month to month.  Understanding the trends allows one to be proactive in determining if the shifts warrant changes to lending policies or cut-off scores.

Population stability is only one area that needs to be monitored; in my next blog I will discuss decision management reports.

 

-- By Wendy Greenawalt

Today, most lenders evaluate tri-bureau credit data when making lending decisions. Credit attributes are the building blocks for creating models, scorecards, segmentation and policy rules. Why is creating tri-bureau attributes so difficult? The main challenges are assessing the bureau data that is available, deriving meaningful information from that data and then equalizing or minimizing the differences inherent to the data available from the credit bureaus.

While this process may seem straight forward, defining an industry designation or a series of attributes within that industry can take months of analysis and careful consideration of trade-offs. Missing even one data element can have a major impact to lending decisions and the portfolio mix of an organization.

For example, let’s look at a very basic attribute like total number of trades. When creating this attribute, an organization has to decide what constitutes a trade. For instance, is a collection account a trade that should be included in the count? Again, this may seem trivial, but could have a significant impact to the risk associated with a consumer when combined with other credit data.

Whether credit attributes are created and managed internally or purchased from an attribute provider, the process of defining and leveling credit bureau data across bureaus requires significant time and resources. Therefore, ensuring the attributes used are statistically accurate and predictive is vital to the long-term success of an organization.
 

-- By Wendy Greenawalt

The US has the most extensive credit bureau data in the world. The available credit data is vast and very complex making it difficult to synthesize the data across bureaus. Transforming tri-bureau data into informed decisions is challenging for most financial institutions. Due to this, many organizations rely on a highly skilled team of credit data experts to create and manage their credit attributes.

Creating or modifying tri-bureau credit attributes requires extensive credit data knowledge. It’s similar to making a cake. Everyone knows it takes certain ingredients to bake a cake but if the measurements are not precise then the cake will not taste good and may even be flat in the middle. Similarly, not knowing all the nuances to bureau data can produce inaccurate results. For an organization to accurately develop tri-bureau attributes, it requires years of analyzing available bureau data, creating attribute definitions and testing the attributes to validate them for accuracy.

This data expertise already exists within the credit bureaus and can easily be leveraged to ensure that the underlying data is accurately evaluated across all bureaus. Data intelligence can assist organizations in interpretation, translation, and manipulation of bureau data, helping them utilize the information to make smarter and more informed decisions. Examples of data intelligence can include tri-bureau attribute leveling, creation of custom attributes, system migrations and auditing of scorecards and/or attributes to validate analytical accuracy. In my next blog I will discuss the specific challenges lenders face when creating tri-bureau and custom attributes.

 

In addition to behavioral models, collections and account management groups need the ability to implement collections workflow strategies in order to effectively handle and process accounts, particularly when the optimization of resources is a priority. While the behavioral models will effectively evaluate and measure the likelihood that an account will become delinquent or result in a loss, strategies are the specific actions taken, based on the score prediction, as well as other key information that is available when those actions are appropriate.

Identifying high-risk accounts, for example, may result in strategies designed to accelerate collections management activity and execute more aggressive actions. On the other hand, identifying low-risk accounts can help determine when to take advantage of cost-saving actions and focus on customer retention programs.  Effective strategies also address how to handle accounts that fall between the high- and low-risk extremes, as well as accounts that fall into special categories such as first payment defaults, recently delinquent accounts and unique customer or product segments.

To accommodate lenders with systems that cannot support either behavioral scorecards or strategies, Experian developed the powerful service bureau solution, Portfolio Management Package, which is also referred to as PMP. To use this service, lenders send Experian customer master file data on a daily basis. Experian processes the data through the Portfolio Management Package system which includes calculating Fast Start behavior scores and identifying special handling accounts and electronically delivers the recommended strategies and actions codes within hours. Scoring and strategy parameters can be easily changed, as well as portfolio segmentation, special handling options and scorecard selections.

PMP also supports Champion Challenger testing to enable users to learn which strategies are most effective. Comprehensive reports suites provide the critical information needed for lenders to design strategies and evaluate and compare the performance of those strategies.
 

1.       Portfolio Management – You should really focus on this topic in 2009.  With many institutions already streamlining the origination process, portfolio management is the logical next step.  While the foundation is based in credit quality, portfolio management is not just for the credit side. 

2.       Review of Data (aka “Getting Behind the Numbers”) – We are not talking about scorecard validation; that’s another subject.  This is more general.  Traditional commercial lending rarely maintains a sophisticated database on its clients.  Even when it does, traditional commercial lending rarely analyzes the data. 

3.       Lowering Costs of Origination – Always a shoe-in for a goal in any year!  But how does an institution make meaningful and marked improvements in reducing its costs of origination? 

4.       Scorecard Validation – Getting more specific with the review of data.  Discuss the basic components of the validation process and what your institution can do to best prepare itself for analyzing the results of a validation.  Whether it be an interim validation or a full-sized one, put together the right steps to ensure your institution derives the maximum benefit from its scorecard.

5.       Turnaround Times (Response to Client) –Rebuild it.  Make the origination process better, stronger and faster.  No; we aren’t talking about bionics here -- nor how you can manipulate the metrics to report a faster turnaround time.  We are talking about what you can do from a loan applicant perspective to improve turnaround time.

6.       Training – Where are all the training programs?  Send in all the training programs!  Worry, because they are not here.  (Replace training programs with clowns and we might have an oldies song.)  Can’t find the right people with the right talent in the marketplace? 

7.       Application Volume/Marketing/Relationship Management – You can design and execute the most efficient origination and portfolio management processes.   But, without addressing client and application volume, what good are they?

8.       Pricing/Yield on Portfolio – “We compete on service, not price.” We’ve heard this over and over again.  In reality, the sales side always resorts to price as the final differentiator.  Utilizing standardization and consistency can streamline your process and drive improved yields on your portfolio.

9.       Management Metrics – How do I know that I am going in the right direction?  Strategize, implement, execute, measure and repeat.  Learn how to set your targets to provide meaningful bottom line results.

10.    Operational Risk Management – Different from credit risk, operational risk and its management, operational risk management deals with what an institution should do to make sure it is not open to operational risk in the portfolio. Items totally in the control of the institution, if not executed properly, can cause significant loss.

What do you think? As the end of April approaches, are these still hot topics in your financial institution?

In addition to behavioral models, collections management and account management groups need the ability to implement strategies in order to effectively handle and process accounts, particularly when the optimization of resources is a priority. While the behavioral models will effectively evaluate and measure the likelihood that an account will become delinquent or result in a loss, strategies are the specific actions taken, based on the score prediction, as well as other key information that is available when those actions are appropriate.

Identifying high-risk accounts, for example, may result in collections strategies designed to accelerate collections activity and execute more aggressive actions and increase collections efficiency. On the other hand, identifying low-risk accounts can help determine when to take advantage of cost-saving actions and focus on customer retention programs. Effective strategies also address how to handle accounts that fall between the high- and low-risk extremes, as well as accounts that fall into special categories such as first-payment defaults, recently delinquent accounts and unique customer or product segments.

To accommodate lenders with systems that cannot support either behavioral scorecards or automated strategy assignments a hosted collections software decisioning system can close the gap. To use these services master file data needs to be transmitted (securely) on a regular basis. The remote decision engine then calculates behavioral scores, identifies special handling accounts and electronically delivers the recommended strategy code or string of actions to drive treatments.
 

This post is a feature from my colleague and guest blogger, Barry Timm, Senior Process Architect in Advisory Services at Baker Hill, a part of Experian.

2008 has proven to be an unbelievably challenging year for the economy as a whole, let alone the financial industry.  Never before have we experienced the type and degree of turmoil that we did in 2008, even since the “Great Depression”.
 

These economic challenges have been quick, severe and widespread; and, from large corporations to the individual consumer, all have been impacted to some degree.  The stock market is down, unemployment up, consumer confidence down, delinquencies up ….not exactly a pleasant roller coaster ride. 
 

And, there is no longer any projecting as to when the “bubble” is going to burst.  It happened.   Decreased real estate values have occurred not only in high impact geographic regions but throughout the country.  While home equity products have traditionally been the “golden child” of consumer loan product offerings, recent economic changes have caused a shift in that perspective.  As a result, tightened underwriting standards have limited the availability of the product as a whole.  In some markets the product offering has even been temporarily halted.
 

We frequently hear the terminology “bailout” being used in the news.  While we all have expectations as it relates to the bailout approach, I thought I would “Google” the word “bailout” to see what would magically appear.  Interestingly enough, the first listing was titled “Walk away from your home”, with a link to the home page for a mortgage default legal team.  This is not exactly what I was expecting to find, but is definitely reflective of the times.
 

And, according to the FDIC, there have been 25 failed financial instituions in the year 2008.  This single year number equates to the total number of failed financial institutions between the prior periods 2001 through 2007. 

Okay … enough doom and gloom.  In spite of all that has occurred within the economy, some financial institutions continue to maintain a strong credit quality position in their consumer portfolios and have maintained profitability throughout all of the market volatility.  

What are the strong survivors doing that differentiates themselves from the others?

1. They understand their portfolio.  
Advisory Services frequently assists clients with various types of portfolio management analysis and often presents those findings to senior management.  We often hear that management is surprised by the results of that analysis. The point is that high-level management reporting is not enough these days. Additional detail and depth are necessary. 

More specifically, as opposed to evaluating payment performance at the portfolio level, it is important to consider the following:

• Do you know your delinquency numbers at the product level? 
• How do delinquencies compare to your product approval rates? 
• Do you routinely compare approval/decline rates and delinquencies to scorecard results and/or credit bureau scores?  
• Do you know where pricing exceptions are being made and are you receiving sufficient return for the level of risk?

2. A focused strategy is in place.
It is important to re-emphasize the specific, strategic direction and focus of your defined market.  Now is not the time to be “pushing the envelope” and extending into untested waters.  There is something to be said about focusing on your strengths, staying within your defined footprint and meeting the needs of your core, proven line of business while following sound financial risk management.

3. The underwriting process is under control.
This does not automatically mean that a “tightening” of underwriting standards is necessary.  It does mean, however, that stronger attention to detail is warranted.  It is important that underwriting criteria is reviewed and that you are sure that defined underwriting practices are consistently applied.  As noted in item number one above, this may require digging a little deeper and reviewing current and past decisioned loans (preferably with a critical eye of an independent third party).  Assessing the underwriting process becomes increasing complex and more critical with a decentralized underwriting approach.

Focus on the positive
Now that 2008 is behind us, let’s continue to focus on the positives to come in 2009.  Reflect on the past, but strive to center your attention on ongoing portfolio monitoring, financial risk management assessments and improvements for the future. 

 

So here it is!  The moment you all have been waiting for--the top ten hot topics of 2009 (in no particular order of importance).

1. Portfolio Risk Management – You should really focus on this topic in 2009.  With many institutions already streamlining the origination process, portfolio management is the logical next step.  While the foundation is based in credit quality, portfolio management is not just for the credit side. 
2. Review of Data (aka “Getting Behind the Numbers”) – We are not talking about scorecard validation; that’s another subject.  This is more general.  Traditional commercial lending rarely maintains a sophisticated database on its clients.  Even when it does, traditional commercial lending rarely analyzes the data. 

3. Lowering Costs of Origination – Always a shoe-in for a goal in any year!  But how does an institution make meaningful and marked improvements in reducing its costs of origination? 

4. Scorecard Validation – Getting more specific with the review of data.  Discuss the basic components of the validation process and what your institution can do to best prepare itself for analyzing the results of a validation.  Whether it be an interim validation or a full-sized one, put together the right steps to ensure your institution derives the maximum benefit from its scorecard.

5. Turnaround Times (Response to Client) –Rebuild it.  Make the origination process better, stronger and faster.  No; we aren’t talking about bionics here -- nor how you can manipulate the metrics to report a faster turnaround time.  We are talking about what you can do from a loan applicant perspective to improve turnaround time.

6. Training – Where are all the training programs?  Send in all the training programs!  Worry, because they are not here.  (Replace training programs with clowns and we might have an oldies song.)  Can’t find the right people with the right talent in the marketplace? 

7. Application Volume/Marketing/Relationship Management – You can design and execute the most efficient origination and portfolio management processes.   But, without addressing client and application volume, what good are they?

8. Pricing/Yield on Portfolio – “We compete on service, not price.” We’ve heard this over and over again.  In reality, the sales side always resorts to price as the final differentiator.  Utilizing standardization and consistency can streamline your process and drive improved yields on your portfolio.

9. Management Metrics – How do I know that I am going in the right direction?  Strategize, implement, execute, measure and repeat.  Learn how to set your targets to provide meaningful bottom line results.

10. Operational Risk Management – Different from credit risk, operational risk and its management, operational risk management deals with what an institution should do to make sure it is not open to operational risk in the portfolio. Items totally in the control of the institution, if not executed properly, can cause significant loss.

Well, that’s it.  We encourage your feedback on this list.  Let us know which of these ten topics is a priority for your institution and what specific areas in each topic you would like to see addressed.