Model risk management What you can do to help
Imagine that you and your entire family are taking the trip of a lifetime to an exotic destination. In preparing to pack your bags, you search Google for the weather and find that it will be amazing the entire time with, sunny skies and mid-80's temperatures. You inform the family that flip flops, shorts and swimsuits are all that is needed! Upon arrival, you're shocked and upset to realize that it's raining and cold - the high is only 58 degrees with similar weather expected all week. You ask yourself what went wrong because the website you searched said it would be nice. Unfortunately, you relied upon a weather model that was flawed, resulting in a new and unexpected wardrobe for the whole family. Not to mention disappointment that no one had prepared themselves for less than perfect weather. You think to yourself that you should have checked a couple of different websites, called the hotel directly, read some prior visitor blogs or packed some different clothing just-in-case.
Much like this example, the models that you rely on at your institution could provide results that paint a sunny picture when they shouldn't. You may find out too late that you should have paid closer attention to understanding the model results and placed a little more effort in double checking the outputs and their possible consequences.
Models can help an institution simplify real world relationships and can be applied to a broad range of activities from underwriting credit and understanding interest rate risk, to determining capital and loan reserve adequacy. Regulators and auditors are encouraging the use of technology and models to improve decision making at institutions. Conversely, however, there is also increased scrutiny from regulators surrounding risk management associated with these same models. Each financial institution regulator has issued specific direction that broadens the scope of prior guidance to include core elements of model risk management and to state the vital role of governance, policies and controls.
Corporate governance, policies and controls
In the simplest sense, a model takes inputs, goes through some kind of processing, and produces output that the institution relies upon for decision-making. Emphasis has been given by regulators and auditors to the importance of having strong governance, policies and controls around model risk management, including a strong tone at the top and understanding of key risks by management. Regardless of the model used, leadership at the institution should take the time to periodically understand the key assumptions and inputs in a model in order to effectively set the applicable policies and procedures. This governance process can occur internally, through an independent institution review or by a third party that is hired to review the significant policies and processes.
To illustrate governance and oversight, we recently reviewed an Allowance for Loan and Lease Losses (ALLL) model at a community bank for consistency with regulatory and accounting guidance, as well as sound risk management practices. The chief credit officer was responsible for calculating the impairment amount for impaired loans. He obtained a new appraisal for a large condominium storage complex and determined that initial value in the calculation should not be the bulk-sale or as-is market value from the appraisal. Instead, the officer used the retail sales value of individual units. He believed that upon foreclosure, the institution would attempt to sell each of the condominium units individually to minimize losses, even though the borrower had been unsuccessful in selling any of the individual units during the prior two years. The difference in the calculated impairment amount using the individual unit retail value was significant at more than $1 million. The oversight committee and other members of management were not aware that the officer had used a value other than the appraised as-is market value when they approved the adequacy of the allowance. Although the officer was not acting maliciously in his use of the individual retail values in estimating losses, he was not familiar with the consequences of holding OREO for long periods of time and made incorrect assumptions. The oversight committee ultimately determined that the as-is market value was a more appropriate value. Using the most applicable appraisal value is a key assumption in the ALLL model and ultimately, in determining the adequacy of the ALLL balance. Governance and oversight could have been more effective if policies required detailed documentation in the calculation worksheets and a second, independent review, as to how initial values were determined in the impairment calculations.
In summary, effective governance requires policies that identify significant models and define the key model assumptions (you know what they are!). The policies should clearly state the expectations for documenting how inputs and assumptions are determined and reported for review at each measurement date.
Other aspects of model risk management include development and implementation, on-going use and periodic valuation.
Model development and implementation
Much like weather prediction models, model development and implementation are often not within your control. Senior management may consult with third party model vendors to determine which model to purchase or may assign internally developed models to a steering committee or specific individuals. However, to the extent you can provide input into establishing the uses and objectives of the model, you should. This can often be accomplished simply by asking good questions that allow senior managers to consider operational aspects and issues they may not have considered. There are many good due-diligence checklists for model selection that are available on the internet as well.
It is important for financial institutions to establish the model's purpose and objective during the early stages of the development process. The model's objective must be aligned with the intended model use and the design should meet the objective. Even with a perfect design in place, financial institutions can run into models that produce erroneous information. Most of the times, such results can be traced to the data used; therefore, it is important to perform a data quality and relevance assessment during the development process. Financial institutions should also keep adequate documentation throughout the development and implementation process. Proper documentation helps unfamiliar parties understand how the model operates, its limitations and its key assumptions.
Finally, pre-implementation testing is another essential component of model development as it assists in determining if the model is performing as intended. Pre-implementation testing should be done under a wide range of scenarios, including testing with extreme or unreasonable values, which help determine the model's limits and may bring to light model weaknesses or gaps.
Using your model
When you arrived at the exotic location and realized the weather website you relied upon was wrong, you likely wanted to call those people and give them an unpleasant piece of your mind. Luckily, users of financial institution models can make a difference by sharing inaccurate or questionable results with those who are running the model. Model users help the entity to monitor and assess model performance over time and as conditions change. Users of the model should challenge the assumptions used in the model, thus opening the lines of communication between developers and users. The key is to look at the model results from different levels and ask whether the results make sense. If not, let the model owner know that there could be a problem. For example, let's say you're looking at an interest rate risk report. The report says that you are currently earning 0.15 percent on interest earning deposits held at other banks. You then flip to the +200 basis point shock scenario and the report says you will earn 2.15 percent interest on those same deposits. Well, chances are the amount your institution earns will not increase at the exact same rate as the fed funds rate. What do you really think the institution will earn if rates increase by 2.0 percent? That information should be shared with the interest rate risk model owner so they can evaluation the assumptions to better reflect earnings on those deposits.
In the OCC model guidance, they promulgate that "validation should be done by people who are not responsible for development or use and do not have a stake in whether a model is determined to be valid." Adding to the independence factor, the personnel performing model validation are required to have the knowledge, skills and expertise necessary, as well as the sufficient authority, so that findings and deficiencies are addressed appropriately and in a timely matter.
Validation activities should be performed on an ongoing basis once the model goes into use. Financial institutions, usually through the internal audit or risk management functions, should perform periodic reviews, based on risk and materiality, of each of the model, determine whether they are working as intended, and whether sufficient validation activities exist.
There are three core elements of an effective validation framework:
|Evaluation of conceptual soundness||Assess the quality of the model design and construction.|
|Ongoing monitoring||Confirm that the model is implemented, used, and performing as intended.|
|Outcome analysis||Compare model outputs with corresponding actual outcomes.|
With many third-party models, the evaluation of conceptual soundness is provided by the vendor through a firm they hired to validate the model. That firm will opine as to the mechanical and mathematical accuracy of the model. In other words, as long as the data going in is good, the results coming out should be good. However, those reports do not validate that the data going in is good, which is often where issues arise. Ongoing monitoring and outcome analysis is the responsibility of all model users at your institution. Nevertheless, an independent party should evaluate the governance and controls around data going in, appropriate mapping for information uploaded into models, reasonableness of key assumptions, and accuracy and usefulness of reports and model outputs, among others.
Models can be complicated and have many moving pieces. Users of various models at all levels can provide value to their institution and help avoid reliance on bad information by helping to establish sound model governance practices, by questioning unusual or unexpected model results, by informing model owners when model results turned out differently than predicted and by having a general knowledge of model risk management and related regulatory expectations. Your involvement can help you be ready for those rainy days!