At S&P Capital IQ, we are fortunate to provide a range of credit risk and capital related solutions to financial and credit lending institutions worldwide. Through these client engagements, there have been many insightful lessons learned – especially pertaining to Model Governance, so much so, that we have developed a checklist of 7 key components that should be considered and robustly documented in a sound model governance framework.
1. Development – Model risk management begins with robust model development, implementation, and use. Regulatory guidance recognizes that model development is not a routine technical process, but one where the experience and judgment of the developers, as much as their technical knowledge, greatly influence the appropriate selection of model components. But from a model governance perspective, it’s vital that every step of the development process is fully documented. That includes the data development set, where there should be rigorous assessments of data quality and relevance, as well as performance testing which is an integral part of model development. Regulatory guidance suggests that model testing should include checking the model's accuracy, demonstrating that the model is robust and stable, assessing potential limitations, and evaluating the model's behavior over a range of input values. We should also assess the impact of assumptions and identify situations where the model performs poorly or becomes unreliable. All this information should be captured in the Model Development Documentation to be reviewed by your Validation Team.
2. Implementation – the key to the implementation process is to generate consistent results. We should note that the development, implementation and use phases of the model life cycle are typically handled by different groups within a company, with the potential for risk increasing as the model is passed between different stakeholders. This is where model implementation documentation, including User’s Guides and operating procedures, are of key importance to help mitigate that risk and reduce key person dependencies.
3. Validation – it is vital to effectively challenge your models at least on an annual basis. That includes a qualitative validation, where model design and methodologies are examined through an independent review of the conceptual soundness of the model. Another example is where the model output will be benchmarked to the output of an alternative model with any divergences investigated. Validations should be conducted internally through a separate and independent group or outsourced to a qualified third party.
4. Approval – when considering the approval process you need to consider the model governance roles and responsibilities.
While we have identified six roles, conceptually, the roles in model risk management can be divided among ownership, controls, and compliance. The checks and balances provided by the Model Advisory Team, Model Validation Group and Model Oversight Committee ensure a rigorous approval process. A fundamental component for each of these roles is their independence, and it is important that reporting lines and incentives be clear, with potential conflicts of interest identified and addressed. Certain regulatory guidance also suggests that senior management is responsible for reporting to the board on significant model risk-- from individual models and in the aggregate -- and on compliance with policy. Board members should ensure that the level of model risk is within their tolerance and direct changes/modifications where appropriate.
5. Modification – the most common breakdown in a model governance framework is the lack of documenting and testing on model modifications. As we have found when we conducted validations with numerous clients, it’s quite common that model “enhancements” are undocumented and informal. Meanwhile, regulators expect institutions to have a formal plan for the on-going performance and risk monitoring. These plans should detail the plan of action for assessing the model’s continued effectiveness (this is, in essence, a formal model validation policy). For example, this could include a Comparison of model prediction against subsequent real-world events (back-testing, out-of-sample-testing, etc.)
6. Retirement – an underrated component of the model governance framework is model retirement, but I want to emphasize that on-going monitoring needs to be done over the life of the model, and what is considered the best or “Champion” model can change over time. As existing models are replaced, resources should be focused on the “Champion” and “Challenger” models, and those models that do not meet that criteria should be retired.
7. Model Inventory – you definitely learn more about an institution’s internal risk rating system when you review their model inventory – the strengths and more importantly where the gaps exist. So a model inventory review is the logical starting point where you would review the overall model governance framework. I want to mention that model inventory is not a one-time exercise, and there should be a process to ensure model inventory is tracking the models by materiality, key dependencies, or assigned model risk scores.
In the current environment of heighted regulatory scrutiny, the need for a sound model governance framework is more crucial than ever. We believe that focusing on these seven components covering all phases of a model life cycle -- development, implementation and on-going performance and risk monitoring -- provides a roadmap towards avoiding the most frequently-seen, critical regulatory concerns on Model Governance.
Interested in our Model Risk Mitigation Services? Click here to get more info.
Thank you for checking out the October installment of the Risk Insight Monthly Video & Blog Series! Check back here each month to read the latest Risk Insight blog and video. Click here if you would like to be added to the mailing list, and, each month, you will receive an email with links to the video, the blog, and a “Get More Info” form. We want this series to be as interactive as our webinars, so please take advantage of this form to ask us questions or simply to get more info on the topic at hand.