The IFRS 9 compliance deadline has passed, but are your models up to scratch? Mark Vickers, Head of IFRS 9, explores the fundamentals of effective model monitoring in this latest blog.
2018 saw firms finalise the first generation of IFRS 9 models, which now drive impairment disclosures. Initially, the main objectives were to satisfy regulatory requirements while addressing the many challenges associated with a successful implementation.
A year on and the focus is shifting towards one of maintenance or ideally, continuous improvement. Many firms acknowledged areas of compromise in the first-generation models, largely as a result of the additional complexities imposed by the new standard and the aggressive timelines associated with its introduction.
Crucially, the now living and breathing models and their outputs need to be understood and, where necessary, remedied in a timely manner. Any firm adopting tactical approaches to this could be forgiven, especially considering the backlog of fixes and explanatory analyses that already dominates the typical post-implementation to-do list.
But the questions and concerns will keep coming; Senior Management will want to see progressively more around what is driving changes in impairment, and the expectation that the health of the models can be readily ascertained will only grow. And the PRA’s recent consultation on the importance of understanding model risk in stress testing and the recent focus from auditors is further elevating the importance of effective model monitoring.
A best practice framework
Regardless of the size of organisation or complexity of the model suite, the fundamental principles of an effective IFRS 9 MI and model Monitoring framework are broadly consistent across the industry. We believe that firms should seek to address the following objectives on a BAU basis:
- Provide confidence the most appropriate ECL are being reported
- Rationalise changes in ECL from previous runs to latest
- Inform as to where management should consider overlays or adjustments to the model output due to potential under performance
- Highlight areas that require redevelopment and prescribe what the scope of redevelopment should be
- Provide sufficient evidence for management to justify proposed ECL to auditors and respond adequately to most concerns
With these intentions in mind, we recommend three distinct areas on which to establish periodic monitoring processes, namely 'Model Inputs', 'Model Outputs' and 'Model Performance'. These are described in the following sections.
- Model Inputs
IFRS 9 caused a step-change in data requirements for many firms. The cliché ‘garbage in, garbage out’ is more prominent than ever before; while the regulator may have taken an accepting approach to the initial implementation, there is now an increased emphasis on whether the data feeding models is an accurate reflection of the state of the business.
Putting aside the increased expectation of strategic above tactical approaches to Data Management, teams running IFRS 9 models should look to establish sufficient confidence in all input data as a precursor to signing off the latest outputs and performing any downstream analysis or reporting. This may largely comprise of standard checks for occurrences of missing values, maxima and minima, default values, averages, field formats, etc, benchmarked according to expectations from previously approved data.
An additional source of input data inherent to IFRS 9 calculations relates to economic indices, in particular their forecasts. The appropriateness of current scenarios should be verified – this may happen via an assessment of actual economic indices observed historically, to either support or challenge how realistic the latest scenario forecasts are. Previous forecast accuracy can also be examined, to help determine any systematic error in the economic forecasting approach that may in turn bring into question the ongoing suitability of the forecasts driving the ECL outputs.
- Model Outputs
Assuming the quality of the input data has been verified, model outputs can be more reliably assessed. The purpose of this strand is to provide meaningful MI that summarises at varying levels of granularity the aggregated outputs of the latest model run, benchmarked against previously reported outputs.
At the highest level, this may show the total ECL overall and across each stage, most typically for a single portfolio. Further breakdowns may arise from various categorisations such as stage subcategories, different portfolio products and grouped delinquency bands. Component-level summaries covering PD, EAD and LGD help to pinpoint drivers of movement trends, with the potential to include their relevant subcomponents for a more targeted diagnosis, should the model infrastructure and materiality dictate.
Creating a breakdown by economic scenario can also be useful to help identify whether a particular scenario is driving significant movement in ECL. Other reports can be designed to look deeper into account-level movements across specific stages over time; this type of analysis can prove useful for highlighting sources of volatility in ECL outputs and assessing the suitability of stage allocation rules.
- Model Performance
By far the trickiest of the 3 areas to establish effective model monitoring for relates to the ongoing performance of the live models. For some firms the infrastructure demands of IFRS 9 have been transformational, while others are able to somewhat leverage existing processes, for example relating to scorecard or IRB monitoring. However, in all cases, the nature of IFRS 9 calls for revolutionary approaches to performance monitoring.
One reason for this is that the forward-looking aspect creates an additional time dimension that is not inherent in previous model monitoring frameworks. For a given metric, the sheer number of charts that can be produced is amplified; experience of IFRS 9 monitoring is crucial to effectively manage this extra dimension and avoid wasting effort creating an unmanageable library of views that end up being ignored.
Another reason is the impact of economic scenarios on those forward-looking predictions. When retrospectively assessing model performance, it is important to decouple forecasting error from model error, in order to answer the question of whether the model remains fit for purpose. Isolating these effects can be easier said than done, depending on the model and data infrastructure.
Practicality is key when considering time horizons over which to monitor IFRS 9 outputs. For some portfolios, ‘lifetime’ poses an obvious problem, in that by the time it is reached the relevance of the verdict may have long since expired. Clearly there is a trade-off between the additional information afforded by longer performance windows and the recency of outputs being assessed. What constitutes a suitable monitoring view also differs across individual ECL components – for example, LGD typically requires longer outcomes and hence alternative data structures and reporting views may be suitable compared to PD.
Tolerances are a key aspect of any performance monitoring framework. They can be either statistically or judgementally based and may take into account the materiality of the component in question. Most importantly, they should be set in such a way that drives tangible actions as opposed to ignorance, with appropriate remedial work being prescribed in light of the particular breach.
In Summary…
The transition of IFRS 9 into something of a BAU requirement has brought with it a need to have a constant finger on the pulse in terms of understanding the models and generating actionable insight.
A three-pronged approach that separately caters for model inputs, outputs and performance provides a best practice framework that helps to ensure the key objectives of IFRS 9 model management are always adhered to. In practice, the complexity should be commensurate to the organisation, considering a range of factors including size, materiality, model structure and sophistication, portfolio and product types and stakeholder requirements.
An in-depth understanding of the regulatory standards, model infrastructure and nuances of IFRS 9 model monitoring is key to ensuring a suitable and effective framework amidst an evolving landscape.
For further guidance on IFRS 9 model monitoring, get in touch.