Traditional model validation assumes you can test a model in isolation, sign it off, and move on. That worked when models were standalone. Today, credit and fraud models are connected. They share data sources and platforms, and their outputs feed decisioning chains. When one component changes, the impact can surface elsewhere, long after a periodic review has passed.
We increasingly see legacy frameworks struggle in this environment because they treat models as static artifacts. In our experience, this approach fails where risk sits in the network, how models interact, how fast they change, and how deeply they link. Risk builds through feedback loops and dependency chains, not single points of failure.
Because change now happens without human input, controls designed for quarterly reviews simply cannot keep up. A model might pass individual validation with flying colours, yet the combined system can still operate dangerously outside tolerance. Component-level assurance does not guarantee system-level control.
Real failures are rarely obvious. A vendor updates training data. An API changes how it classifies transactions. Nothing crashes, monitoring looks fine, and decisions shift anyway.
In distributed architectures, the most dangerous failure is a change in meaning.
|
This article covers: This article looks at why traditional model risk frameworks struggle with interconnected systems, how hidden dependencies create risk, and what continuous assurance and next-generation model risk infrastructure look like. |
Where dependency risk builds
Consider a standard affordability check. A lender uses a cloud-based NLP service to categorise transactions. A BNPL indicator is grouped with similar spend types. At some point, the provider updates its backend training data, perhaps to support a new retail partner. The API version stays the same.
Previously, this indicator sat near "Debt." In the new data, it sits closer to "Lifestyle." The affordability model now treats debt repayments as discretionary spend. Debt-to-income ratios drop artificially. The lender approves riskier loans. MI stays green because the code didn't break; the logic did.
Legacy inventories miss three specific exposure points:
#1. Unreactive to upstream changes
A fraud model feeds a credit model. If the fraud model recalibrates, it changes the data the credit model receives. Because the credit model only tracks inputs, this structural dependency stays invisible to change control.
#2. Masked structural dependencies.
One model feeds another, and the second model’s decisions influence the training data used by the first. The dependency exists, but it is rarely captured explicitly. Over time, the system begins to reinforce its own assumptions.
#3. Self-reinforcing failure modes.
Small biases are amplified through feedback loops. Performance degrades gradually rather than failing outright, making the issue harder to detect and slower to correct.
None of these failure modes sits within a single model, which is why traditional validation frameworks miss them.
The invisible accumulation of Conduct Risk
Material impacts happen when unseen changes cause issues that existing frameworks are slow to identify. Damage appears quickly through customer impacts, financial losses, and regulatory sanctions.
Credit risk (affordability assessments)
An upstream API schema change re-labels risky spend. Automated decision engines approve unaffordable loans at machine speed. Result: immediate exposure accumulation and widespread Consumer Duty breaches.
Fraud detection models
A KYC vendor's model degrades and misclassifies high-risk users. Downstream fraud models receive clean labels for bad actors. Catch rates fall. Fraud losses rise. FCA scrutiny increases.
KYC and sanctions screening
Quality decay in a sanctions list provider leads to large-scale false negatives. The firm onboards prohibited entities. This risks Voluntary Requirements (VREQ) restrictions and personal liability for the SMF holder.
In each case, the damage occurs before periodic controls or manual reviews are able to intervene.
Why assurance has to be engineered
Continuous assurance is not an extension of reporting. It requires a different approach. Review cycles designed for static models break down when behaviour can change at machine speed.
Monitoring needs to operate continuously, using measures that detect material deviation from a model’s expected behaviour. Measures such as population stability index (PSI), rank-order stability, cosine distance and Kullback-Leibler (KL) divergence look at changes in constancy and identify semantic drift.
You also need a defined recovery mechanism. When a threshold triggers a breach, the system enters quarantine mode, testing only verified reference data. New data is locked out until the model is proven safe.
Any fallback must default to a safe mode, such as pausing automated decisions or switching to a static model. Rolling back to a potentially compromised version of the same model simply reintroduces the risk.
What defines a next-generation model-risk infrastructure
To manage these risks, build governance and controls into operations from the start.
Dependency mapping is a core control.
The inventory must capture model-to-model dependencies. If Model A feeds Model B, a change to A automatically triggers a re-validation alert for B.
Action: Treat dependencies as risk exposures in their own right, not as documentation artefacts.
Validation is enforced by construction.
Validation standards and dynamic monitoring exist as executable tests within the CI/CD pipeline. Compliance is enforced, not checked retrospectively.
Action: Identify degradation early to keep dynamic models within appetite.
Critical models require pre-approved fallback strategies.
Models exposed to dynamic risks need recovery paths. In extreme cases, this includes a monitoring-triggered kill-switch mapped to an SMF.
Action: Design resilience into the system rather than dealing with its failure after the fact.
Governance must produce immutable evidence.
Action: Align with SS1/23 expectations. The system generates immutable audit trails.
Logs showing why a kill-switch triggered, or who authorised an override, become primary evidence for audit.
For interconnected systems, periodic validation on its own is no longer sufficient. The most serious failures arise from dependencies that traditional frameworks struggle to see. If your governance framework can’t see the link between a vendor’s API change and your loan book’s default rate, it is documenting risk, rather than managing it.
Rethink model risk for interconnected systems
It’s time to stop validating models in a vacuum and start engineering resilience into the network.
Jaywing Risk works with banks and lenders to design model-risk frameworks that reflect how modern credit and fraud systems operate. If you’d like to discuss how your current approach compares, we’d love to speak to you.