One firm in the FCA's new case studies used narrow guardrails to reduce unexpected AI responses. Its customers then received overly generic outcomes, and some were steered towards inappropriate options. It's the case study I'd put in front of any board setting risk appetite for AI.
In August, I described the FCA Mills Review (published 6 July 2026) as a sensible plan for 2030. I also challenged its response to immediate operational and model risks.
Two months on, the FCA has gone some way to answering that. On 19 October 2026 (amended 20 October), it published AI in financial services: good practice and areas for improvement. The October paper supports several concerns we raised in our August article, including automated output checks, tailored governance, and incident playbooks.
But third-party AI oversight remains difficult. Firms are expected to maintain effective oversight of AI models supplied by external providers, yet smaller and specialist lenders often have little contractual power to audit those models or influence how they operate.
For these lenders, evidence from their own systems becomes particularly important. They need to show how AI is performing, whether customers are receiving appropriate outcomes, and how problems are identified and addressed. That's where I've focused this article.
Key takeaways from this article:
That's the gist. Now let's dig a little deeper. ↓ |
Where the October paper stands in the regulatory framework
First, a word on status. Section 1.2 states that the FCA examples are illustrative and create no new regulatory requirements. In other words, existing rules continue to apply, including the Consumer Duty, the Senior Managers and Certification Regime (SM&CR), and complaints rules (DISP).
It's also worth knowing what the paper leaves out. Section 4 excludes broader frontier AI cyber risks because the FCA examined them separately in its September multi-firm review.
Guardrails versus customer outcomes
The first theme I want to pick up is how firms judge whether an AI tool is working for the customers using it.
Monitoring output meaning and customer outcomes
In August, I noted that many risk teams lack the tools to monitor output changes from a large language model (LLM). Section 3.2 of the October paper backs that up, identifying shortfalls in oversight capability across second-line and third-line teams.
For this article, I'm using semantic monitoring to mean tracking whether the meaning of model outputs changes over time against an approved test set of prompts. Firms should also test whether outputs remain useful and produce good customer outcomes.
Case Study K (Section 3.2) shows what that can look like. One firm evaluated AI-generated claims information against five criteria: completeness, faithfulness to source information, conciseness, adequacy of reasoning, and safety. It combined staff checks with an automated assessment that used an LLM-as-a-judge. Combining human checks with an automated judge reduced the risk that biases shared by models from the same vendor, trained on the same data, would distort the test results.
Guardrails and risk appetite
The flip side shows up in Case Study C (Section 2.1). One firm used narrow technical guardrails to reduce model drift and unexpected responses. However, the tool then gave customers overly generic outcomes, and it steered some consumers towards inappropriate options.
It's easy to see how that happens. Take a hypothetical risk appetite review where the board asks for zero AI errors. Engineers respond by restricting the tool's answers, and testing finds fewer unsupported statements. Meanwhile, customers receive less useful support.
To be fair, Section 2.1 supports abstention, redirection, and staff-led tasks where an AI tool cannot act safely, and a refusal can protect a customer in those cases. But firms must not optimise factual accuracy by excluding necessary customer support from the success criteria.
So instead, I'd point boards to Case Study P (Section 3.5). There, one firm turned its risk appetite into red, amber, and green thresholds with predefined actions. I come back to that idea in the threshold table later in this article.
Vulnerable customers and evidence of decisions
The same tension applies to vulnerable customers. Section 2.1 notes that some firms excluded customers with vulnerability indicators from AI journeys by default. Exclusion may suit some scenarios. Even so, firms should check that these customers can still obtain support or benefits that meet their needs.
Section 2.3 also warns against assuming that an LLM can reliably identify temporary vulnerability from text alone. Firms need to test this capability specifically and have fallback prompts in place where vulnerability may not be detected.
Firms also need evidence of how AI interactions are handled. Under Section 2.2, firms should keep auditable records of consumer prompts, outputs, model versions, applied filters, and escalation decisions. These records support complaints handling and customer redress, and firms should also record retrieved sources as additional evidence.
Fallback models versus approval boundaries
The second theme is resilience, and here the regulatory position depends on what kind of firm you are.
Operational resilience and model risk management
SYSC 15A covers banks and building societies. Some specialist lenders also fall within scope, including enhanced-scope SM&CR firms, so each firm should check its regulatory status before selecting its controls.
For firms in scope, the paper is direct. They must map AI dependencies that support important business services (Section 4.1). And SYSC 15A.2.10G cautions against meeting resilience objectives through another rule breach or increased customer harm.
Model risk sits under a separate regime. PRA SS1/23 (updated 16 April 2026) covers relevant UK-incorporated firms with permission to use internal models for regulatory capital, and other lenders may apply its principles voluntarily.
Supplier concentration and the FCA case studies
Resilience planning also has to reflect how concentrated the supply side has become. When the Bank of England and FCA surveyed 118 firms in 2024, the top three model providers' share of named providers had risen from 18% in 2022 to 44%. Meanwhile, the top three cloud providers accounted for 73% of named cloud providers.
Several case studies show firms designing around that. In Case Study Q (Section 4.1), one firm designed its AI service to avoid dependence on a single model, and it maintained alternative server capacity in another location. In Case Study S (Section 4.2), another firm designed its architecture to avoid dependence on a single provider.
Case Study T shows what happens without that planning. Model retirement forced a firm to redesign its product because it lacked a transition plan. Section 4.2 notes that firms have limited control over vendor changes, and that some have negotiated contractual notice of major updates. In practice, lenders cannot prevent a global provider from changing model weights or retiring a model at short notice. This is particularly difficult for smaller firms, which have little negotiating power with major providers.
Where firms cannot influence the supplier, they need stronger controls over their own technical environment, known as the 'harness’ in the FCA's September cyber review. That includes model routing, automated output checks and monitoring, and validation of fallback models before deployment, so firms can respond safely when monitoring identifies a material change.
Model routers and validation of fallback
A cloud model router can switch a request to a backup model during an outage. However, the validation evidence for the primary model does not automatically apply to the backup. Banks within scope of SS1/23 must apply formal governance and independent challenge to material model changes. Smaller lenders outside its scope should consider the same practical checks, given their obligations under the Consumer Duty to avoid causing foreseeable harm to retail customers.
Before deployment, firms should complete three checks:
- Validate each approved fallback model and its routing rules.
- Test how the system handles incomplete conversations and unfinished actions when switching models.
- Check whether the primary and fallback models rely on the same cloud platform or data retrieval service.
Human oversight, automation bias, and incident thresholds
All that testing covers the technology, and someone still has to review what the AI produces. That brings the October paper back to a point from our August article.
Reviewer scrutiny
In August, we challenged systems that generate both customer decisions and compliance records, or 'marking their own homework'. Human review is the obvious safeguard, and the 2024 Bank and FCA survey found that only 2% of reported use cases involved fully autonomous decisions. Even so, human review does not fully defend against model errors.
Case Study O (Section 3.4) shows why. The reviewer role was poorly defined and became a box-ticking exercise. Over time, reviewers accepted AI recommendations without sufficient scrutiny, and vague escalation criteria allowed uncertain outputs to proceed.
Two other case studies address separate control issues. In Case Study N (Section 3.4), staff assessed affordability cases before seeing the AI recommendation, while the firm was setting up the system. This blind review reduced the risk of staff automatically accepting AI recommendations. Once deployed, firms could apply the same control to a sample of live cases, hiding the AI recommendation until the reviewer has recorded their own decision.
Meanwhile, in Case Study M (Section 3.3), the relevant SMF holder and business areas approved an incident playbook before launch. Red, amber, and green ratings then informed next steps, such as enhanced monitoring or temporary shutdown.
Points for senior managers to test
All of this comes back to accountability. Under Section 3.3 (Case Study L), responsibilities should match the firm's responsibility documents, use case, and separate model approval authority. With that in mind, here are the points I'd want each role to test.
- Chief Risk Officer (SMF4): Do the metrics support board challenge without disguising poor customer outcomes?
- Chief Operations Function (SMF24, where applicable): Can the service remain safe when traffic, staffing, or a supplier fails?
- Head of Model Risk: What evidence supports each configuration, and what changes exceed the existing model approval?
- Responsible business SMF holder: Who owns the customer impact, incident decision, and customer recovery?
An illustrative threshold table
To bring Case Studies M and P together, I've sketched out how thresholds might work for a customer support and early-arrears assessment tool. The thresholds are illustrative controls and carry no FCA regulatory status.
A few ground rules sit behind the table:
- Define minor errors before testing, and use a rolling sample of 200 independently checked outputs within seven days.
- Record the customer groups and model configurations represented.
- Do not assign Green status when evidence is insufficient, and do not treat sample counts as population error limits.
- If you suspect serious harm, pause affected routes while you investigate, and record who can approve restart after remediation and retesting.
|
Status |
Illustrative trigger |
Action and owner |
|
Insufficient evidence |
Fewer than 200 independently checked outputs in seven days, or a missing route sample. |
Withhold Green status. Pause expansion until the model owner completes the sample. |
|
Green |
No Red or Amber event in 200 independently checked outputs in seven days. Four or fewer minor errors, with no unresolved Amber trigger from the previous weekly sample. |
Continue within approved scope. Model owner reviews the monitoring dashboard daily. |
|
Amber (quality or outcomes) |
Five or more minor errors, in two consecutive weekly samples, rising unsuitable options, or meaning drift. |
Pause expansion. Model owner diagnoses the cause within four hours. Validator independently challenges the findings. |
|
Amber (capacity) |
Human review queue exceeds 30 minutes during staffed hours. |
Reduce AI traffic. Operations lead restores capacity within one hour. Otherwise, route users to a tested non-AI channel. |
|
Red (customer harm) |
Credible suspected or confirmed fabricated contract term, failed support, or harmful instruction. |
Block the affected route immediately. Transfer customers to a tested, adequately staffed support channel. Assess whether affected customers need redress. |
|
Red (change control) |
An unapproved model version or routing configuration serves live traffic. |
Block the configuration immediately. Incident manager checks affected outputs. Approving authority signs off before restart. |
Each firm will need to set its own numbers for its own use case. Agreeing the triggers and owners in advance means everyone knows who acts, and how quickly, when a threshold is hit.
A three-step plan for lenders
If you're deciding where to begin, these are the three steps I'd prioritise.
- Measure customer outcomes alongside factual errors. Count unsuitable options, unnecessary refusals, and failed customer support alongside false statements.
- Test the complete system before deployment. Validate the model, system prompts, retrieved data, output filters, and backup model switches as one system before launch.
- Test human reviewers. Use blind sample checks to verify that staff challenge AI outputs, and set clear queue limits to reduce AI traffic when review teams are at capacity.
Building the evidence, one use case first
Those steps are easier to take once you know what evidence the FCA expects to see, and the October paper gives firms a far better sense of that. Much of it picks up the issues we raised in August. For mid-tier and specialist lenders, the next step is producing that evidence from a live service.
My advice is to start with one material AI use case. We can scope a Jaywing AI Governance and Model Risk Readiness Review around that service, so the evidence is agreed up front. That starts with a versioned inventory and comparative fallback test results. It also means an incident playbook, plus a monitoring baseline with triggers for retesting. Finally, we'd define calibration tests for any LLM judge and assign responsibility for independent staff checks.
Throughout, we keep model development separate from independent validation, in line with paragraph 2.14 of SS1/23 where applicable. We also record conflicts, reporting lines, and approval authority.
If you'd like to talk through which use case to start with, get in touch and we can agree the scope and evidence required together.
Further reading. You may also like:
- The Mills Review: a smart long-term plan with a near-term problem
- The PRA's AI roundtable: A governance reality check for UK banks
- Managing model risk using next-gen model-governance infrastructures
FAQs
Can a fallback model rely on the primary model's validation?
When switching to a backup model during an outage, the primary model's validation evidence does not apply to the backup model. Firms should validate each permitted fallback configuration and its routing rules before deployment.
Is excluding vulnerable customers from AI journeys enough?
Exclusion may suit some scenarios, according to Section 2.1. Even so, firms should ensure these customers can still obtain support or benefits that meet their needs.
Does human review protect against AI errors?
Only when reviewers apply real scrutiny. In Case Study O, the reviewer role became a box-ticking exercise and uncertain outputs proceeded. In Case Study N, blind review reduced the risk of automatic deference.