The board approved the programme because the AI was going to sharpen pricing, catch more fraud, and take the routine load off the underwriting team. Eighteen months on, the combined ratio hasn’t moved, the fraud model is flagging the same false positives it always did, and the senior underwriters have quietly gone back to trusting their own read of a risk over the number on the screen.
The instinct at that point is to look at the model. Or the vendor. Or the implementation partner who scoped the rollout. All fair questions – and usually the wrong ones.
MIT’s Project NANDA put a figure on the scale of the problem last year: across more than 300 enterprise deployments, 95% of generative AI pilots delivered no measurable impact on profit or loss[1]. The researchers were clear that this was not a story about weak models. The gap between the 5% that worked and the 95% that didn’t came down to approach – how the technology was integrated, and what it was given to work with. That second part is the one finance functions consistently under-price.
What the model is actually reading
Every decision-making model runs on a body of prior information. For an insurer, that is the accumulated record of the business: policies written, claims paid, underwriting calls made and how they turned out, fraud that was caught and fraud that slipped through, decades of loss experience. It is the institutional version of the judgement an experienced underwriter carries in their head – except the model can only draw on the part of that record which is actually legible to a machine.
Most of it isn’t. A large share sits in documents, broker submissions, loss adjuster reports, correspondence and scanned files that the model can register as existing but can’t read in any useful sense. It sees the file. It cannot pull the decision out of it, connect it to the claim it belongs to, or learn anything from the reasoning inside.
The rest is often technically accessible but inconsistently structured – captured in different formats, under different conventions, across different systems, by teams that have turned over several times. The model can reach it, but it can’t reliably learn from it, because the same concept looks like one thing in a 2011 record and something else in a 2023 one. Train a model on that and it doesn’t learn the business. It learns the gaps in the business’s record and reproduces them, at scale, with complete confidence and no flag to say it might be wrong.
Why this belongs on the CFO’s desk, not just the CTO’s
This is where the framing usually goes wrong. Data readiness gets filed as a technical problem and handed to IT, when it is the single variable that most determines whether the investment pays back.
Gartner expects organisations to abandon 60% of AI projects through 2026 specifically because they aren’t supported by AI-ready data, and found that 63% of organisations either don’t have the right data-management practices for AI or aren’t sure whether they do[2]. The cost of that gap is real money before a single model is trained: Gartner puts the average annual cost of poor data quality at $12.9 million per organisation[3].
For a CFO, the uncomfortable part is what those figures imply about the original business case. The projected improvement in the loss ratio, the fraud recovery, the productivity saving – all of it was underwritten on an assumption about the completeness and quality of the data the model would draw on. In most organisations that assumption was never tested. It was inherited.
In insurance, an underfed model isn’t just wasteful – it’s a conduct risk
There is a sharper edge to this in a regulated market. A model trained on partial or inconsistent history doesn’t simply produce mediocre pricing. It produces confident pricing that nobody can fully explain, on a foundation nobody has audited.
The FCA has already found firms using pricing datasets – some of them bought from third parties – that could contain factors implicitly, and potentially explicitly, related to race or ethnicity, even where it found no evidence of direct discrimination and the firms had no intention of pricing on those lines[4]. Under the Data (Use and Access) Act 2025, automated decisions of that kind are permitted only where the firm can offer meaningful human intervention and let a customer challenge the outcome[4]. Both obligations assume the firm knows what its model learned from and can stand behind it. A model fed on an incomplete, unstructured, unexamined record cannot give you that, and “we didn’t realise what was in the training data” is not a defence that gets better the more decisions the model has made.
What actually closes the gap
The fix is not a better model. It is giving the model you already bought the knowledge base the business case assumed it had. That means taking the unstructured and inconsistently held records – the documents, the correspondence, the scanned claims files, the legacy systems – and turning them into structured, classified, metadata-rich information the model can read and learn from consistently. Dajon’s Data Intelligence Solution does exactly that: reading each document, extracting and standardising what’s inside it, and connecting it to the claims, policies and entities it relates to, so the model finally runs on the full record rather than the fraction that happened to be tidy.
Same model. Same vendor. Different result – because the data underneath it is, at last, in the state everyone assumed it was in from the start.
The question for the next board meeting
When the AI programme comes up and someone asks why the numbers aren’t tracking the business case, there is a question that should come before any question about the model, the vendor or the roadmap.
What is the model actually reading – and is it complete, consistent and defensible enough to trust with a pricing decision, a fraud call, or a regulator’s follow-up question?
If the honest answer is anything short of a confident yes, that isn’t a model problem to solve later. It’s the problem to solve first.
Dajon’s Data Intelligence Solution transforms unstructured organisational data into the structured, classified, AI-ready knowledge base that AI engines need to make decisions that can be trusted. Get in touch to understand where your current data environment might be limiting what your AI investment can deliver.
References