A mid-sized Canadian retailer began using AI to identify customers who might stop buying.
A few months later, the vendor presented the results.
“The model is 92 percent accurate.”
That sounded promising. The dashboard looked clean. The initiative was described as a success.
But nothing important shifted.
Retention stayed about the same.
Marketing began offering broader incentives.
Revenue did not respond in the way leadership had hoped.
From a modeling perspective, performance looked strong.
From a business perspective, the impact was hard to see.
This pattern appears more often than many leadership teams expect. In industries like telecom and banking, churn models frequently report accuracy rates above 85 percent. Yet the measurable improvement in retention is often modest, or nonexistent.
On paper, the system works.
In operations, very little changes.
The disconnect usually begins with what we assume “accuracy” represents.
What Accuracy Actually Tells You
Accuracy measures how often a model’s prediction matches what eventually happens.
If a system makes 100 predictions and 92 are correct, the model has 92 percent accuracy.
That sounds impressive.
But it answers only a narrow question:
Was the prediction correct?
It does not answer the question executives ultimately care about:
Did acting on those predictions improve business outcomes?
Those are different conversations.
Why the Number Can Mislead
In many churn scenarios, the majority of customers were never going to leave in the first place.
Imagine a customer base where 90 percent stay and 10 percent churn.
A model that predicts “everyone will stay” would already be 90 percent accurate.
The number looks strong.
The business gains nothing.
When most customers behave normally, accuracy can look healthy simply because the model is predicting the majority outcome. That does not mean it is identifying the customers where intervention makes a financial difference.
Churn management is about finding the minority of cases that matter, not confirming what was already likely.
Prediction Does Not Automatically Mean Performance
A churn model may correctly flag customers at risk.
But if the retention offer is weak,
if discounts are applied too broadly,
if incentive costs outweigh recovered revenue,
or if high-value customers are missed,
overall performance will not improve.
It is entirely possible for a model to be statistically sound and commercially irrelevant.
Accuracy reflects how closely predictions align with historical data.
ROI reflects whether the business behaves differently and benefits as a result.
Without a change in decisions, there is no change in results.
What Accuracy Leaves Out
1. What You Improved Against
Improvement only has meaning in comparison to something.
Did retention improve relative to how churn was managed before?
Without a clear baseline, it is impossible to separate model impact from normal business variation. A changing number is not the same as a better outcome.
2. What Errors Actually Cost
Accuracy treats every correct or incorrect prediction as equal.
Businesses do not.
Offering a discount to someone who would have stayed anyway reduces margin. Missing a high-value customer who leaves can be far more expensive.
Two models can report identical accuracy and produce very different financial results depending on where mistakes occur.
Accuracy does not capture that nuance.
3. Whether Behavior Changed
A model only creates value when it influences action.
If teams ignore the scores,
if marketing defaults to broad incentives,
or if processes remain largely the same,
the financial outcome will remain largely the same as well.
Accuracy shows alignment with past outcomes.
ROI shows whether decisions improved.
Those are not interchangeable.
Better Questions to Ask
Instead of asking, “What is the model’s accuracy?” leaders might ask:
Did outcomes improve compared to our previous approach?
What is the net financial impact after incentive costs?
Where are the most expensive errors occurring?
How quickly would we detect if performance began to slip?
What level of underperformance would trigger a review?
These questions move the focus from technical performance to decision quality.
That is where value is created.
The BRUKD View
Accuracy is not useless. It simply does not tell the whole story.
It provides insight into prediction quality. It does not tell you whether the organization is stronger because of it.
A system earns real authority only when a clear decision changes and measurable outcomes improve against a defined baseline. It earns trust when error costs are understood, when underperformance can be spotted early, and when the process can be adjusted or paused without disruption.
AI does not create value by being correct in isolation.
It creates value when accurate predictions lead to better decisions under real operating conditions.
Until that link is demonstrated, a high accuracy number is a signal. It is not evidence.
If you would like a structured way to assess whether your organization is ready to rely on AI responsibly, you can begin here:
https://brukdconsultancy.com/assessment.html
If you prefer to discuss your situation directly, you can reach us here:
https://brukdconsultancy.com/contact.html
The Real Test
If your dashboard reports accuracy above 90 percent but revenue has not moved, retention looks unchanged, or costs are rising quietly, it is worth pausing.
If teams are working around the system or applying incentives more broadly just to be safe, that is another sign.
The problem may not be the model itself.
More often, it is the gap between prediction and decision.
Closing that gap requires clarity about ownership, discipline in measurement, and a willingness to compare outcomes honestly.
That is where ROI is actually determined.