Putting gains and costs together: how to do the math on an AI project

The previous two posts worked out the two sides of the enterprise AI accounts.

The first asked:

once AI raises productivity, what do companies actually get?

We followed it all the way from task improvement through workflow and capacity to final value capture.

The second asked:

where should AI’s costs be counted?

Models and APIs of course count, but the full workflow also has data, integration, evals, human review, monitoring, failure handling and long-term maintenance.

At this point, it looks like the two numbers can finally go into a formula:

(gains − costs) ÷ costs = ROI

But having actually taken the research this far, I feel instead:

what needs the most care is precisely giving “enterprise AI” a single answer.

01 | Why is it so hard to get an “average enterprise AI ROI”?

Because “enterprise AI” isn’t itself a concrete enough economic unit.

• A customer service assistant isn’t the same as a coding assistant.
• A pre-sales chatbot for consumers isn’t the same as a knowledge assistant for the finance team.
• A task that happens a million times a day with easy-to-check results, and one that happens a few times a month where a single failure is serious, can’t be compared with the same set of numbers either.
• Even within coding alone, credible studies can reach seemingly opposite results.

A randomized experiment covering three companies and 4,867 developers found that with an AI coding assistant, the number of tasks completed rose by about 26% on average.

In another real-task experiment, METR found that a group of senior developers familiar with mature codebases took about 19% longer using AI, under the tool conditions of the time.

Neither study is necessarily right or wrong.

Different people. Different tasks. Different codebases. Different models and tools. Different measurement methods too.

So one very important conclusion at the end of the research is:

AI’s economic effects depend heavily on the task, baseline, users and workflow.

We can study a specific use case.

But it’s hard to upgrade it directly into:

“how much ROI can AI bring on average?”

The more realistic question may be:

under what conditions is a specific AI workflow worth continuing to scale up?

02 | Before scaling, look at five things first

The full research didn’t turn up a universal ROI formula validated across industries.

But some conditions came up again and again across different evidence.

I’d sort them into five questions.

First, is there real, ongoing demand?

AI easily increases supply.

Generating more copy, writing code faster, answering more questions at once.

But without real demand, more output has no economic value in itself.

In the retail experiment mentioned in the first post, one important reason the pre-sales chatbot brought a clear sales lift was that a large amount of unmet consumer inquiry really did exist.

Once AI lowered the cost of providing the service, that real demand got picked up.

Conversely, if the team has no more tasks, and customers don’t need more output:

even if the productivity looks great, it isn’t necessarily worth expanding the system for “more.”

So the first question is very plain:

if AI helps me do 30% more, does anyone actually need that 30%?

Second, what counts as an “acceptable result,” and can you tell?

An AI demo easily looks good.

After scaling, what’s really troublesome is:

how do you know it got it right every time?

A BCG experiment once produced a very cautionary result.

On tasks suited to the models’ capabilities at the time, AI improved performance.

But on a complex management task beyond the capability frontier, the AI group was on average about 19 percentage points less likely to give the right recommendation.

That isn’t saying “AI isn’t suited to consulting.”

It shows:

fluent, complete and plausible-sounding doesn’t mean the business result is right.

If a workflow can’t even define “what counts as success,” then after scaling it’s very hard to really measure:

• what is the success rate?
• how much needs human checking?
• how much does each acceptable result actually cost?

That’s also why I now think:

acceptance criteria aren’t QA work after an AI project launches, but part of the business case.

Third, can failures be detected and recovered from?

Every system fails.

The more important economic question isn’t:

can it be 100% automatic?

But:

when failure happens, can we tell; and once we know, how much does it cost to deal with?

• A wrong product description may just need fixing.
• A wrongly sent marketing email may affect the brand.
• A wrong financial judgment may cost something completely different.

So with the same 95% accuracy:

the economic meaning of the remaining 5% can differ a lot.

• Some tasks can let AI go first and easily hand off to a person on failure.
• For some tasks, errors are hard to detect and very expensive once they happen.

That’s also why the average success rate on its own is often not enough.

Failure costs and recovery paths also have to enter the economics.

Fourth, can the whole workflow actually use AI’s improvement?

The first post already covered this:

a task getting faster doesn’t mean business results automatically improve.

This matters even more at the scaling stage.

If AI doubles code generation speed but review and testing don’t change at all:

adding more coding-agent capacity won’t necessarily keep increasing product delivery.

If the customer service assistant has already made the waiting queue disappear:

adding another 20% of handling capacity may also have sharply lower marginal value.

So scaling isn’t simply:

it already works → buy more.

You should keep asking:

where is the bottleneck now?

Once AI solves the original bottleneck, it very likely just exposes a new one.

Fifth, once full costs are counted, do the unit economics still hold?

This is where it connects to the last post.

Suppose an AI system:

• really has demand;
• has output that can be verified;
• has errors that can be recovered from;
• has a workflow that can use the new capacity.

In the end you still come back to a very ordinary business question:

is producing one truly acceptable result better than the original way?

“Better” here doesn’t only mean cheaper.

It can be:

• serving more customers at the same cost;
• spending a bit more but clearly increasing revenue;
• about the same cost but fewer errors;
• higher cost, but giving the company a very valuable new capability for the first time.

If you must look at the hardest cash part first, the research took this approach:

net cash benefit = the cash benefits AI really brings − the cash costs AI really adds

One thing I really like here is:

don’t force freed-up employee time into it.

If it really turns into avoided hiring, less overtime or more revenue, then it enters cash benefits.

If it only frees up capacity, record it separately.

That way you don’t turn all the “theoretical salary time” into cash in advance just to make the ROI look good.

03 | Which scenarios have relatively clearer evidence right now?

The research didn’t end with an “AI ROI industry ranking.”

Because we don’t have enough consistent data.

But the maturity of the evidence really does differ across use cases.

Customer service assistance with high volume, stable demand and easily measured results

Here there’s already fairly strong evidence of workflow productivity.

There are also financial results disclosed by real companies, like Klarna:

in its FY2025 20-F, the company said its AI assistant handled about 80% of customer service chats and brought about $59 million in cost savings.

This number is still the company’s own attribution and estimate, not an independent causal audit.

And the public material isn’t enough to recalculate the whole project’s net ROI.

But compared with “how much we expect to save next year,” it’s already closer to real operations.

This kind of scenario is relatively easy to verify because:

• demand persists;
• volume is large;
• results are fairly clear;
• handoffs to people can be designed clearly;
• in the end you can observe queues, volume handled and actual spending.

Retail features that connect directly to real buying behavior

As mentioned in the first post, on one large cross-border platform the pre-sales chatbot lifted sales by about 16.3%, AI search optimization by about 2.9%, while marketing pushes showed no significant sales growth.

I think this set of results is very valuable.

Because it shows:

small, repeatable improvements connected directly to business metrics may make it easier to build a business case than a very dazzling demo.

2.9% doesn’t sound sensational.

But if real volume is large enough, it can be a very considerable gain.

Of course, scale can only amplify unit economics.

If every order was losing money to begin with, a bigger scale won’t magically turn it into a good business.

Software assistance

Here there’s already good task-level evidence.

But because study results differ so much across people and codebases:

the more reasonable approach is still:

test it in your own real development process.

And don’t only measure:

“how much faster is code generated?”

Also look at:

review, testing, maintenance, release and final delivery.

Scenarios where the evidence is still weaker

For example open-ended, cross-system, long-chain autonomous agents.

There are already surveys of real deployments, but not yet enough evidence to support a general positive net ROI for them.

That isn’t saying they have no value.

Just that:

you can’t borrow someone else’s capability showcase directly as your own business case just because agents can complete more and more complex demos.

04 | A higher degree of automation doesn’t mean better economics

There’s another finding from this round of research I really like.

A 2026 study of production agents found, across the 86 production or pilot systems it screened:

68% bring in a person after no more than 10 steps of autonomous execution, and 47% after fewer than 5.

This sample has clear limits.

It comes from production and pilot teams willing to take part in the research, doesn’t represent the whole industry, and can’t be used to calculate an agent success rate.

But it at least serves as a reminder:

real-world agents haven’t all been moving along the progress bar of:

human → human-in-the-loop → fully autonomous

as they move forward.

Many teams choose exactly this:

• AI does a stretch;
• a person judges at key points;
• failures can be taken over;
• then it continues.

From an economic point of view, this may be an entirely reasonable choice.

Because:

reducing human intervention lowers some labor costs, but may also raise error and recovery costs.

So “a higher degree of automation” doesn’t inherently mean better ROI.

The better question is still:

with AI and people put together, is the whole system more effective?

05 | This also made me rethink evals

When talking about evals before, what came to mind first was easily:

accuracy, hallucination, success rate, benchmarks.

These of course matter.

But putting these three posts together, I now think evals for enterprise AI should go one step further.

They aren’t only asking:

did AI get it right?

They should also help answer:

can the whole system reliably complete a piece of genuinely valuable work?

That naturally includes:

• full task success;
• human intervention rate;
• failure and recovery;
• completion time;
• the full cost of each acceptable result;
• and the business outcome that result corresponds to in the end.

Evals of course can’t work out the ROI for the company on their own.

But without this layer at all:

the product easily proves:

model capabilities keep getting better.

Yet never knows:

whether this system, as a business, has actually been getting better.

Having finished three posts of research, what I’m left with in the end is a set of very ordinary questions

When I started this round of research, I really wanted to know:

how should companies actually calculate AI’s ROI?

By the end, I hadn’t gotten a neat industry number.

Nor had I found an answer like:

“AI returns X times on average”

that I’d be willing to believe.

Instead it became three more specific questions.

First, what do companies actually get?

Not stopping at productivity, but following it all the way to whether value is captured.

Second, to get these results, what is paid in full?

Not only looking at the API, but comparing two full workflows.

Third, can this keep holding up in your own real business?

• Is there demand?
• Can the results be verified?
• Can failures be recovered from?
• Can the workflow use the new capacity?
• Once scaled up, do the unit economics still hold?

If I had to fold the three posts into one sentence at the end, I’d now write:

AI being able to do something and a company being right to hand it to AI for the long term are two different questions.

The former is more about capability.

The latter needs:

real demand, business results, reliability, full costs and scale

all brought in together.

So next time I see an AI use case, I might not first ask:

“how advanced is it?”

I’d first ask three plainer questions:

did it really make the result better? To get this result, what did we pay in full? Can these gains and costs keep holding up at real business volume?

If all three can be answered clearly, I think it really starts getting close to AI accounts that can be believed.

About this round of research

This is the third and final post in the research series “The real economics of enterprise AI.”

This round of research is a structured review of public evidence, not a statistical meta-analysis.

One of the biggest limits of the existing evidence is that public research measures productivity, workflow output and some business metrics far more than it discloses full project costs and long-term net returns.

So this series hasn’t tried to give:

• an average ROI for enterprise AI;
• an AI ROI ranking by industry;
• a single pilot failure rate;
• a rule like “the more automation, the more profit”;
• or a scaling formula that applies to every AI project.

The “five questions before scaling” proposed here are an analytical framework drawn from existing experiments, production deployments and cost research, not a causal law validated across industries.

The judgment that’s relatively clearer for now is:

use cases with real demand and enough volume, results that can be verified, failures that can be handled, and workflows that can use the improvement find it easier to build a credible economic model.

But in the end it still has to be verified in the specific business.