Where should the cost of AI stop?

In the last post, I did the first side of the enterprise AI accounts:

once AI raises productivity, what do companies actually get?

More and more evidence in the research can already show that some specific tasks really did get faster, output rose, and some even directly brought sales growth.

But between task improvement and enterprise value, you still have to go through:

task improvement → workflow realization → capacity use → value capture.

Knowing “what we get,” this post wants to work out the other side:

to get these results reliably, what exactly is paid in full?

This is why I wanted to do this round of research in the first place.

After I started building AI products myself, one very direct feeling was:

API prices are easy to look up.

But once you actually get a feature running, the bill doesn’t stop there.

• Before the model there may be search, data preparation and context.
• After the model there may be retries, evals, human checks, monitoring and failure handling.

Once it’s wired into a real product, there are also permissions, system integration, maintenance, and user adoption.

So what this post really wants to answer is:

where should counting AI’s costs stop?

01 | The API bill is real, but it isn’t the whole bill

First, let me clear one thing up.

Saying “AI’s costs aren’t only tokens” doesn’t mean tokens don’t matter.

Describing its multi-agent research system, Anthropic mentioned that the system uses about 15 times as many tokens as an ordinary chat.

That’s engineering experience from one specific Anthropic system, and can’t be read as “every agent costs 15 times more.”

But it at least shows one thing:

individual tokens getting cheaper and cheaper doesn’t mean finishing a task necessarily gets cheaper and cheaper.

A more complex system may call more models and more tools, read more context, and may also keep retrying because of failures or uncertainty.

Meanwhile, in another study of real production deployments, some teams felt:

compared with expensive expert labor, model running costs are almost negligible.

These two seemingly opposite situations don’t actually contradict each other.

If one piece of AI work handles a high-value expert task, tens of dollars in model fees may be small.

If you’re facing millions of low-value requests, even a few extra cents each can become a large operating cost.

So after the research, I think the more reliable conclusion isn’t:

model costs are small.

Nor:

what’s really expensive is integration.

But:

API price can’t represent the full cost. As for which cost is biggest, it depends on the specific workflow.

02 | Where do the costs of an AI system roughly show up?

In the full research, I divided costs quite finely.

Once written in product and business language, I think they can be understood as three stages.

Layer one: first get it running

This includes the parts that come to mind most easily:

Models, software and compute

APIs, licenses, compute, search, tool calls, storage.

Data and retrieval

Cleaning documents, deduplication, permission mapping, knowledge bases, indexing, and keeping all of it updated after launch.

System integration

Connecting CRMs, databases and internal systems; handling identity, permissions, APIs and read/write flows.

A demo only needs one prompt as input.

An AI that really enters a company’s workflow often first needs to know:

what data can I see?

what operations can I perform?

which data must I not touch?

where should the results be written back?

None of this work shows up in the token price.

Layer two: then make the results trustworthy

This is a layer I’ve cared about more and more since building products myself.

AI being able to generate doesn’t mean the results can go straight into the business.

You still need:

Evals and regression testing

What counts as acceptable?

Which cases absolutely must not be wrong?

When the model, prompt, tools or business rules change, do you need to retest?

Monitoring and operations

Can failures be detected?

Why did it fail?

Should it retry, degrade, or hand off to a person?

Security, privacy and governance

Who can access what data?

Which actions need approval?

When something goes wrong, how is it traced?

Human review and exception handling

Which situations go to a person?

Who judges?

How many minutes does checking take?

How much does it cost in the end to remedy one wrong result?

NIST’s Generative AI Profile also places pre-deployment testing, continuous monitoring, provenance verification and governance within the generative AI lifecycle.

It isn’t a “cost report,” but it at least shows:

this work isn’t optional decoration in a production system.

Layer three: keep it alive long term

Some costs are easy to miss in demos and early pilots.

Adoption and organizational change

Training users, changing processes, redividing work, getting the team to actually start using the system.

Maintenance

The model changes, the API changes, the internal knowledge base changes, and evals that used to pass may need rerunning.

Failure, migration and exit

What if the pilot doesn’t succeed?

What about switching vendors?

How do you count data migration, system rollback, and integration costs already sunk?

So “development finished” isn’t the point where costs end.

Some costs happen once.

Some keep happening.

And some only become obvious once you scale up.

03 | There are three accounts that are especially easy to get wrong

Listing every item in a spreadsheet isn’t enough.

Some costs are easy to get wrong by definition.

First, employee time has a cost, but it isn’t new cash spending

Suppose two in-house engineers spend a month completing an AI integration.

Their salaries were being paid anyway.

So that month doesn’t necessarily mean the company suddenly spent an extra two months of salary.

But it can’t be written as:

integration cost = 0

either, because they could have been doing something else.

The more accurate way is to take:

cash costs

and

internal capacity / opportunity cost

and keep them separate.

This is actually the same principle as last post’s “employees save two hours a week.”

Saved employee time can’t all automatically be counted as cash savings.

Spent employee time can’t be counted as free just because there’s no new invoice either.

Second, compare two full workflows

Sometimes talking about AI cost turns into:

AI has model fees, integration fees, evals, errors, human review…

while the manual process on the other side seems to have no costs at all.

That isn’t fair either.

Human work also has:

training, management, rework, errors, waiting, handoffs, overtime.

The really reasonable comparison should be:

at the same quality, service level and business scope, what does the AI workflow need in full, and what did the original workflow need?

Not:

all of AI’s flaws vs. a perfect, zero-error, zero-management-cost human process.

Third, don’t double-count the vendor’s costs

If a company buys APIs from OpenAI, Anthropic or other vendors:

much of the cost of model training and infrastructure is already reflected in the purchase price.

You can’t add the model company’s entire training costs back into your own project costs.

The reverse holds too.

Data preparation, human review and system maintenance borne by the company itself don’t automatically disappear just because they aren’t on the API invoice.

04 | A better unit of cost may not be one call

There’s one judgment from the research I think is especially worth keeping:

what companies really buy is completed work, not tokens.

A concrete example comes from Intercom’s Fin AI Agent.

As of the time of the research, some of its chat and email outcomes were billed at $0.99.

“Charging per outcome” already sounds closer to the business than tokens.

But reading on through the definitions, you find a very interesting problem:

the outcome a vendor defines and the business outcome a company really accepts aren’t necessarily exactly the same.

For example, some handoffs to a human can also count as a billable outcome.

“Resolved” is sometimes explicitly confirmed by the user, and sometimes presumed because the user didn’t ask for more help.

If the user later comes back to the same conversation for more help, the earlier resolution may be clawed back.

What I think is really worth noting here isn’t whether Intercom’s price is high or low.

It’s that three things are best kept separate:

• the vendor’s billing unit
• the company’s business acceptance unit
• the full cost paid to get that result

Take customer service:

• the API successfully returning an answer doesn’t mean the case is resolved.
• a resolved case doesn’t necessarily mean the customer’s problem is solved for good.

The real business outcome may also care about:

• did they get in touch again?
• was it handed to a person?
• how satisfied was the customer?
• in the end, how much did each truly acceptable case cost?

So for many cost-driven AI use cases, I now think the more practical unit is:

what is the full cost of each final result that passes business acceptance?

Not:

how much is one model call?

05 | When counting costs, also look at “when it happens” and “how it changes at scale”

Two more dimensions are easily mixed up.

One-time / ongoing costs

answers:

when does it happen?

For example:

the first system integration is a build investment;

APIs and monitoring keep happening.

The other pair is:

fixed / volume-dependent costs

answers:

how does it change as scale grows?

A yearly platform license can keep happening but basically doesn’t change with each request.

Model calls and human review, though, may rise along with business volume.

These two pairs of concepts are best not mixed.

Because they directly affect how unit economics change after scaling.

If an AI workflow is expensive to build up front but the marginal cost of each acceptable result afterwards is very low, it may get more and more worthwhile as volume rises.

Conversely, if every additional AI case comes with expensive human review, lots of retries and exception handling, scaling up may not naturally improve the economics.

So “AI is cheap” or “AI is expensive” is never quite enough

After doing this part of the research, I actually don’t really want to summarize:

where AI’s really big costs are.

Because the public evidence isn’t enough to support a single answer.

• In some workflows, models and compute are an important cost.
• In some, what really drains the team is data, integration and maintenance.
• In some high-risk scenarios, human judgment and exception handling may not be possible to cut.
• And for some expert work, even if each AI run isn’t cheap, it’s still very worthwhile relative to the original cost of professional labor.

So at the end of this post, I’d change the question from:

how much is one call?

to:

to reliably get one final result the business accepts, what does this workflow pay in total?

When actually evaluating, I’ll keep asking four things:

1. What is the baseline?

What is the full cost of the original process AI is compared against?

2. What do building and long-term running each need?

Don’t only count before launch, and don’t only look at the API invoice after launch.

3. How much model use, tooling, human work and failure handling does each acceptable result still need?

Include rejects, retries and handoffs to people.

4. What are adoption and real business volume?

A system can be very cheap in theory, but if nobody uses it, the upfront fixed investment still can’t be spread out.

At this point, both sides of the enterprise AI accounts finally have a rough outline.

The first post chased:

what do we really get?

The second chased:

to get these results, what is paid in full?

But even knowing both sides, you still can’t directly reach a general answer like “AI ROI = X%.”

Because different use cases differ far too much in tasks, demand, risk, business volume and acceptance criteria.

So in the third post I want to really put the two sides together:

facing a specific AI project, before going from pilot to scale, what should we at least work out?

About this round of research

This is the second post in the research series “The real economics of enterprise AI.”

The cost categories in this post mainly combine the FinOps Foundation’s AI workload cost framework, NIST’s lifecycle governance requirements for generative AI, and real production deployments and engineering cases.

One important limitation is:

existing public material can show very well that “there are costs beyond the API,” but isn’t enough to tell us what share each of these costs usually takes in enterprise AI projects.

So the post doesn’t use claims like “models are only X% of total cost” or “integration is always more expensive than inference.”

Likewise, the roughly 15x token use of Anthropic’s multi-agent system is only used to show that complex systems can amplify inference consumption, not to represent an industry average.

Intercom’s $0.99 outcome pricing is also only used to show that “billing units” and “business value units” can differ, not to suggest customers necessarily get a positive ROI from using it.