How I optimize cost in AI products

The last post said that while testing an AI product, I found one complete flow of use could cost several dollars.

Once I really started optimizing, I found that AI cost control isn’t just shortening prompts, cutting output, or switching to a cheaper model.

In the end I broke it into three questions:

• which money shouldn’t be spent at all;
• for money that must be spent, how to set boundaries;
• once money is saved, whether the results are still worth using.

Step one: first cut the AI calls that shouldn’t happen

The first question in cost optimization isn’t “which model is cheaper,” but:

does this step really need AI?

A lot of work doesn’t need a model.

For example:

• deduplication can use a database and ordinary code;
• format conversion can use a parser;
• comparing salary, location and hard requirements can use rules;
• known job sites can be fetched directly through APIs or fixed page structures;
• simple ranking can use scoring rules first;
• company and web information queried repeatedly can be cached;
• content in a fixed format can be generated from templates.

AI is better suited to the parts ordinary code can’t do reliably, such as:

• judging whether a piece of experience is genuinely relevant to a job requirement;
• combining several pieces of evidence into a conclusion;
• handling information that’s vague, conflicting or missing;
• writing that depends on the specific context.

The old approach might have been:

have AI search for jobs, read web pages, deduplicate, filter, judge and rank.

The new order is:

ordinary code first handles fetching, parsing, deduplication and hard-condition filtering, and only the small remainder that needs judgment goes to AI.

That’s not only cheaper, but more stable and easier to test.

The cheapest call isn’t one that uses a smaller model; it’s one that never needed to happen.

Step two: cut what each call reads

Once the number of calls is down, keep looking at what each call actually reads.

Some of my earlier flows sent the full job posting, the user’s career history, job preferences and past analysis results repeatedly to several model steps.

But not every task needs to know everything.

For example, a LinkedIn message under 200 characters may only need:

• the job title;
• company information;
• one or two of the most relevant experiences;
• the reason for wanting to connect.

It shouldn’t read all the context needed to generate a full résumé and job report.

How I handled it later:

• first put together a reusable evidence pack;
• pick the evidence each task really needs;
• send only the relevant excerpts, not the full history;
• cache job, company and web content;
• avoid different steps searching and reading the same information again.

This step reduces not only tokens, but also the chance of the model being distracted by irrelevant information.

Step three: put a “toll booth” in front of every paid call

Once calls and inputs are reduced, you still need one unified boundary for execution.

I think of it as a toll booth.

Before, every feature could call the model directly:

job scanning had its own calls, fit analysis had its own, material generation went through yet another set.

The problems with that:

• each feature may use different models and parameters;
• some places may forget to set limits;
• the SDK may retry automatically behind the scenes;
• new features may bypass existing cost tracking;
• it’s hard to control the total spend per user or for the whole product.

Later, every paid call had to go through the same entry point first.

Before a call happens, the toll booth checks:

• which feature this is;
• which model it uses;
• the maximum input and output allowed;
• whether search is allowed;
• how many searches at most;
• whether retries or continuations are allowed;
• the most this call could possibly cost;
• whether the user and the system still have budget.

If it doesn’t fit the rules, the call isn’t allowed.

That’s completely different from looking at the bill at month’s end.

A month-end report can only tell me the money is already spent. The toll booth answers:

can this money be spent right now?

Step four: reserve budget before the call, rather than alerting after it’s spent

To make the toll booth really work, I also introduced budget reservations.

The system won’t say, based on an optimistic estimate:

this will probably only cost $0.05, so let’s just run it.

Based on the preset maximum input, output, tool and retry counts, it calculates the worst-case cost and reserves that budget first.

After the call ends, it settles based on actual usage and releases the rest.

The design is a bit like a hotel deposit.

Why not just record the actual cost?

Because several requests may happen at once.

Say the account has $1 left, and three requests “expected to cost $0.40” arrive at the same time. Without reserving in advance, they may all think there’s enough budget, and together go over the limit.

Reservations also prevent immediately repeating a call after a timeout.

A request timing out doesn’t mean the model service definitely didn’t process it. If the system immediately released the budget and ran again, it might pay twice for the same task.

So when it can’t confirm whether the provider accepted the request, the reserved amount stays held until it can be reconciled.

Step five: set a different price for each capability

Different AI features shouldn’t share one vague budget.

A short message, a cover letter, a full résumé and a company research report differ completely in complexity and value to the user.

In my initial setup, different capabilities had different caps per run, for example:

• a short LinkedIn message: $0.03;
• a cover letter: $0.12;
• a tailored résumé: $0.25;
• a full job-fit analysis: $0.40.

On top of that:

• a total daily budget per user;
• a separate, low allowance for automated background tasks;
• a monthly budget for the whole product;
• outer limits on the model provider account.

These numbers aren’t industry standards, just initial boundaries for this product at current model prices.

What matters isn’t the numbers themselves, but:

every capability should have a price tag that matches its value and complexity.

Step six: beyond the toll booth, you need a complete ledger

The toll booth decides whether money can be spent; the ledger records where it went.

With only the total tokens for one logical task, it was hard to answer:

• how many requests were actually sent to the model;
• how much input and output each one had;
• whether a cache read or cache write happened;
• how many searches were used;
• whether the second request was a retry, a continuation or the user clicking again;
• whether costs were incurred before a failure;
• whether the bill after a timeout is still uncertain.

Later, every real provider attempt was recorded on its own.

Even when no product result was produced in the end, it still records:

• usage;
• tool calls;
• the stage where it failed;
• the reason for retrying;
• the actual cost;
• whether settlement has completed.

For the first time, this let me distinguish:

the product result failing and a paid call not happening are two completely different things.

The toll booth controls the future; the ledger explains the past.

Without either one, cost control isn’t complete.

Step seven: once costs drop, prove quality didn’t drop with them

There are actually lots of easy ways to save money:

• give the model less information;
• cut analysis steps;
• shorten the output;
• switch to a weaker model;
• stop searching outside sources.

But if results get worse because of it, that isn’t optimization; it’s just turning API cost into a loss of product quality.

So I also need to compare, on a fixed set of samples, at the same time:

• actual cost;
• response time;
• factual accuracy;
• completeness of the content;
• whether conclusions are stable;
• whether users still find it useful;
• how much ends up needing manual editing.

Three states have to be kept apart here:

• measured: cost, tokens, latency and failure records from real runs;
• engineering projections: savings estimated from fewer calls, smaller context and more caching;
• still unknown: quality changes that need real material and paid testing to confirm.

Until the comparison testing is done, you can’t report projected savings as results already achieved.

Finally

Looking back now, AI cost control comes down to four sentences:

Where you can do without AI, don’t use AI. When you must call it, give it only what it really needs. Every payment goes through the toll booth first, then into the ledger. Once money is saved, confirm that user value wasn’t saved away with it.

Switching to a cheaper model is only a very small step.

Real cost optimization is making every AI call have a reason, a boundary and a record, and be open to evaluation.