Most of us have AI the wrong way round. We ask it to write the strategy and draw the pictures, then spend our own week on the busy work. I put it to a masterclass audience this way: let's use AI to do our dishes so we can do the art.
AI agents are how that happens. An agent can work through a pile of invoices, a backlog of customer emails or a list of new enquiries without ever getting bored. Doing the dishes at that scale raises a question most people never ask until the bill arrives. What does it cost when an agent makes the same kind of decision thousands of times over?
What an agent actually does all day
Most useful agent work is repetition.
Take a business that gets hundreds of supplier invoices a month. An agent reads an invoice. It decides which account code the invoice belongs to. It checks the amount against your approval rules. It flags anything unusual. Then it does the same for the next one.
Every one of those decisions is made against the same reference material: your chart of accounts, your coding rules, your list of approved suppliers. Call that the rulebook. The invoice changes every time. The rulebook barely changes at all.
That is exactly the kind of work where the economics of AI can quietly go wrong.
The hidden cost
AI models charge by the token, which is roughly a word or a piece of a word. They also have no memory between calls.
So without caching, every single decision has to send the whole rulebook again, and you pay full price for the model to read it again. In practice the rulebook is often far bigger than the piece of work being judged. Most of what you pay for is rereading the same instructions.
What prompt caching changes
Prompt caching lets the model keep the fixed part of the prompt ready to reuse.
The rulebook is written to the cache once. Every later call that starts with the same rulebook reads it from the cache at a tenth of the normal input price or less, depending on the model, and pays full price only for the new invoice.
The agent makes exactly the same decision with exactly the same instructions. Only the cost of rereading them changes.
Where caching pays most
Caching cuts the cost of reading. The cost of writing stays the same.
So it pays most where a large, fixed reference is reread for every decision and the answer is short. Sorting invoices against a chart of accounts fits that pattern. So does routing customer emails to the right team, or checking contracts against a standard set of clauses.
It pays least where the reference is small and you ask the model to write a lot back, like drafting a long personalised reply to each customer.
Three design choices
The saving comes from how the job is built.
- Put the fixed rulebook first. The cache only matches from the start of the prompt, so everything that never changes goes at the top and the invoice goes last.
- Group similar decisions together. Run all the invoices of one type in a row, so they all reuse the same part of the rulebook.
- Warm the cache before a big batch. Send one small request first to write the rulebook to the cache, so the thousands of requests that follow read from it instead of each writing their own.
Tidying the rulebook under clear headings also makes it shorter, which makes every call cheaper.
Three questions to ask
You do not need to know how caching works under the bonnet. You do need to know it exists, because it changes the business case for putting agents on repetitive work. When a vendor or your own team proposes an AI agent, ask three questions:
- What is the rulebook, and is it cached?
- How much of each call is the rulebook, and how much is the actual work?
- Are similar decisions grouped so they reuse the same reference?
If nobody can answer, you may be paying full price to reread the same instructions over and over. Get the setup right and the dishes get done for a fraction of the cost. Your people get their week back for the art: the customer conversation, the better plan, the work only they can do.
Common questions
What is prompt caching? It lets an AI model store the fixed part of a prompt, such as a rulebook or reference list, and reuse it on later calls. Those calls pay a fraction of the normal price to read the stored part, and full price only for what is new.
How much does it save? It depends on how much of each call is fixed. The bigger the rulebook compared with the work, the bigger the saving.
Does it change the AI's answers? No. The model receives exactly the same instructions and makes the same decision. Caching only changes what it costs to send those instructions again.
Try this. Pick one repetitive task you would hand to an agent and write down its rulebook on one page. If that page is much longer than a single piece of the work, caching should be part of the plan.