A few cents a run: that was the demo. Three weeks later finance forwarded the invoice with one line highlighted, the same agent in the same workflow, now carrying an extra digit, and a note asking what we had changed. The usual suspects came up innocent. Traffic was flat, the model price had not moved. What changed was that the agent was now doing real work, and real work means loops.
The sales pitch skips over what happens to the cost once the agent starts looping. A single prompt has a cost you can hold in your head: tokens in, tokens out, done. An agent has a cost that compounds. It reads context, decides, calls a tool, reads the result, decides again, and every one of those turns drags the entire conversation so far back through the model. The transcript is the unit of work, and it grows every step. What you bought was not a more expensive prompt but a different cost shape.
The model that actually predicts the bill
Forget per-token pricing for a minute, because it tells you almost nothing about an agent. The unit that predicts the bill is cost per task, and a task is a loop.
# Cost of one agent task, written the way I reason about it on a whiteboard.
# Not the way a billing dashboard slices it (per-token), which hides the loop.
PRICE_IN = 3.00 / 1_000_000 # $ per input token, whatever your model charges
PRICE_OUT = 15.00 / 1_000_000 # $ per output token, output is the expensive side
def cost_per_task(steps, base_context, growth_per_step, out_per_step):
total = 0.0
context = base_context # the system prompt + tools + task. paid EVERY step.
for _ in range(steps):
# the whole transcript so far is the input on every single turn.
# this is the line people forget. step 8 pays for steps 1 through 7 again.
total += context * PRICE_IN
total += out_per_step * PRICE_OUT
context += growth_per_step + out_per_step # tool results + the model's own reply
return total
# a "small" agent: 6 steps, a fat 12k-token system prompt and tool schema,
# each tool dump adds ~2k tokens, model writes ~400 tokens a turn.
print(cost_per_task(steps=6, base_context=12_000,
growth_per_step=2_000, out_per_step=400))Run that and the number comes out bigger than your intuition, not because any single call is expensive but because of the inner accumulation. The base context, the system prompt and the tool definitions and the task, is paid on every step. A twelve-thousand-token tool schema is cheap once and brutal six times. The transcript growth makes it worse: each step appends the tool’s output and the model’s reasoning, so the last step pays for everything that came before it. The cost of a task is closer to quadratic in steps than linear, and that is the whole surprise in one sentence.
The run rate is the easy math: cost per task times tasks per day, and that is honest arithmetic. What is dishonest lives upstream, in two multipliers people leave out of the estimate entirely.
The two multipliers that wreck the estimate
The first multiplier is retries, and it comes from failure. Agents fail mid-loop: a tool times out, the model returns malformed arguments, a guardrail rejects an action, so the sensible design retries. A retry is not a cheap re-poke of the failed step, though. Depending on how you built it, a retry can replay a chunk of the transcript or restart the loop, and now your six-step task is a nine-step task that pays the accumulation tax again. I have seen a “ten percent failure rate” turn into a forty percent cost premium because the retries were the expensive steps, the late ones, with the fattest context.
The second is re-reading, and it hides better. An agent that loses the thread re-fetches a document it already saw, re-lists a directory, re-asks for the same data because the relevant turn scrolled out of its working attention or got summarized away. Every re-read is fresh tokens in and a longer transcript out, so you are paying the model to recover state it already had. (We once found an agent re-reading the same config file eleven times in a single task because nothing told it that it already knew the answer.)
The real model is uglier than tasks times cost-per-task. Write it as tasks, times cost-per-task, times a retry multiplier, times a re-read multiplier. Both multipliers are silent in the demo and load-bearing in production.
The levers, and the one that gets people fired
Now for what you can do about it. You can bring this down a lot, and most of the room is in architecture, not in waiting for prices to drop.
Caching the stable prefix is the single highest-return move and the easiest to miss. Your system prompt and tool schema are identical on every step of every task. If your provider supports prompt caching, that fat unchanging prefix gets billed at a fraction on the repeated reads, and given how many times the loop re-reads it, that is most of your input cost handed back to you. Structure the prompt so the stable prefix really is stable and the variable content sits at the end. People put a timestamp at the top of the system prompt and wreck their own cache hit rate.
Cheaper models can do the sub-steps. The orchestrating model does not need to be the model that summarizes a tool result or classifies which tool to call next. Route the cheap, narrow, high-frequency sub-steps to a small fast model and keep the expensive frontier model for the reasoning that actually needs it. This is the same instinct as putting your hot path on cheaper hardware, and it is the lever with the best return per hour of work.
# Pick the model by the shape of the step, not by habit.
# The reasoning step earns the expensive model. The plumbing does not.
def model_for(step_kind: str) -> str:
if step_kind in ("plan", "decide_next_tool", "final_answer"):
return "frontier" # judgment lives here. pay for it.
if step_kind in ("summarize_tool_output", "classify", "extract_field"):
return "small" # a narrow task a cheap model does fine. and fast.
return "frontier" # when unsure, don't get clever with correctness
def trim_context(transcript, keep_last=8):
# the agent does not need a verbatim replay of every tool dump from 30 steps ago.
# keep recent turns whole, compress the old ones to their decisions and results.
# this is where the quadratic gets bent back toward linear.
head = summarize(transcript[:-keep_last]) if len(transcript) > keep_last else []
return head + transcript[-keep_last:]Context trimming fights the accumulation head-on. The agent rarely needs the verbatim text of a tool result from twenty steps ago; what it needs is the decision it made and the fact it learned. Summarize the old turns down to their conclusions, keep the recent turns intact, and the per-step context stops climbing toward the ceiling. Done carelessly, this is also how you lobotomize the agent: trim the wrong thing and it forgets why it is doing the task, starts over, and re-reads everything, which costs more than you saved. Trim by relevance, not by age.
Step limits are the seatbelt. A hard cap on steps per task will not make the agent smarter; it makes the failure bounded. Without a cap, a confused agent loops until something external stops it, and the steps it burns while confused are the most expensive ones it will ever run. Set the cap, and when a task hits it, fail loudly to a human instead of silently to the invoice.
Batching is the lever everyone forgets because it lives at the fleet level, not the task level. If a hundred tasks each call the same classification sub-step, you do not have to run a hundred separate calls. Many of those independent sub-steps batch, and batched throughput is cheaper than the same work done one request at a time. This only shows up when you are running agents at volume, which is exactly when the bill is large enough to care.
Where this actually lands
Years ago, owning a large infrastructure P&L, I argued that FinOps is just capacity planning with a better hat. The cost was never a dashboard problem but an engineering-culture one: the architecture decided the bill, and the dashboard only reported the verdict after the fact. Agents are the same idea with the dial turned up. You cannot negotiate the model price much. The loop, the context growth, the retries, the re-reads, the model routing, those are architecture, and architecture is yours.
When the bill surprises someone, the honest answer is rarely “the model got expensive.” It is that the agent was allowed to loop without a cap, re-read without a memory, and replay a twelve-thousand-token prefix it could have cached. Each of those is fixable, and none of them require a cheaper model.
An agent survives a budget review not by running on the cheapest tokens but because its creators can draw the cost model on a whiteboard and point at the exact step where the money goes. That drawing is what I now ask for before I approve a new agent, ahead of the demo and ahead of the roadmap. My bet for the next year is that cost per task becomes a first-class design constraint, reviewed right next to latency and correctness, and the teams that learn to draw the loop before they ship it will out-ship the ones still waiting for token prices to fall.