Ever wondered why a 500 token prompt returned two paragraphs of text and you still got charged for something closer to a 128K context? I did. Looking under the hood of how frontier models actually run shows a structural constraint that is easy to miss from the chat UI: these systems do not hold working memory the way humans do.

When you chat with an LLM, the model does not keep a private, durable thought state between tokens. Its active context is reconstructed through attention over a Key Value cache, commonly called the KV cache. In practical terms, the model must re-engage the relevant history, system instructions, tool output, and loaded files as it produces each next token. If that sounds inefficient, that is because it is. We often talk as if models are thinking the way we do. They are producing helpful output. That is not the same as having a human style working memory.

That distinction matters more now than it did in early chatbot days. Coding agents, long context windows, and hidden reasoning traces all expand the amount of state that has to be carried, reloaded, or recomputed. The bill follows the architecture.

Prefill and decode: two different bottlenecks

Inference has two dominant phases.

The first is prefill. The model reads the prompt, builds attention state, and fills the KV cache. The compute cost of attention grows steeply with sequence length. A long system prompt, a large repository slice, or a long chat history makes this phase expensive before the first output token appears.

The second is decode. Tokens are written one by one. For each new token, the runtime has to keep model weights and the growing KV cache available in GPU memory. For a roughly 70 billion parameter model, weight storage alone can sit in the order of 140 GB depending on precision. Add a long KV cache on top of that and memory becomes the scarce resource.

That is why expensive accelerators often look underutilised during generation. They are not always waiting on more FLOPs. They are waiting on memory bandwidth while weights and cache state move through the system. This also helps explain recurring headlines about high end memory scarcity and the energy cost of serving large models at scale. Serving is not only training. Decode traffic is continuous and memory heavy.

The KV cache grows with the conversation

KV cache size scales with context length and with how many concurrent sessions the provider keeps warm. A simple way to see the pressure: for a large model, holding long contexts for even a small number of concurrent users can consume hundreds of gigabytes of GPU memory. Illustratively, keeping around 128K tokens of active context for roughly ten concurrent sessions on a 70B class model can push into the hundreds of gigabytes of memory for cache alone, before counting weights, overhead, and fragmentation. That is multiple top tier cards just to hold active session state.

Hidden reasoning makes this worse. When a model is configured to think before it answers, it may emit thousands of intermediate tokens. Those tokens enter the active sequence. Each additional token expands the work on the next step. The loop is not free. It is paid for in latency, memory, and metered tokens.

Why coding agents amplify the cost

This becomes a financial and operational problem once you leave single turn chat and enter agent harnesses.

A coding agent does not only write code. The harness reads repository files, runs tests, calls tools, and feeds results back into the model. Every action can reintroduce file contents, diffs, stack traces, and prior tool output into the prompt path. Shared prefixes may be cached for a while. New observations usually are not free.

On agent software construction benchmarks such as ProjDevBench style evaluations, published figures in this space often land around 100 plus turns and millions of tokens per task. One commonly cited shape of result is on the order of 138 turns and roughly 4.81 million tokens per task. Run hundreds of those tasks a day and the organisation is repeatedly paying to re ingest overlapping context across loops.

There is also a trust question that operators should ask more often: how exactly are tokens counted across cached prefixes, tool payloads, reasoning traces, and retries? Users see a short final answer. Billing sees the full execution trace. Those are not the same object.

Warm state is scarce, so idle sessions get cold

Providers try to keep calculated KV state in GPU memory so they do not repeat full prefill for every message. That memory is expensive and contended. Cloud fleets serve many users. Warm session state often has a short lifetime.

If a developer pauses for a few minutes to think, their saved state may be evicted to make room for someone else. The next message can force a fresh prefill over a large history. From the product surface this looks like chat continuity. From the serving stack it can look like starting over.

For a long time, some frontier products absorbed much of that cost inside flat subscriptions. That era is ending. Usage based credits, rate limits, and explicit token metering are becoming the default because the hardware economics finally show up in the invoice.

What to do until the architecture changes

Until systems move further away from repeatedly expanding and replaying large active contexts, large generative agents will remain slower and more expensive than the chat UI suggests.

For real engineering work, that has practical consequences:

  • Keep context deliberate. Load only the files and traces the current step needs.
  • Prefer short loops with clear state outside the model over dumping the whole repo into every turn.
  • Treat harness design as cost control. Tool output shaping and memory policy matter as much as model choice.
  • Use small, local autocomplete tools when the job is narrow completion rather than multi step agency.

I wrote a related set of token management practices earlier on LLM ThreatIntel: Claude Code token economics and context window discipline.

There is also a useful distinction between intelligence and intelligent looking output. A clear Computerphile explanation of that gap is worth watching: Computerphile on AI and understanding.

Takeaway

The model is not secretly holding your project in a durable internal notebook. It is operating over an attention based cache that grows with every useful bit of context you give it, every tool result the harness returns, and every hidden reasoning token the system chooses to keep.

That is why a short looking prompt can still cost a long context. The architecture is doing exactly what it was built to do. The invoice is just catching up.