Two terms first, for anyone new to this. An AI model answers from a context window: everything it can see on this turn, including the instructions, the conversation so far, and any data it fetched. A tool call is the model asking for data, a database query for example, and getting the result back into that window.
We were building an agent for a climate data publisher. Verified datasets in a relational database, a tool server in front of them, and an agent that turned a plain English question into tool calls. The tools returned JSON. The JSON was accurate, and very expensive in tokens.
After a certain conversation length the agent's behaviour declined. Instructions were followed less closely, tool calls got sloppier, answers got vaguer. No change to the system prompt stabilised it. This is context rot, and the fix that worked was the least clever one available.
Replace each old tool result with a placeholder, and keep a short note of what mattered.
Profile the window before you change anything
The first step was to see what the model actually received each turn. The window held:
- The system prompt
- Tool definitions
- The user message
- Earlier model responses
- Tool call requests
- Tool call responses
Tool responses were swallowing most of the budget. JSON is token inefficient by construction, all braces, quotes and whitespace, and every old result was carried forward into every new turn. Nothing was thrown away, so the window grew with every call.
Profile first. The fix is obvious once you can see the shape of the waste, and different from the fix you would have guessed.
The fix: a placeholder in the next request
After a tool call completes, we keep the raw JSON for the interface, where the user can still see it. In the next model request we replace it with a short, static line:
This tool call was successful. Re run the tool if you need the data again.
The agent can always get the data back by calling the tool. It just does not carry the payload around in every turn that follows.
What it saved
The first turn, before and after, on one real question:
| Component | Before (tokens) | After (tokens) |
|---|---|---|
| System prompt | 5,000 | 5,000 |
| Tool descriptions | 2,500 | 2,500 |
| Tool request | 50 | 50 |
| Tool response | 5,000 | 100 |
| Model response | 200 | 200 |
| Total | 12,750 | 7,850 |
That is 4,900 tokens saved on turn one, about 38%. The tool response itself shrinks by 98%. With ten tool calls in a conversation, the tool responses alone drop from about 50,000 tokens to about 1,000.
We also measured a real agent loop, where one user message led to many internal steps. Runs were different lengths, so we compared the first 25 steps of each:
| Metric | Before | After | Change |
|---|---|---|---|
| Total input tokens | 507,889 | 316,778 | 191,111 fewer, 37.6% |
| Average tokens per step | 20,315 | 12,671 | 7,644 fewer |
| Median tokens per step | 14,677 | 11,326 | 3,351 fewer |
The shape of the two curves is the point. Without compaction, input tokens climb almost linearly with each step, because nothing is ever removed. With it, the curve stays lower and dips at each compaction point.
The downside, and the notes pattern that covers it
Compaction has one real cost. If the raw result is gone, how does the agent remember what it learned from it?
We use a small notekeeping pattern. Right after a tool returns, the agent writes down what mattered in a structured block. Later turns read the notes instead of the raw data.
For a time series, we ask for five to seven representative points, the source and the last updated date. Then a one line trend, and anything that will matter later:
<notes>
Germany CO2 emissions (time series)
Source: https://data.example.org/emissions/germany
Last updated: 2024-11-12
Key points:
- 2018: 810 Mt
- 2019: 780 Mt
- 2020: 700 Mt
- 2021: 720 Mt
- 2022: 690 Mt
- 2023: 645 Mt
Trend: steady decline 2018 to 2023, a dip in 2020, partial rebound in 2021.
The energy sector drives most of the decline.
</notes>
The note is a few hundred tokens. The payload it replaces was five thousand. The agent keeps its reasoning and loses the bulk.
What this does not solve
Compaction removes stale tool output. It does nothing about a system prompt that is itself most of the window, or a single tool result too large for one turn. Those need a shorter prompt, smaller tool responses, or a sub agent that owns the heavy data and returns a summary.
You do not need any of this on day one. Modern models handle large windows well until a system crosses a complexity threshold. The signal that you have crossed it is behaviour degrading with conversation length while the prompt stays the same.
What to do this week
- Log the token count per component for one real conversation. Prompt, tools, user, responses, tool results.
- Find the biggest block. In a tool heavy agent it is almost always old tool output.
- Replace old results with a placeholder in the next request. Keep the raw data for the interface.
- Add a notes block the agent writes right after each tool returns.
- Measure the same conversation again and keep the before and after. That number is how you explain the change to anyone who did not do the work.
The bottom line
Context rot is not a model problem. It is a window full of things the model no longer needs. Replace old tool results with a placeholder, keep a short note of what mattered, and measure it. On our data agent that was 38% fewer tokens on turn one and 37.6% fewer across a 25 step run. Answer quality held on the checks that gate every release.
