I have noticed something strange with long AI conversations.
At the beginning, everything feels sharp. The model remembers what I am doing, follows the constraints I gave it, and seems to understand the shape of the work.
Then the conversation gets longer. A few hours in, I find myself repeating something I already said. It brings back an old decision I had moved on from. A response is technically fine but somehow less connected to what I am actually trying to do.
My instinct used to be to give it more information. More screenshots. More files. More explanation. More of the previous conversation.
It turns out that can make the problem worse.
What is context rot?
Context rot is the measured decline in an AI model’s output quality as its input grows longer, even when the task itself stays exactly as hard. In a 2025 technical report, the team behind the Chroma vector database tested 18 models by holding the task constant and varying only the amount of context around it. Performance became less reliable as the input grew, including on relatively simple retrieval tasks. Their phrasing: models do not use their context uniformly.
That design is what makes the result useful. Most long-context benchmarks make the questions harder as the input gets longer, so you cannot tell which of the two caused the drop. Holding the question still isolates length as the cause.
It is also easy to confuse context rot with a model simply running out of context. They are different problems. Running out is a hard failure at the token limit. Rot is a gradual slide that starts well below it.
A context window tells you how much information a model can accept. It does not tell you how well the model will use every part of that information. A model with a very large window can technically accept a huge document or conversation while becoming less precise as the material grows.
That changed how I think about context windows. I used to read “200K tokens” as something close to capacity. Now I think of it more like available space. Having the space does not mean filling it is a good idea.
Models do not read every part of a prompt equally
This showed up before the term context rot became popular.
The Stanford-led paper Lost in the Middle tested where information sits inside a model’s context. Models were much better at using information near the beginning or the end, and less reliable when the relevant information sat somewhere in the middle. In one of their tests, GPT-3.5-Turbo answered correctly about 75.8 percent of the time when the relevant document appeared first. That dropped to 53.8 percent when it appeared around the middle.
The number worth sitting with is a third one. Asked the same questions with no documents at all, that model scored 56.1 percent. Burying the answer in the middle of the context was worse than never supplying it.
The information was still there. The model could see it. Its position changed how well the model used it.
A team from Adobe Research and LMU Munich found a related problem with NoLiMa. Instead of asking questions that shared obvious words with the answer, the benchmark made models connect information more indirectly, which is closer to how most real questions work. As the context grew, performance fell sharply. At 32,000 tokens, 11 of the 13 models tested had dropped below half of their strong short-context baseline.
NVIDIA’s RULER benchmark came at it from another angle, testing 17 models that all advertised context windows of at least 32,000 tokens. Only half of them held satisfactory performance at that length, and the shortfall widened as the tasks got harder.
Noise is surprisingly expensive
The Chroma research found something that sounds exactly like normal work. A single piece of material that looks relevant but does not actually contain the answer can reduce accuracy. Add more of those near-misses and performance gets worse.
That is basically what a long working conversation becomes. There is the version you kept and the version you rejected. An old requirement and a new one. A file that mattered twenty minutes ago. A tangent that went nowhere. A correction to something the model said earlier.
Eventually the conversation holds a fairly accurate history of the work. That history is not necessarily the best context for the next task.
That was the part I had been missing. I was treating history and context as though they were the same thing.
They are not. History is everything that happened. Useful context is the material the model needs for the thing I am asking it to do now.
In one of the conversational tests in the Chroma work, models performed noticeably better when given only the relevant history rather than the full transcript. Same underlying question, less around it.
That is a useful correction to the instinct that AI needs to know everything. Sometimes it needs less.
Why a fresh chat feels smarter
Most people who use AI heavily have probably noticed this one. A conversation starts going badly, so you open a new one, give the model a short explanation of what you are doing, and the responses improve.
The model has not changed. The context has.
Anthropic describes context as having a limited attention budget. As more information enters the context, more relationships compete for the model’s attention. Long contexts also appear less often in training than short ones, which contributes to a gradual loss of precision at greater lengths. Their framing is that this is a performance gradient rather than a hard cliff.
Long sessions add a second problem on top of that. They accumulate stale information. Something that was correct at 10am may have been superseded by noon, and both versions are still sitting in the conversation with nothing to mark which one you still mean.
A fresh chat clears that away.
I used to avoid starting again because I did not want to lose everything I had already given the model. I now think that was the wrong trade. The useful material does not have to live inside the conversation.
I stopped using the chat as storage
This has changed my workflow more than any prompting trick.
When I have a useful prompt, a requirement, a reference file, a snippet of code or a decision I know I will need again, I keep it outside the conversation. Then I can start another session whenever I need to without losing the material.
I give the model the current task, the relevant material, and enough background to understand what is happening. Everything else stays outside.
It also makes me think harder about what I paste. Does the model actually need the whole document? Does it need every screenshot from the project? Does it need three previous answers, or does it need the conclusion I reached after reading them?
That small bit of editing before I send something often does more for the response than another paragraph of instructions. It is the same habit I wrote about in context engineering for humans, pointed at a smaller target.
I keep those pieces in Tansei, which is the shelf I ended up building for this. It sits at the edge of the screen and holds what I copy, so a prompt I spent an hour getting right is still there the next morning, pinned, rather than buried forty messages up a conversation I have already given up on.
Starting a clean conversation no longer means starting my work again. The screenshot that explains the bug sits beside the code I am changing, and when a requirement changes I replace it on the shelf instead of leaving both versions in a chat log to argue with each other.
Then I can hand the model those pieces when they are relevant, without dragging the whole history of the project along with them.
That is what I mean by working context. It is the material that is still in play, and it is worth keeping separate from the conversation itself.
Smaller is not always better
There are obviously jobs where a model genuinely needs a lot of information. Asking questions across a long contract, comparing dozens of documents, working through a large codebase. Cutting aggressively there will remove something that mattered.
The research does not say long context is useless. It says size alone is a poor proxy for quality.
The practical question is whether each piece of information is helping the model do the task in front of it. That is a better question than asking how much will fit.
For me the habit has become simple. When a conversation starts feeling muddy, I start another one, and I carry across the current decision, the source material that still matters, and whatever instructions are still true. Everything else can stay outside.
We spent years pushing for bigger context windows. It turns out the skill worth having is knowing when not to fill them.
Tansei is a one-time purchase for Mac and Windows, and everything you keep on the shelf stays on your machine.
Frequently asked questions
What is context rot?
Context rot is the measured decline in an AI model’s output quality as its input grows longer, even when the task itself stays exactly as hard. Researchers at Chroma tested 18 models in 2025 by holding the task constant and increasing only the surrounding context. Performance became less reliable as the input grew, including on simple retrieval tasks.
Is context rot the same as running out of context window?
No. Running out of context is a hard failure at the token limit. Context rot is a gradual slide that starts well below it. A context window tells you how much information a model can accept. It does not tell you how well the model will use every part of that information.
Does a bigger context window fix the problem?
Not on its own. NVIDIA’s RULER benchmark tested 17 models that all advertised 32,000 tokens or more, and only half of them held satisfactory performance at that length. The advertised window is the space available, not a promise about how well the model uses it.
Why do AI answers get worse in long conversations?
Two things happen at once. The input grows, so more material competes for the model’s attention, and Anthropic describes this as a limited attention budget. At the same time a long session accumulates stale information: an old decision, a rejected version, a requirement that has since changed. Both versions are still sitting in the conversation, and the model cannot tell which one you still mean.
Does where I put information in a prompt matter?
Yes. The Stanford-led Lost in the Middle paper found models use information best when it sits near the beginning or the end of the input and worst when it sits in the middle. In one test, GPT-3.5-Turbo answered correctly about 75.8 percent of the time when the relevant document appeared first, dropping to 53.8 percent when it appeared mid-context. Asked the same questions with no documents at all, it scored 56.1 percent, so burying the answer mid-context was worse than never supplying it. The information was still there. Its position changed how well the model used it.
How do I avoid context rot in practice?
Give the model the material the current task needs rather than the full history of the work. Start a fresh session when a conversation starts drifting, carrying across only the current decision and the source material that still matters. Keep near-miss material out, since research found that something which looks relevant but does not contain the answer measurably reduces accuracy. When the input genuinely has to be long, put the most important parts at the start and the end.
Does Tansei manage an AI model’s context for me?
No. Tansei is a local shelf that holds the things you are working with: text, links, screenshots, files, code and colors. It does not connect to a model, edit prompts, or manage a context window. Its role is to keep your working material outside the conversation, so starting a clean session does not mean losing it. You still decide what the model sees.
Back to top