KVEraser Makes an LLM Forget a Passage Without Reprocessing
Researchers from Georgia Institute of Technology and Meta train a module that rewrites a deleted span's cached states, so a model can drop stale context without recomputing what follows.
TL;DR
- KVEraser, from Georgia Institute of Technology and Meta researchers, makes a model forget one processed span by rewriting only that span's cached states [^1].
- In a needle test it matched full recomputation's accuracy from 1K to 32K tokens, while its latency grew 24 percent against 17.6 times for recomputation [^1].
Background
When a language model reads a prompt, it stores keys and values for every token in a KV cache, so new tokens can attend to earlier ones without recomputing them [^1]. Each token's cached state depends on everything before it, so deleting a span invalidates the cache of every token after it [^1]. The exact fix reuses the cache up to the span and reprocesses the rest. Its cost grows with the length of that suffix, not of the deleted text [^1].
What happened
The paper, posted to arXiv on June 15, starts from cases where a problem is spotted only after the context has been processed: a stale retrieved fact, a wrong tool result, a user preference that was taken back, or a harmful prompt injection [^1]. KVEraser keeps the cache before and after the span unchanged and replaces only the span's own key and value states with learned "steering states" [^1]. These come from an eraser module, a trainable copy of the model's backbone, which reads the span on top of the preceding cache and outputs keys and values for every layer and attention head at those positions [^1]. The original model stays frozen. The eraser is trained so that, starting from the edited cache, the model produces the same continuation it would produce from the prompt with the span deleted. Because the cache keeps its length, no positions shift [^1].
Training runs in two stages. First come 80K samples from Wikipedia, in which a 100-token chunk is inserted into a long document and the model must retrieve the text next to an anchor string. Then the eraser is fine-tuned on about 7.5K samples from a needle-in-a-haystack task and three question-answering datasets [^1]. The generator is Qwen3-8B, which supports 32K-token contexts. Training used two 80 GB A100 GPUs; pre-training took less than a day and fine-tuning about 8 hours [^1].
In the needle test, two facts with the same key but different values sit in filler text, and the earlier one has to be erased [^1]. KVEraser reached near-perfect exact match at every size from 1K to 32K tokens, matching full recomputation [^1]. The baselines deleted the span's states and shifted the rest, told the model to ignore the span, or recomputed 15 percent of the suffix; they either started poorly or degraded as contexts grew [^1]. On three QA datasets unseen in training, 2WikiMultiHopQA, MuSiQue and IIRC, the span to erase was a misleading Wikipedia passage. KVEraser had the highest exact match of the approximate methods, while full recomputation was more accurate but took 3 to 4 times as long [^1].
Why it matters
The paper's case against the cheap fix is architectural. Once the suffix has been processed, its cached states already carry the deleted span, so an instruction to forget it "can only compete with stored evidence at decoding time rather than rewrite the cached states themselves" [^1]. The tests bear that out: as contexts grew, the instruction-only baseline increasingly answered with both the erased and the retained value [^1]. KVEraser ties the cost of the fix to the length of the span rather than to everything after it, which pays off most when a short passage sits early in a long context [^1].
The limits are clear in the paper. It handles one contiguous span; several spans are left to future work [^1]. On natural QA it still trails full recomputation, a gap the authors suspect may partly reflect the scale and diversity of their training data [^1]. The eraser has to be trained for the model it serves and is a copy of that model's backbone [^1]. And while prompt injection is one of the motivating cases, the experiments test planted numbers and misleading factual passages, not injected instructions [^1].
Practical example
Say Priya runs the support chatbot for a small accounting-software company. The bot runs on Qwen3-8B and answers from long contexts built from help-center articles, the customer's ticket history and tool output. One afternoon a customer asks about export formats. Retrieval pulls in a short passage from an outdated article early in the context, and a freshness check flags it only after the whole context has been processed.
Priya has three options. Telling the model to ignore the passage is cheapest, but in the paper's tests that approach increasingly let the erased value slip into answers as contexts grew, because the cached states after the passage were computed with it in view [^1]. Reprocessing everything after the passage gives the exact result, but its cost scales with the thousands of tokens of ticket history that follow [^1]. With KVEraser, the eraser produces replacement states for just the flagged passage's positions and the rest of the cache is reused as it is [^1]. The bot then answers as if the article had never been retrieved, at a cost set by the passage's length rather than by what follows [^1].
The catch: Priya's team would first have to train an eraser for their model, which took the authors two A100 GPUs, less than a day of pre-training and about 8 hours of fine-tuning [^1]. On the paper's QA tests, the eraser was still right less often than full recomputation, so for an answer that must be correct, the slower exact path remains the safer one [^1].
Related gear
We recommend this book because it explains how transformer models and their attention mechanism work, the machinery whose cached states KVEraser edits.