Random Eviction Works Just as Well ── Random Attention challenges the need for scoring in KV cache compression

1. The Problem: Longer Thought, Larger Memory Bill

Reasoning-capable large language models unfold long chains of thought before arriving at an answer. Each generated token adds a Key-Value (KV) pair to the cache. The longer the reasoning, the larger the cache grows, consuming GPU memory and reducing the number of concurrent requests a server can handle. This directly raises operating costs.

Existing KV cache compression methods share a common paradigm: estimate a score for each cached token predicting how much it will matter later, then keep the highest-scoring ones. The scoring method differs from one approach to another, but the structure is the same: select the important tokens and discard the rest. Building and running these scorers adds computational overhead, which partly offsets the memory savings the compression was meant to deliver.

2. The Proposal: No Scoring at All

Random Attention does two things and nothing more:

• Keep the prompt: The KV cache entries corresponding to the input prompt are never evicted.
• Evict reasoning tokens uniformly at random: Within each attention head, tokens generated during reasoning are dropped with equal probability.

The authors report that across four models and six reasoning tasks, this approach matches the strongest prior evictor while serving 32-43% higher throughput in vLLM deployment. The throughput gain comes not only from eliminating the scoring overhead but also from the simplicity of the eviction itself, which requires only compaction (repacking the remaining entries) rather than any priority computation.

3. Why It Works (According to the Abstract)

The authors present two findings from controlled experiments.

First, the prompt is the fragile part of the cache. They argue that most of the performance gap between existing selectors can be explained by whether their scoring signal happened to preserve the prompt. In other words, what looked like the benefit of sophisticated scoring was, they claim, merely a side effect of incidentally protecting the prompt.

Second, the reasoning trace is self-protecting. At the text level, the model restates information it still needs as it works. At the attention head level, each head maintains its own copy of the trace. This dual redundancy means that once the prompt is safe, a random draw retains enough copies of what the model still needs. No scoring is required to select them.

This paper is a preprint and has not undergone peer review.

4. An Analogy: Sticky Notes on a Textbook

5. What Is Not New, and Where the Limits Are

KV cache compression is an active field with numerous prior methods, including ThinKV and others. What Random Attention contributes is a negative finding (scoring is unnecessary) and an experimental explanation for it (reasoning trace redundancy). Reporting a negative result head-on, and then digging into why the established approach was unnecessary, is a mode of inquiry that challenges the field's assumptions in a constructive way.

Limits are clear. First, the abstract specifies four models and six tasks but does not name them. The generality of the finding depends on the diversity of those models and tasks, which the reader cannot assess from the abstract alone. Second, for tasks where reasoning traces lack sufficient redundancy (very short reasoning, or patterns with little repetition), the behavior is unknown. Following the authors' own logic, low-redundancy reasoning should be more vulnerable to random eviction. Third, the 32-43% throughput improvement is measured specifically in vLLM; reproducibility in other serving frameworks is not addressed in the abstract.

6. Why This Topic Is Clustering Now

The proliferation of reasoning-focused models (the o1 family, DeepSeek-R1, and others) has turned the memory cost of long chains of thought into an urgent operational concern. The selection keywords include "kv cache compression," "reasoning trace," "redundancy," and "vllm throughput." The trade-off is stark: better reasoning requires longer thought, and longer thought requires more memory. Compression research is moving on multiple fronts simultaneously. Random Attention stands out within that wave by asking whether the scoring machinery everyone builds is necessary at all.

The paper has 170 Hugging Face upvotes and 49 GitHub stars. These indicate that the question resonated within the research community, but they do not validate the correctness of the claims.

7. What This Connects to in Pharma and Regulatory Work

KV cache compression directly affects the cost of running LLMs in production. For pharmaceutical companies deploying LLMs internally for literature review assistance, submission document drafting, or pharmacovigilance triage, the number of concurrent requests and the memory cost per request influence deployment decisions. Whether to add another GPU or to apply a compression technique is a choice that shapes cost structure.

If Random Attention's findings reproduce, they would mean that equivalent output quality can be maintained with less GPU memory by simply dropping the scoring computation. The caveat is that the method's effectiveness depends on reasoning trace redundancy, which may not hold uniformly across tasks. In a pharmaceutical context, short summarization tasks and complex drug-interaction reasoning may behave differently under this eviction strategy. Task-specific validation would be needed before any adoption. The broader takeaway, though, is that techniques reducing LLM serving costs without degrading output quality are worth watching closely, because they lower the barrier to deploying these models in resource-constrained enterprise settings.