The Model Shouldn't Have to Reread the Room

Don't Reread the Room

12:40

There's a new AI paper that sounds, at first glance, like the kind of thing only an inference engineer could love. It's called "Cross-Model KV Cache Transfer in LLM Families." I know. Riveting.

But buried inside that title is an idea that could become extremely important to how AI systems actually operate.

The problem the researchers are attacking is simple. Modern AI systems increasingly use more than one model.

A smaller model might handle the cheap, easy work. A larger model gets called when something becomes difficult. Another model may specialize in coding. Another in reasoning. Another may be deployed somewhere closer to the edge.

That kind of routing makes sense economically. There's just one annoying problem. Every time you hand the conversation to another model, the new model generally has to read the room again.

Technically, it has to process the accumulated context and build its own KV cache before it can continue generating. That initial pass through the context is called prefill, and the cost grows with both the size of the model and the amount of context it has to ingest.

If you've got a long-running agent with a massive context window, that isn't trivial.

The new paper asks a fascinating question: What if we could translate the first model's KV cache into something the second model could use instead?

In other words, instead of giving the new model the entire conversation and saying: Here. Read all of this so you know what's going on.

You hand it something closer to: Here's the computational state the other model already built from reading it.

That distinction matters.

First, what the hell is a KV cache? I'll spare you the graduate seminar.

When a transformer processes text, it produces internal key and value representations that its attention mechanism uses to keep track of the context it has already processed.

During generation, those values are cached so the model doesn't have to recompute the entire conversation every time it produces the next token. That cache is one of the reasons modern LLM inference is practical at all.

But KV caches have historically been largely model-specific. A Qwen model's internal representation isn't automatically usable by another Qwen model of a different size, much less some completely different architecture.

The researchers wanted to know whether those representations were actually as incompatible as we've assumed. Apparently, not always.

Their experiments found substantial linear structure between the KV representations of different-sized models within the same family. For Qwen3 14B to Qwen3 32B, a single source layer could explain a meaningful portion of the target model's KV structure, and combining multiple source layers pushed that much higher. That led them to build a surprisingly simple translator based largely on ridge regression.

Not another giant neural network.

Not some giant new training operation.

A mathematical mapping trained on a relatively small calibration set.

And it worked.

Across six model pairs spanning three families, four of the pairs retained between 73% and 98% of the receiving model's normal standalone accuracy across the evaluated benchmarks. Two pairs performed badly with the linear method, although a nonlinear MLP recovered substantial performance in those harder cases.

So this is absolutely not universal. There are architectural compatibility requirements. Some transfers work much better than others. There's still accuracy loss in many cases.

Keep those caveats in giant letters.

But then comes the number that made me sit up. For the successful small-to-large transfers, converting the KV cache was 2.7 to 25 times faster than making the receiving model perform the prefill again. And the researchers report that the approach remained stable across multi-turn handoffs. That's where this stops being an obscure optimization trick.

The handoff tax

Think about the way we're beginning to build AI systems.

You don't necessarily want a giant frontier model answering every question. That would be like hiring a neurosurgeon to answer the phone. You want cheap intelligence handling cheap cognition and expensive intelligence appearing when the situation warrants it.

That means routing.

Maybe a small model is talking to you and suddenly realizes: This just got complicated.

The orchestrator moves the task upstream to a larger model.

Today, part of the cost of that move is that the larger model has to reconstruct the accumulated context for itself. The NVIDIA-affiliated research team's work suggests there may be cases where we can begin reducing that reconstruction tax. Several of the authors currently work at NVIDIA, including Taekyung Heo, Ritika Borkar and Bita Darvish Rouhani.

That has obvious consequences... Lower compute cost... Lower time-to-first-token... More practical model cascading... Better economics for long-running agents... More aggressive routing between models.

Those alone are significant. But they're probably not the most interesting part.

Spend the latency

Here's where I start extrapolating.

And I want to be clear: the paper does not demonstrate the future I'm about to describe.

It provides one technical primitive that points in an interesting direction.

We normally think of latency improvements as: same answer, faster.

But AI gives us another option. We can spend the savings.

If the system saves several hundred milliseconds or several seconds by not rebuilding context, perhaps it doesn't return those milliseconds to me. Perhaps it uses them to think.

Call another specialist.

Run a verification pass.

Move to a larger model.

Explore several approaches in parallel.

Check its own work.

And still answer inside roughly the same conversational window.

That gets particularly interesting with voice.

Modern voice AI is already getting close to the timing required for natural conversation. But there's a fundamental tension between responsiveness and depth.

Humans will tolerate only so much dead air before the interaction starts feeling like we're waiting for a computer again. Reduce enough of the machinery happening between my sentence and the system's answer, and some of that latency budget can become reasoning budget. The result wouldn't necessarily feel faster. It could feel smarter at the same speed. That's a much more important threshold.

Up, down and sideways

There's another implication.

The paper specifically describes bidirectional transfer within compatible model families.

Small to large means you can potentially move up when you need more capability. Large to small means you can potentially move down when the difficult work is finished and you'd rather stop paying for the giant model. That alone creates something resembling elastic intelligence. But eventually, this kind of work raises a broader question.

What about sideways?

A general model hands off to a coding model. A reasoning model hands context to another specialist. An edge model hands a task to a cloud model. One agent hands useful computational state to another.

Again, we are well beyond what this particular paper proves. But the underlying research question becomes fascinating: How portable can useful computational context become?

Because historically, when one intelligent system communicates with another, we serialize everything back into language. One model thinks. It emits tokens. Another model reads the tokens. Then it reconstructs its own internal representation of whatever the first model already figured out. That works. But it's almost hilariously inefficient when you think about it.

We're making intelligence repeatedly translate its internal state back into human-readable language so another machine can translate it back into machine state.

Perhaps that won't always be necessary.

Transportable "understanding"

I'm deliberately putting quotation marks around the word understanding here.

A KV cache isn't a soul. It isn't consciousness. It isn't a portable mind. But operationally, it contains part of the representation a model has computed after processing context.

So there's an important conceptual shift hiding in this research.

We've spent decades making data portable.

Files.

Documents.

Databases.

APIs.

Cloud storage.

Now researchers are beginning to explore whether portions of computed context can become portable too.

Not simply: Here is the information.

But something incrementally closer to: Here is the state produced by processing that information.

That is a very different primitive. And if that primitive keeps improving, I suspect it eventually becomes foundational to serious agent systems.

Persistent cognition

This is the phrase I keep coming back to.

Today we tend to think of the model as the persistent thing. You're talking to GPT-whatever. Claude-whatever. Gemini-whatever. But long term, that may be backward. The model may become temporary. The persistent thing may be the cognitive state of the task.

One model works on it for a while. Another takes over. A specialist branches off.

Something cheap handles the boring portions. Something enormous shows up for thirty seconds when the problem becomes difficult.

The models become interchangeable compute resources applied to a persistent thread of cognition.

That would make agents look considerably less like a collection of chatbots with job titles and considerably more like an actual distributed cognitive system.

And then an interesting prediction falls out of that: The persistent unit of AI may eventually stop being the model. It may become the state.

We're nowhere near proving that. But this paper makes it a more legitimate question to ask.

And then things get weird

If useful cognitive state becomes increasingly portable, a whole second class of problems appears.

Who owns it?

Who can read it?

Which agent is allowed to inherit which portion?

Can it be copied?

Can it be poisoned?

Can you revoke access?

Can another model carry forward something derived from sensitive information without ever seeing the original document?

What happens when a malicious system inherits trusted context?

We already have enormous security problems around data and credentials. Portable computational state could create another category entirely.

The upside is equally large.

Imagine an agent that spends hours understanding a software repository and doesn't require every specialist it calls to reread the entire codebase. Or a research system whose working context can move between different levels of intelligence depending on the task. Or an edge device that can hand a problem to cloud intelligence and then continue locally without rebuilding everything from scratch. Or real-time multimodal systems where models appear and disappear behind the scenes while the user experiences one continuous interaction.

None of those are products described in this paper. They're simply the kinds of architectures that become easier to imagine once state itself starts moving.

The boring papers are usually the dangerous ones

That's why I love research like this.

There's no humanoid robot doing backflips. No giant AGI announcement. No cinematic demo.

Just some researchers asking whether one model's KV cache can be mathematically translated into another model's KV cache. And finding that, under the right conditions, it can. Faster than rereading the context. Sometimes much faster.

Maybe this remains an important but narrow inference optimization. That's entirely possible. But I've been around technology long enough to pay attention when someone makes state portable across a boundary where we previously assumed recomputation was necessary. Those boundaries have a habit of becoming architecture. And if this one does, the destination isn't merely faster AI.

It's a world where intelligence can change size, specialization and eventually perhaps location without continually forgetting where it was. That's a much more interesting future. Because the breakthrough may not be giving every model perfect memory.

It may be making the memory of the work less dependent on which model happens to be doing it. And that's the beginning of something I suspect we're going to hear a lot more about: Persistent cognition.

---

Rich Washburn is a technologist and strategist working at the intersection of AI, infrastructure, and capital. He is Managing Partner and Chief AI Officer at Eliakim Capital.