128K の文書は、読ませ直さずに使い回せるのか Can a 128K Document Be Reused Without Reading It Again?

Introduction

The conclusion of the previous post was “128K works, but reading it in takes 21–24 minutes”.

But that’s the first time. If you ask many questions about the same document, from the second time on it should be able to reuse the state it has already read. llama-server has several mechanisms for that.

This model has a quirk, though. As I found last time, it’s a hybrid of SSM and full attention.

An SSM state is just a single “current state” built up from the tokens read so far. Unlike a KV cache, you can’t “roll it back to token 100”. So it might not be reusable the way an ordinary model’s state is.

If it can be reused, under what conditions? I measured it.

How I Measured

I judge by timings.prompt_n (the number of tokens actually reprocessed) returned by the server, not by feel. Not “it feels faster”, but “how many tokens did it re-read”.

The document is this series’ work logs concatenated together. I first tried 6 scenarios at 32K to understand the mechanism, then confirmed at 128K.

Results at 32K

Just changing the question takes 4.7 seconds. That’s 1/34 of the first time. Even with another document in between, the state stayed in RAM and it came back in 0.7 seconds.

On the other hand, editing one line in the middle meant re-reading everything, and restoring from disk also meant re-reading everything.

Why: There Are Only Two Places to Go Back To

With verbose logging (-lv 4), the reason was written right there.

created context checkpoint 1 of 32 (n_tokens = 31466, size = 124.990 MiB)
created context checkpoint 2 of 32 (n_tokens = 31978, size = 126.002 MiB)

Checkpoints (saved SSM states) are created only at two places near the end of the prompt: 512 tokens before the end, and 4 tokens before the end. None are created along the way at 8K, 16K, 24K and so on.

That explains all four results.

• Same prompt: goes back to 4 before the end and processes just 4 tokens
• Different question: goes back to 512 before the end and processes just under 500 tokens at the end of the document plus the question
• Edit in the middle: there’s no place to go back to before the edit, so it processes from the start
• Restore from disk: explained in the next section

A KV cache alone could go back to any position. What can’t go back is the SSM state: it can only return to a point that was saved.

Restoring from Disk Only Brings Back the KV

Both saving (0.68 GiB, 3.6 s) and restoring (0.3 s) succeeded. The server even recognized that the restored content matched exactly.

selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 1.000

The server itself printed why it re-read anyway:

forcing full prompt re-processing due to lack of cache data
(likely due to SWA or hybrid/recurrent memory, ...)

The save file contains only the KV; checkpoints aren’t included. Even without changing the question, generating a response requires going back 4 tokens. It can’t go back those 4 tokens, so it ends up re-reading from the start.

Just in case, I also tried sending the exact same question. That re-read everything too.

With this model, saving to disk is no help at all.

Confirming at 128K

Now that I understood the mechanism, at 128K I measured only E1, E2 and E4. Editing the middle and restoring from disk are certain to re-read everything for the same reasons as at 32K.

Even at 128K, just changing the question takes 10.7 seconds. That’s roughly 1/100 of the first time. It’s slower than at 32K (4.7 s) because the 515 re-read tokens sit deep in the context, and each token attends to 130,000 tokens’ worth.

E4 worked too. The state for 128K (about 2.8 GiB of KV + 2 checkpoints) fit in the RAM cache (8 GiB limit), so even after reading another 128K document it came back in 2.1 seconds.

The First-Read Time Moved Again

This time the first read took 17.6 minutes. Clearly faster than the two runs I measured last time (21.5 and 24.2 minutes). I confirmed there were no interruptions or sleeps. The PC had been on standby for 10 hours right before, so maybe it was cool.

Three measurements, 17.6–24.2 minutes. Nearly a 40% spread with the same settings. The previous post said “21–24 minutes”, so I added an update to it.

A Mistake in an Earlier Post I Found Along the Way

The server’s startup log had this line:

srv    load_model: cache_reuse is not supported by multimodal, it will be disabled

The --cache-reuse 256 in my serve.ps1 had been disabled from the start. This model is the multimodal version that also handles images, and I was loading the mmproj for images.

So does it work if I drop the mmproj? When I started without it, the reason changed but it was still disabled.

srv    load_model: cache_reuse is not supported by this context, it will be disabled

--cache-reuse shifts the KV to reuse chunks that match partway through. That doesn’t go together with an SSM state, which can’t be shifted. It’s structurally unusable with this model.

The problem is that in part 2 I had written this:

Evidence That --cache-reuse Works

cache_n  : 425    ← キャッシュから再利用したトークン
prompt_n : 21     ← 実際に計算したトークン

The reuse really did happen. But it came from --cache-prompt, which is on by default: the basic mechanism that skips recomputing the part whose beginning matches.

The effect was real; I had attributed it to the wrong setting. I connected “I added the setting → it got faster → thanks to that setting” without checking whether the setting was actually enabled.

I added a correction to the part 2 post. I kept --cache-reuse in serve.ps1, with a fixed comment, since it may work with other models and does no harm.

How to Use It

In short: append to the end of the document, put the question last, and don’t stop the server. Stick to those three and you only pay the reading wait the first time.

On the other hand, there’s one thing to watch in my llm setup. In the CLI post, I added a setting that unloads the model from memory after 12 hours unused (--sleep-idle-seconds 43200). I haven’t measured what happens after unloading, but it’s natural to assume the cache goes with it. On days when I work with long documents, I need to keep this setting in mind.

Summary

• Just changing the question takes a few seconds. From the second time on, the same document needs almost no wait
• Editing the middle re-reads everything. The only places to go back to (checkpoints) are two near the end
• Saving to disk doesn’t help. The KV comes back, but the SSM state doesn’t
• The reason for all of these is the hybrid architecture. An SSM state can only return to a point that was saved
• Part 2’s “--cache-reuse works” was wrong. What worked was the default --cache-prompt

The previous post’s “21–24 minutes” turned out to be a wait you only pay the first time. That assumes, though, that you don’t edit the document and don’t stop the server.