RidgeFill: the short cut through the long wait

RidgeFill makes long prompts answer faster by not computing most of what the model's upper layers would have done. It works because of two properties of transformers that are easy to miss: prefill and decode are bottlenecked on different things, and most of what the late layers compute per prompt token exists only to produce a cache entry. This is those mechanics: why the shortcut is possible, what its error looks like, and where the guardrails go.

There is a mirror image that makes the idea click. Speculative decoding already cheats on the answer: a cheap guess runs ahead of the model and gets verified almost for free, because decoding has spare compute and nothing else to do with it. RidgeFill cheats on the read the same way, guessing what the upper layers would write instead of computing it, spending the bandwidth prefill leaves idle. The difference is what happens after the guess. A guessed token is checked and corrected; a guessed cache entry has no checker short of running the layers you skipped. So instead of verifying, RidgeFill decides in advance where a wrong guess would hurt, and refuses to guess there.

Two phases, two bottlenecks

Prefill pushes the whole prompt through every layer at once. Each weight load is amortized across hundreds of rows, so the GPU runs out of arithmetic long before it runs out of memory bandwidth. The wait on a long document is compute-bound, full stop.

Decoding is the inverse: one token at a time, every step streams all the weights for a single row, and the hardware is mostly idle between loads. Speculative decoding exploits that idleness: guess ahead with spare compute, verify cheaply. RidgeFill exploits the opposite property of the other phase: prefill has spare bandwidth, and the shortcut spends that to buy back arithmetic. Reading one extra table per layer from memory is nearly free when the machine is waiting on multiplies.

The second property: a layer's real deliverable for a prompt token, in the upper half of the network, is the key/value entry that token leaves in the cache. After prefill, the prompt's influence lives almost entirely in those entries: every generated token reads the cache, not the layers. So a great deal of late-layer arithmetic is scaffolding whose only job is to produce an entry. Scaffolding can be predicted.

Predict the input, skip the work

Transformers refine, they don't replace. Each layer adds a small correction to the token's running vector (the residual stream) rather than rewriting it. By a layer about a third of the way in, that vector already carries most of what the rest of the network will use; the layers above mostly add refinements on top. Layer forty's input is approximately layer sixteen's state plus a small update, and if you can predict that update's effect, you never have to compute it.

That is the whole method. Pick the split layer. For every layer above it, learn a single linear map from the split state to that layer's input. At prefill time, a prompt token's row skips the layers entirely: the map produces its input at layer forty, the layer runs just the projections that write the cache entry from that input (the keys, the values, and in a linear-attention layer the state update), and the attention block and MLP for that row never fire.

The maps are regression, not training. Run a few hundred documents through the unmodified model, record the paired hidden states at the split and at each late layer, and solve for the best linear reconstruction: one matrix and one bias per layer, in closed form, in minutes on a CPU, with no gradient steps and no touched weights. A ridge penalty keeps the solution from memorizing the calibration text. The map is a side file; the model is exactly the model.

Why it mostly dissolves, and when it doesn't

A predicted input is wrong in a way that is small and, from one token to the next, nearly independent. That combination is the good case for a model: attention is a weighted average over the cache, so independent per-entry errors largely cancel in the output, while a systematic bias would survive the averaging. Measured as an effect on the answer, most of the fill's error averages away.

The failure mode is the verbatim. When the answer quotes something (a part number, an identifier, a name), there is no averaging step; whatever the entry said is what comes out. Early needle tests showed exactly this signature: the model returned the buried number with only the last digit wrong. That result did more for the design than any loss curve, because it said where the error matters, which an average error never will.

And there is no cheap check. Speculative decoding gets to guess because the full model verifies each guess almost for free. A filled cache has no verifier: the only model that can certify an entry is the full model running the full row, which is precisely the cost the fill was hired to avoid. So the goal inverts: not "make the estimate right everywhere", which converges to just running the layers, but "find where the estimate does damage, and don't estimate there."

Where exactness goes

The first guardrail is recency. The tail, the last two to four thousand tokens, runs every layer exactly. Questions are conditioned on the end of the conversation, so the most recent context is where an error is most likely to surface; the tail is also the fixed, predictable part of the budget.

The second guardrail is the one with real ideas in it. Attention weights are not just bookkeeping; they bound influence. A token's effect on the output is proportional, up to softmax normalization, to the attention it receives; a row nobody looks at can be wrong about a lot and change almost nothing, while a row the final queries stare at hard has its error amplified straight into the answer. So: put the question at the end of the prompt, watch where the last few hundred query rows attend during prefill, and run those rows exactly. The rest of the prompt can be estimated at your convenience. The pointer is the trick, not the compute: pick the same count of rows at random and most of the protection disappears.

The third guardrail is simply the answer phase: every generated token runs the full model, full depth, no maps. The shortcut applies to the read, never the write.

Two settings, one trade

The split depth and the tail length decide where you sit on a curve that cannot be moved. Deeper splits mean more exact work per token, but the prediction also gets easier, because the split state sits closer to what the late layers actually receive. Shallower splits mean more speed and a harder guessing problem, and on models that don't normalize their keys, a shallow split is where the arithmetic of attention lets errors through. Longer tails mean fewer surprises in the recent context and less left to the map.

No setting is free: a method that skips part of prefill cannot be both lossless on every prompt and faster than exact; the rows it doesn't skip are the argument. The published operating points sit around a quarter off the wait with recall essentially unchanged; on the long end, a 120,000-token prompt goes from about two minutes and eight seconds to about one minute and forty on a single GPU, the one number worth remembering.

What the shortcut does not save is memory. Every token still keeps its cache entry (the entries are the whole point), and the maps ride alongside the weights as a gigabyte-scale side file. This is a compute technique wearing cache clothing; anyone comparing it to token-dropping or cache-eviction methods is comparing different papers.

Three sentences