Attention Windowing

The instrument the program is built on. Replace one attention head with a fixed sink-plus-local window, renormalize, and read off the perplexity it costs: a causal, per-head map of how much of each head is load-bearing. Across GPT-2, Pythia, and Qwen from 124M to 7.6B parameters, 81 to 99.9% of heads come back near-free to window, and the fraction climbs with scale. What's windowable is predicted by locality, not by how peaky a head looks, and grouped-query attention, the dominant KV-cache trick, doesn't consume the headroom (a pre-registered prediction, falsified by a permutation test).

Cashing the map out end-to-end on Qwen2.5-7B, with the honest correction that makes it deployable: per-head-free is not jointly-free, and a map measured at short context over-promises at long context. Measure at the context you actually run at, window whole KV groups, and it becomes a Pareto improvement: 25% of groups lossless, 50% for about 2× faster attention (an exact FlexAttention kernel) and 2× less KV cache at matched quality, no fine-tuning. A healing-gated frontier runs past that.

The open question, and an honest one. Perplexity is blind to long-range retrieval: a windowed model can read as lossless and still have forgotten how to find a needle in a haystack. A passkey suite with six arms tests whether the perplexity-expensive heads the map keeps dense are the same heads retrieval needs. If they are, this is a long-context compression result that beats the KV-cache baselines; if they aren't, it's a clean edge-AI story instead. I'm waiting on GPU credits to run it.

Where the compression compounds: windowing is orthogonal to weight quantization and stacks with it. On 4-bit weights it adds about 0.2 perplexity, and the KV-cache saving is structural, roughly the windowed fraction regardless of context. A small QLoRA heal recovers the frontier. Four-bit weights, half the groups windowed, an optional adapter: a 7-to-8B model at long context on commodity hardware.