Streaming the layers

This is the change everything else rests on. A transformer's weights dwarf its activations, so the standard runtime — load the whole file resident — makes peak memory scale with model size: 1.1 GB for a 1.1 B Q8 model, tens of GB for a 70 B one, which never fits on a phone. But a decoder only ever touches one transformer block at a time. We built the loader so exactly one block is resident and its memory is reused for the next, which makes peak weight RSS track a layer's width, not the model's depth.

The problem: memory scaled with the wrong number

When we started, "run a bigger model" meant "buy more RAM." Every weight was resident for the whole run even though block L's weights are needed only while block L executes. The math never needed them all at once — the loader was forcing it. If we could make block L's memory reusable for block L+1, the dominant term (weights) would stop scaling with the number of layers entirely.

The fix: a seam the math can't see through

The transformer never opens a file. It talks to a WeightSource (include/llm/weight_source.h) that exposes only three things: a tensor directory (tensors(), find()), typed metadata, and positional reads (read_raw() / read_raw_at(), a pread that never loads the whole file). Both the real GGUF parser (GgufFile) and the toy .llmw reader implement it, so the entire forward pass is oblivious to where bytes come from. That seam is what lets us change residency strategy without touching a line of arithmetic.

LayerLoader (include/llm/loader.h, src/loader.cpp) drives blocks through the seam. Transformer::forward (src/transformer.cpp) loops:

embed_token(token)              -> x   (one row streamed, or resident if tied)
for layer L in 0 .. n_layers-1:
    loadLayer(L)                -> make block L's weights resident
    block(L, pos)               -> RMSNorm -> QKV -> RoPE -> GQA attn -> proj -> RMSNorm -> SwiGLU
    unloadLayer()               -> release block L for reuse
final RMSNorm + project_output  -> logits

Each block asks for weights by Role (AttnNorm, AttnQ, AttnK, AttnV, AttnOut, FfnNorm, FfnGate, FfnUp, FfnDown at the core), which the loader maps to GGUF names like blk.<L>.attn_q.weight. Two mechanisms keep only a layer's worth of weights live:

1. Per-layer residency. loadLayer(L) reads block L into a slot buffer; unloadLayer() releases it for reuse. With the synchronous single-buffer path exactly one block is resident.
2. Weights stay quantized; dequant is per-row. In Residency::Quantized the loader keeps each weight's raw on-disk bytes resident and never bulk-expands them. matmul_quant (src/quant.cpp) dequantizes one output row into a tiny scratch buffer, dots it with the activation, and moves on — so a whole layer stays ~4 bits/weight in RAM instead of 32. Residency::FP32 (dequant on load) survives only as the numeric oracle for the equivalence tests. This is the subject of Flat resident weights.

Measured: the same model, dialed from resident to streamed

TinyLlama-1.1B Q8_0, --ctx 512, greedy, 4 threads, warm cache — one model, one dial:

Resident, we use 2.1× less RAM at ~88% of llama.cpp's decode; fully streamed, 13× less RAM for the same output. The streamed path is disk-bound — that tradeoff is the whole point of Running bigger than RAM — and --ram-budget buys the speed back linearly.

What it unlocked

The seam is why every later article is possible: streaming is switchable per-backend, the async prefetch ring overlaps I/O with compute, single-pass prefill stops re-streaming, and streaming the LM head removes the last resident floor. The reference for the seam and its three interchangeable backends is the streaming layer loader architecture page.