Benchmarks & validation

North Star scorecard

The project's single measured source of truth, refreshed at the end of every optimization wave (charter Rule 1):

Bigger than RAM — the defining capability

16 GB Mac, ~3 GB free (loading these resident is impossible), --stream-lm-head --no-async --ctx 512, greedy:

Peak RSS grows with layer width, never model depth/total size. See Bigger than RAM.

Half the RAM, comparable speed

TinyLlama-1.1B Q8_0, --ctx 512, greedy, 4 threads, warm cache:

2.1× less RAM at ~88% of llama.cpp's decode, or 13× less RAM streaming — a smooth RAM↔speed dial, not a fixed point.

Golden validation vs llama.cpp

For the same model and prompt, SipLLM dumps every transformer block's residual stream and the final logits and diffs them against llama.cpp's own values (captured through its eval callback). Cross-engine outputs are never bit-exact — summation order and rounding differ — so the comparison is numerical: per-layer max|Δ|, cosine similarity, and argmax/top-k agreement.

Prompt "The capital of France is" → both engines greedily predict " Paris" (TinyLlama-1.1B, 22 layers, dim 2048, GQA 32/4):

Two things fall straight out, both exactly what theory predicts:

1. F16 is numerically identical to llama.cpp (cosine 1.000000) — the compute graph is correct; every residual difference in the quantized rows is pure quantization error.
2. Error grows monotonically as quantization coarsens (F16 ≪ Q8_0 < Q5_K_M < Q4_K_M). A bug would produce erratic, layer-localized divergence; this smooth accumulation is the signature of a faithful implementation.

The golden matrix currently covers the Llama path; newer architectures are unit-tested and validated against llama.cpp as models are added. See Quantization and the LM-head journal.

RAM-budget sweep

bench_ram_budget.sh, M3, warm, ctx 512:

Pinning is a pure cache: logits + KV are bit-identical across every budget, and peak RSS ≤ budget at every point. See Memory planner.

Streamed LM head

TinyLlama-1.1B Q4_K_M, M3, warm, t=4:

Streaming the output projection nearly halves peak RSS for a negligible decode cost — the head is often the single largest tensor. See the LM-head journal.

Methodology

• Peak RSS is read from /usr/bin/time -l (maximum resident set size) — the authoritative, cross-runtime figure, independent of any engine's self-report.
• Median-of-3 for every timing (TTFT, decode, prefill tok/s); warm cache so the file is in the page cache.
• Cross-engine correctness is numerical, not bit-exact — see the golden matrix above and golden/README.md for why (summation order + rounding differ between engines).
• Every report states model · quantization · RAM budget · hardware · compiler · commit SHA alongside the metrics, and compares against the previous baseline (charter Rule 1).

Reproduce it

# peak RSS / TTFT / decode tok/s harness
scripts/bench.sh

# decode-tok/s + peak-RSS vs --ram-budget sweep
scripts/bench_ram_budget.sh

# cross-engine numerical validation vs llama.cpp
python3 golden/validate_matrix.py --prompt "The capital of France is"

# the dependency-free unit suite
make test

Latest runs are committed under bench/results/ (demo-v0.4-*.json, bigger-than-ram-*.json, ram-budget-*.json) with the full test log in bench/results/test-results-2026-07-27.txt. See the performance history for the wave-by-wave trend.