← voidwest ember road to ember 1.0

v0.3: quantized on disk, quantized in memory

draft · part 3 of the road to Ember 1.0 series · not yet published
mohammed al-thobaiti · 2026-08-04
Ember quantization GGUF memory AVX2 LLaMA Qwen

draft status

complete draft with explicit citations to repository artifacts. figures are embedded as SVG (dark/light theme variants under figures/) with captions citing the evidence. the performance and memory numbers cited here were re-derived from tagged builds on 2026-08-04 and recorded in artifacts/benchmark-v03/part3-binary-memory.md; release-time values are cited from artifacts/benchmark-v03/bench-summary.json and docs/validation.md.

the file said Q4. the runtime said f32. the model was smaller on disk and just as large in RAM, until v0.3 stopped the runtime from undoing the model's compression.

the problem: the inversion

a quantized checkpoint is a promise about size. Q4_K_M means roughly four bits per weight on disk, and the file delivers: Llama-3.2-1B at Q4_K_M is about 39% smaller than the same model at Q8_0. the promise breaks when the runtime loads it. Ember's loader before v0.3 dequantized every K-quant tensor to f32 at load time and kept the expanded buffers resident, then ran everything through generic f32 kernels. the checkpoint stayed Q4 on disk; the running model was f32 in memory.

the consequences were measured before v0.3, in the frozen baseline note (docs/research-notes/the-file-said-q4-runtime-said-f32.html), on the same machine that produced the release benchmarks:

model	file	eager decode tok/s	eager peak RSS	runtime representation
Qwen2.5-1.5B	Q4_K_M 1066 MB	2.6	8712 MB	persistent f32 expansion
Qwen2.5-1.5B	Q6_K 1396 MB	2.5	8713 MB	persistent f32 expansion
Llama-3.2-1B	Q4_K_M 770 MB	3.2	6965 MB	persistent f32 expansion
Llama-3.2-1B	Q6_K 974 MB	2.9	6965 MB	persistent f32 expansion
Llama-3.2-1B	Q8_0 1260 MB	149.6	2685 MB	compressed-resident

across the four Q4/Q6 rows, peak RSS was 2.59×–4.49× higher than the Q8_0 baseline and decode was 15.6×–51.6× slower. the only axis on which the lower-bit files were smaller was the file itself. Qwen Q4_K_M is about 41% smaller on disk than Q8_0 and uses about 4.49× the peak RSS while decoding about 15.6× more slowly.

that is the question this release answers: what is the point of a Q4 model if the runtime expands it back to f32?

the naive solution: dequantize everything, honestly

the dequant-to-f32 loader was not lazy. it was the correct engineering choice for a research engine whose first job is numerical honesty: one representation, one generic kernel path, no per-dtype execution to validate. the v0.3 contract (docs/v03-execution-contracts.md) states the cost plainly in section 3: resident weight memory is 4× the compressed size, Llama-3.2-1B Q6_K is 1.02 GB on disk and ~4.1 GB as resident f32, and every matmul is a dense f32 gemm.

the problem was never that the reference path existed. it was that the reference path was the only path. a Q4 file that must be expanded to f32 before it can run is not a low-memory deployment; it is a research loader that happens to read Q4 files.

why it had to change

three measurements made the change unavoidable. first, the eager path's peak RSS tracked the expanded weights, not the file: ~6.99 GB for Llama Q4_K_M/Q6_K and ~8.83 GB for Qwen2.5-1.5B on this machine (reproduced 2026-08-04, see section 2 of artifacts/benchmark-v03/part3-binary-memory.md). second, the pinned llama.cpp reference ran the same files at 1.08–1.77 GB peak RSS (artifacts/benchmark-v03/bench-summary.json). third, the eager K-quant decode was slower than the model's own Q8_0 run on the same machine by one to two orders of magnitude. none of this is a claim that Q4 is "worse" than Q8, it is a statement about what a runtime does with a file.

the design constraint

the constraint, frozen before implementation (docs/v03-execution-contracts.md, status line): the weights stay packed and mmap-resident, and dequantization happens only at block or tile granularity inside the matmul kernels. no persistent full f32 expansion exists for native-path tensors. the contract also froze the limits of the change:

the eager-f32 path is retained as a selectable reference and never removed;
no silent fallback: every per-tensor execution choice is recorded;
native kernels exist for Q4_K and Q6_K only, Q2_K/Q3_K/Q5_K/Q8_K stay eager-only;
per-tensor workspace is bounded (≤ 9 KiB per thread);
the six semantic hook boundaries (before-layer, after-attention, after-mlp, after-layer, before-logits, after-logits) are guaranteed identical to the reference path, because K-quant models keep the single generic forward path.

Eager expansion vs compressed residency: the reference path expands the whole weight matrix to f32; the v0.3 path keeps packed blocks mmap-resident and runs native K-quant matvec — figure 1, eager expansion vs compressed residency. peak RSS on the v0.3.0 tag build: 6,989,724–8,837,276 KB (eager) vs 844,216–1,266,684 KB (compressed) across Llama-3.2-1B and Qwen2.5-1.5B at Q4_K_M/Q6_K (artifacts/benchmark-v03/part3-binary-memory.md).

Eager expansion vs compressed residency (light theme) — figure 1, eager expansion vs compressed residency. peak RSS on the v0.3.0 tag build: 6,989,724–8,837,276 KB (eager) vs 844,216–1,266,684 KB (compressed) across Llama-3.2-1B and Qwen2.5-1.5B at Q4_K_M/Q6_K (artifacts/benchmark-v03/part3-binary-memory.md).

what the GGUF layout forces

K-quants are not one format. a Q4_K tensor stores 8 blocks per super-block with per-block scales and offsets; Q6_K stores 16 blocks per super-block with int8 scales; a "Q4_K_M" model is a per-tensor mix, the Llama-3.2-1B Q4_K_M file contains 96 Q4_K tensors, 17 Q6_K tensors (attention value + FFN down in selected layers, plus the tied embedding), and 34 F32 tensors for norms and biases (inventory in docs/v03-execution-contracts.md section 5). a native path therefore has to dispatch per tensor, not per file, and Q4_K_M forces both Q4_K and Q6_K kernels to exist.

the loader also has to respect the 2024 dtype-numbering shift (Q2_K=10 … Q6_K=14 in GGUF) , silently misreading a K-tensor type would corrupt every weight without an error (dtype range documented in docs/v03-execution-contracts.md section 3). and the per-tensor reality means "Q4" is never claimed from the file name; the inventory is parsed from the GGUF headers.

Q4_K and Q6_K super-block anatomy: 8 blocks of 32 weights with per-block 2-bit scale factors for Q4_K; 16 blocks of 16 weights with int8 scales for Q6_K — figure 4, Q4_K and Q6_K super-block anatomy (conceptual). dispatch is per tensor, not per file: the llama-3.2-1b Q4_K_M checkpoint mixes 96 Q4_K + 17 Q6_K + 34 F32 tensors (docs/v03-execution-contracts.md section 5).

Q4_K and Q6_K super-block anatomy (light theme) — figure 4, Q4_K and Q6_K super-block anatomy (conceptual). dispatch is per tensor, not per file: the llama-3.2-1b Q4_K_M checkpoint mixes 96 Q4_K + 17 Q6_K + 34 F32 tensors (docs/v03-execution-contracts.md section 5).

the implementation

the new representation (KQuantWeight in src/quant_k.rs) mirrors the existing Q8_0 compressed-resident template: raw block storage backed by the mmap, with checked construction (block alignment, byte length, mapped range). dequantization reuses the already-reference-validated dequant_q4_k/dequant_q6_k functions at block granularity. on top of that:

scalar kernels, the numerical oracle: block dequant + f32 dot, no SIMD.
AVX2 kernels, the same math, vectorized; selected when the AVX2+FMA+F16C feature set is present.
per-tensor dispatch, --k-strategy eager-f32|scalar|x86|auto; unsupported dtypes under an explicit compressed strategy hard-fail naming the tensor unless --k-allow-fallback is given; auto chooses x86 when available and records the decision (contract section 6).
provenance inventory, capture manifests gain a per-tensor execution record: GGUF dtype, resident representation, strategy, kernel, CPU feature requirement, fallback reason, workspace bytes (contract section 10). an artifact now says what actually executed.
embeddings and tied heads, K-quant embedding rows are dequantized per lookup into a small thread buffer and never held resident; the tied LM head of the Llama file routes through the same path (contract sections 5 and 7).

the kernel shape battery in the unit tests is deliberately wide, rows {1,2,8,32} × in {256,512,1536,2048,8960} × out {128,512,2048}, plus zero-scale, negative-min, saturated, and non-aligned edges (contract section 9, gate a).

Anatomy of a K-quant matvec: decode one block into a small per-thread buffer, multiply by the activation slice, accumulate; the whole matrix never becomes f32-resident — figure 2, anatomy of a K-quant matvec. one block is decoded into the [f32; 256] per-thread buffer at a time; per-thread workspace for the compressed path is ≤ 9 KiB and no model-scale f32 buffer is created (docs/v03-execution-contracts.md section 7).

Anatomy of a K-quant matvec (light theme) — figure 2, anatomy of a K-quant matvec. one block is decoded into the [f32; 256] per-thread buffer at a time; per-thread workspace for the compressed path is ≤ 9 KiB and no model-scale f32 buffer is created (docs/v03-execution-contracts.md section 7).

the validation gates

the gates were pre-registered in the contract before implementation and could only be tightened, never loosened to fit results. four are numerical; the fifth is the causal workflow:

gate a, kernel parity: scalar and AVX2 kernels vs the eager dequant-then-gemm oracle, max_abs ≤ 1e-4·scale across the shape battery and edge cases; AVX2 vs scalar within tolerance.
gate b, model parity: compressed vs eager-f32 on the fresh ladder, both families × Q6_K/Q4_K_M; per-layer max_abs ≤ 5e-4·scale, cosine ≥ 1−1e-6, logits ≤ 1e-2 (qwen amended to 2e-2), greedy token sequences identical on the frozen prompt set. a token flip is a failure to investigate, not a threshold to relax.
gate c, golden logits: final-position logits vs the pinned llama.cpp CLI on all six ladder rungs; 100% top-1 agreement on every rung, with the recorded envelopes: llama q8/q6/q4 = max abs diff 0.59/0.81/0.65, cosine 0.9995/0.9989/0.9992; qwen q8/q6/q4 = 0.82/1.36/1.74, cosine 0.9991/0.9975/0.9963 (docs/validation.md v0.3 section). Top-1 agreement is agreement on the argmax, not equality of logits.
gate d, x86 vs scalar: gate a numbers at kernel level, gate b at model level.
causal workflow: capture → compare → patch → frozen verdict on the compressed path, intervention flips 8/8 records and the patch restores every captured tensor bit-exactly (docs/validation.md).

Validation ladder: scalar packed kernel, AVX2 packed kernel, planned model execution, per-layer hidden-state parity, final logits, greedy tokens, pinned llama.cpp reference; capture and intervention hooks attach at the same six semantic sites — figure 3, the validation ladder. kernel parity (gate a) → model parity (gates b/d) → external golden against pinned llama.cpp (gate c), with the six semantic hook sites attached at the same call sites on both paths (docs/v03-execution-contracts.md sections 8–9; docs/validation.md).

Validation ladder (light theme) — figure 3, the validation ladder. kernel parity (gate a) → model parity (gates b/d) → external golden against pinned llama.cpp (gate c), with the six semantic hook sites attached at the same call sites on both paths (docs/v03-execution-contracts.md sections 8–9; docs/validation.md).

the honest part of gate c is its amendment history, which the contract records verbatim. the first drafted bound (max abs ≤ 1e-2 on final logits) misread the existing golden standard, the pilot's own Q8_0 report showed max abs 0.364 against llama.cpp's integer accumulation kernels, so no accumulation-order change could reach 1e-2 on this family. the evidence-based standard became top-1 100%, cosine ≥ 1−1e-3, mean ≤ 0.1, max ≤ 1.0; a second amendment moved the single-element max gate from 0.5 to 1.0 after one extreme element of 128k per sample proved quantizer- and order-sensitive; a third set the final per-family envelopes from the full six-rung run. none of these changed the primary gate , top-1 agreement stayed 100% throughout.

the result

re-derived on 2026-08-04 from the v0.3.0 tag build (full commands and raw values in artifacts/benchmark-v03/part3-binary-memory.md), the same protocol as the release matrix (scripts/bench_v03.sh, 64 tokens, 1 warmup, 3 reps, 8 threads, whole-process peak RSS):

model	strategy	median tok/s	peak RSS
Llama-3.2-1B Q4_K_M	eager-f32	0.98	6,989,724 KB (6.67 GiB)
Llama-3.2-1B Q4_K_M	compressed (x86)	2.01	844,216 KB (0.81 GiB)
Llama-3.2-1B Q6_K	eager-f32	0.98	6,989,252 KB (6.67 GiB)
Llama-3.2-1B Q6_K	compressed (x86)	2.21	1,053,524 KB (1.00 GiB)
Qwen2.5-1.5B Q4_K_M	eager-f32	0.87	8,828,632 KB (8.42 GiB)
Qwen2.5-1.5B Q4_K_M	compressed (x86)	1.66	987,360 KB (0.94 GiB)
Qwen2.5-1.5B Q6_K	eager-f32	0.85	8,837,276 KB (8.43 GiB)
Qwen2.5-1.5B Q6_K	compressed (x86)	1.67	1,266,684 KB (1.21 GiB)

the compressed path cut peak RSS by 6.6×–8.9× versus eager on the same binary, and decode improved 1.9×–2.3× in this session. the release-time matrix recorded similar speedups (1.94×–2.75×) and the resident-weight footprint at file size: 799.6 MB compressed vs ~3.2 GB expanded for Llama Q4_K_M, 1457.6 MB vs ~5.83 GB for Qwen Q6_K (artifacts/benchmark-v03/bench-summary.json, docs/v03-execution-contracts.md section 5). the pinned llama.cpp reference ran the same files at 1.08–1.77 GB peak RSS, Ember's compressed path now sits in the same neighborhood, which is where a quantized model should live.

the eager arms of this reproduction match the frozen pre-v0.3 baseline within 0.4% (6,989,724 KB vs the recorded 6,965 MB for Llama Q4_K_M), which is the sanity check that the comparison is measuring the representation change, not machine drift (artifacts/benchmark-v03/part3-binary-memory.md section 2).

binary cost: measured growth from v0.2.0 to v0.3.0 is 820,480 b (~801 KiB) on the current toolchain (rustc 1.95.0). an internal estimate of "~190 KB" does not reproduce under this toolchain; the measured value is recorded in the evidence file. the release added roughly 0.8 MiB of kernels, dispatch, and provenance to the executable, against 6–8 GiB of memory it removed.

what still does not work

Q2_K/Q3_K/Q5_K/Q8_K tensors have no native kernel in v0.3; under an explicit compressed strategy they hard-fail unless fallback is allowed.
K-quant models are slow to load (13–16 s on this machine) and ineligible for the Q8 workspace fast path; the scalar kernels are reference-only (~0.3–0.4 tps).
the extraction tokenization path carries a pre-existing byte-offset limitation with non-ASCII prompts (the golden ladder uses English prompts).
Gemma 4's numerical gap is unchanged and out of v0.3 scope; "supported" still does not mean numerically validated for every row.
llama.cpp remains roughly an order of magnitude faster on decode; Ember is not a throughput competitor, and this post makes no parity claim.

what this unlocked

keeping weights packed fixed the memory problem, but decode still ran the generic hooked path with per-token dispatch rediscovery and per-token scratch. the next release, v0.4, plans that execution once per model, and the plan interpreter runs on exactly this compressed substrate. the memory the native path saved is the memory the plan-driven runtime then refuses to reallocate.