v0.4: plan-driven decode without sacrificing hooks

most of the work in decoding one token is deciding what to do, not doing it. v0.4 moves that decision out of the loop, without moving the hooks with it.

the problem: the loop re-decides everything

v0.3 left decode on the generic hooked forward path: for every token, for every layer, the same work was rediscovered. which tensor does this projection read? which kernel handles its dtype? where does the output go? does any hook target this stage? the v0.4 contract (docs/v04-execution-contract.md) froze the reference sequence before touching it , embedding, per-layer RMSNorm → q/k/v → RoPE → KV store → attention → output projection → residual → MLP → residual → final norm → head (contract section 3), and then named the waste (section 9): per-token shape/dispatch rediscovery, dynamic hook checks against a registry, and per-token scratch allocation.

none of that work changes between tokens. the model does not get a new architecture at step 42. the only thing that changes per token is the token itself.

the design constraint

the v0.4 release could have taken the usual route: keep the generic path, add a fast path beside it, and hope the fast path matches the slow one. the contract forbade that in two ways. first, the out-of-scope list includes no second hidden runtime that bypasses hooks, an optimized path that researchers cannot hook is not an optimization, it is a different program. second, kernels are not duplicated: the v0.3 scalar and AVX2 Q4_K/Q6_K matvecs are the only matvec implementations, and the planned path resolves a kernel per tensor through the same resolve_kernel function the legacy path uses, asserted equal by tests (contract decision D6).

so the constraint, stated once: plan the work once per model; execute it per token; keep the six semantic hook sites observing the exact same tensors at the exact same call sites.

the implementation

• immutable execution plan. built once, immediately after load, by the model backend that owns the tensors. it stores stable indices and validated metadata, never raw pointers, so weight access is structural and cannot dangle (contract D2). the plan carries the tensor table, per-layer operation lists, scratch layout, KV layout, hook-site resolution, dispatch decisions, and provenance (contract section 10, schema v04-plan/1). ember inspect-plan prints it; benchmarked runs write execution-plan.json.
• three execution concepts. --execution reference keeps the v0.3 generic hooked path as the readable oracle; planned runs the identical operation sequence through the plan interpreter with no fusion; planned-fused applies the frozen fusion set (contract section 9). the three stay separable for validation.
• reusable scratch arena. one aligned allocation per decode session, sized by the plan, 64-byte aligned, with regions proven non-overlapping by lifetime and an arena report listing every overlap and its proof (contract section 11). the steady-state token loop performs zero heap allocations.
• frozen fusion set F1–F5. RMSNorm+projection; residual+RMSNorm; QKV orchestration; RoPE inside attention; output projection accumulating into the residual destination (contract section 6). each fusion names what it eliminates, why that is safe, and the parity test it must pass.
• hook-driven de-fusion. F5 eliminates the standalone o tensor , which is the after_attention hook site. if that hook is active on a layer, the planner selects the unfused route for that layer, and the reason is recorded in the plan and in provenance (contract sections 6–7). a hook must never silently observe a different semantic tensor because fusion changed the graph.
• column-parallel matvec. large single-row decode matvecs split their output dimension across the rayon pool, each column accumulating identically to the serial kernel , bit-identical by test (gate a), not approximately equal.
• operator profiling. bench-decode --profile-operators records per-operator timing on the planned path, so the next optimization is chosen from a table, not a guess.

when the CPU became an unreliable witness

the measured speedup was real. getting the measurement to be real took six separate failures, and they are part of the story because each one is a place a faster runtime could have lied to its own authors.

1. thermal noise made fused execution appear slower. this machine runs a powersave governor, recorded in every benchmark's run_metadata (artifacts/benchmark-v03/bench-summary.json), so absolute tokens/second is thermally variable, and early fused runs landed on a throttled window and looked like a regression. the fix was protocol, not code: back-to-back arms per model so all execution modes share the machine state, medians over 5 measured repetitions, no cross-session comparisons (artifacts/benchmark-v04/2026-08-04/SUMMARY.md).
2. a benchmark mode accidentally executed the reference path. the changelog records it plainly: "planned-fused silently ran the reference path until planned_decode_eligible accepted the mode" (CHANGELOG.md, v0.4 fixed). a flag that names an execution mode but does not gate it is a flag that can lie.
3. fusion tests passed without exercising the intended route. reconstructed from the same changelog entry and the parity tests (tests/k_parity.rs): the model-level gate b tests for planned/fused exist and pass, but a fused run requested through the CLI could fall through to the reference path before the fix, so a test suite can be green while the named mode never runs. the response was the eligibility gate plus parity tests that set the execution mode explicitly and assert the greedy tokens (k_parity.rs, v04_planned_matches_reference_real_model).
4. an allocation gate tested a shadowed trait path. the gate e test comment is the evidence: "trait path (ForwardModel) so v0.4 execution-mode dispatch runs; the inherent Llama method would shadow it" (tests/k_parity.rs). an inherent method call can bypass the plan interpreter entirely; the allocation test only meant something once it routed through the trait that actually dispatches.
5. pinned-core runs distorted performance. pinning to four physical cores understates the column-parallel matvec, which is the performance lever. the final protocol runs the full 8-thread machine with no taskset, and the summary says so explicitly (artifacts/benchmark-v04/2026-08-04/SUMMARY.md).
6. attention deduplication later exposed additional false allocation behavior. after the release, a code-redundancy pass merged the triplicated attention kernels and removed per-task Vec allocations "that could surface as a spurious allocation on a rayon-stealing thread" (commit 624a216, main, 2026-08-04, after v0.4.0, hence "later"). the gate e numbers were right; the accounting had not yet caught every allocation site that only a warm thread pool visits.

the common thread: every one of these is a case where the harness, not the engine, was the thing being benchmarked. a faster runtime is not trustworthy unless the benchmark harness and the semantic execution route are also validated.

the validated outcome

gates A–G were pre-registered in the contract before implementation (contract section 13) and could only be tightened:

• gate a (kernel/op parity), planned and fused outputs vs the independent reference path, max_abs ≤ 1e-4, over the standard shape battery; the column-parallel matvec is bit-identical to the serial kernel (same per-column accumulation order), and the planned dispatch kernel equals the legacy dynamic dispatch (debug assert + tests).
• gate b (model parity), reference/planned/planned-fused produce identical greedy token sequences on the six frozen prompts (English + Arabic) within the frozen logit envelopes, on all four primary combinations (tests/k_parity.rs, env-gated real-model test).
• gate c (hooks), the six semantic sites fire at the same call sites with the same stages/layers/shapes on the planned path; inactive hooks are bit-identical to disabled; a zero-layer-output intervention lands identically; planned-fused de-fuses F5 when after_attention is active so the hook sees the materialized o tensor.
• gate d (memory), packed K-quant weights stay mmap-resident on the planned path; peak RSS reference 844,840 KB vs planned 847,952 KB on Llama-3.2-1B Q4_K_M at release (+0.37%), the named reusable arena being the whole difference.
• gate e (allocation), after warmup, the planned decode loop performs at most 3 heap allocations per token, all of them the documented logits tensor materialization (shape + strides + data), verified on a real model with the counting allocator (src/llama.rs, planned_decode_is_zero_steady_state_allocation).
• gate f (performance), the matrix below; floor was ≥1.75× on ≥3/4 combinations, achieved on 4/4.
• gate g (external), the v0.3 golden ladder is preserved; planned and fused paths reproduce the reference greedy outputs within the frozen envelopes, so the 100% top-1 agreement with pinned llama.cpp carries over by transitivity.

the release-time matrix (artifacts/benchmark-v04/2026-08-04/SUMMARY.md; 64-token single-token decode, 1 warmup, 5 reps, medians, 8 threads, full machine):

that is the "roughly 2.0–2.7×" claim, stated precisely: vs the v0.3 reference path on the same binary, same protocol, four primary combinations. it is not a claim against llama.cpp, the v0.4 matrix records no llama.cpp arm, and the contract's final framing repeats that no competitive parity is claimed unless evidence unexpectedly demonstrates it (contract section 19).

reproduced on 2026-08-04 from the v0.4.0 tag build (binary sha 23322cd3…, reproducible, the rebuild matched; full record in artifacts/benchmark-v04/2026-08-04/part4-reproduction.md), all four primary combinations with the same protocol:

peak RSS across the same arms: Llama Q4_K_M 843,952 → 847,284 KB (+0.39%), Llama Q6_K 1,053,944 → 1,056,984 KB (+0.29%), Qwen Q4_K_M 986,988 → 992,404 KB (+0.55%), Qwen Q6_K 1,267,412 → 1,271,992 KB (+0.42%), effectively flat, confirming gate d on all four combinations. the reproduction's absolute tps is higher than the release matrix on the same machine (machine state differs), but the shape is identical: all four combinations clear the gate f floor (≥1.75× planned on ≥3/4), with planned ratios 1.79×–2.19×. deterministic semantic output is not timing reproducibility, identical tokens and identical hidden states do not imply identical wall-clock times, which is exactly what the unreliable-witness section is about.

what still does not work

• fused mode must de-fuse per layer whenever a hook needs a fused-away tensor; fusion is a privilege granted by the hook set, not a default.
• the v0.3 Q8_0 native fast path keeps precedence and is never rerouted through the plan, the plan's speedup applies to the four K-quant combinations, not to every path.
• two of the six witness failures (fusion-route and shadowed-trait) are reconstructed from tests, the changelog, and git history rather than a written post-mortem; the reconstruction is marked in the text and cites its evidence.
• the scalar kernels remain reference-only; there is no ARM/NEON path; attention is still single-query at decode.
• Gemma 4 remains numerically untrusted and out of the planned scope; "supported" still does not mean golden-validated.

what this unlocked

the v0.5 experiment machinery rides the plan interpreter: capture and intervention specs resolve to the plan's hook-site resolution, and the frozen fusion set is what the v0.5 de-fusion policy reasons about. the token is now planned once; the next release makes the plan itself an artifact that can be verified, compared, and reproduced by someone who never reads Rust.