the ecosystem already has llama.cpp. Ember exists because a model that runs is not the same as a model you can inspect, intervene on, and reproduce. this series follows that distinction release by release, failures included.
if your goal is tokens per second, the honest answer to "why build another inference engine" is that you shouldn't. llama.cpp exists, it is fast, and it is the reference for a reason. this series is about a different goal.
Ember started as a means to a research end. the question was about Arabic morphology: how and where do neural language models represent non-concatenative morphology, roots and patterns, and why does representational availability sometimes fail to produce generation behavior? answering it requires hidden states at chosen layers and token positions, the ability to change those states and observe the consequence, and a procedure repeatable enough that a second researcher can run it and trust the result.
the tools that move tokens through a model are excellent at moving tokens. they are not built to answer "what was this tensor, at this layer, at this position, at this moment, and how do i change it, and prove the change mattered?"
llama.cpp is the external reference for Ember, for correctness and for performance. the
golden-logit ladder validates Ember's output against a pinned llama.cpp revision
(scripts/validate_golden_ladder.sh; reference harness tools/logits_dump.c),
and the benchmark artifacts record llama.cpp's numbers on the same files
(artifacts/benchmark-v03/llama-3.2-1b-q4_k_m.llama-bench.json).
but the external reference has a different job. llama.cpp is engineered for breadth and throughput; Ember is engineered for the inverse property: an execution path that is deliberately readable, where every operator, tensor, and hook site is explicit, plus instruments for capturing, patching, comparing, and restoring states with provenance attached. that is not a critique of llama.cpp. it is a division of labor: llama.cpp is the oracle, and Ember validates its own path against it.
Ember does not claim comparable breadth or throughput, and this series will not argue otherwise. the reason to read on is not speed; it is that the two runtimes answer different questions.
three constraints, fixed from the start:
these constraints cost performance, and the series will show exactly how much, in what places, and what was done to reduce the cost without giving the constraints up.
| release | core capability | the question it answered |
|---|---|---|
| v0.1 | inspectable CPU inference over GGUF: prefill, decode, KV cache, tracing, hidden-state extraction, deterministic benchmarks | can inference be made legible enough for model-internals research? |
| v0.2 | activation capture, intervention hooks, patching between runs, exact restoration, structured artifacts with provenance | can Ember support causal experiments rather than only descriptive observation? |
| v0.3 | native Q4_K/Q6_K execution: packed mmap-resident weights, scalar + AVX2 kernels, per-tensor dispatch, external parity | what is the point of a Q4 model if the runtime expands it back to f32? |
| v0.4 | plan-driven decode: immutable per-model execution plans, scratch arenas, a frozen fusion set with hook-driven de-fusion, column-parallel matvec, profiling | can execution be planned and optimized without hiding the tensors researchers need to inspect? |
| v0.5.1 | reproducible experiment bundles: strict specs, byte-exact Arabic token selection, semantic hooks, interventions, offline verification, comparison, reproduction | can a researcher who did not write Ember run and verify a model-internals experiment without touching Rust? |
one number per release, so the shape of the series is visible up front. v0.3 took the resident
memory of a quantized model from gigabytes of expanded f32 down to roughly the size of the file:
6,989,724 KB → 844,216 KB peak RSS on Llama-3.2-1B Q4_K_M
(artifacts/benchmark-v03/part3-binary-memory.md). v0.4 more than doubled
reference-path decode throughput on the four primary model/precision combinations, roughly
2.0–2.7×, with peak RSS effectively flat
(artifacts/benchmark-v04/2026-08-04/SUMMARY.md). v0.5 made the
whole experiment pipeline deterministic and verifiable offline. the caveats attached to each number
are part of the story, not footnotes.
"supported" in Ember's model tables means an execution path exists. it does not mean the row is numerically validated. the distinction is the ladder: smoke (the command ran), golden logits (output compared against a trusted reference), activation-reference checks (internal states compared), probes (decodability), interventions (causal use). each rung is a different claim, and the series will not slide between them.
current state, in one breath: the llama and Qwen2.5 family rows have golden-logit agreement with the pinned llama.cpp reference (100% top-1 across the six ladder rungs, agreement on the argmax, not equality of the logits, with envelope max abs differences up to 1.74 (Qwen q4 rung) recorded); Qwen3 rows are promising but golden validation is pending; Gemma 4 loads but is numerically untrusted (single-token cosine ~0.87 against the reference and eroding across layers); activation-reference checks are pending for several rows. deterministic semantic output is not timing reproducibility: identical tokens and identical hidden states do not mean identical wall-clock times.
the fastest way to see the constraints in action is the memory story. a Q4 file that arrives in memory as f32 has, in a precise sense, been decompressed by the runtime that was supposed to run it. the next article is v0.3: quantized on disk, quantized in memory, and the validation ladder that kept the compressed path honest.