v0.1: an inference engine that can explain what it did

extracting a tensor is the easy part. knowing what it means, which layer, which position, which execution route, which kernel, is the engineering.

the problem: probes need hidden states with known origins

Ember's reason to exist is a research question about Arabic morphology: where in a transformer are roots and patterns represented, and why does representational availability sometimes fail to produce generation behavior? every form of that question starts with the same act: take a tensor out of a running model and say something about it. the act is only meaningful if the tensor's origin is known exactly, which model, which layer, which token position, which execution path, which kernel, and what came before it.

a hidden-state dump without that context is not evidence; it is a number. the central claim of this post is the v0.1 release's: a hidden-state dump is not useful unless its exact semantic and numerical origin is known. v0.1 built the engine that makes that origin explicit.

the naive solution: dump whatever the model computes

the naive approach is to instrument at the end: run the model, intercept the outputs, write the tensors to disk, and let the researcher sort out what they are. it produces files quickly and meaning slowly. three things are lost:

• tensor identity, which layer produced this buffer? was it pre- or post-residual? which norm fed it?
• position, a prompt has many tokens; a decode run has many steps. a dump that does not say which position a row belongs to cannot be aligned to anything.
• route, which kernel executed this projection, and from what representation (packed Q8 blocks or expanded f32)? two runs can produce the same filename and different numerics.

why it failed: the loader already had the answers

the v0.1 loader (CHANGELOG.md, v0.1: GGUF v3 loading for F32/F16/BF16/Q8_0; mmap-backed Q8_0 weights; SIMD decode kernels; tiled prefill) knew everything a researcher needs: the dtype, the shape, the layout, the file the tensor came from. the problem was that this knowledge stopped at the loader boundary. the extraction path had to re-discover it, and the GGUF format makes that re-discovery genuinely hard:

• f32/f16 linear and embedding tensors are stored [in, out] with the first dim contiguous; linears need a row-major fixup and embeddings a dims-swap, and a tied LM head needs a real transpose, get any of these wrong and every downstream tensor is silently corrupted (repository gotchas, documented against the loader in docs/architecture.md and the loader's layout helpers).
• tokenizer offsets are bytes, not characters. an extraction that assumes character offsets breaks on the first Arabic prompt, which, for this project, is every prompt.
• dtype handling is versioned (the 2024 K-quant renumbering); a loader that assumes the old numbers misreads every K tensor without an error.

the consequence: obtaining a tensor and knowing what it means are different problems, and the second one is the actual engineering.

the design: a reference path that is readable by construction

v0.1 kept a deliberately unoptimized reference execution path as the correctness oracle alongside the packed Q8 fast path: same model, same inputs, generic kernels, explicit intermediates. the two paths must agree on greedy output within the recorded envelopes (controlled a/b regression tests, trace-fingerprint and hidden-state tests , CHANGELOG.md v0.1). the reference path is not a fallback; it is the definition of what the fast path is allowed to change.

on top of that: KV-cached generation with deterministic greedy decoding, structured operation tracing, and hidden-state extraction (CHANGELOG.md v0.1). "deterministic" here is semantic determinism: temperature 0 with a fixed seed produces the same tokens and the same hidden states on the same binary, not necessarily the same wall-clock times.

what the engine says it did

the trace is a per-operation journal. on 2026-08-04, the release binary traced a 6-token prefill on Llama-3.2-1B Q8_0 into 275 events, each carrying the op name and kind, the layer index, input/output shapes and byte counts, duration, and an estimated FLOP count (artifacts/benchmark-v01/part1-legibility.md section 2):

{"duration_ns": 5896, "estimated_flops": 0, "input_bytes": 49152,
 "input_shape": [6, 2048], "layer": 18446744073709551615,
 "name": "embedding", "op_kind": "Embedding", "output_bytes": 49152}

every layer, projection, norm, and attention op appears with its shapes and bytes; the decode phase is traced per step. the trace is the engine saying what it did, the origin of every tensor it produced, in order.

the extraction path writes that origin next to the tensors. a direct-mode extraction on the same model produced a run directory with 16 per-layer shards (one 2048-dim f32 row per layer, the final prompt position) plus a manifest that records the model architecture, tokenizer, backend, sample hashes, config hash, and checksums (artifacts/benchmark-v01/part1-legibility.md section 3). the artifact is self-describing: schema_version, layout, tensor_contract, dtype, output_format, and the full model/tokenizer records are in the manifest, not assumed.

the validation ladder

v0.1 also established the vocabulary the whole series uses (docs/validation.md): smoke (the command ran) < golden logits (output compared against a trusted reference) < activation-reference checks (internal states compared) < probes (decodability) < interventions (causal use). each rung is a different claim. v0.1 shipped golden-logit and trace-fingerprint tests for the paths it supported; the ladder is what makes "supported" distinct from "numerically validated on every row".

what still does not work

• at v0.1, activation-reference checks were pending for several rows; "supported" meant an execution path exists, not that every architecture was golden-validated.
• Qwen2.5 support was explicitly experimental; Gemma 4 was dense text-only, and its numerical gap remains untrusted to this day.
• the --arch default silently fell back to gpt2 when a model needed an explicit architecture, a trap that shipped and was only fixed later with --arch auto.
• the tokenizer's byte offsets were the source of repeated non-ASCII bugs (fixed in v0.5); the extraction tokenization path still carried a byte-offset limitation at v0.1.
• llama.cpp remains the external performance reference; v0.1 made no throughput claim, and neither does this post. the v0.1 release page calls its own numbers "host-specific validation numbers... rather than a leaderboard".

what this unlocked

knowing where a tensor came from is observation. the next step is changing it and observing the consequence, which requires the capture, patch, and restore machinery of v0.2.