ember v0.1

notes on a small CPU-first Rust inference engine whose execution path is meant to remain visible.

source experiment guide hook contract MIT · commit 663a979

a runtime for work that needs an audit trail

ember is for the part of inference research where the representation boundary, execution order, and exact intervention point need to be accounted for.

it loads GGUF models, runs prefill and KV-cached decode on CPU, keeps Q8 weights compressed, exposes structured tracing and hidden states, and retains generic kernels as correctness oracles beside optimized paths.

llama.cpp remains the external performance reference. ember keeps a generic correctness path beside optimized kernels so a research change can be localized and checked.

the experiment boundary in v0.1

the release fixes the scope at one observer and one intervention. both pass through the same narrow, statically compiled Rust hook lifecycle. they show what the boundary can support without turning it into a general plugin platform.

built-in 01 · observation-only

activation-stats

records L2 norms, absolute maxima, and lightweight fingerprints at every approved activation boundary. the result is ordered JSON you can diff, plot, or inspect with jq.

ember --arch qwen3 \
  --model Qwen3-0.6B-Q8_0.gguf \
  --tokenizer tokenizer-qwen3.json \
  --prompt "The capital of France is" \
  --max-tokens 4 --temperature 0 \
  --activation-stats stats.json

built-in 02 · mutates execution

zero-layer-output

chooses a layer and zeros its attention, MLP, or completed layer output during prefill and decode. ember counts every intervention and makes the modified run obvious on stderr.

ember --arch gemma4 \
  --model gemma-4-E2B-it.Q8_0.gguf \
  --tokenizer tokenizer-gemma4.json \
  --prompt "The capital of France is" \
  --max-tokens 4 --temperature 0 \
  --zero-layer-output 4:attention

the artifact is the interface

observation records carry phase, semantic stage, layer, token range, shape, dtype, norm, maximum, and fingerprint. no token buffers or model metadata are cloned per hook.

the same generated text still lands on stdout. experiment notices stay on stderr. run manifests identify whether an experiment modified execution.

experiment guide hook contract

activation-stats.json

{
  "experiment": "activation-stats",
  "observation_only": true,
  "records": [{
    "phase": "prefill",
    "stage": "after_attention",
    "layer_index": 4,
    "sequence_length": 5,
    "shape": [5, 1024],
    "dtype": "f32",
    "l2_norm": 6.117406,
    "abs_max": 0.782913,
    "fingerprint": 152836286671...
  }]
}

validation record

the experiment hooks were admitted only after the unmodified path cleared real-model parity, artifact, allocation, optimized-code, and controlled A/B checks.

check	surface	result
real-model parity	Qwen and Gemma logits; three-family generation	matched frozen artifacts
inspection parity	Gemma layer dumps and normalized trace fingerprints	byte-identical
allocation behavior	warmed no-experiment layer execution	no per-layer growth
optimized code	representative LLaMA block	disabled dispatch erased
controlled A/B	pinned prefill and single-token decode	inside measurement noise

release measurements

these are host-specific validation numbers, included to make the release state inspectable rather than to claim a leaderboard.

release workload	decode	prefill	read
Qwen3 0.6B Q8_0	40.88 tok/s	52 tokens · 489 ms	local i5-1135G7 validation host
LLaMA 3.2 1B Q8_0	33.10 tok/s	53 tokens · 943 ms	local i5-1135G7 validation host
Gemma 4 E2B Q8_0	10.05 tok/s	52 tokens · 2,006 ms	local i5-1135G7 validation host

host-specific release validation, not a cross-engine speed claim. llama.cpp remains the external performance reference.

support boundary

surface	included	boundary
models	GPT-2, LLaMA, Qwen3, dense text-only Gemma 4	Qwen2.5 remains experimental
GGUF tensors	F32, F16, BF16, Q8_0	no K-quants
inspectability	structured tracing, value fingerprints, hidden-state extraction	active experiments do not enter dump/probe surfaces yet
Q8 execution	mmap weights, scalar/AVX2/AVX-512, tiled and packed paths	generic path stays available as oracle

non-goals

dynamic .so or .dll plugins
WASM or Python hooks
runtime experiment discovery
multiple concurrent experiments
configuration DSLs or registries
async hooks or broad event buses
arbitrary model-weight mutation
a claim to production inference breadth

release coordinates

tag: v0.1.0
commit: 663a979
license: MIT
toolchain: Rust 1.92
guide: experiments
reference: hook contract
changes: changelog
source: repository

experiment API: unstable in v0.1 · linux CPU validation