← voidwest    research notes

so, you wanna build an inference engine?

2026-10-06 · GPT-2 small · about 700 lines of Rust · 19 todo!()s, 17 tests

You have probably seen the inference hype and wondered what those people actually do all day. A good part of it happens down in the systems-programming mines.

The fastest way I know to understand that work is to build an engine yourself. So I took the first version of my own engine, cut it down to the core, and removed the important parts. You write them back.

the short version

ember-mini is an educational Rust repo: a GPT-2 inference engine of about 700 lines, with 19 functions left as todo!(). Each one keeps its signature and has hints in the comments, and 17 tests tell you when it works. The full code is on the solution branch. It is not a library, and it is not meant for production.

what an inference engine is

You know models, weights and tokens. Between a model file and the tokens you read, there is a fair amount of machinery. That machinery is the inference engine, and its core fits in a few lines:

load weights
tokenize input
loop:
    run the transformer on the newest token
    get logits (one score for every token in the vocabulary)
    pick the next token
    append it

Put simply, it is where the math behind a model actually runs and turns into the outputs we all know.

The inference loop: the text “The capital of France is” becomes the token ids 464, 3139, 286, 4881 and 318, GPT-2 runs on them, and softmax gives next-token probabilities. The token “ the” wins with 6.1%; “ Paris” is sixth with 3.1%. An arrow carries the picked token back to the ids. The inference loop: the text “The capital of France is” becomes the token ids 464, 3139, 286, 4881 and 318, GPT-2 runs on them, and softmax gives next-token probabilities. The token “ the” wins with 6.1%; “ Paris” is sixth with 3.1%. An arrow carries the picked token back to the ids.
Real numbers from ember-mini on GPT-2 small with Q8_0 weights.

For The capital of France is, GPT-2 small gives " the" 6.1% and " Paris" only 3.1%, in sixth place. Greedy decoding picks " the" and writes “the capital of France, and it is the capital of France.”

GPT-2 small is not smart, and that is not the point. It is small enough to hold in your head, and every part of a modern engine is already in it.

why build one yourself

You can read about transformers for a long time and still not feel how one runs. Write the loop yourself and you get concrete parts you can take apart and follow step by step. These only clicked for me after I wrote it the first time:

the repo

ember-mini is my original GPT-2 engine, stripped down so that you can code it back together. It covers the fundamentals only, with no abstractions to hide behind: every operation is a plain function on f32 slices, and the only dependencies are tokenizers and half.

Every missing function keeps its signature. The comments say what to compute and what usually goes wrong, but not how to write it.

The softmax exercise on the main branch: the signature, comment hints that explain the formula and the overflow problem, and todo!("softmax") where the code goes. Next to it, the solution from the solution branch. The softmax exercise on the main branch: the signature, comment hints that explain the formula and the overflow problem, and todo!("softmax") where the code goes. Next to it, the solution from the solution branch.
One of the 19 exercises on the main branch, next to its solution.

There are six exercises, in order: the math, bytes to numbers, the KV cache, the GGUF reader, the forward pass and the generation loop. Each one has its own test command.

A checklist of the six exercises: ops.rs, quant.rs, kv_cache.rs, loader.rs, model.rs and main.rs, each with what it covers and its test command, from cargo test ops to cargo run. A checklist of the six exercises: ops.rs, quant.rs, kv_cache.rs, loader.rs, model.rs and main.rs, each with what it covers and its test command, from cargo test ops to cargo run.

You start at 0 of 17 passing tests and stop when GPT-2 talks.

Two terminal excerpts. On the main branch, cargo test reports 0 passed and 17 failed with not yet implemented: dot. After the exercises, 17 tests pass and cargo run continues “The capital of France is” with GPT-2 text. Two terminal excerpts. On the main branch, cargo test reports 0 passed and 17 failed with not yet implemented: dot. After the exercises, 17 tests pass and cargo run continues “The capital of France is” with GPT-2 text.
Real output from the main branch at the start, and from the solution branch.

what each line of the loop hides

load weights

GGUF, the llama.cpp file format, has four parts: a header, metadata, tensor infos, then the tensor data. The GPT-2 file has 1.7 MB of metadata before the first weight, because the whole tokenizer vocabulary lives there.

The layout of gpt2.Q8_0.gguf: a header with version 3, 149 tensors and 17 keys; metadata such as gpt2.block_count = 12; tensor infos such as blk.0.attn_qkv.weight with dimensions [768, 2304]; and 175.9 MB of tensor data from byte 1,775,072. Below, one Q8_0 block: a 2-byte scale and 32 one-byte values. The layout of gpt2.Q8_0.gguf: a header with version 3, 149 tensors and 17 keys; metadata such as gpt2.block_count = 12; tensor infos such as blk.0.attn_qkv.weight with dimensions [768, 2304]; and 175.9 MB of tensor data from byte 1,775,072. Below, one Q8_0 block: a 2-byte scale and 32 one-byte values.
Values read from gpt2.Q8_0.gguf.

Places where you can blow up:

run the transformer

One forward pass: take the token's row of the token embedding plus its position's row of the position embedding, and you have 768 numbers. Run 12 blocks of x = x + attention(norm(x)) then x = x + mlp(norm(x)). A final norm and the output layer give 50,257 logits. In ember-mini the whole forward pass is 51 lines.

One forward pass: the embedding feeds the residual stream x, which runs through a layernorm and attention block and a layernorm and MLP block, each adding its result back to x, repeated 12 times, then a final layernorm and the output layer give 50,257 scores. One forward pass: the embedding feeds the residual stream x, which runs through a layernorm and attention block and a layernorm and MLP block, each adding its result back to x, repeated 12 times, then a final layernorm and the output layer give 50,257 scores.

get logits and pick the next token

GPT-2's raw logits for that prompt all sit near −100. In f32, exp(−100.7) is 1.8e−44: a subnormal number with about 4 bits of precision left. Below −104, exp returns exactly 0. So a naive softmax on GPT-2 is quietly broken. Subtract the maximum first. The other direction breaks too: exp(1000) is infinity, and infinity divided by infinity is NaN.

ember-mini picks with argmax: greedy decoding, no sampling, no temperature.

append and repeat

Without a cache, every new token runs the whole text again, so 6 tokens cost 21 token passes. With a KV cache, the model stores the key and value of each token once and reads them back later, so 6 tokens cost 6 passes. For GPT-2 small at its full context of 1,024 tokens, the cache is 72 MiB.

Two grids of six steps over the tokens of “The capital of France is the”. Without a cache, every step runs all tokens so far: 21 passes. With a KV cache, each step runs only the new token and reads the earlier keys and values: 6 passes. Two grids of six steps over the tokens of “The capital of France is the”. Without a cache, every step runs all tokens so far: 21 passes. With a KV cache, each step runs only the new token and reads the earlier keys and values: 6 passes.

what I learned the hard way

ember-mini makes about 9 tokens per second on one thread, because its dot product is a plain scalar loop. Faster engines hit a different wall: memory. Decode reads every weight once per token and does only a couple of operations with each one. Ember runs Llama-3.2-1B at Q8_0 at about 58 tokens per second on the CPU, which is roughly 1.3 GB × 58 ≈ 75 GB/s of weights moving through it. At that point the question is no longer how fast your multiply is. It is how fast you can move bytes.

A lot went wrong while I built Ember. Some examples:

A faster kernel made the engine slower. I wrote a 4-column Q4_K tile, where each activation load serves four weight columns. In a single-thread kernel test it was 25–30% faster. In the full engine at 4 threads, decode went from 68.2 to 64.9 tokens per second, so I rejected it.

The biggest win had nothing to do with the kernel. The K-quant matrix-vector products ran on a Rayon fork/join tree, efficiency cores included, with one fork/join per projection. I moved them to a small spin-waiting team on the performance cores, with the same serial kernel, bit for bit. Q4_K_M went from 61.4 to 68.2 tokens per second at 4 threads, and from about 70 to about 87 at 8.

Two changes to Ember's Q4_K decode. Attempt 1, a 4-column tile: 25 to 30% faster in a one-thread kernel test, but the whole engine at 4 threads went from 68.2 to 64.9 tokens per second, so it was rejected. Attempt 2, the same kernel with better scheduling: 61.4 to 68.2 tokens per second at 4 threads and about 70 to 87 at 8, so it shipped. Two changes to Ember's Q4_K decode. Attempt 1, a 4-column tile: 25 to 30% faster in a one-thread kernel test, but the whole engine at 4 threads went from 68.2 to 64.9 tokens per second, so it was rejected. Attempt 2, the same kernel with better scheduling: 61.4 to 68.2 tokens per second at 4 threads and about 70 to 87 at 8, so it shipped.
Llama-3.2-1B Q4_K_M decode on an M1 Pro.

Prefill and decode want different things. The spin-waiting team is good for decode, which is many tiny tasks. When I put prefill's large matrix tiles on it, prefill got 15–25% slower at 8 threads than with Rayon's work stealing.

Layout is not a detail. An interleaved weight layout for the MLP was faster in every standalone test. In the full process it made every operation slower, even the output layer, which it did not touch. I never found out why, so it did not ship.

Your baseline can lie to you. llama-bench uses Metal by default. My “CPU against CPU” comparison read 112 against 66 tokens per second until I added -ngl 0.

So can your laptop. Under normal desktop load, 8-thread runs dropped 20–30% at random. Now every A/B comparison is interleaved and repeated.

Correct and fast are separate problems. Ember promises bit-identical replay, because it is a research tool, so a change that moved RoPE's sine and cosine by one ulp was reverted, even though nobody would ever see it in the text. And my own GPT-2 code in Ember uses the exact erf GELU, while GPT-2 was trained with the tanh approximation. ember-mini uses tanh, and with F32 weights it matches llama.cpp token for token for 100 tokens on four prompts. With the Q8_0 file, llama.cpp also squeezes the activations to 8 bits, and the greedy text drifts after a few tokens. The same model does not mean the same numbers.

Where that leaves Ember: llama.cpp is still ahead on the CPU. On Llama-3.2-1B, an M1 Pro and 4 threads in early October, it was 57.8 against 65.8 tokens per second at Q8_0, and 67.5 against 88.6 at Q4_K_M. Profile before you touch anything, measure the whole engine, and do not trust a number you measured once. A little paranoia about measurement goes a long way.

where it gets scary

ember-mini is one model, one format and one thread. Real engines grow in every direction at once:

Ember is about 140,000 lines of Rust today, across src/ and the ember-core crate. Not all of it is inference: there is a GUI, research tooling and audio too. This is where the small educational loop turns into a lot of lines of suffering, with some help from Claude.

go build the tiny version

Building an engine is worth it even if you never ship one. It gives you a feel for ML systems that papers and high-level libraries do not: where the time goes, where the memory goes, and why the same model can give you different text.

Clone ember-mini, download GPT-2 (the README has the command and the checksum), and run cargo test ops.

If you have ever wondered what happens between a model file and “hello world”, build the tiny version once. And if you got this far: thank you for reading.

where the numbers come from

related work on this site