You have probably seen the inference hype and wondered what those people actually do all day. A good part of it happens down in the systems-programming mines.
The fastest way I know to understand that work is to build an engine yourself. So I took the first version of my own engine, cut it down to the core, and removed the important parts. You write them back.
ember-mini is an educational Rust repo: a GPT-2 inference engine of about 700 lines, with 19 functions left as todo!(). Each one keeps its signature and has hints in the comments, and 17 tests tell you when it works. The full code is on the solution branch. It is not a library, and it is not meant for production.
You know models, weights and tokens. Between a model file and the tokens you read, there is a fair amount of machinery. That machinery is the inference engine, and its core fits in a few lines:
load weights
tokenize input
loop:
run the transformer on the newest token
get logits (one score for every token in the vocabulary)
pick the next token
append it
Put simply, it is where the math behind a model actually runs and turns into the outputs we all know.
For The capital of France is, GPT-2 small gives " the" 6.1% and " Paris" only 3.1%, in sixth place. Greedy decoding picks " the" and writes “the capital of France, and it is the capital of France.”
GPT-2 small is not smart, and that is not the point. It is small enough to hold in your head, and every part of a modern engine is already in it.
You can read about transformers for a long time and still not feel how one runs. Write the loop yourself and you get concrete parts you can take apart and follow step by step. These only clicked for me after I wrote it the first time:
ember-mini is my original GPT-2 engine, stripped down so that you can code it back together. It covers the fundamentals only, with no abstractions to hide behind: every operation is a plain function on f32 slices, and the only dependencies are tokenizers and half.
Every missing function keeps its signature. The comments say what to compute and what usually goes wrong, but not how to write it.
There are six exercises, in order: the math, bytes to numbers, the KV cache, the GGUF reader, the forward pass and the generation loop. Each one has its own test command.
You start at 0 of 17 passing tests and stop when GPT-2 talks.
GGUF, the llama.cpp file format, has four parts: a header, metadata, tensor infos, then the tensor data. The GPT-2 file has 1.7 MB of metadata before the first weight, because the whole tokenizer vocabulary lives there.
Places where you can blow up:
[768, 2304]. Forget to flip it and nothing lines up. I did this more than once.One forward pass: take the token's row of the token embedding plus its position's row of the position embedding, and you have 768 numbers. Run 12 blocks of x = x + attention(norm(x)) then x = x + mlp(norm(x)). A final norm and the output layer give 50,257 logits. In ember-mini the whole forward pass is 51 lines.
GPT-2's raw logits for that prompt all sit near −100. In f32, exp(−100.7) is 1.8e−44: a subnormal number with about 4 bits of precision left. Below −104, exp returns exactly 0. So a naive softmax on GPT-2 is quietly broken. Subtract the maximum first. The other direction breaks too: exp(1000) is infinity, and infinity divided by infinity is NaN.
ember-mini picks with argmax: greedy decoding, no sampling, no temperature.
Without a cache, every new token runs the whole text again, so 6 tokens cost 21 token passes. With a KV cache, the model stores the key and value of each token once and reads them back later, so 6 tokens cost 6 passes. For GPT-2 small at its full context of 1,024 tokens, the cache is 72 MiB.
ember-mini makes about 9 tokens per second on one thread, because its dot product is a plain scalar loop. Faster engines hit a different wall: memory. Decode reads every weight once per token and does only a couple of operations with each one. Ember runs Llama-3.2-1B at Q8_0 at about 58 tokens per second on the CPU, which is roughly 1.3 GB × 58 ≈ 75 GB/s of weights moving through it. At that point the question is no longer how fast your multiply is. It is how fast you can move bytes.
A lot went wrong while I built Ember. Some examples:
A faster kernel made the engine slower. I wrote a 4-column Q4_K tile, where each activation load serves four weight columns. In a single-thread kernel test it was 25–30% faster. In the full engine at 4 threads, decode went from 68.2 to 64.9 tokens per second, so I rejected it.
The biggest win had nothing to do with the kernel. The K-quant matrix-vector products ran on a Rayon fork/join tree, efficiency cores included, with one fork/join per projection. I moved them to a small spin-waiting team on the performance cores, with the same serial kernel, bit for bit. Q4_K_M went from 61.4 to 68.2 tokens per second at 4 threads, and from about 70 to about 87 at 8.
Prefill and decode want different things. The spin-waiting team is good for decode, which is many tiny tasks. When I put prefill's large matrix tiles on it, prefill got 15–25% slower at 8 threads than with Rayon's work stealing.
Layout is not a detail. An interleaved weight layout for the MLP was faster in every standalone test. In the full process it made every operation slower, even the output layer, which it did not touch. I never found out why, so it did not ship.
Your baseline can lie to you. llama-bench uses Metal by default. My “CPU against CPU” comparison read 112 against 66 tokens per second until I added -ngl 0.
So can your laptop. Under normal desktop load, 8-thread runs dropped 20–30% at random. Now every A/B comparison is interleaved and repeated.
Correct and fast are separate problems. Ember promises bit-identical replay, because it is a research tool, so a change that moved RoPE's sine and cosine by one ulp was reverted, even though nobody would ever see it in the text. And my own GPT-2 code in Ember uses the exact erf GELU, while GPT-2 was trained with the tanh approximation. ember-mini uses tanh, and with F32 weights it matches llama.cpp token for token for 100 tokens on four prompts. With the Q8_0 file, llama.cpp also squeezes the activations to 8 bits, and the greedy text drifts after a few tokens. The same model does not mean the same numbers.
Where that leaves Ember: llama.cpp is still ahead on the CPU. On Llama-3.2-1B, an M1 Pro and 4 threads in early October, it was 57.8 against 65.8 tokens per second at Q8_0, and 67.5 against 88.6 at Q4_K_M. Profile before you touch anything, measure the whole engine, and do not trust a number you measured once. A little paranoia about measurement goes a long way.
ember-mini is one model, one format and one thread. Real engines grow in every direction at once:
Ember is about 140,000 lines of Rust today, across src/ and the ember-core crate. Not all of it is inference: there is a GUI, research tooling and audio too. This is where the small educational loop turns into a lot of lines of suffering, with some help from Claude.
Building an engine is worth it even if you never ship one. It gives you a feel for ML systems that papers and high-level libraries do not: where the time goes, where the memory goes, and why the same model can give you different text.
Clone ember-mini, download GPT-2 (the README has the command and the checksum), and run cargo test ops.
If you have ever wondered what happens between a model file and “hello world”, build the tiny version once. And if you got this far: thank you for reading.
gpt2.Q8_0.gguf from QuantFactory/gpt2-GGUF (SHA-256 0afd77d8…), one thread on an Apple M1 Pro. The llama.cpp comparison used an F32 copy of the same weights made with llama-quantize --allow-requantize, with --temp 0 and -ngl 0.