When Does Multithreading Help LLM Decode? Finding the CPU Crossover Point in Ember

most LLM inference engines give you one knob for parallelism: a thread count. set it to 4 or 8 or 16 and hope.

but does single-token decode actually benefit from more threads?

the answer depends on model size, kernel implementation, and the CPU. this note finds the crossover point where thread parallelism starts helping, and shows why a fixed global thread count is the wrong abstraction.

the question

during decode, a transformer produces one token at a time. each token passes through every layer: attention projections, MLP projections, RMS norms, rotary embeddings. most of that work is matrix multiplication — small matmuls, one after another, each processing a single row of activations against a quantized weight matrix.

those matmuls are naturally parallelizable: split the output dimension across threads, compute partial dot products, reduce. but launching threads has a fixed cost. if the per-matmul work is too small, the overhead dominates and parallelism makes things slower.

so the question becomes quantitative:

at what per-matmul FLOP count does thread parallelism become net-positive for single-token Q8_0 decode on a consumer CPU?

the experiment

two experiments, same CPU (i5-1135G7, 4 physical / 8 logical cores), same Q8_0 kernel:

model-level: run three real models at 1, 2, 4, and 8 threads. measure decode throughput via Ember's per-operation tracer.

kernel-level: run a synthetic sweep of matmul_q8_0_decode at six matrix sizes with a fixed 1:3 aspect ratio (embed_dim : inter_dim ≈ 1:3). 100 measurements per cell, median and coefficient-of-variation reported.

all code is in Ember.

model-level results

each cell: median of 2–3 measured runs after 1–2 warmup runs. the engineering rule below rests primarily on the synthetic sweep (100 measurements per cell); treat these model-level numbers as directional confirmation, not high-confidence point estimates.

at 6.3 and 23.1 MFLOPs per matmul, threads are net-negative or break-even. Ember's should_parallel_q8_decode gate returns false at these sizes — the matmul stays single-threaded while the larger thread-pool configuration provides no useful parallel work and slightly regresses end-to-end throughput.

at 50.3 MFLOPs, 2 threads give 1.46×, 4 threads give 1.56×. 8 threads regress to 1.48× with high variance — the CPU has only four physical cores; hyperthreads contend for cache and memory bandwidth.

across all models, matmul_q8_0 stays at 99.4–99.6% of runtime regardless of thread count. element-wise ops (RMS norm, RoPE, SiLU) remain below 1% and never become the bottleneck.

synthetic sweep

to sample the transition region, a kernel-only sweep was run on synthetic Q8_0 weights at six MFLOPs levels. the column ms/op is median single-kernel latency — this is not full-model throughput.

speedup = ms/op(1 thread) / ms/op(n threads). each cell: 100 measurements. the synthetic harness bypasses Ember's per-call should_parallel_q8_decode gate — it shows what the parallel kernel would deliver if always activated, not what a real model run currently does.

the 2-thread line crosses 1.0× between 14 and 25 MFLOPs. four and eight threads both outperform two threads across most sizes above 25 MFLOPs. eight-thread results are less stable, with higher relative dispersion (stdev/median up to 15% vs 6–14% for 4-thread), so four physical cores provide the safer default on this CPU.

scheduling overhead

rayon thread-pool dispatch cost, measured with an empty par_iter().for_each() loop:

at 14.2 MFLOPs, matmul latency is ~1.7 ms — scheduling overhead (~4 µs for 2 threads) is small in absolute terms. the real issue was the gate used at the time: it kept these small real-model calls serial. the current Q8×Q8 implementation has since lowered that gate substantially; this paragraph describes the historical run.

the engineering rule

on this CPU and isolated Q8_0 kernel, the first observed median speedup depends on thread count. two and eight threads cross between the 14.2 and 25.2 MFLOP samples; four threads are marginally positive at 14.2 MFLOPs. this sweep does not by itself justify a universal threshold, and Ember's current Q8×Q8 gate is calibrated separately.

in the real-model experiment, performance peaked at four physical cores. in the isolated synthetic sweep, eight threads occasionally produced slightly higher median speedups, but with less stable results. the safe recommendation is to pin threads to physical cores.

this rule is specific to this CPU and this kernel. it should be re-tested on:

• a different CPU (more cores, larger caches),
• different quantization levels (Q4_0, Q6_K),
• prefill (batch matmul reaches the threshold at smaller dimensions).

what this means for Ember

a fixed global thread count is the wrong abstraction.

different operations inside the same decode step have wildly different matrix sizes. the attention Q projection is 2× the K/V projection. the MLP up/gate/down projections are 3× the attention output. the LM head is tens of times wider than the per-layer projections. assigning the same thread count to all of them wastes cores on small matmuls and underutilizes them on large ones.

the next step is an adaptive scheduler:

if estimated_work < crossover:
    serial SIMD
else:
    physical-core parallel path

the crossover data collected here provides the calibration point. this is where the project stops being "i profiled my engine" and starts becoming runtime design.

related work

the closest published system is ProfInfer (2026), which attaches eBPF probes to inference engines for fine-grained operator profiling. Ember's tracer provides the model-semantic counterpart: per-operation timing with exact layer, shape, and FLOP identity, plus output fingerprints for numerical parity checks.

llama.cpp's llama-bench gives end-to-end throughput numbers but does not break down per-operation scaling. the synthetic sweep here is complementary — it measures the kernel in isolation to find the exact crossover.

all raw data and the tracer source are in the Ember repository.