the newest packed Q8 path raises median full-model decode from 21.968 to 30.888 tok/s on the measured Llama-3.2-1B workload. that is a 40.6% gain. the first implementation also appeared to need almost another model-sized resident copy. the speed number was real; the first memory interpretation was not complete.
the engineering question became narrower: can Ember keep the decode-oriented layout, evict the source projection pages after prefill, and preserve deterministic output? on this one model and CPU, yes. that is an engineering result, not a general claim about Q8 inference.
| measured result | 21.968 → 30.888 tok/s for median full-model decode, +40.6%. |
| best selective result | 30.199 tok/s while excluding the down projections: 92.9% of the full measured gain. |
| startup cost | all-projection packing took about 450 ms; the selective configuration took 350 ms. |
| claim boundary | one Llama-3.2-1B Q8_0 model, one 4-core Intel laptop, three randomized repetitions. |
the preceding runtime pass removed work around the matrix kernels. single-token decode now reuses a cache-aligned workspace instead of rebuilding intermediate tensors, stores the KV cache as FP16, shares one activation quantization across Q/K/V and gate/up projections, and uses an interleaved VNNI layout for the large LM head. the row-contiguous VNNI kernel also widened from 4 to 8 output rows and gained explicit prefetching.
the latest pass changes the per-layer projection layout itself. eligible Llama Q8_0 weights are packed in tiles of 16 output rows. for every block of 32 input coordinates, bytes for the same four coordinates across all 16 rows sit together. an AVX-512 VNNI kernel can broadcast four activation bytes, load one contiguous 64-byte weight group, and accumulate 16 outputs.
row-contiguous source
row 0 [scale][q0 q1 ... q31]
row 1 [scale][q0 q1 ... q31]
...
packed 16-row tile
[rows 0..15, q0..q3]
[rows 0..15, q4..q7]
...
[rows 0..15, q28..q31]
[16 fp16 scales]one packed tile/block record is 544 bytes, exactly the encoded size of 16 ordinary Q8_0 blocks. packing does not make the weights denser. it changes which bytes arrive together. unsupported CPUs, Qwen's split-half RoPE path, biased projections, and an explicit EMBER_LLAMA_PACKED_Q8=0 switch stay on the generic path.
each trial ran in a fresh process on an Intel i5-1135G7. four Rayon workers were pinned to physical CPUs 0–3, SMT siblings were excluded, and the process started at or below 80 °C package temperature. a six-token prompt ran through generic prefill, then the model generated 128 deterministic greedy tokens. the ten lifecycle and projection-selection modes were shuffled with a fixed seed across three repetitions.
all 30 accepted trials produced the same generated-token hash: fnv1a64:9f8e8158645ba677. an earlier mode-grouped run was rejected because two identical configurations landed 13% apart. in the randomized run, those controls were only 0.2% apart.
packing everything reached 30.827 tok/s in the projection-selection matrix. packing gate/up plus Q/K/V/O, but leaving down row-contiguous, reached 30.199 tok/s. that smaller 714 MiB selection retained 92.9% of the measured all-projection gain while cutting packing time from 452.9 to 350.3 ms and transient peak RSS from 3,104.5 to 2,827.4 MiB.
down is the useful negative result. adding it to gate/up packed another 272 MiB but added only 0.393 tok/s in the median. the operator profile tells the same local story: Q, O, gate, and up improved by roughly 1.7–1.8x inside their projection calls; down was 1.03x and the separately interleaved LM head was 1.01x. this does not yet explain why. hardware counters and another model size are still missing.
the GGUF loader keeps row-contiguous Q8_0 weights in a read-only memory map. packed weights are anonymous memory. immediately after packing, Ember can issue MADV_DONTNEED for the mapped source ranges. a decode-only run then looks close to replacement storage.
ordinary generation has a prefill phase first. generic prefill reads the row-contiguous source and faults those pages back into memory. without another eviction at the phase boundary, the process keeps both layouts resident. the initial low-overhead decode result did not describe that normal lifecycle.
in the durable all-projection mode, file-backed PSS fell from 1,265.1 MiB after prefill to 279.0 MiB after re-eviction and stayed there through 127 packed decode evaluations. final RSS was 2,113.6 MiB. the deliberate duplicate-layout mode ended at 3,099.7 MiB and did not decode faster. that is the result that supports replacement-style residency.
faster steady-state decode does not guarantee faster short generations. the durable all-projection mode moved about 450 ms of packing into time-to-first-token and broke even at approximately 36 generated tokens for this prompt. excluding down reduced the measured break-even to 29 tokens.
packing after prefill can emit token one before the packing pause, but then creates a roughly 438 ms first-to-second-token gap. that is a timing relocation, not a latency win. the right policy depends on whether a process is long-lived and how many tokens it will produce.
the narrow result is enough: the packed kernel made decode faster, and the phase boundary made that speed compatible with durable source-page eviction on this workload. the next test is whether the same tradeoff survives Llama-3.2-3B before changing the kernel again.