Ember currently loads K-quants correctly, then discards their deployment advantage. Q6_K and Q4_K_M are decoded from the GGUF file into resident f32 weights and sent through generic f32 kernels; Ember has no native K-quant matmul or packed execution path for them yet. this is a reference loader, not a low-memory K-quant deployment.
that makes the current path unsuitable as a low-memory or low-latency deployment backend. this note freezes that pre-optimization baseline before native K-quant execution is implemented. it is not a claim that Q4_K_M or Q6_K are inherently inefficient: the result comes from Ember's current loader and backend policy and should not be generalized to GGUF, K-quants, or low-bit quantization as a whole.
pre-optimization engineering baseline only. the table covers two small model families on one local machine. it is not a novel quantization result, a production deployment claim, or a comparison that makes Ember faster than llama.cpp.
does a smaller quantized checkpoint necessarily produce a smaller and faster local deployment?
the frozen deployment benchmark measures file size, load plus prefill time, decode throughput, and peak RSS. the last column is the important one for interpreting the table: it records what Ember keeps in memory and executes, not only what the filename says.
| model | quantization | file size | load + prefill | decode tok/s | peak RSS | runtime representation |
|---|---|---|---|---|---|---|
| Qwen2.5-1.5B-Instruct | Q8_0 | 1807 MB | 0.97 s | 40.6 | 1941 MB | compressed-resident |
| Qwen2.5-1.5B-Instruct | Q6_K | 1396 MB | 14.4 s | 2.5 | 8713 MB | persistent f32 expansion |
| Qwen2.5-1.5B-Instruct | Q4_K_M | 1066 MB | 13.0 s | 2.6 | 8712 MB | persistent f32 expansion |
| Llama-3.2-1B | Q8_0 | 1260 MB | 2.03 s | 149.6 | 2685 MB | compressed-resident |
| Llama-3.2-1B | Q6_K | 974 MB | 16.4 s | 2.9 | 6965 MB | persistent f32 expansion |
| Llama-3.2-1B | Q4_K_M | 770 MB | 15.9 s | 3.2 | 6965 MB | persistent f32 expansion |
Qwen Q4_K_M is about 41% smaller on disk than Q8_0, but uses about 4.49× the peak RSS and decodes about 15.6× more slowly. Llama Q4_K_M is about 39% smaller, but uses about 2.59× the peak RSS and decodes about 46.8× more slowly.
across the four Q4/Q6 comparisons, decode is 15.6× to 51.6× slower and peak RSS is 2.59× to 4.49× higher. the 51.6× endpoint is Llama Q6_K; the 15.6× endpoint is Qwen Q4_K_M.
the inversion is consistent across both families: the only axis on which the lower-bit files are smaller is the file itself. load time also rises from roughly one or two seconds for Q8_0 to roughly thirteen to sixteen seconds for Q6_K and Q4_K_M.
checkpoint quantization describes how tensors are encoded in the file. it does not guarantee how a runtime stores those tensors or which kernels execute them. Ember's Q8_0 path keeps quantized blocks resident and uses its packed Q8 execution path.
the current K-quant path is a reference loader, not a native K-quant backend. the loader dequantizes the weights to f32 at load time, keeps the expanded buffers resident, and sends the work through generic f32 execution. there is no K-specific matmul or packed kernel in Ember's current backend. the checkpoint is still Q4 or Q6 on disk; the ordinary in-memory deployment is not.
Q8_0:
GGUF Q8 blocks
→ compressed-resident weights
→ packed Q8 kernels
Q4_K_M / Q6_K today (reference path):
GGUF K-quant blocks
→ full f32 expansion
→ generic f32 executionif the mapped compressed file also remains resident, RSS can reflect both the source mapping and the expanded representation. that is one reason a single RSS number needs a residency breakdown before it is treated as a complete memory explanation.
a filename or GGUF suffix does not determine process memory, startup cost, decode speed, temporary workspace, kernel behavior, or whether the source mapping remains resident. those properties are decisions made by the loader, tensor layout, dispatch policy, and backend.
a quantized checkpoint is only a storage format until the runtime decides how to materialize and execute it.
that is the conceptual boundary in this audit. “Q4” is true as a statement about the file. it is incomplete as a statement about the deployment.
llama.cpp is the external correctness and performance reference for this work. its optimized CPU backend keeps K-quant weights compressed in ordinary paths and uses native quantized vector-dot kernels for Q4_K, Q6_K, and Q8_0, rather than retaining a whole-model f32 copy for ordinary execution.
that is the relevant contrast here. the external reference preserves a low-bit runtime representation; Ember's current K-quant research path preserves low-bit storage but materializes f32 weights.
the benchmark used the same CPU affinity and thread setting across configurations. that improves comparability inside this table, but it does not turn the run into a broad hardware study.
Ember's next relevant systems task is to compare eager whole-model f32 expansion, one bounded-memory alternative, native compressed-resident K-quant execution, and llama.cpp as the external baseline.
any optimized path also has to preserve Ember's activation tracing, capture, and patching semantics. lower memory or higher throughput is useful only if the research instrument still observes and intervenes on the same computation.
the exact frozen deployment output and runner exist locally in
research/pilots/arabic_quantization_001/deployment_benchmark.{json,csv}
on the local pilot-001 branch. that branch is not a
public path, so this page does not pretend there is a public link to
those measurements.
the public implementation context is available in
src/quant_k.rs,
src/loader.rs,
src/residency.rs,
and the packed-Q8 engineering notes
packed-q8-research-memo.md
and
packed-q8-lifecycle.md.
the checkpoint said Q4. the runtime chose f32. the deployment cost followed the runtime.