complete draft with explicit citations. the causal-localization reproduction in section 5
was run live on 2026-08-04 and is recorded in
artifacts/road-to-1.0/part6-null-result.md. the null is scoped to the tested
matrix and is not externally reproduced; the causal result is validated on the rows with
completed golden checks (Qwen2.5-1.5B and Llama-3.2-1B families). the figure is embedded as
SVG (dark/light theme variants under figures/).
we set out to show that quantization degrades Arabic specifically. the data said no. the tools we built to check said something more useful.
the pilot began with a paper-driven question: does low-bit quantization degrade Arabic morphology and dialect behavior more than it degrades the languages the quantizers were tuned on? the mechanism story was plausible, Arabic tokenization is unusual, the writing system is high-entropy, and a compressed representation could plausibly lose the fine distinctions that morphology probing depends on. the hypothesis, stated before the runs: an Arabic-selective quantization degradation exists at Q4/Q6 relative to Q8.
two model families, Qwen2.5-1.5B and Llama-3.2-1B, at three precisions each, Q8_0, Q6_K,
Q4_K_M, with a matched, hash-verified quantization ladder and a 32-item instrument balanced
across four categories: Arabic morphology, dialect and register, code-switching and script,
and controls (English + MSA). roughly 500 deterministic runs, temperature 0
(docs/validation.md; the item set and scoring live in the pilot directory,
local branch, not published).
the headline hypothesis did not survive. behaviorally: Q4 was not worse than Q8 (q8 19/25, q6 20/25, q4 20/25 correct on the scored items), 30 of 32 items produced identical output across all three quants, and the two differences both favored lower precision. internally: the q8-vs-q4 activation drift was about 5× the q8-vs-f16 drift and was uniform, dialect items sat inside the control range, with no Arabic-selective amplification (PILOT_REPORT.md, pilot directory).
a falsified hypothesis is not a failed experiment. the null was informative in two ways. it removed a wrong story, Arabic is not specially fragile at 4-bit in this matrix, and it forced the question that could be answered: the quantization boundary does occasionally break, and where it breaks, can we find out where and why?
the surviving failures were rare and specific. one Arabic morphology item (sound plural, correct at f16 and q6, wrong at q8 on Qwen2.5-1.5B) was the case that mattered: a quantization-boundary failure that a single-layer activation patch restored exactly, by patching in the activation from either the f16 or the q6 run. the causal locus was layer 7 of 28, one layer before the visible divergence ramp. the same pattern transferred to Llama-3.2-1B: an adjective-number item failed at q6, and patching q6's layer-1 after-mlp activation from q8 restored the correct output, with the answer-position divergence gap collapsing from rel-L2 0.4286 to 0.0927 (4.6×); single-layer patches at layers 2 and 4 did not restore. the causal layer sits one before the ramp in both families (PILOT_REPORT.md).
the mechanism consistent with all of it is a near-threshold flip: quantization noise crossing the model's smallest decision margin, not a broad representational collapse. two failure classes did not localize at a single layer, first-token divergence (both families) and one control item that needed a multi-layer patch for partial restoration, which is itself a useful boundary on the claim.
reproducing the demo was its own lesson. the committed end-to-end script
(research/pilots/arabic_quantization_001/causal_demo.sh) patches the degraded
quant with the reference quant's captured activation, different model files. on 2026-08-04
the current binary refused it, fail-closed, because the patch source's model SHA-256 must now
match the current model (the rule was added 2026-08-03 in the hardening pass). and the
pilot's headline revision panicked on the llama Q6_K file (a tied-embedding orientation bug
fixed the same day the pilot ran). running the demo with the revision it was written for ,
in the window after the head fix and before the SHA rule, reproduced the finding:
baseline compare: differs 0 identical / 85 aligned
patched run applied (5 targets)
answer-step logits: baseline rel_l2 vs reference 0.4279 | patched 0.0989
criterion: divergence reduced 4.3x
VERDICT: RESTORED
the reproduction's numbers track the pilot's recorded numbers almost exactly (0.4286 → 0.0927
recorded vs 0.4279 → 0.0989 measured, 4.6× vs 4.3×). both harness findings are recorded in
artifacts/road-to-1.0/part6-null-result.md: the stricter model-SHA contract is
scientifically defensible (a patch is only meaningful when source and target are the same
model), and the as-committed demo simply predates it.
docs/validation.md and the README.the question "does quantization hurt Arabic?" became "where, exactly, does quantization break?", and answering the second question required the exact instruments this series has been describing: capture with provenance (v0.2), compressed execution that keeps semantics identical (v0.3), hooks that survive optimization (v0.4), and experiments another researcher can verify (v0.5). the null result did not produce a research conclusion to publish; it produced the toolchain that makes the next, better-posed question answerable.