arabic benchmarks are valuable. they make it possible to compare systems, track progress, and ask whether models handle arabic tasks beyond english-centered evaluation. but aggregate accuracy can hide measurement artifacts.
this note is a small diagnostic pilot on ArabicMMLU-style multiple-choice evaluation. the question is not whether ArabicMMLU is bad. the question is what else should be reported with an accuracy score, especially when evaluating small local decoder-only models.
in the strongest case, Qwen3-0.6B scored 0.680 when the correct answer was labeled A, but 0.020 when the same setup placed the correct answer under B. that does not make ArabicMMLU invalid, but it does show why raw accuracy is not enough.
treat this as a measurement note, not a model ranking. the item count is small, the models are small/local decoder-only models, and the 3B run is a scale-check rather than a full scaling study.
the core finding is narrow: in these runs, prompt formatting, answer parsing, option-label preference, and correct-answer position were associated with large changes in measured accuracy. scaling from roughly 0.6B/1B to 3B improved accuracy and robustness, but did not eliminate option-position sensitivity.
in multiple-choice evaluation, a model is not only solving the underlying task. it is also following the answer format, choosing from labels, and interacting with a parser.
a multiple-choice benchmark looks simple: ask a question, show A/B/C/D options, parse one answer, and compute accuracy. in practice, several interface behaviors can enter the measurement.
these are not reasons to discard multiple-choice benchmarks. they are reasons to report diagnostic metadata alongside raw accuracy.
pilot 1 used 50 ArabicMMLU source items and 6 prompt variants per item, for 300 prompts per model. the models were Qwen3-0.6B, Llama-3.2-1B, and Llama-3.2-3B. the metrics were overall accuracy, accuracy on parseable outputs, parse-failure rate, semantic consistency across variants, strict-output accuracy, strict-output parse failure, and predicted label distribution.
here, semantic consistency means the proportion of source items for which the model's parseable answers map to the same underlying answer choice across prompt variants or option permutations.
| model | acc all | acc parseable | parse fail | semantic consistency | strict acc | strict parse fail |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | 0.273 | 0.381 | 0.283 | 0.200 | 0.380 | 0.000 |
| Llama-3.2-1B | 0.303 | 0.313 | 0.030 | 0.080 | 0.300 | 0.000 |
| Llama-3.2-3B | 0.430 | 0.433 | 0.007 | 0.340 | 0.400 | 0.000 |
Qwen3-0.6B had high parse failure under non-strict variants. strict output prompting eliminated parse failures for all three models, but fixing parseability did not fully solve robustness. Llama-3.2-3B improved accuracy, parseability, and semantic consistency, which is evidence that scale helps in this setup. still, semantic consistency remained 0.340, meaning most source items did not receive a stable semantic answer across all prompt variants.
for the Llama-3.2-3B pilot 1 run, the aggregate predicted label distribution was A/B/C/D/E/PF = 59/87/92/58/2/2. that distribution is not collapsed to one label, but it is still useful context for interpreting the accuracy number.
pilot 2 used 50 ArabicMMLU source items with 4 permutations per item, for 200 prompts per model. the prompt used strict-output instruction only:
Answer with exactly one character: A, B, C, or D. Do not explain.each source item was permuted so that the correct answer appeared exactly once under A, B, C, and D. the question content and answer choices were preserved; only labels and order changed. this isolates option-label and correct-position effects after parseability has already been controlled.
label-bias score is computed as the largest predicted-label proportion minus 0.25, where 0.25 corresponds to a uniform A/B/C/D distribution.
| model | acc all | parse fail | label-bias score | semantic consistency | correct-position range |
|---|---|---|---|---|---|
| Qwen3-0.6B | 0.275 | 0.000 | 0.380 | 0.120 | 0.660 |
| Llama-3.2-1B | 0.300 | 0.000 | 0.120 | 0.160 | 0.220 |
| Llama-3.2-3B | 0.410 | 0.000 | 0.145 | 0.260 | 0.320 |
| model | correct=A | correct=B | correct=C | correct=D |
|---|---|---|---|---|
| Qwen3-0.6B | 0.680 | 0.020 | 0.060 | 0.340 |
| Llama-3.2-1B | 0.200 | 0.380 | 0.200 | 0.420 |
| Llama-3.2-3B | 0.300 | 0.460 | 0.600 | 0.280 |
| model | A | B | C | D | PF | other |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | 126 | 4 | 9 | 61 | 0 | 0 |
| Llama-3.2-1B | 29 | 57 | 40 | 74 | 0 | 0 |
| Llama-3.2-3B | 25 | 62 | 79 | 34 | 0 | 0 |
strict-output prompting eliminated parse failures for all models. that is useful, but it also makes the remaining instability clearer. Qwen3-0.6B showed an extreme A-label preference: 126 of 200 predictions were A. its accuracy changed sharply depending on where the correct answer was placed: 0.680 when correct=A versus 0.020 when correct=B.
Llama-3.2-1B was less extreme, but still unstable. Llama-3.2-3B improved overall accuracy and semantic consistency, but still showed non-trivial label bias and a 0.320 correct-position accuracy range. in this pilot, scale helped, but raw accuracy still needed diagnostic context.
Llama-3.2-3B improved several measurements at once. in pilot 1, it had higher accuracy than the two smaller models, lower parse failure, and higher semantic consistency. in pilot 2, it also had higher overall accuracy and better semantic consistency than Llama-3.2-1B.
the important caveat is that the 3B result is a robustness check, not a scaling study. it adds one larger model size under the same audit setup. it does not establish how larger models in general behave, and it does not replace evaluation on stronger arabic-capable systems. larger-model evaluation is future work.
the practical reading is cautious: scaling improved robustness here, but option-position sensitivity remained visible after strict formatting removed parse failures.
this pilot does show:
this pilot does not show:
a benchmark score is more useful when it comes with the interface diagnostics needed to interpret it.
for small or local decoder-only models, i would not report raw ArabicMMLU multiple-choice accuracy alone. a more useful report would include:
prompts, responses, score files, and summaries are available in the
public Ember repo under
artifacts/benchmark-audit/arabicmmlu/.
the actual runs were performed in a private local testing directory;
the repo copy is the published artifact bundle used to make the
numbers on this page inspectable.
the main takeaway is not that one model is better than another. the model ranking is not the point of this audit. the point is that raw benchmark accuracy can conflate task competence with benchmark interface behavior.
arabic evaluation should not only ask whether a model gets the right answer. it should also ask whether the answer survives harmless changes in prompt format, option labels, option order, and arabic text normalization.