Moving the correct answer changed measured accuracy sharply in this ArabicMMLU audit. Qwen3-0.6B scored 0.680 with the answer under A and 0.020 under B. The small local-model sample measures interface sensitivity; it cannot rank Arabic capability broadly.
this note is a small diagnostic pilot on ArabicMMLU-style multiple-choice evaluation. the question is not whether ArabicMMLU is bad. the question is what else should be reported with an accuracy score, especially when evaluating small local decoder-only models.
Qwen3-0.6B scored 0.680 with the correct answer under A and 0.020 under B in the same setup. That sensitivity makes answer position essential context for the accuracy score.
This small-sample audit covers local decoder-only models. The 3B run adds one scale check; broader rankings and scaling conclusions require more models and items.
the core finding is narrow: in these runs, prompt formatting, answer parsing, option-label preference, and correct-answer position were associated with large changes in measured accuracy. scaling from roughly 0.6B/1B to 3B improved accuracy and robustness, but did not eliminate option-position sensitivity.
Multiple-choice evaluation combines task solving, answer-format compliance, label selection, and parsing.
a multiple-choice benchmark looks simple: ask a question, show A/B/C/D options, parse one answer, and compute accuracy. in practice, several interface behaviors can enter the measurement.
these are not reasons to discard multiple-choice benchmarks. they are reasons to report diagnostic metadata alongside raw accuracy.
pilot 1 used 50 ArabicMMLU source items and 6 prompt variants per item, for 300 prompts per model. the models were Qwen3-0.6B, Llama-3.2-1B, and Llama-3.2-3B.
the metrics were overall accuracy, accuracy on parseable outputs, parse-failure rate, semantic consistency across variants, strict-output accuracy, strict-output parse failure, and predicted label distribution.
here, semantic consistency means the proportion of source items for which the model's parseable answers map to the same underlying answer choice across prompt variants or option permutations.
| model | acc all | acc parseable | parse fail | semantic consistency | strict acc | strict parse fail |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | 0.273 | 0.381 | 0.283 | 0.200 | 0.380 | 0.000 |
| Llama-3.2-1B | 0.303 | 0.313 | 0.030 | 0.080 | 0.300 | 0.000 |
| Llama-3.2-3B | 0.430 | 0.433 | 0.007 | 0.340 | 0.400 | 0.000 |
Qwen3-0.6B had high parse failure under non-strict variants. strict output prompting eliminated parse failures for all three models, but fixing parseability did not fully solve robustness.
Llama-3.2-3B improved accuracy, parseability, and semantic consistency, which is evidence that scale helps in this setup. still, semantic consistency remained 0.340, meaning most source items did not receive a stable semantic answer across all prompt variants.
Llama-3.2-3B’s aggregate label distribution in pilot 1 was A/B/C/D/E/PF = 59/87/92/58/2/2. Predictions spread across labels, with B and C most frequent.
pilot 2 used 50 ArabicMMLU source items with 4 permutations per item, for 200 prompts per model. the prompt used strict-output instruction only:
Answer with exactly one character: A, B, C, or D. Do not explain.each source item was permuted so that the correct answer appeared exactly once under A, B, C, and D. the question content and answer choices were preserved; only labels and order changed. this isolates option-label and correct-position effects after parseability has already been controlled.
label-bias score is computed as the largest predicted-label proportion minus 0.25, where 0.25 corresponds to a uniform A/B/C/D distribution.
| model | acc all | parse fail | label-bias score | semantic consistency | correct-position range |
|---|---|---|---|---|---|
| Qwen3-0.6B | 0.275 | 0.000 | 0.380 | 0.120 | 0.660 |
| Llama-3.2-1B | 0.300 | 0.000 | 0.120 | 0.160 | 0.220 |
| Llama-3.2-3B | 0.410 | 0.000 | 0.145 | 0.260 | 0.320 |
| model | correct=A | correct=B | correct=C | correct=D |
|---|---|---|---|---|
| Qwen3-0.6B | 0.680 | 0.020 | 0.060 | 0.340 |
| Llama-3.2-1B | 0.200 | 0.380 | 0.200 | 0.420 |
| Llama-3.2-3B | 0.300 | 0.460 | 0.600 | 0.280 |
| model | A | B | C | D | PF | other |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | 126 | 4 | 9 | 61 | 0 | 0 |
| Llama-3.2-1B | 29 | 57 | 40 | 74 | 0 | 0 |
| Llama-3.2-3B | 25 | 62 | 79 | 34 | 0 | 0 |
strict-output prompting eliminated parse failures for all models. that is useful, but it also makes the remaining instability clearer.
Qwen3-0.6B showed an extreme A-label preference: 126 of 200 predictions were A. its accuracy changed sharply depending on where the correct answer was placed: 0.680 when correct=A versus 0.020 when correct=B.
Llama-3.2-1B was less extreme, but still unstable. Llama-3.2-3B improved overall accuracy and semantic consistency, but still showed non-trivial label bias and a 0.320 correct-position accuracy range. in this pilot, scale helped, but raw accuracy still needed diagnostic context.
Llama-3.2-3B improved several measurements at once. in pilot 1, it had higher accuracy than the two smaller models, lower parse failure, and higher semantic consistency. in pilot 2, it also had higher overall accuracy and better semantic consistency than Llama-3.2-1B.
The 3B run adds one larger model under the same audit setup. Generalizing the finding requires a broader size range and stronger Arabic-capable systems.
the practical reading is cautious: scaling improved robustness here, but option-position sensitivity remained visible after strict formatting removed parse failures.
this pilot does show:
this pilot does not show:
a benchmark score is more useful when it comes with the interface diagnostics needed to interpret it.
for small or local decoder-only models, i would not report raw ArabicMMLU multiple-choice accuracy alone. a more useful report would include:
prompts, responses, score files, and summaries are available in the
public Ember repo under
artifacts/benchmark-audit/arabicmmlu/.
the actual runs were performed in a private local testing directory;
the repo copy is the published artifact bundle used to make the
numbers on this page inspectable.
Raw benchmark accuracy can combine task competence with sensitivity to the evaluation interface. Report parsing and position effects alongside the score.
Arabic evaluation should measure whether correct answers survive harmless changes to prompt format, labels, option order, and text normalization.