I now freeze the benchmark inputs and scoring rules before inference. The record links each item to its rendered prompt, output, parser, and score. This is a reproducibility method, with no new accuracy result on this page.
which model was used? which seed? which decoding settings? what score did it get?
The evaluation interface needs the same care.
Reproducing a score requires the exact items, prompt template, rendered prompts, parser, scoring rules, and metadata linking them to the model.
this matters most when the interface is part of the experiment.
Prompt templates, parsers, and item manifests are experimental conditions: changing any of them can change the score.
for my Arabic benchmark audit work, every run now has a frozen artifact trail before inference starts:
the goal is simple: a reported number should be traceable back to the exact experiment that produced it.
not roughly the same dataset. not approximately the same prompt. not a regenerated version that should be equivalent.
the exact item rows, exact rendered inputs, exact parser rules, and exact scoring artifacts.
The prompt template fixes what the model is asked to do. Save its exact rendered output with the experiment.
in a benchmark evaluation, a prompt template defines what the model actually sees. small changes can alter parseability, answer preferences, or consistency across equivalent variants.
so each template in the audit now has:
every rendered prompt row carries the template ID, template checksum, and renderer version that produced it. that makes the prompt traceable in both directions: from a result back to the exact template, and from a template forward to the rendered prompts and responses it produced.
the same model output can be scored differently depending on the parser. for example, an exact parser may reject:
The answer is A.while a permissive parser may recover A.
those are not the same measurement.
the exact parser measures format compliance. the permissive parser measures recoverable answer behavior. both can be useful, but they answer different questions. reporting one without naming the parser regime makes the score harder to interpret.
in the audit, i report both:
strict_exact: exact-format compliance,permissive_extract: recoverable answer behavior.that separation prevents parse failures from being silently mixed with task failures. it is the same distinction that became visible in my ArabicMMLU answer-position audit: fixing parseability clarified the measurement, but did not make the remaining prompt and position sensitivity disappear.
A benchmark result depends on:
if any of those change, the result may change.
Can someone reconstruct the exact experiment that produced the number?
that includes the boring files. especially the boring files.
the manifest. the template checksum. the response JSONL. the parser mode. the validation report.
those are the parts that keep a benchmark result from becoming a number with a backstory nobody can verify. this is also the principle behind Atlas: the artifact trail does not make an interpretation correct, but it makes the path to it inspectable.
Arabic benchmark evaluation adds more places where the interface can matter.
that does not mean Arabic benchmarks are broken. it means Arabic benchmark scores need provenance and robustness metadata around them.
aggregate accuracy is still useful. it is just not enough by itself.
reproducibility starts before inference.
before the model answers anything, the evaluation should already know which items are being used, which templates render them, which perturbations are applied, which parser will score them, and which artifacts prove that nothing silently changed.
a benchmark score without that trail is still a number. it is just a much harder number to trust.