benchmark reproducibility is often discussed after inference has already happened.
which model was used? which seed? which decoding settings? what score did it get?
those details matter, but they are not enough.
a benchmark result is only as reproducible as the interface that produced it. that means freezing not only the model and the score, but also the exact item set, prompt template, rendered prompts, parser, scoring rules, and metadata connecting all of them.
this matters most when the interface is part of the experiment.
if changing the prompt format changes the score, the prompt template is not a formatting detail. it is an experimental condition. if changing the parser changes the accuracy, the parser is not an implementation detail. it is part of the measurement procedure. if changing the sampled items changes the conclusion, the item manifest is not just a convenience file. it is part of the result.
for my Arabic benchmark audit work, every run now has a frozen artifact trail before inference starts:
the goal is simple: a reported number should be traceable back to the exact experiment that produced it.
not roughly the same dataset. not approximately the same prompt. not a regenerated version that should be equivalent.
the exact item rows, exact rendered inputs, exact parser rules, and exact scoring artifacts.
prompt templates are easy to treat as small formatting helpers. they are not.
in a benchmark evaluation, a prompt template defines what the model actually sees. small changes can alter parseability, answer preferences, or consistency across equivalent variants.
so each template in the audit now has:
every rendered prompt row carries the template ID, template checksum, and renderer version that produced it. that makes the prompt traceable in both directions: from a result back to the exact template, and from a template forward to the rendered prompts and responses it produced.
the same model output can be scored differently depending on the parser. for example, an exact parser may reject:
The answer is A.while a permissive parser may recover A.
those are not the same measurement.
the exact parser measures format compliance. the permissive parser measures recoverable answer behavior. both can be useful, but they answer different questions. reporting one without naming the parser regime makes the score harder to interpret.
in the audit, i report both:
strict_exact: exact-format compliance,permissive_extract: recoverable answer behavior.that separation prevents parse failures from being silently mixed with task failures. it is the same distinction that became visible in my ArabicMMLU answer-position audit: fixing parseability clarified the measurement, but did not make the remaining prompt and position sensitivity disappear.
a benchmark result is not only a function of the model. it is a function of:
if any of those change, the result may change.
so the reproducibility question is not only: can someone rerun the script? it is: can someone reconstruct the exact experiment that produced the number?
that includes the boring files. especially the boring files.
the manifest. the template checksum. the response JSONL. the parser mode. the validation report.
those are the parts that keep a benchmark result from becoming a number with a backstory nobody can verify. this is also the principle behind Atlas: the artifact trail does not make an interpretation correct, but it makes the path to it inspectable.
Arabic benchmark evaluation adds more places where the interface can matter.
that does not mean Arabic benchmarks are broken. it means Arabic benchmark scores need provenance and robustness metadata around them.
aggregate accuracy is still useful. it is just not enough by itself.
reproducibility starts before inference.
before the model answers anything, the evaluation should already know which items are being used, which templates render them, which perturbations are applied, which parser will score them, and which artifacts prove that nothing silently changed.
a benchmark score without that trail is still a number. it is just a much harder number to trust.