Jais 1 → Jais 2: What Changed in Arabic Morphology Probing?

Jais 2 improved number recovery in the full-metadata held-out comparison. Number macro-F1 rose from 0.7177 to 0.9245; gender barely changed and POS declined. The two frozen configurations differ in several ways, so the comparison cannot isolate a model-revision effect.

both runs use the same 4,701-example Modern Standard Arabic testbed, derived from Universal Dependencies Arabic-PADT and processed with CAMeL Tools. they use BF16 hidden-state captures, the same four feature interfaces, three tasks, random/lemma-heldout/root-heldout splits, five outer folds, and the same fixed probe:

StandardScaler + RidgeClassifier(alpha=1.0)

These separately frozen experiments change model revision, architecture, tokenizer, parameter count, and base-versus-chat training status together. Their deltas describe the evaluated configurations; the contribution of each change remains unresolved.

the two model configurations

Jais-13B uses inceptionai/Jais-13B at revision prefix 0d5eb87f; JAIS-2-8B-Chat uses inception42/Jais-2-8B-Chat at revision prefix da0e1639. both were requested and saved as BF16.

the tokenizer revisions differ too: Jais 1 has 5,149 target-span tokens across the examples, while Jais 2 has 5,118. every example aligned successfully in both runs.

absolute layer numbers therefore should not be compared as though the networks had the same depth.

the headline result is task-dependent

Number recovery improves most in the full_prompt_final comparison. The table averages lemma-heldout and root-heldout macro-F1, with each split averaged over five outer folds. Deltas are Jais 2 minus Jais 1. Accuracy is shown alongside macro-F1 because class imbalance can hide minority-class changes.

number is the clear separation. Jais 1's full-metadata held-out number macro-F1 is low despite 0.9362 accuracy, a sign that the majority class is doing much of the work.

Jais 2 raises both metrics, and its macro-F1 gain is much larger than its accuracy gain. gender is nearly unchanged in this aggregate, while POS declines modestly.

those summaries conceal an important interface effect.

metadata changes the comparison

the striking result is that the Jais 2 number gain survives without metadata, but is smaller there. gender loses about 0.077 macro-F1 in the metadata-free interface even though it is unchanged with the full prompt.

full_prompt_final reads the final state of a prompt that contains surface, target, lemma, root, and pattern. the fields after the target are available at that endpoint. metadata_free_prompt_final removes the explicit lexical fields but remains a prompt-final representation.

the following table uses only the two grouped splits:

this is why “Jais 2 is better at morphology” is too broad: the answer depends on which information is present at the extraction endpoint.

readout position

the two target-local interfaces read the final token overlapping the target or the mean of every target-overlapping token state. averaged over tasks and grouped splits, their macro-F1 comparison is:

Jais 2 is strongest on the target-final interface in this aggregate, narrowly ahead of full prompt-final. Jais 1's best aggregate here is metadata-free prompt-final.

that reversal is not evidence that one pooling rule is universally correct; it shows that the probe's estimand includes where the state is read, what the causal prefix contains, and how a multi-token span is pooled.

held-out transfer is still not random interpolation

for the full-metadata interface, random, lemma-heldout, and root-heldout macro-F1 means across the three tasks are:

Both models lose performance on unseen lemma or root groups. Jais 2 improves grouped number scores substantially, while random splitting remains an optimistic interpolation reference.

controls and scope

each run selected layers on development data only, refit on train-plus-development, and evaluated test once per outer fold. all 4,701 examples aligned successfully for both models.

the Jais 2 archive contains 180 shuffled-label cells; none exceeds its train-only majority baseline. the larger Jais 1 archive contains 360 shuffled cells for its two designated primary models (Qwen3-0.6B and Llama-3.2-1B), not for Jais-13B itself.

these controls check the pipeline; they are not a causal test of model behavior.

the result is therefore precise but narrow: under this prompt, lexical split, alignment rule, target interface, pooling rule, selected layer, and linear estimator, Jais 2 makes number more linearly recoverable than Jais 1 in these frozen captures.

that does not establish better Arabic generation, universal Arabic competence, or that either model uses the decoded feature during generation.

frozen sources

the comparison is computed from the immutable result summaries, not from manuscript tables. the two source records are:

freezes/bf16-analysis-20260826-v2/snapshot/analysis/results/summary.csv
freezes/bf16-analysis-jais2-20260829-v1/snapshot/analysis/results/summary.csv

their analysis decisions record the five-fold, development-only protocol and the weight-free post-capture contract. the corresponding frozen model metadata and bundle manifests preserve the checkpoint revisions, tokenizer revisions, successful alignments, and BF16 representation semantics. both freezes are evidence records; neither should be regenerated or overwritten.

the broader study is documented in the arabic-morphology-probing-lre repository. the paper, “Lexical Split Design and Representation-Interface Validity in Decoder-Only Morphology Probing: An Arabic Evaluation Study,” is currently under editorial assignment at Language Resources and Evaluation.