Jais 1 → Jais 2: What Changed in Arabic Morphology Probing?

Jais 2 is not simply a uniformly better version of Jais 1 in this probe. it is much better on number, roughly similar on gender and POS in the main full-metadata interface, and changes more when the explicit lexical metadata is removed. the useful comparison is not a single leaderboard number; it is the pattern across tasks, lexical splits, and representation interfaces.

both runs use the same 4,701-example Modern Standard Arabic testbed, derived from Universal Dependencies Arabic-PADT and processed with CAMeL Tools. they use BF16 hidden-state captures, the same four feature interfaces, three tasks, random/lemma-heldout/root-heldout splits, five outer folds, and the same fixed probe:

StandardScaler + RidgeClassifier(alpha=1.0)

these are two separately frozen experiments, not a controlled intervention on one checkpoint. model revision, architecture, tokenizer, parameter count, and base-versus-chat training status all change together. the deltas below describe the two evaluated model configurations; they do not isolate a causal “Jais 2 effect.”

the two model configurations

Jais-13B uses inceptionai/Jais-13B at revision prefix 0d5eb87f; JAIS-2-8B-Chat uses inception42/Jais-2-8B-Chat at revision prefix da0e1639. both were requested and saved as BF16. the tokenizer revisions differ too: Jais 1 has 5,149 target-span tokens across the examples, while Jais 2 has 5,118. every example aligned successfully in both runs. absolute layer numbers therefore should not be compared as though the networks had the same depth.

the headline result is task-dependent

the table compares the main full_prompt_final interface, averaging each task over the lemma-heldout and root-heldout test splits. each underlying split value is itself a mean across five outer folds. the delta is Jais 2 minus Jais 1. accuracy is included because the number task is imbalanced across classes and its accuracy can hide a change in minority-class recovery.

number is the clear separation. Jais 1's full-metadata held-out number macro-F1 is low despite 0.9362 accuracy, a sign that the majority class is doing much of the work. Jais 2 raises both metrics, and its macro-F1 gain is much larger than its accuracy gain. gender is nearly unchanged in this aggregate, while POS declines modestly. those summaries conceal an important interface effect.

metadata changes the comparison

full_prompt_final reads the final state of a prompt that contains surface, target, lemma, root, and pattern. the fields after the target are available at that endpoint. metadata_free_prompt_final removes the explicit lexical fields but remains a prompt-final representation. the following table uses only the two grouped splits:

the striking result is that the Jais 2 number gain survives without metadata, but is smaller there. gender loses about 0.077 macro-F1 in the metadata-free interface even though it is unchanged with the full prompt. this is why “Jais 2 is better at morphology” is too broad: the answer depends on which information is present at the extraction endpoint.

where the representation is read also matters

the two target-local interfaces read the final token overlapping the target or the mean of every target-overlapping token state. averaged over tasks and grouped splits, their macro-F1 comparison is:

Jais 2 is strongest on the target-final interface in this aggregate, narrowly ahead of full prompt-final. Jais 1's best aggregate here is metadata-free prompt-final. that reversal is not evidence that one pooling rule is universally correct; it shows that the probe's estimand includes where the state is read, what the causal prefix contains, and how a multi-token span is pooled.

held-out transfer is still not random interpolation

for the full-metadata interface, random, lemma-heldout, and root-heldout macro-F1 means across the three tasks are:

both models lose performance when familiar lemma or root groups are kept out of training. Jais 2's number result makes its grouped scores much higher, but the lexical gap does not disappear. random splitting remains an optimistic interpolation reference, not a measure of transfer to unseen lexical groups.

controls and scope

each run selected layers on development data only, refit on train-plus-development, and evaluated test once per outer fold. all 4,701 examples aligned successfully for both models. the Jais 2 archive contains 180 shuffled-label cells; none exceeds its train-only majority baseline. the larger Jais 1 archive contains 360 shuffled cells for its two designated primary models (Qwen3-0.6B and Llama-3.2-1B), not for Jais-13B itself. these controls check the pipeline; they are not a causal test of model behavior.

the result is therefore precise but narrow: under this prompt, lexical split, alignment rule, target interface, pooling rule, selected layer, and linear estimator, Jais 2 makes number more linearly recoverable than Jais 1 in these frozen captures. that does not establish better Arabic generation, universal Arabic competence, or that either model uses the decoded feature during generation.

frozen sources

the comparison is computed from the immutable result summaries, not from manuscript tables. the two source records are:

freezes/bf16-analysis-20260826-v2/snapshot/analysis/results/summary.csv
freezes/bf16-analysis-jais2-20260829-v1/snapshot/analysis/results/summary.csv

their analysis decisions record the five-fold, development-only protocol and the weight-free post-capture contract. the corresponding frozen model metadata and bundle manifests preserve the checkpoint revisions, tokenizer revisions, successful alignments, and BF16 representation semantics. both freezes are evidence records; neither should be regenerated or overwritten.

the broader study is documented in the arabic-morphology-probing-lre repository. the paper, “Lexical Split Design and Representation-Interface Validity in Decoder-Only Morphology Probing: An Arabic Evaluation Study,” is currently under editorial assignment at Language Resources and Evaluation.