What Does a Morphology Probe Measure? Lexical Splits and Representation Interfaces

i ended up treating a probe score as a property of a representation under a particular prompt, split, alignment rule, pooling rule, selected layer, and estimator. the useful question here is narrow: what changes when familiar lexical identities stay out of training?

the testbed is 4,701 Modern Standard Arabic stimuli derived from Universal Dependencies Arabic-PADT and CAMeL Tools processing. the tasks are part of speech, gender, and number. all results below are about linear recoverability under named conditions; broader Arabic competence and model behavior are separate questions.

the measurement unit

the matrix crosses eleven decoder-only models with three split conditions and four representation interfaces. every combination uses five outer folds. the probe is fixed:

StandardScaler + RidgeClassifier(alpha=1.0)

for each outer fold, layer selection is performed on the training data and a development partition carved from the outer-training portion. the chosen layer is then refit on train plus development and evaluated once on test. test metrics never select the layer, preprocessing, model, or hyperparameter. macro-F1 is reported with accuracy and a majority reference computed from training labels only.

the three split conditions put different lexical units on the other side of the boundary: random instance is an optimistic reference for interpolation among a familiar lexical inventory; lemma-heldout tests transfer across lemma groups; and root-heldout tests transfer across root groups, a broader and different grouping key.

direct overlap checks find no shared applicable key across the grouped partitions. Semantic, orthographic, morphological, source-document, and tokenizer correlations can still travel across those boundaries.

random is an optimistic reference

the compact comparison has one row for each model, task, and interface: 132 rows in total. the quantities are:

delta_lemma = macro_f1_random - macro_f1_lemma_heldout
delta_root  = macro_f1_random - macro_f1_root_heldout

all 132 delta_lemma values are positive. all 132 delta_root values are positive. across the matrix, random minus lemma-heldout ranges from +0.0127 to +0.2863, with mean +0.1059. random minus root-heldout ranges from +0.0255 to +0.2796, with mean +0.1179.

taken as a reference, random splitting measures interpolation among instances from a familiar lexical inventory. the grouped runs ask transfer questions, and their effect sizes vary by model, task, representation interface, and grouping key. the table below maps that heterogeneity rather than collapsing it into one correction.

these means map heterogeneity across the matrix. architecture, tokenizer, parameter count, training mixture, and instruction status all vary together, so the table cannot isolate a size or family effect. Qwen3-8B has a slightly larger lemma gap than root gap, while most rows go the other way.

prompt context changes the interface

the full-metadata prompt is:

Arabic morphology token probe.
Surface: {surface_dediac}
Token: {target}
Lemma: {lemma}
Root: {root}
Pattern: {abstract_pattern}
Predict the token morphology.

the metadata-free prompt removes the explicit lexical fields:

Arabic morphology token probe.
Surface: {surface_dediac}
Token: {target}
Predict the token morphology.

full-metadata prompt-final and target-final differ along two axes. the final state of the complete prompt can attend to later lemma, root, and pattern fields. target-final subtoken comes from a causal prefix ending at the selected final target-overlapping token, where those later fields have not appeared. metadata-free prompt-final narrows the information-availability difference while retaining a prompt-final context.

the examples go in opposite directions. Qwen favors the full prompt, while the Jais number example favors the metadata-free version. the ordering changes because position and available information move together, so these rows are construct-validity comparisons. a prompt-final score belongs to that endpoint and its context; the target-word question uses a different interface.

pooling is part of the construct

every target-overlapping token state is archived. target-final subtoken uses the state of the final token overlapping the target; target-mean span takes the arithmetic mean over every overlapping token state. both are local operations on the same span.

averaged over the full matrix, the split gaps are:

pooling changes absolute scores and gap size. in this matrix all 132 random-minus-grouped comparisons stay positive, and complete-span storage makes the check available without another model forward pass.

alignment before probing

target position comes from validated offsets rather than a fixed [-1] index, row order, or an accidental text match. each per-example record carries the rendered prompt and target text, character and UTF-8 byte spans, token IDs and token pieces, target token indices and count, and the selected final overlapping token.

alignment requires contiguous complete span coverage. all 22 model-by-prompt-condition bundles contain 4,701 examples and 4,701 successful alignments. the probe pipeline enters analysis only after identity, ordering, shape, and finite-value checks pass.

capture once, analyze locally

extraction stores the embedding output, the prompt-final state for every transformer layer, and all target-span token states for every transformer layer. full-sequence activations are omitted. the post-capture analysis contract is effectively:

weights_required = false

with those arrays and manifests in hand, split audits, pooling analyses, probe reruns, layer curves, fragmentation checks, and error analysis can run locally. model inference is the expensive boundary; the later analyses do not depend on keeping a cloud instance alive or reloading model weights.

controls are pipeline checks

shuffled-label controls cover the two primary models (Qwen3-0.6B and Llama-3.2-1B), four interfaces, three tasks, three splits, and five folds: 360 cells in total. labels are shuffled independently within train, development, and test, and the full layer-selection pipeline is rerun. none exceeds its corresponding train-only majority baseline.

the control checks whether removing the label relationship still produces comparable performance. its scope ends there: causal use of the real-label signal would require intervention evidence. shuffled labels and the real-label probe are both linear recoverability measurements.

the reportable unit

a result should be stated with enough mechanics that its estimand is recoverable. the record should name the lexical grouping rule, prompt information available at the extraction site, target-to-token alignment, representation position, subtoken pooling rule, probe estimator, layer-selection procedure, development/test boundary, metric, and majority reference.

the precise claim is therefore of this form:

under this prompt, lexical split, aligned token span,
pooling rule, selected layer, and linear estimator,
feature X is recoverable at score Y.

that wording keeps the claim tied to the probe interface. causal use, generation behavior, and broader linguistic competence are separate measurements.

artifacts and status

the code, manifests, generated result tables, validation reports, and reproducibility artifacts are in the arabic-morphology-probing-lre repository. the frozen captures preserve the evidence needed for the stated post-extraction analyses; model weights are not part of the public artifact package.

the paper, “Lexical Split Design and Representation-Interface Validity in Decoder-Only Morphology Probing: An Arabic Evaluation Study,” is currently under editorial assignment at Language Resources and Evaluation.