tokenization in Arabic embedding models

the result

The study reports gains from morphology-aware tokens with Word2Vec and BERT. It reports a 60% smaller vocabulary and evaluates NER, sentiment, and POS tagging. The gains belong to those experimental settings.

results

• 60% smaller vocab, morphology-aware tokenization collapsed surface forms into their constituent morphemes
• better OOV handling, unseen words composed from known morphemes were representable
• SOTA without retraining, the modified BERT achieved SOTA on two Arabic datasets despite using a frozen pretrained model

context

this worked because Arabic's morphological system has a finite (and relatively small) set of roots, patterns, and affixes.

a morphology-aware tokenizer maps surface forms to these primitives, reducing the effective vocabulary from ~1M surface forms to ~20K morphemes, and since the embedding layer accounts for a large fraction of parameter count in smaller models, shrinking the embedding table while keeping semantic compositionality intact produces gains.