The inherited exam
When a new model claims Arabic capability, the receipts are usually familiar: a handful of famous English evaluation suites, machine-translated into Arabic, scored, and reported to two decimal places. The precision is real. The measurement is not, or not of what it claims. An exam translated is an exam transformed, and the transformation is never neutral.
We learned this the expensive way, twice, which is why two of our releases are not models at all.
What translation carries
Three things ride along with a translated benchmark. Assumptions: questions written for one culture test the knowledge distribution of that culture, so the Arabic version measures familiarity with translated trivia as much as language understanding. Artifacts: machine translation has its own phrasing habits, and models trained on machine-translated text recognize the translator’s dialect, inflating scores without inflating capability. And direction: a suite conceived in English tests Arabic only as a target, never as a source, so whole categories, diacritics, dialect variation, morphological richness, simply have no rows in the table.
The result is a benchmark that cannot fail in the places Arabic actually fails, which makes it worse than no benchmark: it is reassurance manufactured at scale.
Two confessions
Sadeed needed a diacritization benchmark, and the public options skewed toward classical religious text, overlapped with everyone’s training corpora, and compressed all errors into one number that treated a missed vowel and an inverted case ending as equals. So SadeedDiac-25 exists: registers separated, difficulty graded, sentences a reviewer would defend one by one.
Mutarjim needed a translation benchmark, and the honest ones were either narrow or already inside training sets. So Tarjama-25 exists: five thousand sentence pairs, expert-reviewed, deliberately bidirectional, because a model that translates into Arabic beautifully and out of it poorly is hiding an asymmetry the old suites never asked about. Building the exam cost a meaningful fraction of building the model. It was the fraction that made the model’s numbers mean something.
Writing from Arabic, not into it
The alternative is not exotic; it is just work. Author natively: questions conceived in Arabic by people who live in it, then translated outward if comparison is wanted, reversing the direction of inheritance. Spread the registers: classical, modern standard, and dialect, because a model serving this region meets all three before lunch. Protocol the contamination: document sources, hold out unseen material, and say so in print. And publish an error taxonomy, because a single score is a press release while a taxonomy is an instrument.
None of this requires a frontier lab. It requires believing the language is worth examining on its own terms, which is a budget line, not a breakthrough.
The inheritance test
Four questions for any Arabic benchmark
- Origin: written in Arabic, or translated into it, and by whom?
- Coverage: which registers and dialects have rows, and which have none?
- Exposure: what is the documented overlap with public training corpora?
- Resolution: does it report an error taxonomy, or one flattering number?
A capability score means exactly as much as the exam behind it. Ask about the exam first.
Coda
There is an old courtesy in this region about inherited things: you keep the gift, you honor the giver, and you do not pretend it was made for you. Translated benchmarks deserve the same courtesy. Keep them for what they are, comparisons of convenience, and write our own exams for the questions that decide anything.
The house rule at Misraj is that every model ships with its benchmark. The quieter half of the rule is that the benchmark ships with its provenance. Measurement you cannot audit is marketing with a decimal point.
Sara Chrouf
@misc{chrouf2026inheritance,
author = {Chrouf, Sara},
title = {A Benchmark Translated Is Not a Benchmark Inherited},
year = {2026},
url = {https://sarachrouf.com/essay-a-benchmark-translated.html}
}