LLM Evaluation and Benchmark Development in Pashto, Dari, and 22 More Afghan Languages

• Translated test sets measure the translator

  When an English benchmark is machine-translated into Pashto, a low score can mean the model failed or the translation failed, and the report cannot tell you which. The 2026 red-team result above quantifies the cost: automated translation suppressed the measured jailbreak rate by roughly sixteen points.

• Dialect and variety collapse

  Afghanistan Dari is scored as Iranian Persian. Hazaragi, a variety of Dari with its own lexicon, is absent from the test set entirely. Afghan Turkmeni is evaluated against the standard Turkmen of Turkmenistan. Each substitution moves the score away from the population the model will actually serve.

• Script fidelity failures that pass a metric

  Pashto-specific letters — ټ ډ ړ ږ ځ څ ڼ ښ — are dropped or normalized toward Urdu or Persian forms. Output can clear a word-error-rate threshold and still be unreadable to a Pashto speaker, because the metric never looked at the orthography.

• Harm taxonomies written somewhere else

  Adversarial content in Pashto and Dari draws on the Afghan war, displacement, local political vocabulary and community-specific coded speech. An English harm taxonomy does not contain those categories, so the red team never writes the prompt that would have found the failure.

1. 01

   Scope the decision, not the dataset

   We start from the decision the score has to support — a release gate, a language expansion, a system-card claim, a contract acceptance test — and work backward to sample size, task families and agreement thresholds.

2. 02

   Design the taxonomy and rubric in-language

   Rubrics are written in Pashto or Dari first and back-checked into English, so the categories match how the language behaves rather than how English does.

3. 03

   Author items with named native speakers

   Degree-holding native speakers of the specific variety write every item. Source, author, rights status and consent are recorded per item at the point of authoring.

4. 04

   Double-annotate blind, then adjudicate

   Every item is annotated independently twice. A third senior reviewer resolves each disagreement, and the adjudication is logged rather than silently overwritten.

5. 05

   Validate statistically

   Krippendorff's alpha or Cohen's kappa per task, confidence intervals on every reported figure, and a written sample-size justification. Below the threshold set at scoping, the task goes back to rubric design.

6. 06

   Deliver, then re-test on your cadence

   Versioned dataset, datasheet, scoring harness and findings memo. The held-out split stays under separate custody so your next release can be measured against an uncontaminated set.

• AimaqایماقTier 3
• BalochiبلوچیTier 2
• BrahuiبراهویTier 3
• DariدریTier 1
• GawarbatiگواربتیTier 3
• HazaragiهزارگیTier 1
• IshkashimiاشکاشمیTier 3
• KatiکتیTier 2
• KyrgyzقرغیزیTier 3
• MunjiمنجیTier 3
• NuristaniنورستانیTier 2
• OrmuriارمړيTier 3
• ParachiپراچیTier 3
• Pashayiپشه‌ییTier 2
• PashtoپښتوTier 1
• PrasunپارونTier 2
• SanglechiسنگلیچیTier 3
• ShughniشغنیTier 3
• TirahiتیراهیTier 3
• TurkmeniترکمنیTier 2
• UzbekiازبیکیTier 2
• WaigaliویگلیTier 2
• WakhiوخیTier 3
• YidghaیدغهTier 3

Tier 1   Tier 2   Tier 3

Hazaragi is a variety of Dari and is listed as one; we staff it separately because the lexical distance is large enough to change a score. Dari deliverables are Afghanistan Dari, not Iranian Persian. Turkmeni deliverables are Afghan Turkmeni, not the standard Turkmen of Turkmenistan.

• Frontier and applied AI labs

  Pre-release evaluation, system-card and model-card evidence, and the data to decide whether a language is ready to claim.

• Federal agencies and prime contractors

  Acceptance testing of language AI before it reaches mission use, with documentation written for a contracting file.

• Model and data platform companies

  An Afghan-language evaluation bench you do not have to recruit, train and govern yourself.

• Research institutions and benchmark consortia

  Co-authored, citable datasets with datasheets, licensing and reproducible scoring.

• Health systems and humanitarian organizations

  Evaluation of AI already deployed to Afghan patients, Afghan refugees and Afghan diaspora communities before it carries clinical or protection consequences.

• 01

  We do not machine-translate a benchmark and call it coverage

  Every item in every deliverable is authored in the target language by a native speaker of the specific variety. This is the single largest source of error in published low-resource evaluation, and it is the one we removed first.

• 02

  We resolve the variety, not the language code

  Afghanistan Dari is not Iranian Persian. Afghan Turkmeni is not the Turkmen of Turkmenistan. Hazaragi is a variety of Dari with a lexicon of its own. We staff, score and report at that resolution, because a model shipped to Afghan users will be judged at that resolution.

• 03

  Scholars, not bilinguals

  We do not send you a bilingual. Evaluators hold university degrees; the people designing the instruments hold graduate degrees from leading universities. No U.S. certification exists for Afghan-language professionals, so Ariana Nexus wrote its own qualification standard, trains Afghan-language interpreters inside and outside the firm to it, and holds every evaluator on this bench to the same standard.

• 04

  Every number arrives with its agreement statistics

  Alpha per task, confidence intervals, sample-size justification, adjudication log. A score without its reliability evidence is an opinion, and an opinion is not what you are buying.

• 05

  One firm, one point of accountability

  Every part of the engagement is produced by our own people. No subcontractors, no brokered specialists, no crowd marketplace behind the curtain — which also means no unexplained drift in who is labeling your data.

• 06

  A compliance line we do not cross

  Ariana Nexus routes no documents, data or inquiries through channels controlled by the de facto authorities in Afghanistan. Contributors are diaspora-based and pseudonymized. For adversarial work, that is a safety requirement, not a preference.

• 07

  We publish, and we are citable

  The firm's language research is published under its own name so that a reviewer, a regulator or an AI search system can trace a claim back to a document rather than to a sales page.