Research
Arabic breaks silently. This is the measurement.
Independent, reproducible evaluation of the Arabic language technology that production work depends on — published from Abu Dhabi. Every finding ships with an open test set, the scorer that produced it, the raw per-item results, and a plain statement of what the measurement does not establish.
Is the Arabic that AI models train on already broken?
The recipe that corrupts Arabic appears in 3,168 indexed Python files. If it ever ran before a corpus was saved, every model trained on that corpus learned from text no Arabic reader would accept. Nobody had measured whether it did.
276 datasets readable, 26,318 rows, 119,517 text fields. One stores contextual
glyph forms in every label sampled — 1,000 of 1,000, across ten windows
spanning its whole split — and it is an OCR set, so the affected field is the
ground‑truth label, which is the path from a rendering bug into model weights.
Only the shaping half of the recipe ran there, so word order is intact and
NFKC alone recovers the text; I first reported it as reordered too, which
was wrong, and the correction is on the dataset card.
Twenty more carry isolated stray glyph forms: not the same defect, but measured against
four real vocabularies, one stray form in a four‑letter word is enough for an
Arabic‑specific tokenizer to emit [UNK] and discard the word rather
than split it. To be precise about scale, that dataset has 17 downloads. This
shows the mechanism reaches training data. It is not evidence that widely‑used
Arabic corpora are affected.
65 datasets could not be read, and the viewer fails most often on the largest corpora,
so this covers the mid and small end well and the head of the distribution poorly. The
scanner ships with the results, so the numbers can be re‑derived rather than trusted.
The audit is deposited and citable:
doi:10.5281/zenodo.22733934, An audit of presentation-form corruption
in 341 public Arabic datasets, 13 September 2026 — so a paper can cite the
version it actually read rather than a repository that may have moved on.
arabic-corpus-audit
·
doi.org/10.5281/zenodo.22733934
·
arabic-lint (pip install arabic-lint)
One of these reads ٢٠٢٦ as a year. The other reads it as noise.
Arabic is written with two numeral systems and both appear constantly in real copy. The same sentence written with Arabic-Indic digits rather than Western drops from 73% to 7% intelligible on one leading speech engine. A second engine handles both digit forms at 87%, so this is a missing normalisation step rather than a hard problem. It goes unnoticed because voice demos are prose, and prose has no numbers.
The Arabic fix everyone recommends is now the thing breaking your Arabic
arabic_reshaper + python-bidi is the advice in nearly every
tutorial. On any renderer that already does complex-text layout it produces
مكب ابحرم where مرحبا بكم belongs, because
the work happens twice. 15/15 correct without it, 0/15 with it. Filed
upstream against a package downloaded 5.4 million times a month.
Your Arabic PDF is fine. What reads it is not.
A PDF looks right, so it is trusted — but almost nothing that consumes one looks at it. Portals parse it, ATS systems read it, compliance tools scan it. Across four generation routes and three extractors, reversed, ligature-mangled and space-collapsed are three distinct failure modes, not one.
Upstream contributions
The findings above did not stay on this site. Where one turned out to be a defect in a library rather than a fact about Arabic, it was filed against that library — which is the only version of this work anyone can check without taking my word for it, and the only place it can be rejected by someone who maintains the code.
pypdf extracted Arabic-Indic digits backwards. The codepoints
U+0660–U+0669 sat inside the library's right-to-left
character range, so each digit was prepended to the output instead of appended:
١٢٣٤ came out of a PDF as ٤٣٢١. Arabic
runs right to left; Arabic numbers do not, and that one-line distinction is the
whole bug. Reviewed, approved and merged by stefan6419846. The full
walk-through is
written
up here.
To be precise about the size: the library change is two lines. The other
81 are a new test.
And it is not in a release yet: the newest tag, 6.18.1, predates the
merge, so nobody has this fix by installing pypdf today.
What this is, and what it is not: it is my first merged code
contribution. The matplotlib pull request below is a thirty-line documentation change, and
I would rather say that here than let a reader take the two for the same thing. What makes
this one worth listing is not its size — it is that a maintainer of a library with
over ten thousand stars read the diff, approved it and merged it. That is someone else's
judgement, not my own measurement of my own claim.
py-pdf/pypdf #4077 · merged · not yet in a released version
matplotlib 3.11 shapes Arabic itself. That silently reverses the
arabic_reshaper + python-bidi workaround that every tutorial,
Stack Overflow answer and blog post still recommends — no error, no warning, just
backwards labels on a chart nobody who reads Arabic was asked to check. The upgrade guide
now tells 3.11 upgraders to remove that workaround.
To be precise about what was merged: the pull request is a documentation
change, thirty lines. The work behind it was establishing that the reversal happens at
all, that it is unconditional, and that the fix everyone recommends had become the bug.
Three maintainers, including the project lead, engaged with the finding.
matplotlib #32263 · merged
The same class of defect one layer down: under the Raqm layout engine, pre-shaped Arabic is corrupted rather than drawn. Compounded by a trap that costs people days — whether Raqm is active is a property of the runtime, not of the Pillow version, so the identical wheel behaves differently on two machines depending on the host libraries. The patch warns at run time and documents that the layout engine is a build-time decision.
Pillow #9925 · open, 8 of 8 checks passing, no maintainer response since 5 September
A fuller account — including four bug reports filed against projects that ship the broken recipe, of which two were confirmed and fixed by a maintainer and two have had no reply at all — is on the work and evidence page.
Method
Findings come out of production, not a lab: each began with Arabic breaking in delivered work. Test sets are small and public, scorers are published alongside the results, and every claim is reproducible from the repository.
In two of the three benchmark studies the measurement contradicted the assumption it started from, and the write-ups say so. In one, a pass/fail scorer reported a broken configuration as working; adding a middle band is what surfaced the real result. Designing a test that can prove you wrong is most of the work.
Corrections
When four faults were found in the scoring code behind the speech study, a documented correction was published with old-to-new figures per cell rather than a silent revision. A measurement that quietly changes its numbers is worth less than one that never published them.
Elsewhere
Datasets: huggingface.co/syamjithnk · Code: github.com/genviz-ai · Work & evidence
Press and partnership enquiries: get in touch.