"Arabic" is not one language, and most data vendors treat it as one.

A Levantine speaker, an Egyptian speaker and a Gulf speaker can read the same sentence and disagree about whether it sounds natural. Most pipelines hide that disagreement by averaging it away or letting one reviewer decide.

We do the opposite. Every batch we deliver gets a report that shows how often independent reviewers agreed, using Cohen's kappa, a standard measure that corrects for agreement by chance.

Why it matters: if two reviewers keep disagreeing on the same items, the problem is usually the task, the guideline or the dialect fit. The disagreement tells you where the data is weak before a model trains on it.

Most vendors show you a quality percentage. We think you should see the disagreements.

If you build or evaluate Arabic models, I'd like to hear how you check dialect quality today.