FIELD GUIDE / AZION LABS
How to evaluate Arabic AI for real Saudi workflows
A fluent Arabic answer is only the beginning. A useful evaluation asks whether the model understood the professional register, used the right evidence, and stayed within the limits of the question. This guide explains how to read AQWAL and turn benchmark results into a focused product evaluation.
By AZION LabsUpdated:
Who this is forProduct teams, Arabic AI researchers, procurement teams, and developers comparing models for Saudi business use.
Start with the task, then read the track
For an Arabic support assistant, test whether a casual customer message becomes a clear professional response without losing its meaning. For a research assistant, test whether a claim is supported by the cited source. For a policy workflow, distinguish retrieving a rule from interpreting how it applies. These are different evaluation questions even when they share the same language.
AQWAL separates professional register (DAMJ), cross-regulator knowledge (NITHAM), and national strategy context (RU’YA). Use the relevant track as a starting point, then construct an additional evaluation from the documents, audiences, and decisions your own product will encounter.
Make comparisons that answer the same question
Before comparing two scores, align the model version, prompt, dataset split, tool access, grading rubric, and generation settings. Record how refusals, invalid responses, and missing runs enter the denominator. A score on a public development split cannot be interpreted as if it came from an unseen test set.
Keep model quality separate from deployment fit. Response time, cost per completed task, data handling, and human review effort may change the operating choice even when benchmark scores are close. Document the trade-off, then test the complete application with its actual retrieval and permission controls.
Source:AI Risk Management Framework
Use benchmark coverage as evidence of scope
The charts below describe the frozen v0.1.0 item inventory, not a ranking of models. The single-answer multiple-choice subset has 118 items. The full set also includes multi-select questions, short answers, and register-transfer tasks. A category with more items is a larger slice of this benchmark, not a claim that it matters more to every organization.
Read the detailed benchmark for results, cohort dates, and pending evaluations. Keep a change log for your own test set and retain examples of both successful answers and failures. Repeat evaluation when the model, prompts, source documents, or product behavior changes.
Workflow
From use case to a reviewable model decision
- 01 →
Define the decision
Name the user, task, acceptable answer, and consequences of a wrong response.
- 02 →
Build an evidence set
Collect representative prompts and reference answers with dated sources.
- 03 →
Run comparable trials
Fix the settings, track exclusions, and keep each model version identifiable.
- 04 →
Review failures and repeat
Inspect error types and rerun the full application after material changes.
An evaluation workflow, not a certification or guarantee of production readiness.
Decision guide
Which evaluation evidence do you need?
| Question | Useful evidence | Check before comparing |
|---|---|---|
| Arabic style | Register-transfer examples | Meaning retained; appropriate audience |
| Factual reliability | Source-bound answers | Source date and claim boundaries |
| Operational reliability | End-to-end task evaluation | Tools, permissions, refusal handling |
| Model selection | Matched trials and error review | Same scope, scoring, and denominators |
Choose evidence for the decision you are making; a single aggregate score does not cover every use case.
Metrics explained
- Sample size (n)
- The number of evaluated items in a stated subset. Report it beside the score, including exclusions.
- Accuracy
- Correct scored responses divided by scored responses under a fixed rubric. It is not deployment safety.
- Coverage
- Which tasks, domains, and formats are represented. Counts describe the test inventory, not business importance.
- Held-out evaluation
- Evaluation on items kept separate from development. Keep public examples and private test results identifiable.
Data & charts
How to read these figures
Every chart identifies its period, unit, and source. Published market data describes its stated market; product inventories describe features or catalog entries. Neither is a measurement of customer results.
Single-answer multiple-choice coverage by track
118 multiple-choice items distributed across three Saudi professional tracks.
Items
Scale starts at zero
Counts reconcile to 118. This chart excludes the 28 items in other formats and does not compare model performance.
Source:AQWAL methodology, cohorts, and results
View the data table
| Category | Value |
|---|---|
| DAMJ · professional register | 31 |
| NITHAM · regulation | 54 |
| RU’YA · national strategy | 33 |
Full benchmark composition by response format
A total of 146 items, including written tasks as well as selected-answer questions.
Items
Scale starts at zero
Short answers combine 9 NITHAM and 7 RU’YA items. The categories are mutually exclusive and sum to 146; counts are not grading weights.
Source:AQWAL methodology, cohorts, and results
View the data table
| Category | Value |
|---|---|
| Single-answer multiple choice | 118 |
| Multiple select | 7 |
| Short answer | 16 |
| Register transfer | 5 |
Questions, answered
What does AQWAL measure?
AQWAL evaluates Arabic AI on professional language, Saudi regulatory context, and national strategy knowledge. Read track-level results and item formats before using an aggregate comparison.
Does a high score mean a model is ready for production?
It is useful evidence within the benchmark scope. Production decisions also require evaluation of your documents, users, tools, latency, permissions, and human review process.
Can I compare different cohorts directly?
Only after confirming matching scope and settings. A preview on public development items must remain separate from a full held-out run. The benchmark page states these distinctions.
How should I cite an AQWAL result?
Include the release, cohort date, model identifier, track, sample size, and link to the published method. Retain qualifications about missing, refused, or ungraded responses.
References & further reading
- 01AQWAL methodology, cohorts, and results ↗
AZION Labs
Primary benchmark publication. Charts reproduce the frozen v0.1.0 item counts from the June 2026 results table; later preview results remain separate.
- 02AI Risk Management Framework ↗
NIST
A voluntary reference for considering trustworthiness across AI design, development, use, and evaluation. It does not certify AQWAL or any model.