
Most AI language models are trained mainly on English and a few other big languages. Very few tests exist to check how well they work in Burmese.
We built a Burmese evaluation dataset, had it checked by human reviewers, and used it to test four small, open AI models on everyday language tasks.
The dataset follows the format and task list of SEA-HELM, the Southeast Asian language benchmark from AI Singapore, so our results can be compared with other Southeast Asian languages.
How the Evaluation Data Was Created
We built the data in four steps:
Step 1 – Start from SEA-HELM (source data). We downloaded the SEA-HELM datasets. Some tasks already have Burmese items, for example sentiment, toxicity and instruction following. We kept all of them.
Step 2 – Add more Burmese data (extended data). To make the tests bigger and more varied, we added public Burmese data: FLORES+ translation sentences, a Burmese sentiment dataset and two Burmese hate-speech datasets from Hugging Face.
Step 3 – Translate what is missing. Some SEA-HELM tasks have no Burmese items. We translated them from Thai, Indonesian, Tamil, Vietnamese, Chinese, Tagalog or English using Gemini. Items that only make sense in another country's culture were not translated.
Step 4 – Check by hand. Human reviewers checked the items, gave each one a score and corrected mistakes (see “Human Review” below).
Examples from the Dataset
Here are some real items from the dataset. The English line is a translation to help readers.
Task: Sentiment analysis
example:မိုးအေးအေးနဲ့ အိမ်ထဲမှာ ကော်ဖီပူပူလေးသောက်ရင်း စာဖတ်ရတာ အရမ်းဇိမ်ရှိတာပဲ
“Reading at home with a hot coffee on a cool rainy day is so relaxing.”
Correct Answer: Positive
Task: Machine translation
Example: English: 28-year-old Vidal had joined Barça three seasons ago, from Sevilla.
Burmese reference: အသက် ၂၈နှစ်အရွယ် ဗီဒယ်လ်သည် လွန်ခဲ့သော သုံးရာသီတွင် ဆီဗီးလားမှ ဘာကာသို့ ပြောင်းလာခဲ့သည်။
Model's translation is compared with the reference
Task: Causal reasoning
Example: ထိုပစ္စည်းကို လေဖောင်းပလတ်စတစ်ဖြင့် ထုပ်ပိုးထားသည်။ (အကြောင်းအရင်း)
“The item was packed in bubble wrap.” What is the reason? (A) The item is fragile. (B) The item is small.
Correct Answer: A
Human Review (Manual Verification)
Machine translation and automatic tools can make mistakes, so human reviewers from our QC team checked the data in Excel review sheets. They read each item and gave it a score from 1 to 5 for each of these points:
- Completeness: nothing is missing from the text, question or answer.
- Fluency: the Burmese reads naturally and smoothly.
- Sensibility: the item makes sense and is logical.
Text clearance: correct spelling and font, and no Zawgyi (old Burmese encoding) problems.
Some tasks had extra points. Translation was also scored for grammar and faithfulness to the original meaning. Summaries were scored for relevance and coherence, and instructions for clarity. Reviewers also marked whether the given answer or label was correct (yes or no). When an item had a problem, they wrote a corrected version and a comment.
On the 1 to 5 scale, 5 means fully correct and natural, 4 means mostly good, 3 means partly good, and 1 or 2 means serious problems.
How We Tested the Models
We tested four small open models that can run on a single GPU:
Gemma-4-E2B-it (Google)
Gemma-SEA-LION-v4.5-E2B-IT (AI Singapore, built on Gemma)
Qwen3.5-4B (Alibaba)
Llama-3.2-3B-Instruct (Meta)
Each model was run with three different generation settings (fixed, shared, and each maker's recommended setting). This shows whether a score comes from the model itself or only from the setting. The scoring methods were adjusted for Burmese, which has no spaces between words, so we count characters or syllables instead of words. For multi-turn chat, an AI judge (Gemini) rated each answer from 1 to 10.
Results
The following list shows each model's average score over the three settings. Higher is better.
Sentiment Analysis (Accuracy) Gemma - 76%, SEA-LION-76%, QWEN- 25%, LLAMA-1%
Machine Translation (chrF) Gemma - 49.6%, SEA-LION-49.8%, QWEN- 29.6%, LLAMA- "-"
Question Answering (Char-F1) Gemma - 45.7%, SEA-LION-44.1%, QWEN- 32.9%, LLAMA- "-"
Metaphor understanding (Accuracy) Gemma - 74%, SEA-LION-73%, QWEN- 51%, LLAMA-47%
Instruction following (Pass rate) Gemma - 70%, SEA-LION-67%, QWEN- 44%, LLAMA-25%
Multi-turn chat (Judge score /10) Gemma - 5.5%, SEA-LION-5.5%, QWEN- 2.1%, LLAMA-1.1%
Summarization (ROUGE-L) Gemma - 17.6%, SEA-LION-17.6%, QWEN- 13.3%, LLAMA-"-"
“–” means the model was not tested on that task. Results for reading, toxicity, inference and causal reasoning, and for the four new datasets, will be added later.
Key findings
- Gemma and SEA-LION are clearly the best for Burmese and score almost the same on every task. Qwen is weaker, and Llama is the weakest.
- The ranking is real. It stayed the same under all three generation settings. The settings only made a small difference in open-ended chat.
- Even the best models have gaps. They rarely detect neutral sentiment, they write summaries that are too long, they struggle with exact rules such as “end with this phrase”, and they reach only about 5.5 out of 10 in conversation.
- Smaller is faster, not better. Llama and Qwen use less memory (2.5 GB and 3.4 GB) and run faster, but Gemma and SEA-LION (about 6.9 GB) give much better Burmese answers.
Next Steps
- Apply the reviewers' corrections, starting with the question-answering and reading datasets.
- Review the Burmese grammar dataset with a Burmese linguist.
- Test the models on the four new Burmese datasets.
- Add the results for the remaining four tasks.
References
These are the published research papers behind the benchmark, data and scoring methods used in this work.
Benchmarks and datasets
1.Susanto, Y. et al. (2025). SEA-HELM: Southeast Asian Holistic Evaluation of Language Models. arXiv. https://arxiv.org/abs/2502.14301
The benchmark our dataset follows.
2.Bandarkar, L. et al. (2024). The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. ACL 2024. https://arxiv.org/abs/2308.16884
Source of the multiple-choice reading task.
3.NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv. https://arxiv.org/abs/2207.04672
Introduced the FLORES-200 translation benchmark, which FLORES+ is based on.
Evaluation methods
4.Popović, M. (2015). chrF: character n-gram F-score for automatic MT evaluation. WMT 2015. https://aclanthology.org/W15-3049
Translation score (chrF).
5.Rei, R. et al. (2020). COMET: A Neural Framework for MT Evaluation. EMNLP 2020. https://aclanthology.org/2020.emnlp-main.213
Translation score (COMET).
6.Juraska, J. et al. (2024). MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. WMT 2024. https://aclanthology.org/2024.wmt-1.35
Translation score (MetricX-24).
7.Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. https://aclanthology.org/W04-1013
Summary score (ROUGE-L).
8.Rajpurkar, P. et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. https://aclanthology.org/D16-1264
Question-answering scores (Exact Match and F1).
9.Zhou, J. et al. (2023). Instruction-Following Evaluation for Large Language Models (IFEval). arXiv. https://arxiv.org/abs/2311.07911
Instruction-following test.
10.Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. https://arxiv.org/abs/2306.05685
Multi-turn chat test and AI-judge scoring.