Overall leaderboard
| # | Model | Help | Score | Score and 95% range | chrF++ | Answers |
|---|---|---|---|---|---|---|
| 1 | Gemini FlashAPI (online) | Few-shot | 91.9% | 49.3 | 41 | |
| 2 | Gemini FlashAPI (online) | Alone | 90.2% | 48.8 | 41 | |
| 3 | Gemini FlashAPI (online) | RAG | 87.8% | 50.5 | 41 | |
| 4 | Gemini Flash-LiteAPI (online) | Few-shot | 75.6% | 43.2 | 41 | |
| 5 | Gemini Flash-LiteAPI (online) | RAG | 72.4% | 42.4 | 41 | |
| 6 | Mistral Large 4API (online) | Few-shot | 71.5% | 41.1 | 41 | |
| 7 | Mistral Large 4API (online) | RAG | 65.0% | 41.5 | 41 | |
| 8 | Ministral 3 8BOpen weights, run locally | Fine-tuning | 44.7% | 33.0 | 41 | |
| 9 | Gemini Flash-LiteAPI (online) | Alone | 38.2% | 31.2 | 41 | |
| 10 | Mistral Large 4API (online) | Alone | 36.6% | 29.2 | 41 | |
| 11 | Qwen3.5 4BOpen weights, run locally | RAG | 30.1% | 28.7 | 41 | |
| 12 | Qwen3.5 4BOpen weights, run locally | Fine-tuning | 27.6% | 28.2 | 41 | |
| 13 | Qwen3.5 4BOpen weights, run locally | Few-shot | 24.4% | 28.5 | 41 | |
| 14 | Ministral 3 8BOpen weights, run locally | RAG | 24.4% | 21.8 | 41 | |
| 15 | Ministral 3 8BOpen weights, run locally | Few-shot | 23.6% | 25.2 | 41 | |
| 16 | Ministral 3 8BOpen weights, run locally | Alone | 0.8% | 2.7 | 41 | |
| 17 | Qwen3.5 4BOpen weights, run locally | Alone | 0.0% | 9.2 | 41 |
What each kind of help adds
Each row is a model, each column a way to help it. The gain is measured test by test against the model alone.
| Model | Alone | Few-shot | RAG | Fine-tuning |
|---|---|---|---|---|
| Gemini FlashAPI (online) | 90.2% | 91.9%+1.6 pts | 87.8%−2.4 pts | not possible |
| Gemini Flash-LiteAPI (online) | 38.2% | 75.6%+37.4 pts | 72.4%+34.1 pts | not possible |
| Mistral Large 4API (online) | 36.6% | 71.5%+35.0 pts | 65.0%+28.5 pts | not possible |
| Ministral 3 8BOpen weights, run locally | 0.8% | 23.6%+22.8 pts | 24.4%+23.6 pts | 44.7%+43.9 pts |
| Qwen3.5 4BOpen weights, run locally | 0.0% | 24.4%+24.4 pts | 30.1%+30.1 pts | 27.6%+27.6 pts |
Fine-tuning is only possible for open-weight models: Gemini and Mistral Large 4 do not allow it.
From the model alone to the helped model
The grey dot is the model alone; each colored mark is the same model with one kind of help.
AloneFew-shotRAGFine-tuning
Gemini Flash
Alone · 90.2%Few-shot · 91.9%RAG · 87.8%
Gemini Flash-Lite
Alone · 38.2%Few-shot · 75.6%RAG · 72.4%
Mistral Large 4
Alone · 36.6%Few-shot · 71.5%RAG · 65.0%
Ministral 3 8B
Alone · 0.8%Few-shot · 23.6%RAG · 24.4%Fine-tuning · 44.7%
Qwen3.5 4B
Alone · 0.0%Few-shot · 24.4%RAG · 30.1%Fine-tuning · 27.6%
0%25%50%75%100%
By category
The benchmark centres on translation; the other categories, with 2 to 4 tests each, are first indicators. The number of tests is shown under each heading: read it as a trend, not a ranking.
| Model | Translation French → Kabyle25 tests | Translation Kabyle → French4 tests | Comprehension3 tests | Culture3 tests | Writing2 tests | Conversation4 tests |
|---|---|---|---|---|---|---|
| Gemini FlashFew-shot | 97 | 100 | 100 | 100 | 100 | 33 |
| Gemini FlashAlone | 96 | 100 | 100 | 100 | 67 | 42 |
| Gemini FlashRAG | 92 | 100 | 100 | 89 | 100 | 33 |
| Gemini Flash-LiteFew-shot | 85 | 100 | 22 | 100 | 50 | 25 |
| Gemini Flash-LiteRAG | 80 | 83 | 33 | 89 | 100 | 17 |
| Mistral Large 4Few-shot | 76 | 83 | 100 | 78 | 83 | 0 |
| Mistral Large 4RAG | 69 | 83 | 67 | 67 | 100 | 0 |
| Ministral 3 8BFine-tuning | 60 | 50 | 44 | 0 | 0 | 0 |
| Gemini Flash-LiteAlone | 29 | 25 | 78 | 78 | 50 | 42 |
| Mistral Large 4Alone | 25 | 92 | 33 | 56 | 83 | 17 |
| Qwen3.5 4BRAG | 33 | 42 | 11 | 0 | 50 | 25 |
| Qwen3.5 4BFine-tuning | 36 | 42 | 11 | 11 | 0 | 0 |
| Qwen3.5 4BFew-shot | 28 | 33 | 33 | 22 | 0 | 0 |
| Ministral 3 8BRAG | 27 | 42 | 0 | 33 | 33 | 0 |
| Ministral 3 8BFew-shot | 25 | 58 | 0 | 33 | 0 | 0 |
| Ministral 3 8BAlone | 0 | 0 | 0 | 0 | 17 | 0 |
| Qwen3.5 4BAlone | 0 | 0 | 0 | 0 | 0 | 0 |
Does the automatic score agree with the human grade?
Spearman rank correlation of 0.70 between the human grade and chrF++ over 576 answers: strong agreement. The automatic metric ranks answers in the same direction as the author, but cannot replace her grade: it does not understand meaning.
ρ = 0.70Spearman · 576 answers