Results

Every grade is given by hand, from 0 to 3, without knowing which model answered, then converted to a percentage. Ranges show the 95% uncertainty margin.

Overall leaderboard

#ModelHelpScoreScore and 95% rangechrF++Answers
1Gemini FlashAPI (online)Few-shot91.9%49.341
2Gemini FlashAPI (online)Alone90.2%48.841
3Gemini FlashAPI (online)RAG87.8%50.541
4Gemini Flash-LiteAPI (online)Few-shot75.6%43.241
5Gemini Flash-LiteAPI (online)RAG72.4%42.441
6Mistral Large 4API (online)Few-shot71.5%41.141
7Mistral Large 4API (online)RAG65.0%41.541
8Ministral 3 8BOpen weights, run locallyFine-tuning44.7%33.041
9Gemini Flash-LiteAPI (online)Alone38.2%31.241
10Mistral Large 4API (online)Alone36.6%29.241
11Qwen3.5 4BOpen weights, run locallyRAG30.1%28.741
12Qwen3.5 4BOpen weights, run locallyFine-tuning27.6%28.241
13Qwen3.5 4BOpen weights, run locallyFew-shot24.4%28.541
14Ministral 3 8BOpen weights, run locallyRAG24.4%21.841
15Ministral 3 8BOpen weights, run locallyFew-shot23.6%25.241
16Ministral 3 8BOpen weights, run locallyAlone0.8%2.741
17Qwen3.5 4BOpen weights, run locallyAlone0.0%9.241

Test set 1.0.4 · SHA-256 2004fad2988a… · results exported on November 16, 2026

What each kind of help adds

Each row is a model, each column a way to help it. The gain is measured test by test against the model alone.

ModelAloneFew-shotRAGFine-tuning
Gemini FlashAPI (online)90.2%91.9%+1.6 pts87.8%−2.4 ptsnot possible
Gemini Flash-LiteAPI (online)38.2%75.6%+37.4 pts72.4%+34.1 ptsnot possible
Mistral Large 4API (online)36.6%71.5%+35.0 pts65.0%+28.5 ptsnot possible
Ministral 3 8BOpen weights, run locally0.8%23.6%+22.8 pts24.4%+23.6 pts44.7%+43.9 pts
Qwen3.5 4BOpen weights, run locally0.0%24.4%+24.4 pts30.1%+30.1 pts27.6%+27.6 pts

Fine-tuning is only possible for open-weight models: Gemini and Mistral Large 4 do not allow it.

From the model alone to the helped model

The grey dot is the model alone; each colored mark is the same model with one kind of help.

AloneFew-shotRAGFine-tuning
Gemini Flash
Alone · 90.2%Few-shot · 91.9%RAG · 87.8%
Gemini Flash-Lite
Alone · 38.2%Few-shot · 75.6%RAG · 72.4%
Mistral Large 4
Alone · 36.6%Few-shot · 71.5%RAG · 65.0%
Ministral 3 8B
Alone · 0.8%Few-shot · 23.6%RAG · 24.4%Fine-tuning · 44.7%
Qwen3.5 4B
Alone · 0.0%Few-shot · 24.4%RAG · 30.1%Fine-tuning · 27.6%
0%25%50%75%100%

By category

The benchmark centres on translation; the other categories, with 2 to 4 tests each, are first indicators. The number of tests is shown under each heading: read it as a trend, not a ranking.

ModelTranslation French → Kabyle25 testsTranslation Kabyle → French4 testsComprehension3 testsCulture3 testsWriting2 testsConversation4 tests
Gemini FlashFew-shot9710010010010033
Gemini FlashAlone961001001006742
Gemini FlashRAG921001008910033
Gemini Flash-LiteFew-shot85100221005025
Gemini Flash-LiteRAG8083338910017
Mistral Large 4Few-shot768310078830
Mistral Large 4RAG698367671000
Ministral 3 8BFine-tuning605044000
Gemini Flash-LiteAlone292578785042
Mistral Large 4Alone259233568317
Qwen3.5 4BRAG33421105025
Qwen3.5 4BFine-tuning3642111100
Qwen3.5 4BFew-shot2833332200
Ministral 3 8BRAG2742033330
Ministral 3 8BFew-shot255803300
Ministral 3 8BAlone0000170
Qwen3.5 4BAlone000000

Does the automatic score agree with the human grade?

Spearman rank correlation of 0.70 between the human grade and chrF++ over 576 answers: strong agreement. The automatic metric ranks answers in the same direction as the author, but cannot replace her grade: it does not understand meaning.

ρ = 0.70Spearman · 576 answers