The test set
Every Kabyle text in the test set (references, comprehension texts, prompts written in Kabyle) is handwritten by the benchmark's author, a native Kabyle speaker. None is AI-generated.
The set has 52 tests: 41 public ones, used for these results, and 11 kept private for future evaluations safe from any leak into training data.
The set deliberately centres on French → Kabyle translation (25 of 41 tests), the task that is easiest to assess objectively. The five other categories are first indicators and will be expanded in the next version.
| Category | Task | Tests |
|---|---|---|
| Translation French → Kabyle | Translate a sentence from French into Kabyle. | 25 |
| Translation Kabyle → French | Translate a sentence from Kabyle into French. | 4 |
| Comprehension | Read a short Kabyle text and answer a question about it. | 3 |
| Culture | Answer a question about Kabyle culture and traditions. | 3 |
| Writing | Write in Kabyle, in the Latin script or in Tifinagh. | 2 |
| Conversation | Reply naturally to a line, as in a conversation. | 4 |
Protocol
Same prompt for all
Every model receives exactly the same prompt, no system message, temperature 0. Each answer is stored with its exact parameters.
Blind grading
Answers are drawn at random across all models. The author grades without knowing which model answered; the model is revealed only after grading.
No leakage
Few-shot examples, the RAG index and training data are checked automatically: no sentence from the test set appears in them.
Grading scale
The three kinds of help
Few-shot
5 Kabyle–French sentence pairs, the closest to the question in the Tatoeba corpus, are added to the prompt.
RAG (retrieval-augmented)
At answer time, the model receives the 8 passages closest to the question, found by semantic search (bge-m3) in a Kabyle–French corpus.
Fine-tuning (QLoRA)
The model is lightly retrained on tens of thousands of Kabyle–French pairs, on a free GPU. Only possible for open-weight models.
Models tested
| Model | Maker | Access | Size | Configurations |
|---|---|---|---|---|
| Gemini Flash | API (online) | not disclosed | ||
| Gemini Flash-Lite | API (online) | not disclosed | ||
| Mistral Large 4 | Mistral AI | API (online) | not disclosed | |
| Ministral 3 8B | Mistral AI | Open weights, run locally | 8 billion | |
| Qwen3.5 4B | Alibaba (Qwen) | Open weights, run locally | 4 billion |
Glossary
- Human score
- Mean hand-given grade (0 to 3), scaled to 100: 2.4 / 3 is 80%. It is the main measure.
- 95% range
- With 41 tests, a score is approximate. The range gives where the model's true level very likely lies (bootstrap). If two ranges largely overlap, the gap is not significant.
- Gain (points)
- Difference between the helped model's score and the model alone, measured on the same tests. +38 points: from 37% to 75%.
- chrF++
- Automatic score (0 to 100) comparing the answer to the reference character by character and word by word. Useful as a check, but it does not understand meaning: a good translation phrased differently is penalized.
- Model alone (base)
- The model as is, with the plain prompt.
- Open weights
- A downloadable model you can run on your own machine and retrain. Here: Qwen3.5 4B and Ministral 3 8B.
Limitations
- A single annotator: grades reflect one variety of Kabyle and particular spelling choices. Several references per test reduce this bias without removing it.
- Kabyle varies from one region of Kabylia to another: vocabulary, phrasing, pronunciation. The author accepted forms from the different regions as far as possible, but she may not know every variant: a correct answer in another local variety may have been under-graded.
- 41 public tests: ranges are wide, and small gaps between models are not significant. Per-category scores (2 to 4 tests outside translation) are indicative.
- The 11 private tests were not used here: free API tiers may reuse requests, so only public tests were sent to them.
- For Gemini Flash, the model alone was queried through gemini-flash-latest, but few-shot and RAG through gemini-3.6-flash, as the “latest” version was overloaded for days. Their gaps with the model alone therefore mix the effect of the help with that of the version.
- Models change fast: results hold for the versions and date shown.
Everything is open
Code, public test set, prepared data, training notebooks and raw results are published on GitHub: every number on this site can be checked and reproduced.