Method

How the benchmark was built, what the numbers mean, and what they do not say.

The test set

Every Kabyle text in the test set (references, comprehension texts, prompts written in Kabyle) is handwritten by the benchmark's author, a native Kabyle speaker. None is AI-generated.

The set has 52 tests: 41 public ones, used for these results, and 11 kept private for future evaluations safe from any leak into training data.

The set deliberately centres on French → Kabyle translation (25 of 41 tests), the task that is easiest to assess objectively. The five other categories are first indicators and will be expanded in the next version.

CategoryTaskTests
Translation French → KabyleTranslate a sentence from French into Kabyle.25
Translation Kabyle → FrenchTranslate a sentence from Kabyle into French.4
ComprehensionRead a short Kabyle text and answer a question about it.3
CultureAnswer a question about Kabyle culture and traditions.3
WritingWrite in Kabyle, in the Latin script or in Tifinagh.2
ConversationReply naturally to a line, as in a conversation.4

Protocol

  1. Same prompt for all

    Every model receives exactly the same prompt, no system message, temperature 0. Each answer is stored with its exact parameters.

  2. Blind grading

    Answers are drawn at random across all models. The author grades without knowing which model answered; the model is revealed only after grading.

  3. No leakage

    Few-shot examples, the RAG index and training data are checked automatically: no sentence from the test set appears in them.

Grading scale

0Wrong or other language
1Partly correct
2Correct, minor errors
3Correct and natural

The three kinds of help

Few-shot

5 Kabyle–French sentence pairs, the closest to the question in the Tatoeba corpus, are added to the prompt.

RAG (retrieval-augmented)

At answer time, the model receives the 8 passages closest to the question, found by semantic search (bge-m3) in a Kabyle–French corpus.

Fine-tuning (QLoRA)

The model is lightly retrained on tens of thousands of Kabyle–French pairs, on a free GPU. Only possible for open-weight models.

Models tested

ModelMakerAccessSizeConfigurations
Gemini FlashGoogleAPI (online)not disclosedAloneFew-shotRAG
Gemini Flash-LiteGoogleAPI (online)not disclosedAloneFew-shotRAG
Mistral Large 4Mistral AIAPI (online)not disclosedAloneFew-shotRAG
Ministral 3 8BMistral AIOpen weights, run locally8 billionAloneFew-shotRAGFine-tuning
Qwen3.5 4BAlibaba (Qwen)Open weights, run locally4 billionAloneFew-shotRAGFine-tuning

Glossary

Human score
Mean hand-given grade (0 to 3), scaled to 100: 2.4 / 3 is 80%. It is the main measure.
95% range
With 41 tests, a score is approximate. The range gives where the model's true level very likely lies (bootstrap). If two ranges largely overlap, the gap is not significant.
Gain (points)
Difference between the helped model's score and the model alone, measured on the same tests. +38 points: from 37% to 75%.
chrF++
Automatic score (0 to 100) comparing the answer to the reference character by character and word by word. Useful as a check, but it does not understand meaning: a good translation phrased differently is penalized.
Model alone (base)
The model as is, with the plain prompt.
Open weights
A downloadable model you can run on your own machine and retrain. Here: Qwen3.5 4B and Ministral 3 8B.

Limitations

  • A single annotator: grades reflect one variety of Kabyle and particular spelling choices. Several references per test reduce this bias without removing it.
  • Kabyle varies from one region of Kabylia to another: vocabulary, phrasing, pronunciation. The author accepted forms from the different regions as far as possible, but she may not know every variant: a correct answer in another local variety may have been under-graded.
  • 41 public tests: ranges are wide, and small gaps between models are not significant. Per-category scores (2 to 4 tests outside translation) are indicative.
  • The 11 private tests were not used here: free API tiers may reuse requests, so only public tests were sent to them.
  • For Gemini Flash, the model alone was queried through gemini-flash-latest, but few-shot and RAG through gemini-3.6-flash, as the “latest” version was overloaded for days. Their gaps with the model alone therefore mix the effect of the help with that of the version.
  • Models change fast: results hold for the versions and date shown.

Everything is open

Code, public test set, prepared data, training notebooks and raw results are published on GitHub: every number on this site can be checked and reproduced.

See the code on GitHub →