ⵣ Azul!

Do AI models speak Kabyle?

17 language-model configurations, 41 tests written by the author, a native Kabyle speaker, 697 answers graded blind. What models really know about Kabyle, and what we can teach them.

Test set 1.0.4 · November 16, 2026

Overall leaderboardScore
  1. 1Gemini FlashFew-shot91.9%
  2. 2Gemini FlashAlone90.2%
  3. 3Gemini FlashRAG87.8%
  4. 4Gemini Flash-LiteFew-shot75.6%
  5. 5Gemini Flash-LiteRAG72.4%
  6. 6Mistral Large 4Few-shot71.5%
Full leaderboard (17) →
90.2%best score, with no help at allGemini Flash
+43.9 ptslargest improvement, from fine-tuningMinistral 3 8B
697answers graded by hand, blind17 configurations
41tests across 6 categorieswritten by the author, a native speaker

Summary

Key takeaways

  1. Only one model already knows Kabyle

    Gemini Flash reaches 90.2% with no help. No other model comes close, even with the best help. And help does not improve it significantly: Few-shot +1.6 pts, RAG −2.4 pts.

  2. For large models, 5 examples are enough

    Adding a few translated sentences to the prompt (few-shot) gains 35 to 37 points for Gemini Flash-Lite and Mistral Large 4. It is the simplest and most effective help for them.

  3. Small models start from zero

    Ministral 3 8B and Qwen3.5 4B score 0.8% and 0.0%: they answer in another language, or make up Kabyle-looking words.

  4. Fine-tuning transforms a small model

    Trained on Kabyle–French pairs, Ministral 3 8B goes from 0.8% to 44.7%. It then beats much larger models used without help: Gemini Flash-Lite and Mistral Large 4.

  5. Conversation remains the weak spot

    Translation is within reach of the best models. Holding a natural conversation in Kabyle is not, for any of them: it is the lowest category everywhere.

Taqbaylit

Why Kabyle?

Kabyle (Taqbaylit) is a Berber language spoken by several million people in Kabylia, northern Algeria, and by a large diaspora, especially in France. It is mostly written in the Latin script, with specific letters such as ɛ, ɣ, ḥ or ṭ, and also in Tifinagh (ⵜⵉⴼⵉⵏⴰⵖ).

It is a low-resource language: little digital text, few tools, variable spelling. Language models, trained mostly on English, often confuse it with Arabic, French or other Berber languages.

Open data

Everything is open

Code, public test set, prepared data, training notebooks and raw results are published on GitHub: every number on this site can be checked and reproduced.