ⵣ Azul!
Do AI models speak Kabyle?
17 language-model configurations, 41 tests written by the author, a native Kabyle speaker, 697 answers graded blind. What models really know about Kabyle, and what we can teach them.
- 1Gemini FlashFew-shot91.9%
- 2Gemini FlashAlone90.2%
- 3Gemini FlashRAG87.8%
- 4Gemini Flash-LiteFew-shot75.6%
- 5Gemini Flash-LiteRAG72.4%
- 6Mistral Large 4Few-shot71.5%
Summary
Key takeaways
Only one model already knows Kabyle
Gemini Flash reaches 90.2% with no help. No other model comes close, even with the best help. And help does not improve it significantly: Few-shot +1.6 pts, RAG −2.4 pts.
For large models, 5 examples are enough
Adding a few translated sentences to the prompt (few-shot) gains 35 to 37 points for Gemini Flash-Lite and Mistral Large 4. It is the simplest and most effective help for them.
Small models start from zero
Ministral 3 8B and Qwen3.5 4B score 0.8% and 0.0%: they answer in another language, or make up Kabyle-looking words.
Fine-tuning transforms a small model
Trained on Kabyle–French pairs, Ministral 3 8B goes from 0.8% to 44.7%. It then beats much larger models used without help: Gemini Flash-Lite and Mistral Large 4.
Conversation remains the weak spot
Translation is within reach of the best models. Holding a natural conversation in Kabyle is not, for any of them: it is the lowest category everywhere.
Taqbaylit
Why Kabyle?
Kabyle (Taqbaylit) is a Berber language spoken by several million people in Kabylia, northern Algeria, and by a large diaspora, especially in France. It is mostly written in the Latin script, with specific letters such as ɛ, ɣ, ḥ or ṭ, and also in Tifinagh (ⵜⵉⴼⵉⵏⴰⵖ).
It is a low-resource language: little digital text, few tools, variable spelling. Language models, trained mostly on English, often confuse it with Arabic, French or other Berber languages.
Open data
Everything is open
Code, public test set, prepared data, training notebooks and raw results are published on GitHub: every number on this site can be checked and reproduced.