Ask a popular model the same question in English and in Marathi and you may notice differences in speed, cost and quality. A key reason hides in a step most users never see: tokenization.
What is a token?
Language models do not read letters or words; they read tokens, pieces of text from a fixed vocabulary. Common English words are often a single token. If a tokenizer was built mostly from English text, text in other scripts gets split into many tiny pieces, sometimes down to individual bytes.
Why that matters
- Cost: APIs usually charge per token, so the same sentence can cost several times more in an under-served language.
- Speed: more tokens mean more steps to generate a response.
- Context: a fixed context window holds less actual content.
- Quality: fragmented tokens make it harder for the model to learn meaning, morphology and grammar.
The complexity of Indic scripts
Devanagari, Tamil, Telugu, Bengali and other Indic scripts use combining vowel signs, conjunct consonants and, in some cases, rich inflection. Good tokenizers and models must handle these units sensibly, and handle code-mixing such as Hinglish, which is how many people really type.
What it takes to do better
- Train or extend the tokenizer on balanced, multilingual Indic text so common words and morphemes become single tokens.
- Curate quality data across languages, domains and registers, with consent and licensing handled properly.
- Evaluate honestly with native-speaker benchmarks, not only translated English tests.
- Support speech and vision, because many users prefer voice and many documents are scanned.
- Make it efficient so that models run affordably, including on local hardware.
Why this is central to Bharat AI
If AI is to serve every citizen, tokenization and data cannot be an afterthought. Language-efficient models cut cost, speed up responses and raise quality, which is precisely what our sovereign model effort is aimed at.
Working on Indic language AI?
We would love to collaborate on data, evaluation and models.
Get in touch →