Do Non-English Prompts Really Cost More Tokens? A Method to Measure It

Teams that run AI agents in a language other than English keep having the same argument. One side says every non-English prompt burns more tokens. The other side says modern tokenizers fixed that, and the real question is answer quality. Both sides usually cite one figure, such as "40% more tokens", that someone measured once on some model.

Both are partly right, and a single figure settles nothing. This post looks at two papers that cover the two halves of the argument, then turns them into a method: measure the token premium on your provider, measure quality on your own tasks, and combine both into cost per solved task.

Half One: Tokenizer Cost

The paper Language Model Tokenizers Introduce Unfairness Between Languages (Petrov, La Malfa, Torr and Bibi, NeurIPS 2023) tokenized the same parallel text in many languages with 17 tokenizers. The text is 2,000 sentences from the FLORES-200 corpus, so every language says the same thing. The abstract states that "the same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases".

The raw counts are in the project repository. We divided each language's total by the English total from tokenization_lengths.csv. English is 1.00 by definition:

Two things stand out. First, the premium is real: under cl100k_base, German text costs about 1.6 times English and Hindi almost 4.8 times. Second, the premium belongs to the tokenizer, not to the language. Russian is 2.49 under cl100k_base and 1.17 under XLM-RoBERTa, a tokenizer built for many languages. So any claim like "Russian costs 40% more" is true for one tokenizer at best. It tells you nothing about the model you run today.

Tokenizers also change between model versions, even for English. Anthropic's pricing page says that Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text", and that "the exact increase depends on the content and workload shape". The OneRuler paper gives a cross-vendor example: one Tamil document is 42,124 tokens with Gemini's tokenizer and 103,990 tokens with Qwen's. A ratio you measured last year is out of date after a model upgrade.

The paper names three costs of a high premium: the price of the API call, "processing time and latency", and "the amount of content that can be provided as context". The third one matters for agents. A 4.8 times premium shrinks the context window by the same factor for that text.

Half Two: Answer Quality

Cheap tokens do not mean good answers. One ruler to measure them all (Kim, Russell, Karpinska and Iyyer, 2025) built OneRuler, a long-context benchmark in 26 languages. It tests needle-in-a-haystack retrieval and word aggregation at 8K, 32K, 64K and 128K tokens, on models such as Gemini 1.5 Flash, o3-mini-high, Qwen 2.5, Llama 3.1 and Llama 3.3.

The results do not follow the tokenizer table:

• English is not first. The abstract says English ranked "6th out of 26", "with Polish emerging as the top language". At 64K and 128K, Polish averaged 88% on the retrieval tasks across all models. English, in sixth place, averaged 83.9%.
• Low-resource languages fall behind as context grows. The gap between the five largest and five smallest languages (by Wikipedia size) grows "from 11% with a context length of 8K to 34% with context length of 128K".
• The instruction language matters. Accuracy "can fluctuate by up to 20% depending on the instruction language".
• Models fail in different ways. o3-mini-high scored 67% in English at 128K and 92% in Polish. The authors note it often wrongly answered that the needle was absent, even in high-resource languages.

So the language with the lowest token premium is not always the one with the best answers. Tokenizer cost and answer quality are two separate measurements. Merging them into "English is cheaper, so English is better" gives wrong routing decisions.

These models are from 2024 and 2025. Treat the ranking as a reason to test, not as a ranking for the model you run now.

The Method

Step 1: Build a Parallel Sample From Your Own Traffic

Take 20 to 50 real texts from your workload: alert descriptions, runbook sections, chat messages from engineers. Have each one translated by a person into every language you serve, so the meaning stays the same. FLORES-200 is a good public fallback, but your own text is better. Runbooks full of hostnames and metric names tokenize differently from news sentences.

Step 2: Count Tokens on the Exact Model

Count, do not estimate. Each vendor has a counter tied to its tokenizer:

# Anthropic: free endpoint, counts under the tokenizer of the model you pass
curl -s https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-5-5",
       "messages": [{"role": "user", "content": "<runbook text in one language>"}]}'

# OpenAI-family models: tiktoken, offline
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode(text)))

The Anthropic token counting docs say the endpoint is free within its own rate limit, returns an estimate that can differ "by a small amount" from billing, and that you should recount prompts on the model you plan to use. Gemini has a countTokens method (docs), and tiktoken covers OpenAI encodings such as o200k_base. For open models, load the tokenizer that ships with the weights.

Divide each language's total by the English total. That is your premium per language, per model.

Step 3: Weight the Premium by What Is Actually in the Context

An agent's context is mostly not human language. In an incident investigation, most tokens are tool output: logs, JSON from APIs, kubectl tables, metric names. Those look the same whatever language the team speaks. Only the human part pays the premium.

So compute a blended ratio:

blended input ratio = (1 - h) + h × premium

h is the share of input tokens that is human language. Measure it from logged prompts, do not guess. With the cl100k_base numbers above as an example, and h = 0.15:

• German (1.58): 0.85 + 0.15 × 1.58 = 1.09, so about 9% more input tokens
• Hindi (4.79): 0.85 + 0.15 × 4.79 = 1.57, so about 57% more

Output is different. If the agent answers in the team's language, the whole answer pays the premium, and output tokens are usually priced higher than input tokens. They also add latency, because the model generates them one by one.

Step 4: Measure Quality Separately

Run the same tasks in each language, with known correct answers. Also try the cross-lingual setup: instructions and system prompt in English, data and answers in the local language. OneRuler shows that the instruction language alone can move accuracy by up to 20%, so this test is cheap and worth it.

Step 5: Combine Into Cost per Solved Task

cost per solved task = cost per run / success rate

An illustrative case, not measured data: English costs 1.00 per run and solves 90% of tasks, so 1.11 per solved task. Language A costs 1.20 per run and also solves 90%, so 1.33. Language B also costs 1.20 per run but solves 70%, so 1.71. Same token premium, very different results. Quality, not the tokenizer, decides the final number. This is the same reason we compare models by cost per root cause on SRE Bench and not by price per token.

Decision Table

Where This Fits in an Agent Stack

The premium is one more input to model routing, next to price, speed and accuracy. Our post on where AI agent tokens go shows that context re-reads dominate an agent bill, so a premium on long, repeated context hurts more than a premium on one short question. In Akmatori you can save several model configurations in Settings → AI models and switch the active one, so you can run your multilingual eval on two providers and keep the one with the lowest cost per solved task. The getting started guide lists the supported providers.

Conclusion

Yes, non-English prompts usually cost more tokens, but the premium belongs to the tokenizer and changes between model generations. Answer quality per language is a separate number, and it can matter more. Count tokens on your exact model with parallel text from your own workload, weight the premium by the share of human language in the context, measure quality on its own, and plan with cost per solved task.

If your team wants AI-assisted incident workflows on your own infrastructure, Akmatori helps SRE teams investigate alerts, coordinate response, and automate safe infrastructure actions. Powered by Gcore for global infrastructure reliability.