Study: non-English languages can consume far more AI tokens, with the most expensive using 1.5x-5x more than the cheapest, resulting in a “tokenizer tax”
restofworld.org via Techmeme
Some languages require far more tokens for AI models to process, raising costs and reducing context for the people who use them.
A short brief from Techmeme’s public feed, in the source language. The full original article has not been read.
More from Techmeme
Lumaril, which develops AI software for optimizing industrial production by connecting to existing control systems, raised a $66M Series A at a $400M valuation
Techmeme 2 h ago
DeepSeek open sources Super-Agent as a research prototype, a proactive coding agent it says can monitor group chats for coding issues and suggest solutions
Techmeme 2 h ago
Insurance software maker Guidewire agrees to acquire Hyperexponential, which makes AI-powered insurance underwriting and pricing software, for $1.7B in cash
Techmeme 3 h ago
Tests: chatbots rely on three obscure AI-built voter sites for US midterm answers; Copilot cited the error-prone Ballot Brief in 75% of tests and ChatGPT in 22%
Techmeme 7 h ago