Tokenizer comparison
Text type
Tokenizers and vocabulary sizes
| Tokenizer | Vocab size | Models using it |
|---|---|---|
| Claude 4.7+ | ~16,384* | Claude Opus 4.7 and later |
| Claude ≤4.6 | ~49,152* | Claude 3 through Opus 4.6 |
| Laguna S 2.1 | 100,352 | Laguna S 2.1 |
| cl100k (GPT-3.5/4) | 100,261 | GPT-3.5, GPT-4, OLMo 3, OLMo 3.1 |
| EuroLLM-22B | 128,000 | EuroLLM-22B |
| DeepSeek-V4 | 129,280 | DeepSeek-V3, DeepSeek-V4 |
| Mistral Medium 3.5 | 131,072 | Mistral Medium 3.5, Small 4, Large 3 |
| Nemotron 3 | 131,072 | Nemotron 3, Nemotron 3.5, SOOFI |
| Apertus | 131,072 | Apertus |
| OpenEuroLLM 128k | 131,072 | OpenEuroLLM (TBD) |
| GLM-5.3 | 154,856 | GLM 5, 5.1, 5.2, 5.3 |
| Kimi K3 | 163,840 | Kimi K2, Kimi K3 |
| o200k | 201,088 | GPT-4o, o-series, GPT-5 family, gpt-oss, Inkling |
| Muse Glimmer | 202,048 | Muse Glimmer, Muse Spark 1.1, 1.2, ~ Llama 4 |
| Qwen3.8 | 248,077 | Qwen 3.5, Qwen 3.6, Qwen 3.8 |
| Grok 4.6 | ≥248,151* | Grok 4.5, Grok 4.6 |
| Command A+ | 255,032 | Command A+ |
| Gemini 3.x / Gemma 4 | 262,144 | Gemini 3.x, Gemma 4 |
| OpenEuroLLM 256k | 262,144 | OpenEuroLLM (TBD) |
* Claude vocab sizes are based on Land’s reconstruction. 248,151 is the highest token id we observed on the Grok API during our tests.
Winner by language
Winner by language
Tokens needed relative to o200k on the same text. Lower is better.
Compare languages and text types
Tokens needed relative to o200k on the same text. Lower is better.
Vocabulary composition
| Tokenizer | Vocab | English | 7 EU languages | East Asian scripts | Structural |
|---|---|---|---|---|---|
| Laguna S 2.1 | 100,352 | 22.7% | 2.4% | 2.9% | 39.8% |
| cl100k | 100,261 | 21.6% | 4.4% | 1.3% | 39.9% |
| DeepSeek-V4 | 129,280 | 15.8% | 4.2% | 28.8% | 20.9% |
| Command A+ | 255,032 | 15.1% | 7.7% | 9.4% | 28.8% |
| GLM-5.3 | 154,856 | 14.0% | 5.0% | 19.3% | 29.0% |
| Gemini 3.x / Gemma 4 | 262,144 | 13.8% | 7.3% | 11.0% | 27.9% |
| Kimi K3 | 163,840 | 13.1% | 1.7% | 42.9% | 23.6% |
| Nemotron 3 | 131,072 | 12.2% | 10.5% | 7.2% | 25.6% |
| Mistral Medium 3.5 | 131,072 | 12.2% | 10.5% | 7.2% | 25.6% |
| Apertus | 131,072 | 12.2% | 10.5% | 7.2% | 25.6% |
| o200k | 201,088 | 11.7% | 8.8% | 5.1% | 30.4% |
| Muse Glimmer | 202,048 | 10.6% | 8.8% | 9.3% | 28.0% |
| OpenEuroLLM 128k | 131,072 | 10.0% | 12.5% | 0.2% | 17.3% |
| OpenEuroLLM 256k | 262,144 | 8.8% | 14.2% | 0.1% | 13.1% |
| Qwen3.8 | 248,077 | 8.8% | 7.4% | 26.5% | 20.6% |
| EuroLLM-22B | 128,000 | 6.6% | 15.6% | 10.8% | 15.7% |
Tokenizer timeline
Best effort estimates, no guarantee for correctness.
Vocabulary sizes over time
Best effort estimates, no guarantee for correctness.