From 45 Tokens to 27: Measured Korean Token Reduction in Recent Models and the Reason Why

The decisive reason GPT-4o and Llama 3 were able to dramatically improve token efficiency for non-English languages, and Korean in particular, is that they massively expanded the vocabulary size of the tokenizer and poured in large volumes of multilingual training data. The figures below are based on OpenAI's official announcement and Meta's published specifications. (These cite public sources rather than operator-environment measurement, so the numbers you feel in practice will vary with the model version and the input sentence.)

1. GPT-4o: a twofold vocabulary expansion and a 1.7x efficiency gain

When OpenAI released GPT-4o it discarded the tokenizer used by GPT-4 and GPT-3.5 (cl100k_base) and introduced a newly designed o200k_base tokenizer.

• Vocabulary size expansion: the number of word and character combinations the tokenizer can hold grew twofold, from about 100,000 (cl100k) to 200,000 (o200k).
• Korean token reduction: by OpenAI's official figures, processing a Korean sentence of identical meaning uses about 1.7 times fewer tokens (roughly a 40 percent reduction).

Measured comparison (OpenAI's official example)

Sentence: "Hello, my name is GPT-4o. I am a new type of language model, nice to meet you!" - GPT-4 (previous cl100k_base): 45 tokens consumed - GPT-4o (o200k_base): 27 tokens consumed - Result: the same sentence is processed in 18 fewer tokens (a 1.7x efficiency gain)

2. Llama 3: a fourfold vocabulary and the end of byte fragmentation

Meta's Llama 2 was English-centric, so Korean performance and token efficiency suffered. With Llama 3, however, a structural change arrived.

• Tokenizer method and vocabulary size change:

- Llama 2: SentencePiece based / 32,000 (32k) vocabulary - Llama 3: Tiktoken-style BPE based / 128,000 (128k) vocabulary

• Byte fragmentation (byte fallback) mitigated: in Llama 2, Korean words absent from the vocabulary were often split into several byte tokens per character. In Llama 3, thanks to the 128k vocabulary, Korean syllables and common words map to a single token far more often.

3. How exactly they improved it (technical mechanism)

The core principle: whole-string registration of frequent sequences

The BPE (Byte Pair Encoding) algorithm is the method of merging strings that frequently co-occur into a single token.

In the past the vocabulary had insufficient capacity, so English was filled in first and Korean was left to be split into small fragments. Expanding the vocabulary to 128k–200k made it possible for frequent Korean word and particle combinations to be registered directly as one token in the vocabulary, and this is the direct cause of the drop from 45 tokens to 27.

4. Three changes users and developers can feel

1. Lower API cost: the number of tokens consumed when handling Korean prompts and responses itself drops, so the cost of the same task decreases.
2. Better response latency: AI outputs one token at a time, sequentially. When the number of tokens to emit falls from 45 to 27, the time to complete a generated sentence is physically shorter.
3. Longer documents fit in one call: even with the same context window (for example 128k tokens), you can feed a larger amount of Korean text in one go for summarization and analysis.

References

• OpenAI GPT-4o announcement (o200k tokenizer, the Korean 45→27 token example): OpenAI official blog and tokenizer documentation
• Meta Llama 3 model card (128k vocabulary, Tiktoken-based BPE): Meta's official model card