Build a BPE tokenizer

Sources and honesty

Sennrich et al.: Neural Machine Translation of Rare Words with Subword Units
GPT-2 encoder and regex
RFC 3629: UTF-8
Hugging Face Tokenizers documentation

This simulation trains a small byte-level BPE tokenizer on a fixed English corpus. It uses GPT-2-style splitting, not the GPT-2 vocabulary or token IDs.

Next lessonBuild a semantic search index →