Let's build the GPT Tokenizer
5 人推荐 · 跨 5 个月 · 2024–2025
中文导读
Andrej Karpathy 亲授的从零实现 GPT Tokenizer 硬核教程：手写实现 Byte Pair Encoding (BPE) 以及完整的 encode / decode，讲透 Unicode 编码，揭秘了 LLM 许多诡异行为的底层分词根源。
为什么 HN 推它
讲解分词器最透彻的标杆教程，任何从事大模型研发或应用落地的工程师都值得花两小时彻底吃透。
「只要你跟大模型打交道，花 2 小时把底层逻辑彻底吃透绝对超值。」HN 网友原话：Worth spending 2h for the deeper understanding if you deal with LLMs.
英文原摘要（HackerTube / Claude 整理）
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.
Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
HN 上的提及
check out Andrej Karpathy's latest video if you have 2 hours for a deep dive
For reference, Let's build the GPT Tokenizer
People tried inputting UTF8 directly into transformers, but it doesn't work that well. Karpathy explains why
Notable example of high quality notebook
this video is a great resource