Skip to Content
04 TokenReadme Tokenizers

HuggingFace Tokenizers - Complete Learning Module

Fast, blazing-fast tokenization with the 🤗 Tokenizers library

This module provides a complete, hands-on guide to the HuggingFace Tokenizers library - the fastest tokenization library available, with full alignment tracking and support for all major tokenization algorithms.

📚 What You’ll Learn

  • Build tokenizers from scratch
  • Train custom tokenizers on your data
  • Use pretrained tokenizers
  • Understand BPE, WordPiece, and Unigram algorithms
  • Master the tokenization pipeline
  • Optimize for production use

🚀 Quick Start

Installation

pip install tokenizers

Your First Tokenizer (2 minutes)

from tokenizers import Tokenizer
 
# Load pretrained BERT tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
 
# Encode text
output = tokenizer.encode("Hello, world!")
print(output.tokens)  # ['hello', ',', 'world', '!']
print(output.ids)     # [7592, 1010, 2088, 999]

📖 Learning Path

1. Read the Guide (1 hour)

Start with the comprehensive guide:

📘 10_huggingface_tokenizers_guide.md

This guide covers:

  • Introduction and installation
  • Quick start examples
  • Tokenization pipeline explained
  • All components (Normalizers, PreTokenizers, Models, PostProcessors, Decoders)
  • Training custom tokenizers
  • Working with pretrained models
  • Advanced features (padding, truncation, batch encoding)
  • Complete examples
  • Best practices

Sections:

  1. Introduction - What and why
  2. Quick Start - Get running in 5 minutes
  3. Tokenization Pipeline - Understanding the process
  4. Components Deep Dive - Each component explained
  5. Training Custom Tokenizers - Build your own
  6. Pretrained Tokenizers - Load existing models
  7. Advanced Features - Encoding, padding, truncation
  8. Complete Examples - Full implementations
  9. Best Practices - Tips and tricks

2. Run Quick Start Examples (30 minutes)

Practice with the quick start script:

🎯 01_tokenizers_quickstart.py

10 Interactive Examples:

  1. Load Pretrained Tokenizer - Use BERT tokenizer
  2. Build from Scratch - Create a simple BPE tokenizer
  3. Understanding Encoding - Explore the Encoding object
  4. Batch Encoding - Process multiple sequences
  5. Padding & Truncation - Handle variable lengths
  6. Encode & Decode - Round-trip conversion
  7. Sentence Pairs - Work with pairs (for NLI, QA)
  8. Vocabulary Inspection - Explore token mappings
  9. Special Tokens - Add custom tokens
  10. Performance Comparison - Batch vs single encoding
# Run all examples
python 01_tokenizers_quickstart.py
 
# Or run individual examples in Python
from tokenizers_quickstart import example_1_pretrained_tokenizer
example_1_pretrained_tokenizer()

3. Train Your Own Tokenizers (45 minutes)

Learn to train custom tokenizers:

🏋️ 02_tokenizers_training.py

7 Training Examples:

  1. BPE Tokenizer (GPT-2 Style) - Byte-level BPE
  2. WordPiece (BERT Style) - Classic BERT tokenizer
  3. Unigram (Multilingual) - SentencePiece-style for multiple languages
  4. Code Tokenizer - Domain-specific for programming languages
  5. Train from Files - Use actual text files
  6. Compare Tokenizers - See differences between models
  7. Fine-tune Tokenizer - Add tokens to existing models
# Run all training examples
python 02_tokenizers_training.py

Outputs:

  • Trained tokenizers saved in ./tokenizers/ directory
  • Ready to use in your projects
  • Comparison reports

4. Advanced Training Methods (45 minutes)

Master different training patterns:

🎓 03_advanced_training_methods.py

7 Advanced Patterns:

  1. Train from List - Simple Python lists/tuples
  2. Train from Iterables - Tuples, generators, any iterable
  3. 🤗 Datasets Library - Batch iterators for efficiency
  4. Gzip Files - Read compressed files directly
  5. Batch Efficiency - Compare single vs batch performance
  6. Custom Iterators - Filter, transform, multi-source patterns
  7. Progress Tracking - Monitor training with length parameter
# Run all advanced training examples
python 03_advanced_training_methods.py

Key Learnings:

  • Batch iterators are 10-20x faster
  • Use generators for memory efficiency
  • Progress tracking with length parameter
  • Train from any Python iterator

5. Production Guide (30 minutes)

Learn production-level considerations:

🏭 06_production_guide.md

Critical Topics:

  • Performance optimization (batch processing, parallelization)
  • Memory management (streaming, lazy loading)
  • Error handling & edge cases
  • Security considerations (input sanitization, rate limiting)
  • Monitoring & debugging
  • Common production issues & solutions

6. Tokenizer Comparison (20 minutes)

Understand different algorithms and choose the right one:

📊 07_tokenizer_comparison.md

Comparisons:

  • BPE vs WordPiece vs Unigram vs WordLevel
  • GPT vs BERT vs T5 vs LLaMA tokenizers
  • Performance benchmarks (speed, memory)
  • Language support comparison
  • Use case recommendations

Includes decision tree to help you choose!

7. Integration Guide (30 minutes)

Connect tokenizers to your ML workflow:

🔌 08_integration_guide.md

Integrations:

  • 🤗 Transformers (AutoTokenizer, models)
  • PyTorch & TensorFlow (custom datasets)
  • FastAPI / Flask (REST APIs)
  • Database storage (SQLite, PostgreSQL)
  • Streaming applications
  • Complete working examples

🎯 Learning Objectives

By the end of this module, you will be able to:

  • ✅ Load and use pretrained tokenizers
  • ✅ Build custom tokenizers from scratch
  • ✅ Choose the right tokenization algorithm for your task
  • ✅ Train tokenizers on your own data
  • ✅ Understand the full tokenization pipeline
  • ✅ Use advanced features (padding, truncation, batching)
  • ✅ Optimize tokenization for production
  • ✅ Debug tokenization issues
  • ✅ Compare different tokenization approaches

📊 File Structure

1-token/
├── README_TOKENIZERS.md                   # This file - Complete guide
├── 10_huggingface_tokenizers_guide.md        # Detailed reference (1 hour)
├── 01_tokenizers_quickstart.py            # Quick start (30 min)
├── 02_tokenizers_training.py              # Training examples (45 min)
├── 03_advanced_training_methods.py        # Advanced patterns (45 min)
│
├── 02_intro.md                               # Tokenization basics
├── tiktoken_example.py                    # tiktoken examples
├── token_exploration.py                   # Token analysis
├── token_exercises.py                     # Practice exercises
│
└── tokenizers/                            # Output directory
    ├── bpe_gpt2_style.json               # Trained BPE tokenizer
    ├── wordpiece_bert_style.json         # Trained WordPiece
    ├── unigram_multilingual.json         # Trained Unigram
    ├── code_tokenizer.json               # Code-specific tokenizer
    ├── finetuned_tokenizer.json          # Fine-tuned model
    └── list_trained.json                 # List-trained example

🔑 Key Concepts

Tokenization Algorithms

AlgorithmUse CaseExamples
BPEGeneral purpose, EnglishGPT-2, GPT-3, RoBERTa
WordPieceBERT-style modelsBERT, DistilBERT, ELECTRA
UnigramMultilingual, probabilisticT5, ALBERT, XLNet
WordLevelSimple baselineBasic models

Tokenization Pipeline

Why HuggingFace Tokenizers?

  1. Speed ⚡

    • 10-20x faster than pure Python implementations
    • Optimized Rust core with Python bindings
    • Can tokenize 1GB of text in seconds

  2. Full Alignment 🎯

    • Track exact character positions
    • Map tokens back to original text
    • Essential for span-based tasks (NER, QA)

  3. All Algorithms 🧰

    • BPE (GPT-2 style)
    • WordPiece (BERT style)
    • Unigram (SentencePiece)
    • WordLevel (baseline)

  4. Production Ready 🚀

    • Used by Transformers library
    • Battle-tested in production
    • Easy to serialize/deserialize

💡 Quick Reference

Load Pretrained

from tokenizers import Tokenizer
 
# From Hugging Face Hub
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
 
# From local file
tokenizer = Tokenizer.from_file("my-tokenizer.json")

Train Custom

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
 
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]"])
 
# Train from iterator
tokenizer.train_from_iterator(texts, trainer=trainer)
 
# Or train from files
tokenizer.train(["data.txt"], trainer)

Encode & Decode

# Encode
output = tokenizer.encode("Hello, world!")
print(output.tokens)  # Token strings
print(output.ids)     # Token IDs
print(output.offsets) # Character positions
 
# Decode
text = tokenizer.decode([7592, 1010, 2088])

Batch Processing

# Encode batch
outputs = tokenizer.encode_batch(["Text 1", "Text 2", "Text 3"])
 
# With padding
tokenizer.enable_padding(pad_id=0, pad_token="[PAD]")
outputs = tokenizer.encode_batch(texts)
 
# With truncation
tokenizer.enable_truncation(max_length=512)
outputs = tokenizer.encode_batch(texts)

🎓 Exercises

Beginner

  1. Load BERT tokenizer and encode 5 sentences
  2. Count tokens for different models on same text
  3. Find which tokens correspond to unknown words
  4. Compare BPE vs WordPiece on your text

Intermediate

  1. Train a BPE tokenizer on your domain data
  2. Add 50 domain-specific tokens to BERT tokenizer
  3. Build a tokenizer pipeline with custom normalizer
  4. Implement padding and truncation strategy

Advanced

  1. Train multilingual tokenizer (3+ languages)
  2. Build code-specific tokenizer for your language
  3. Optimize vocab size for your task
  4. Compare tokenizer performance on 1GB corpus

📈 Performance Tips

  1. Use Batch Encoding - 10-20x faster than loops
  2. Enable Padding Efficiently - Pad to multiple of 8 for GPU
  3. Choose Right Vocab Size - Larger = better coverage, slower
  4. Reuse Tokenizers - Don’t reload every time
  5. Save in JSON - Fast serialization/deserialization

🔍 Debugging Tips

# Check vocabulary
vocab = tokenizer.get_vocab()
print(f"Vocab size: {len(vocab)}")
 
# Inspect tokens
for token, id in list(vocab.items())[:10]:
    print(f"{token} -> {id}")
 
# Track alignment
output = tokenizer.encode(text)
for i, token in enumerate(output.tokens):
    start, end = output.offsets[i]
    print(f"{token} came from: {text[start:end]}")
 
# Test special tokens
print(tokenizer.token_to_id("[MASK]"))
print(tokenizer.id_to_token(0))

🌟 Next Steps

After completing this module:

  1. Integrate with Transformers - Use with 🤗 Transformers models
  2. Build NLP Pipeline - Tokenize → Model → Decode
  3. Production Deployment - Optimize for speed and memory
  4. Custom Algorithms - Implement your own tokenizer
  5. Multilingual Systems - Build language-agnostic pipelines

📚 Additional Resources

Official Documentation

Community

⏱️ Time Estimates

ActivityTimeDifficulty
Read guide1 hourBeginner
Quick start examples30 minBeginner
Training examples45 minIntermediate
Advanced training methods45 minIntermediate
Production guide30 minAdvanced
Comparison guide20 minIntermediate
Integration guide30 minAdvanced
Practice exercises2 hoursMixed
Total~7-8 hoursBeginner-Advanced

🎯 Success Criteria

You’ve mastered this module when you can:

  • Explain the tokenization pipeline
  • Choose appropriate algorithm for your task
  • Train tokenizer on your data (>95% success rate)
  • Use all encoding features (padding, truncation, batching)
  • Debug tokenization issues independently
  • Optimize for production deployment
  • Integrate with ML models

🤝 Contributing

Found an issue or want to add examples?

  1. Fork the repository
  2. Add your improvements
  3. Submit a pull request

📝 Notes

  • All examples use Python 3.7+
  • Requires tokenizers library (pip install tokenizers)
  • Some examples download pretrained models (requires internet)
  • Output files saved in ./tokenizers/ directory

❓ FAQ

Q: Which tokenizer algorithm should I use? A: Use BPE for general English, WordPiece for BERT-style, Unigram for multilingual.

Q: How much data do I need to train? A: Minimum 1MB text for basic vocab, 10MB+ for production quality.

Q: Can I use with other frameworks? A: Yes! Works standalone or with Transformers, FastAI, etc.

Q: Is it faster than tiktoken? A: Yes, generally 2-5x faster due to Rust core.

Q: How do I handle unknown words? A: All tokenizers have unk_token that represents unknown tokens.

Q: Can I add new tokens later? A: Yes! Use add_tokens() or add_special_tokens().

Happy Tokenizing! 🚀

Built with ❤️ by the AI/ML learning community

Last updated on
Readme Tiktoken05 Embeddings