Huggingface tokenizers

--- name: huggingface-tokenizers description: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training. version: 1.0.0 author: Orchestra Research license: MIT tags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production] dependencies: [tokenizers, transformers, datasets] --- # HuggingFace Tokenizers - Fast Tokenization for NLP Fast, production-ready tokenizers with Rust performance and Python ease-of-use. ## When to use HuggingFace Tokenizers Use HuggingFace Tokenizers when: - Need extremely fast tokenization (<20s per GB of text) - Training custom tokenizers from scratch - Want alignment tracking (token → original text position) - Building production NLP pipelines - Need to tokenize large corpora efficiently Performance: - Speed: <20 seconds to tokenize 1GB on CPU - Implementation: Rust core with Python/Node.js bindings - Efficiency: 10-100× faster than pure Python implementations Use alternatives instead: - SentencePiece: Language-independent, used by T5/ALBERT - tiktoken: OpenAI's BPE tokenizer for GPT models - transformers AutoTokenizer: Loading pretrained only (uses this library internally) ## Quick start ### Installation ``bash # Install tokenizers pip install tokenizers # With transformers integration pip install tokenizers transformers ` ### Load pretrained tokenizer `python from tokenizers import Tokenizer # Load from HuggingFace Hub tokenizer = Tokenizer.from_pretrained("bert-base-uncased") # Encode text output = tokenizer.encode("Hello, how are you?") print(output.tokens) # ['hello', ',', 'how', 'are', 'you', '?'] print(output.ids) # [7592, 1010, 2129, 2024, 2017, 1029] # Decode back text = tokenizer.decode(output.ids) print(text) # "hello, how are you?" ` ### Train custom BPE tokenizer `python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import Whitespace # Initialize tokenizer with BPE model tokenizer = Tokenizer(BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = Whitespace() # Configure trainer trainer = BpeTrainer( vocab_size=30000, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], min_frequency=2 ) # Train on files files = ["train.txt", "validation.txt"] tokenizer.train(files, trainer) # Save tokenizer.save("my-tokenizer.json") ` **Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB ### Batch encoding with padding `python # Enable padding tokenizer.enable_padding(pad_id=3, pad_token="[PAD]") # Encode batch texts = ["Hello world", "This is a longer sentence"] encodings = tokenizer.encode_batch(texts) for encoding in encodings: print(encoding.ids) # [101, 7592, 2088, 102, 3, 3, 3] # [101, 2023, 2003, 1037, 2936, 6251, 102] ` ## Tokenization algorithms ### BPE (Byte-Pair Encoding) **How it works**: 1. Start with character-level vocabulary 2. Find most frequent character pair 3. Merge into new token, add to vocabulary 4. Repeat until vocabulary size reached **Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa `python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import ByteLevel tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>")) tokenizer.pre_tokenizer = ByteLevel() trainer = BpeTrainer( vocab_size=50257, special_tokens=["<|endoftext|>"], min_frequency=2 ) tokenizer.train(files=["data.txt"], trainer=trainer) ` **Advantages**: - Handles OOV words well (breaks into subwords) - Flexible vocabulary size - Good for morphologically rich languages **Trade-offs**: - Tokenization depends on merge order - May split common words unexpectedly ### WordPiece **How it works**: 1. Start with character vocabulary 2. Score merge pairs: frequency(pair) / (frequency(first) × frequency(second)) 3. Merge highest scoring pair 4. Repeat until vocabulary size reached **Used by**: BERT, DistilBERT, MobileBERT `python from tokenizers import Tokenizer from tokenizers.models import WordPiece from tokenizers.trainers import WordPieceTrainer from tokenizers.pre_tokenizers import Whitespace from tokenizers.normalizers import BertNormalizer tokenizer = Tokenizer(WordPiece(unk_token="[UNK]")) tokenizer.normalizer = BertNormalizer(lowercase=True) tokenizer.pre_tokenizer = Whitespace() trainer = WordPieceTrainer( vocab_size=30522, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], continuing_subword_prefix="##" ) tokenizer.train(files=["corpus.txt"], trainer=trainer) `` Advantages: - Prioriti