Huggingface tokenizers

--- name: huggingface-tokenizers description: Fast tokenizers optimized for research and production. The Rust-based implementation tokenizes 1 GB in less than 20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, and handle padding and truncation. Integrates seamlessly with Transformers. Use when you need high-performance tokenization or custom tokenizer training. version: 1.0.0 author: Orchestra Research license: MIT tags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production] dependencies: [tokenizers, transformers, datasets] --- # HuggingFace Tokenizers - Fast Tokenization for NLP Fast, production-ready tokenizers that combine Rust’s performance with Python’s ease of use. ## When to use HuggingFace Tokenizers Use HuggingFace Tokenizers when: - You need extremely fast tokenization (<20 seconds per GB of text) - You’re training custom tokenizers from scratch - You want alignment tracking (token → original text position) - You’re building production NLP pipelines - You need to tokenize large corpora efficiently Performance: - Speed: <20 seconds to tokenize 1 GB on CPU - Implementation: Rust core with Python/Node.js bindings - Efficiency: 10:100× faster than pure Python implementations Use alternatives instead: - SentencePiece: Language-independent, used by T5/ALBERT - tiktoken: OpenAI’s BPE tokenizer for GPT models - transformers AutoTokenizer: Loads pretrained models only (uses this library internally) ## Quick Start ### Installation ``bash # Install tokenizers pip install tokenizers # With transformers integration pip install tokenizers transformers ### Load pretrained tokenizer python from tokenizers import Tokenizer # Load from HuggingFace Hub tokenizer = Tokenizer.from_pretrained("bert-base-uncased") # Encode text output = tokenizer.encode("Hello, how are you?") print(output.tokens) # ['hello', ',', 'how', 'are', 'you', '?'] print(output.ids) # [7592, 1010, 2129, 2024, 2017, 1029] # Decode back text = tokenizer.decode(output.ids) print(text) # "hello, how are you?" ### Train custom BPE tokenizer python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import Whitespace # Initialize tokenizer with BPE model tokenizer = Tokenizer(BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = Whitespace() # Configure trainer trainer = BpeTrainer( vocab_size=30000, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], min_frequency=2 ) # Train on files files = ["train.txt", "validation.txt"] tokenizer.train(files, trainer) # Save tokenizer.save("my-tokenizer.json") **Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB ### Batch encoding with padding python # Enable padding tokenizer.enable_padding(pad_id=3, pad_token="[PAD]") # Encode batch texts = ["Hello world", "This is a longer sentence"] encodings = tokenizer.encode_batch(texts) for encoding in encodings: print(encoding.ids) # [101, 7592, 2088, 102, 3, 3, 3] # [101, 2023, 2003, 1037, 2936, 6251, 102] ## Tokenization algorithms ### BPE (Byte-Pair Encoding) **How it works**: 1. Start with character-level vocabulary 2. Find most frequent character pair 3. Merge into new token, add to vocabulary 4. Repeat until vocabulary size reached **Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa python from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import ByteLevel tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>")) tokenizer.pre_tokenizer = ByteLevel() trainer = BpeTrainer( vocab_size=50257, special_tokens=["<|endoftext|>"], min_frequency=2 ) tokenizer.train(files=["data.txt"], trainer=trainer) `` Advantages: - Handles out-of-vocabulary (OOV) words well (breaks them into subwords) - Flexible vocabulary size - Good for morphologically rich languages Trade-offs: - Tokenization depends on merge order - May split common words unexpectedly ### WordPiece How it works: 1. Start with character vocabulary 2. Score merge