domains / families / research / sentencepiece

sentencepiece

separate

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

When to use

• Building multilingual models (no language-specific rules)
• Working with CJK languages (Chinese, Japanese, Korean)
• Need reproducible tokenization (deterministic vocabulary)
• Want to train on raw text (no pre-tokenization needed)
• Require lightweight deployment (6MB memory, 50k sentences/sec)
• **Speed**: 50,000 sentences/sec
• **Memory**: ~6MB for loaded model
• **Languages**: All (language-independent)

When not to use

• OpenAI models (GPT-3.5/4)
• English-centric tasks
• Tasks requiring faster training than SentencePiece

Skill page

Source