{"id":"sentencepiece","name":"sentencepiece","summary":"言語に依存しないトークナイザーで、テキストを生のUnicodeとして扱います。BPEおよびUnigramアルゴリズムをサポートしています。","body":"# SentencePiece - Language-Independent Tokenization\n\nUnsupervised tokenizer that works on raw text without language-specific preprocessing.\n\n## When to use SentencePiece\n\n**Use SentencePiece when:**\n- Building multilingual models (no language-specific rules)\n- Working with CJK languages (Chinese, Japanese, Korean)\n- Need reproducible tokenization (deterministic vocabulary)\n- Want to train on raw text (no pre-tokenization needed)\n- Require lightweight deployment (6MB memory, 50k sentences/sec)\n\n**Performance**:\n- **Speed**: 50,000 sentences/sec\n- **Memory**: ~6MB for loaded model\n- **Languages**: All (language-independent)\n\n**Use alternatives instead**:\n- **HuggingFace Tokenizers**: Faster training, more flexibility\n- **tiktoken**: OpenAI models (GPT-3.5/4)\n- **BERT WordPiece**: English-centric tasks\n\n## Quick start\n\n### Installation\n\n```bash\n# Python\npip install sentencepiece\n\n# C++ (requires CMake)\ngit clone https://github.com/google/sentencepiece.git\ncd sentencepiece\nmkdir build && cd build\ncmake .. && make -j $(nproc)\nsudo make install\n```\n\n### Train model\n\n```bash\n# Command-line (BPE with 8000 vocab)\nspm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe\n\n# Python API\nimport sentencepiece as spm\n\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='m',\n    vocab_size=8000,\n    model_type='bpe'\n)\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus\n\n### Encode and decode\n\n```python\nimport sentencepiece as spm\n\n# Load model\nsp = spm.SentencePieceProcessor(model_file='m.model')\n\n# Encode to pieces\npieces = sp.encode('This is a test', out_type=str)\nprint(pieces)  # ['▁This', '▁is', '▁a', '▁test']\n\n# Encode to IDs\nids = sp.encode('This is a test', out_type=int)\nprint(ids)  # [284, 47, 11, 1243]\n\n# Decode\ntext = sp.decode(ids)\nprint(text)  # \"This is a test\"\n```\n\n## Language-independent design\n\n### Whitespace as symbol (▁)\n\n```python\ntext = \"Hello world\"\npieces = sp.encode(text, out_type=str)\nprint(pieces)  # ['▁Hello', '▁world']\n\n# Decode preserves spaces\ndecoded = sp.decode_pieces(pieces)\nprint(decoded)  # \"Hello world\"\n```\n\n**Key principle**: Treat text as raw Unicode, whitespace = ▁ (meta symbol)\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='bpe_model',\n    vocab_size=16000,\n    model_type='bpe'\n)\n```\n\n**Used by**: mBART\n\n### Unigram (default)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='unigram_model',\n    vocab_size=8000,\n    model_type='unigram'\n)\n```\n\n**Used by**: T5, ALBERT, XLNet\n\n## Training configuration\n\n### Essential parameters\n\n```python\nspm.SentencePieceTrainer.train(\n    input='corpus.txt',\n    model_prefix='m',\n    vocab_size=32000,\n    model_type='unigram',\n    character_coverage=0.9995,  # 1.0 for CJK\n    user_defined_symbols=['[SEP]', '[CLS]'],\n    unk_piece='<unk>',\n    num_threads=16\n)\n```\n\n### Character coverage\n\n| Language Type | Coverage | Rationale |\n|---------------|----------|-----------|\n| English       | 0.9995   | Most common chars |\n| CJK (Chinese) | 1.0      | All characters needed |\n| Multilingual  | 0.9995   | Balance |\n\n## Encoding options\n\n### Subword regularization\n\n```python\n# Sample different tokenizations\nfor _ in range(3):\n    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)\n    print(pieces)\n\n# Output (different each time):\n# ['▁token', 'ization']\n# ['▁tok', 'en', 'ization']\n```\n\n**Use case**: Data augmentation for robustness.\n\n## Common patterns\n\n### T5-style training\n\n```python\nspm.SentencePieceTrainer.train(\n    input='c4_corpus.txt',\n    model_prefix='t5',\n    vocab_size=32000,\n    model_type='unigram',\n    user_defined_symbols=[f'<extra_id_{i}>' for i in range(100)],\n    unk_id=2,\n    eos_id=1,\n    pad_id=0\n)\n```\n\n### Integration with transformers\n\n```python\nfrom transformers import T5Tokenizer\n\n# T5 uses SentencePiece internally\ntokenizer = T5Tokenizer.from_pretrained('t5-base')\ninputs = tokenizer('translate English to French: Hello', return_tensors='pt')\n```\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus | BPE (16k) | Unigram (8k) |\n|--------|-----------|--------------|\n| 100 MB | 1-2 min   | 3-4 min      |\n| 1 GB   | 10-15 min | 30-40 min    |\n\n### Tokenization speed\n\n- **SentencePiece**: 50,000 sentences/sec\n- **HF Tokenizers**: 200,000 sentences/sec (4× faster)\n\n## Supported models\n\n**T5 family**: `t5-base`, `t5-large` (32k vocab, Unigram)\n**ALBERT**: `albert-base-v2` (30k vocab, Unigram)\n**XLNet**: `xlnet-base-cased` (32k vocab, Unigram)\n**mBART**: `facebook/mbart-large-50` (250k vocab, BPE)\n\n## References\n\n- **[Training Guide](references/training.md)** - Detailed options, corpus preparation\n- **[Algorithms](references/algorithms.md)** - BPE vs Unigram, subword regularization\n\n## Resources\n\n- **GitHub**: https://github.com/google/sentencepiece ⭐ 10,000+\n- **Paper**: https://arxiv.org/abs/1808.06226 (EMNLP 2018)\n- **Version**: 0.2.0+","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/02-tokenization/sentencepiece","license":"MIT","category":"document","lang":"en","tokens":1427,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/algorithms.md","size":4212,"sha256":"6927a326b067ed3aa25712a33a2601e24704f55d2146e451efb2a4619bf4fb52"},{"path":"references/training.md","size":6240,"sha256":"d1061c220d74d4e8889ebec06f6abe99a0ff3f6706435fff37b58c3eaf2931a9"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["arxiv.org"]}}