{"id":"nanogpt","name":"nanogpt","summary":"教育用GPTの実装は~300行で完成します。OpenWebText上でGPT-2(124M)を再現しています。","body":"# nanoGPT - Minimalist GPT Training\n\n## Quick start\n\nnanoGPT is a simplified GPT implementation designed for learning and experimentation.\n\n**Installation**:\n```bash\npip install torch numpy transformers datasets tiktoken wandb tqdm\n```\n\n**Train on Shakespeare** (CPU-friendly):\n```bash\n# Prepare data\npython data/shakespeare_char/prepare.py\n\n# Train (5 minutes on CPU)\npython train.py config/train_shakespeare_char.py\n\n# Generate text\npython sample.py --out_dir=out-shakespeare-char\n```\n\n**Output**:\n```\nROMEO:\nWhat say'st thou? Shall I speak, and be a man?\n\nJULIET:\nI am afeard, and yet I'll speak; for thou art\nOne that hath been a man, and yet I know not\nWhat thou art.\n```\n\n## Common workflows\n\n### Workflow 1: Character-level Shakespeare\n\n**Complete training pipeline**:\n```bash\n# Step 1: Prepare data (creates train.bin, val.bin)\npython data/shakespeare_char/prepare.py\n\n# Step 2: Train small model\npython train.py config/train_shakespeare_char.py\n\n# Step 3: Generate text\npython sample.py --out_dir=out-shakespeare-char\n```\n\n**Config** (`config/train_shakespeare_char.py`):\n```python\n# Model config\nn_layer = 6          # 6 transformer layers\nn_head = 6           # 6 attention heads\nn_embd = 384         # 384-dim embeddings\nblock_size = 256     # 256 char context\n\n# Training config\nbatch_size = 64\nlearning_rate = 1e-3\nmax_iters = 5000\neval_interval = 500\n\n# Hardware\ndevice = 'cpu'  # Or 'cuda'\ncompile = False # Set True for PyTorch 2.0\n```\n\n**Training time**: ~5 minutes (CPU), ~1 minute (GPU)\n\n### Workflow 2: Reproduce GPT-2 (124M)\n\n**Multi-GPU training on OpenWebText**:\n```bash\n# Step 1: Prepare OpenWebText (takes ~1 hour)\npython data/openwebtext/prepare.py\n\n# Step 2: Train GPT-2 124M with DDP (8 GPUs)\ntorchrun --standalone --nproc_per_node=8 \\\n  train.py config/train_gpt2.py\n\n# Step 3: Sample from trained model\npython sample.py --out_dir=out\n```\n\n**Config** (`config/train_gpt2.py`):\n```python\n# GPT-2 (124M) architecture\nn_layer = 12\nn_head = 12\nn_embd = 768\nblock_size = 1024\ndropout = 0.0\n\n# Training\nbatch_size = 12\ngradient_accumulation_steps = 5 * 8  # Total batch ~0.5M tokens\nlearning_rate = 6e-4\nmax_iters = 600000\nlr_decay_iters = 600000\n\n# System\ncompile = True  # PyTorch 2.0\n```\n\n**Training time**: ~4 days (8× A100)\n\n### Workflow 3: Fine-tune pretrained GPT-2\n\n**Start from OpenAI checkpoint**:\n```python\n# In train.py or config\ninit_from = 'gpt2'  # Options: gpt2, gpt2-medium, gpt2-large, gpt2-xl\n\n# Model loads OpenAI weights automatically\npython train.py config/finetune_shakespeare.py\n```\n\n**Example config** (`config/finetune_shakespeare.py`):\n```python\n# Start from GPT-2\ninit_from = 'gpt2'\n\n# Dataset\ndataset = 'shakespeare_char'\nbatch_size = 1\nblock_size = 1024\n\n# Fine-tuning\nlearning_rate = 3e-5  # Lower LR for fine-tuning\nmax_iters = 2000\nwarmup_iters = 100\n\n# Regularization\nweight_decay = 1e-1\n```\n\n### Workflow 4: Custom dataset\n\n**Train on your own text**:\n```python\n# data/custom/prepare.py\nimport numpy as np\n\n# Load your data\nwith open('my_data.txt', 'r') as f:\n    text = f.read()\n\n# Create character mappings\nchars = sorted(list(set(text)))\nstoi = {ch: i for i, ch in enumerate(chars)}\nitos = {i: ch for i, ch in enumerate(chars)}\n\n# Tokenize\ndata = np.array([stoi[ch] for ch in text], dtype=np.uint16)\n\n# Split train/val\nn = len(data)\ntrain_data = data[:int(n*0.9)]\nval_data = data[int(n*0.9):]\n\n# Save\ntrain_data.tofile('data/custom/train.bin')\nval_data.tofile('data/custom/val.bin')\n```\n\n**Train**:\n```bash\npython data/custom/prepare.py\npython train.py --dataset=custom\n```\n\n## When to use vs alternatives\n\n**Use nanoGPT when**:\n- Learning how GPT works\n- Experimenting with transformer variants\n- Teaching/education purposes\n- Quick prototyping\n- Limited compute (can run on CPU)\n\n**Simplicity advantages**:\n- **~300 lines**: Entire model in `model.py`\n- **~300 lines**: Training loop in `train.py`\n- **Hackable**: Easy to modify\n- **No abstractions**: Pure PyTorch\n\n**Use alternatives instead**:\n- **HuggingFace Transformers**: Production use, many models\n- **Megatron-LM**: Large-scale distributed training\n- **LitGPT**: More architectures, production-ready\n- **PyTorch Lightning**: Need high-level framework\n\n## Common issues\n\n**Issue: CUDA out of memory**\n\nReduce batch size or context length:\n```python\nbatch_size = 1  # Reduce from 12\nblock_size = 512  # Reduce from 1024\ngradient_accumulation_steps = 40  # Increase to maintain effective batch\n```\n\n**Issue: Training too slow**\n\nEnable compilation (PyTorch 2.0+):\n```python\ncompile = True  # 2× speedup\n```\n\nUse mixed precision:\n```python\ndtype = 'bfloat16'  # Or 'float16'\n```\n\n**Issue: Poor generation quality**\n\nTrain longer:\n```python\nmax_iters = 10000  # Increase from 5000\n```\n\nLower temperature:\n```python\n# In sample.py\ntemperature = 0.7  # Lower from 1.0\ntop_k = 200       # Add top-k sampling\n```\n\n**Issue: Can't load GPT-2 weights**\n\nInstall transformers:\n```bash\npip install transformers\n```\n\nCheck model name:\n```python\ninit_from = 'gpt2'  # Valid: gpt2, gpt2-medium, gpt2-large, gpt2-xl\n```\n\n## Advanced topics\n\n**Model architecture**: See [references/architecture.md](references/architecture.md) for GPT block structure, multi-head attention, and MLP layers explained simply.\n\n**Training loop**: See [references/training.md](references/training.md) for learning rate schedule, gradient accumulation, and distributed data parallel setup.\n\n**Data preparation**: See [references/data.md](references/data.md) for tokenization strategies (character-level vs BPE) and binary format details.\n\n## Hardware requirements\n\n- **Shakespeare (char-level)**:\n  - CPU: 5 minutes\n  - GPU (T4): 1 minute\n  - VRAM: <1GB\n\n- **GPT-2 (124M)**:\n  - 1× A100: ~1 week\n  - 8× A100: ~4 days\n  - VRAM: ~16GB per GPU\n\n- **GPT-2 Medium (350M)**:\n  - 8× A100: ~2 weeks\n  - VRAM: ~40GB per GPU\n\n**Performance**:\n- With `compile=True`: 2× speedup\n- With `dtype=bfloat16`: 50% memory reduction\n\n## Resources\n\n- GitHub: https://github.com/karpathy/nanoGPT ⭐ 48,000+\n- Video: \"Let's build GPT\" by Andrej Karpathy\n- Paper: \"Attention is All You Need\" (Vaswani et al.)\n- OpenWebText: https://huggingface.co/datasets/Skylion007/openwebtext\n- Educational: Best for understanding transformers from scratch","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/01-model-architecture/nanogpt","license":"MIT","category":"document","lang":"en","tokens":1810,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/architecture.md","size":11716,"sha256":"0a5bef9e725f8fdb0fe649688168562c69cd82deea279d289939df4a5aec23d2"},{"path":"references/data.md","size":11212,"sha256":"316bcdacc6e8538b7315d8e079bd71a00815bec435fad39b42e85be1a5f3a0de"},{"path":"references/training.md","size":13551,"sha256":"82cba9bf57cb82a2120f637f839badc1c54920be5ca41a00fd22042c45665119"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[{"code":"code.eval","kind":"dangerous-code","where":"references/training.md:515","excerpt":"eval(","message":"evaluates code at runtime","severity":"warn"}],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["d4mucfpksywv.cloudfront.net","huggingface.co"]}}