{"id":"openraiser-accelerate","name":"huggingface-accelerate","summary":"最もシンプルな分散トレーニングAPIです。4行でPyTorchスクリプトに分散サポートを追加できます。","body":"# HuggingFace Accelerate - Unified Distributed Training\n\n## Quick start\n\nAccelerate simplifies distributed training to 4 lines of code.\n\n**Installation**:\n```bash\npip install accelerate\n```\n\n**Convert PyTorch script** (4 lines):\n```python\nimport torch\n+ from accelerate import Accelerator\n\n+ accelerator = Accelerator()\n\n  model = torch.nn.Transformer()\n  optimizer = torch.optim.Adam(model.parameters())\n  dataloader = torch.utils.data.DataLoader(dataset)\n\n+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n\n  for batch in dataloader:\n      optimizer.zero_grad()\n      loss = model(batch)\n-     loss.backward()\n+     accelerator.backward(loss)\n      optimizer.step()\n```\n\n**Run** (single command):\n```bash\naccelerate launch train.py\n```\n\n## Common workflows\n\n### Workflow 1: From single GPU to multi-GPU\n\n**Original script**:\n```python\n# train.py\nimport torch\n\nmodel = torch.nn.Linear(10, 2).to('cuda')\noptimizer = torch.optim.Adam(model.parameters())\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=32)\n\nfor epoch in range(10):\n    for batch in dataloader:\n        batch = batch.to('cuda')\n        optimizer.zero_grad()\n        loss = model(batch).mean()\n        loss.backward()\n        optimizer.step()\n```\n\n**With Accelerate** (4 lines added):\n```python\n# train.py\nimport torch\nfrom accelerate import Accelerator  # +1\n\naccelerator = Accelerator()  # +2\n\nmodel = torch.nn.Linear(10, 2)\noptimizer = torch.optim.Adam(model.parameters())\ndataloader = torch.utils.data.DataLoader(dataset, batch_size=32)\n\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)  # +3\n\nfor epoch in range(10):\n    for batch in dataloader:\n        # No .to('cuda') needed - automatic!\n        optimizer.zero_grad()\n        loss = model(batch).mean()\n        accelerator.backward(loss)  # +4\n        optimizer.step()\n```\n\n**Configure** (interactive):\n```bash\naccelerate config\n```\n\n**Questions**:\n- Which machine? (single/multi GPU/TPU/CPU)\n- How many machines? (1)\n- Mixed precision? (no/fp16/bf16/fp8)\n- DeepSpeed? (no/yes)\n\n**Launch** (works on any setup):\n```bash\n# Single GPU\naccelerate launch train.py\n\n# Multi-GPU (8 GPUs)\naccelerate launch --multi_gpu --num_processes 8 train.py\n\n# Multi-node\naccelerate launch --multi_gpu --num_processes 16 \\\n  --num_machines 2 --machine_rank 0 \\\n  --main_process_ip $MASTER_ADDR \\\n  train.py\n```\n\n### Workflow 2: Mixed precision training\n\n**Enable FP16/BF16**:\n```python\nfrom accelerate import Accelerator\n\n# FP16 (with gradient scaling)\naccelerator = Accelerator(mixed_precision='fp16')\n\n# BF16 (no scaling, more stable)\naccelerator = Accelerator(mixed_precision='bf16')\n\n# FP8 (H100+)\naccelerator = Accelerator(mixed_precision='fp8')\n\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n\n# Everything else is automatic!\nfor batch in dataloader:\n    with accelerator.autocast():  # Optional, done automatically\n        loss = model(batch)\n    accelerator.backward(loss)\n```\n\n### Workflow 3: DeepSpeed ZeRO integration\n\n**Enable DeepSpeed ZeRO-2**:\n```python\nfrom accelerate import Accelerator\n\naccelerator = Accelerator(\n    mixed_precision='bf16',\n    deepspeed_plugin={\n        \"zero_stage\": 2,  # ZeRO-2\n        \"offload_optimizer\": False,\n        \"gradient_accumulation_steps\": 4\n    }\n)\n\n# Same code as before!\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n```\n\n**Or via config**:\n```bash\naccelerate config\n# Select: DeepSpeed → ZeRO-2\n```\n\n**deepspeed_config.json**:\n```json\n{\n    \"fp16\": {\"enabled\": false},\n    \"bf16\": {\"enabled\": true},\n    \"zero_optimization\": {\n        \"stage\": 2,\n        \"offload_optimizer\": {\"device\": \"cpu\"},\n        \"allgather_bucket_size\": 5e8,\n        \"reduce_bucket_size\": 5e8\n    }\n}\n```\n\n**Launch**:\n```bash\naccelerate launch --config_file deepspeed_config.json train.py\n```\n\n### Workflow 4: FSDP (Fully Sharded Data Parallel)\n\n**Enable FSDP**:\n```python\nfrom accelerate import Accelerator, FullyShardedDataParallelPlugin\n\nfsdp_plugin = FullyShardedDataParallelPlugin(\n    sharding_strategy=\"FULL_SHARD\",  # ZeRO-3 equivalent\n    auto_wrap_policy=\"TRANSFORMER_AUTO_WRAP\",\n    cpu_offload=False\n)\n\naccelerator = Accelerator(\n    mixed_precision='bf16',\n    fsdp_plugin=fsdp_plugin\n)\n\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n```\n\n**Or via config**:\n```bash\naccelerate config\n# Select: FSDP → Full Shard → No CPU Offload\n```\n\n### Workflow 5: Gradient accumulation\n\n**Accumulate gradients**:\n```python\nfrom accelerate import Accelerator\n\naccelerator = Accelerator(gradient_accumulation_steps=4)\n\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n\nfor batch in dataloader:\n    with accelerator.accumulate(model):  # Handles accumulation\n        optimizer.zero_grad()\n        loss = model(batch)\n        accelerator.backward(loss)\n        optimizer.step()\n```\n\n**Effective batch size**: `batch_size * num_gpus * gradient_accumulation_steps`\n\n## When to use vs alternatives\n\n**Use Accelerate when**:\n- Want simplest distributed training\n- Need single script for any hardware\n- Use HuggingFace ecosystem\n- Want flexibility (DDP/DeepSpeed/FSDP/Megatron)\n- Need quick prototyping\n\n**Key advantages**:\n- **4 lines**: Minimal code changes\n- **Unified API**: Same code for DDP, DeepSpeed, FSDP, Megatron\n- **Automatic**: Device placement, mixed precision, sharding\n- **Interactive config**: No manual launcher setup\n- **Single launch**: Works everywhere\n\n**Use alternatives instead**:\n- **PyTorch Lightning**: Need callbacks, high-level abstractions\n- **Ray Train**: Multi-node orchestration, hyperparameter tuning\n- **DeepSpeed**: Direct API control, advanced features\n- **Raw DDP**: Maximum control, minimal abstraction\n\n## Common issues\n\n**Issue: Wrong device placement**\n\nDon't manually move to device:\n```python\n# WRONG\nbatch = batch.to('cuda')\n\n# CORRECT\n# Accelerate handles it automatically after prepare()\n```\n\n**Issue: Gradient accumulation not working**\n\nUse context manager:\n```python\n# CORRECT\nwith accelerator.accumulate(model):\n    optimizer.zero_grad()\n    accelerator.backward(loss)\n    optimizer.step()\n```\n\n**Issue: Checkpointing in distributed**\n\nUse accelerator methods:\n```python\n# Save only on main process\nif accelerator.is_main_process:\n    accelerator.save_state('checkpoint/')\n\n# Load on all processes\naccelerator.load_state('checkpoint/')\n```\n\n**Issue: Different results with FSDP**\n\nEnsure same random seed:\n```python\nfrom accelerate.utils import set_seed\nset_seed(42)\n```\n\n## Advanced topics\n\n**Megatron integration**: See [references/megatron-integration.md](references/megatron-integration.md) for tensor parallelism, pipeline parallelism, and sequence parallelism setup.\n\n**Custom plugins**: See [references/custom-plugins.md](references/custom-plugins.md) for creating custom distributed plugins and advanced configuration.\n\n**Performance tuning**: See [references/performance.md](references/performance.md) for profiling, memory optimization, and best practices.\n\n## Hardware requirements\n\n- **CPU**: Works (slow)\n- **Single GPU**: Works\n- **Multi-GPU**: DDP (default), DeepSpeed, or FSDP\n- **Multi-node**: DDP, DeepSpeed, FSDP, Megatron\n- **TPU**: Supported\n- **Apple MPS**: Supported\n\n**Launcher requirements**:\n- **DDP**: `torch.distributed.run` (built-in)\n- **DeepSpeed**: `deepspeed` (pip install deepspeed)\n- **FSDP**: PyTorch 1.12+ (built-in)\n- **Megatron**: Custom setup\n\n## Resources\n\n- Docs: https://huggingface.co/docs/accelerate\n- GitHub: https://github.com/huggingface/accelerate\n- Version: 1.11.0+\n- Tutorial: \"Accelerate your scripts\"\n- Examples: https://github.com/huggingface/accelerate/tree/main/examples\n- Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries","author":"@OpenRaiser","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/OpenRaiser/NanoResearch/tree/main/skills/vendor-ai-research/accelerate","license":"MIT","category":"coding","lang":"en","tokens":1943,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/custom-plugins.md","size":11781,"sha256":"1b2f1cf5d70a155f7792cad12a466be4315b787fe21e01b36d32bf374693c020"},{"path":"references/megatron-integration.md","size":11252,"sha256":"69dedcc3c78c197defa86531412541b96a624090c0afd787a967cf865117273d"},{"path":"references/performance.md","size":12569,"sha256":"559ea051c23773d3a7227610018429a0e12e49795949a6e9adfa91b305182a37"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["docs.nvidia.com","huggingface.co","pytorch.org","www.deepspeed.ai"]}}