{"id":"simpo","name":"simpo-training","summary":"LLMアライメントのためのシンプルな優先最適化。DPOのリファレンスフリー代替で、より良いパフォーマンス(AlpacaEval 2.0で+6.4ポイント)です。","body":"# SimPO - Simple Preference Optimization\n\n## Quick start\n\nSimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.\n\n**Installation**:\n```bash\n# Create environment\nconda create -n simpo python=3.10 && conda activate simpo\n\n# Install PyTorch 2.2.2\n# Visit: https://pytorch.org/get-started/locally/\n\n# Install alignment-handbook\ngit clone https://github.com/huggingface/alignment-handbook.git\ncd alignment-handbook\npython -m pip install .\n\n# Install Flash Attention 2\npython -m pip install flash-attn --no-build-isolation\n```\n\n**Training** (Mistral 7B):\n```bash\nACCELERATE_LOG_LEVEL=info accelerate launch \\\n  --config_file accelerate_configs/deepspeed_zero3.yaml \\\n  scripts/run_simpo.py \\\n  training_configs/mistral-7b-base-simpo.yaml\n```\n\n## Common workflows\n\n### Workflow 1: Train from base model (Mistral 7B)\n\n**Config** (`mistral-7b-base-simpo.yaml`):\n```yaml\n# Model\nmodel_name_or_path: mistralai/Mistral-7B-v0.1\ntorch_dtype: bfloat16\n\n# Dataset\ndataset_mixer:\n  HuggingFaceH4/ultrafeedback_binarized: 1.0\ndataset_splits:\n  - train_prefs\n  - test_prefs\n\n# SimPO hyperparameters\nbeta: 2.0                  # Reward scaling (2.0-10.0)\ngamma_beta_ratio: 0.5       # Target margin (0-1)\nloss_type: sigmoid          # sigmoid or hinge\nsft_weight: 0.0             # Optional SFT regularization\n\n# Training\nlearning_rate: 5e-7         # Critical: 3e-7 to 1e-6\nnum_train_epochs: 1\nper_device_train_batch_size: 1\ngradient_accumulation_steps: 8\n\n# Output\noutput_dir: ./outputs/mistral-7b-simpo\n```\n\n**Launch training**:\n```bash\naccelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \\\n  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml\n```\n\n### Workflow 2: Fine-tune instruct model (Llama 3 8B)\n\n**Config** (`llama3-8b-instruct-simpo.yaml`):\n```yaml\nmodel_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct\n\ndataset_mixer:\n  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0\n\nbeta: 2.5\ngamma_beta_ratio: 0.5\nlearning_rate: 5e-7\nsft_weight: 0.1             # Add SFT loss to preserve capabilities\n\nnum_train_epochs: 1\nper_device_train_batch_size: 2\ngradient_accumulation_steps: 4\noutput_dir: ./outputs/llama3-8b-simpo\n```\n\n**Launch**:\n```bash\naccelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \\\n  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml\n```\n\n### Workflow 3: Reasoning-intensive tasks (lower LR)\n\n**For math/code tasks**:\n```yaml\nmodel_name_or_path: deepseek-ai/deepseek-math-7b-base\n\ndataset_mixer:\n  argilla/distilabel-math-preference-dpo: 1.0\n\nbeta: 5.0                   # Higher for stronger signal\ngamma_beta_ratio: 0.7       # Larger margin\nlearning_rate: 3e-7         # Lower LR for reasoning\nsft_weight: 0.0\n\nnum_train_epochs: 1\nper_device_train_batch_size: 1\ngradient_accumulation_steps: 16\n```\n\n## When to use vs alternatives\n\n**Use SimPO when**:\n- Want simpler training than DPO (no reference model)\n- Have preference data (chosen/rejected pairs)\n- Need better performance than DPO\n- Limited compute resources\n- Single-node training sufficient\n\n**Algorithm selection**:\n- **SimPO**: Simplest, best performance, no reference model\n- **DPO**: Need reference model baseline, more conservative\n- **PPO**: Maximum control, need reward model, complex setup\n- **GRPO**: Memory-efficient RL, no critic\n\n**Use alternatives instead**:\n- **OpenRLHF**: Multi-node distributed training, PPO/GRPO\n- **TRL**: Need multiple methods in one framework\n- **DPO**: Established baseline comparison\n\n## Common issues\n\n**Issue: Loss divergence**\n\nReduce learning rate:\n```yaml\nlearning_rate: 3e-7  # Reduce from 5e-7\n```\n\nReduce beta:\n```yaml\nbeta: 1.0  # Reduce from 2.0\n```\n\n**Issue: Model forgets capabilities**\n\nAdd SFT regularization:\n```yaml\nsft_weight: 0.1  # Add SFT loss component\n```\n\n**Issue: Poor preference separation**\n\nIncrease beta and margin:\n```yaml\nbeta: 5.0            # Increase from 2.0\ngamma_beta_ratio: 0.8  # Increase from 0.5\n```\n\n**Issue: OOM during training**\n\nReduce batch size:\n```yaml\nper_device_train_batch_size: 1\ngradient_accumulation_steps: 16  # Maintain effective batch\n```\n\nEnable gradient checkpointing:\n```yaml\ngradient_checkpointing: true\n```\n\n## Advanced topics\n\n**Loss functions**: See [references/loss-functions.md](references/loss-functions.md) for sigmoid vs hinge loss, mathematical formulations, and when to use each.\n\n**Hyperparameter tuning**: See [references/hyperparameters.md](references/hyperparameters.md) for beta, gamma, learning rate selection guide, and model-size-specific recommendations.\n\n**Dataset preparation**: See [references/datasets.md](references/datasets.md) for preference data formats, quality filtering, and custom dataset creation.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100/H100 recommended\n- **VRAM**:\n  - 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)\n  - 8B model: 2× A100 40GB\n  - 70B model: 8× A100 80GB\n- **Single-node**: DeepSpeed ZeRO-3 sufficient\n- **Mixed precision**: BF16 recommended\n\n**Memory optimization**:\n- DeepSpeed ZeRO-3 (default config)\n- Gradient checkpointing\n- Flash Attention 2\n\n## Resources\n\n- Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)\n- GitHub: https://github.com/princeton-nlp/SimPO\n- Models: https://huggingface.co/princeton-nlp\n- Alignment Handbook: https://github.com/huggingface/alignment-handbook","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/06-post-training/simpo","license":"MIT","category":"coding","lang":"en","tokens":1531,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/datasets.md","size":10889,"sha256":"885d747d64d810b5b0173fd897d003a01633d34c079a399118966901c6f3a150"},{"path":"references/hyperparameters.md","size":8502,"sha256":"56f2ad2aa946254a5ade4a1ed980b29ccde4d96894fd8fa92287a7a355d56f3d"},{"path":"references/loss-functions.md","size":7067,"sha256":"a197433caba8cd89b37c0732bdd06e4cf68d4029dbee2961aa80729fc35721f4"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["arxiv.org","huggingface.co","pytorch.org"]}}