{"id":"openraiser-ml-training-recipes","name":"ml-training-recipes","summary":"LLM、視覚、拡散、医療画像、タンパク質/新薬発見、空間オミクス、ゲノミクスなど、あらゆる分野向けの実戦実証済みPyTorchトレーニングレシピ。","body":"# ML Training Recipes\r\n\r\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\r\n(Karpathy's autoresearch/nanochat, torchvision, HuggingFace) and modern training practice.\r\n\r\n## Reference files (read when needed)\r\n\r\n- `references/architecture.md` — Transformer/LLM architecture code patterns, weight init\r\n- `references/optimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\r\n- `references/domain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\r\n- `references/scaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\r\n- `references/biomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\r\n- `references/experiment-loop.md` — Autonomous experiment loop (autoresearch keep/discard/revert)\r\n\r\n---\r\n\r\n## Architecture Selection\r\n\r\nPick the right model by **data type** and **data scale**:\r\n\r\n| Data Type | < 10K samples | 10K-100K | > 100K |\r\n|-----------|--------------|----------|--------|\r\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\r\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT/LLaMA (LoRA) | Pretrain from scratch |\r\n| **Tabular** | XGBoost/LightGBM | Still XGBoost | Neural viable |\r\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\r\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\r\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\r\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR / MedSAM |\r\n\r\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\r\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\r\n\r\nFor biomedical domains, see `references/biomedical.md`.\r\nFor sequence model selection and compute planning, see `references/scaling-and-selection.md`.\r\n\r\n---\r\n\r\n## Scaling Laws\r\n\r\n### Chinchilla rule (Hoffmann et al., 2022)\r\n\r\nCompute-optimal training: **~20 tokens per parameter**.\r\n\r\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\r\n|-----------|----------------|--------------------------|\r\n| 125M | 2.5B tokens | 12.5B tokens |\r\n| 1B | 20B tokens | 100B tokens |\r\n| 7B | 140B tokens | 700B tokens |\r\n\r\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\r\n\r\n---\r\n\r\n## Training Loop\r\n\r\n```python\r\nimport gc, time, torch\r\n\r\ntorch.manual_seed(42)\r\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\n\r\ngrad_accum_steps = total_batch_size // (batch_size * seq_len)\r\nstep = 0\r\n\r\nwhile not done:\r\n    t0 = time.time()\r\n    for micro_step in range(grad_accum_steps):\r\n        with autocast_ctx:\r\n            loss = model(x, y)\r\n        (loss / grad_accum_steps).backward()\r\n        x, y = next(train_loader)\r\n\r\n    update_lr(optimizer, progress)\r\n    optimizer.step()\r\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\r\n\r\n    if loss.item() > 100:  # fast-fail on divergence\r\n        print(\"FAIL: loss exploded\"); exit(1)\r\n\r\n    torch.cuda.synchronize()\r\n    if step == 0:\r\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\r\n    step += 1\r\n```\r\n\r\n### Key principles\r\n\r\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\r\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\r\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\r\n- **Time-based budgets** make experiments comparable across hardware.\r\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\r\n\r\n---\r\n\r\n## Optimizer Configuration\r\n\r\nModern LLM training uses different optimizers per parameter group:\r\n\r\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\r\n|---------------|-----------|-----------|--------------|\r\n| 2D weight matrices | Muon | 0.04 | 0.2 |\r\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\r\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\r\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\r\n\r\n**LR scaling by dimension**: `lr * (d_model / 768)^(-0.5)` — keeps dynamics stable across sizes.\r\n\r\n### Rules of thumb\r\n\r\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\r\n- Weight decay scheduling: linearly decay WD to 0 over training.\r\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\r\n\r\nFor Muon details (polar express orthogonalization, NorMuon), see `references/optimizers.md`.\r\n\r\n---\r\n\r\n## Learning Rate Scheduling\r\n\r\n### Time-based (autoresearch style)\r\n\r\n```python\r\ndef get_lr_multiplier(progress):  # progress = elapsed_time / time_budget\r\n    if progress < warmup_ratio:\r\n        return progress / warmup_ratio\r\n    elif progress < 1.0 - warmdown_ratio:\r\n        return 1.0\r\n    else:\r\n        cooldown = (1.0 - progress) / warmdown_ratio\r\n        return cooldown + (1 - cooldown) * final_lr_frac\r\n```\r\n\r\n### Cosine decay\r\n\r\n```python\r\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\r\n    if step < warmup_steps:\r\n        return max_lr * step / warmup_steps\r\n    progress = (step - warmup_steps) / (total_steps - warmup_steps)\r\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\r\n```\r\n\r\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\r\n\r\n### Guidance\r\n\r\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\r\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\r\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\r\n\r\n---\r\n\r\n## Mixed Precision & Compilation\r\n\r\n```python\r\nimport os\r\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\r\n\r\nimport torch\r\ntorch.set_float32_matmul_precision(\"high\")\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\nmodel = torch.compile(model, dynamic=False)\r\n```\r\n\r\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\r\n- **fp16**: needs GradScaler. Use only on V100 or older.\r\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\r\n- First steps are slow (JIT) — exclude from timing.\r\n\r\n---\r\n\r\n## Memory & Performance\r\n\r\n### Meta device init (large models)\r\n\r\n```python\r\nwith torch.device(\"meta\"):\r\n    model = GPT(config)          # zero memory\r\nmodel.to_empty(device=\"cuda\")\r\nmodel.init_weights()\r\n```\r\n\r\n### MFU (Model FLOPs Utilization)\r\n\r\n```python\r\nachieved_flops = model_flops_per_token * batch_tokens / step_time\r\nmfu = achieved_flops / gpu_peak_flops\r\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\r\n```\r\n\r\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\r\n\r\n### OOM solutions (in order)\r\n\r\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\r\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\r\n3. `model.zero_grad(set_to_none=True)`\r\n4. Meta device init → `to_empty`\r\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\r\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\r\n\r\n---\r\n\r\n## Hyperparameter Search\r\n\r\n### Priority order (tune first → last)\r\n\r\n1. **Learning rate** — most impactful. Always tune first.\r\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\r\n3. **Weight decay** — 0.01-0.1 for AdamW.\r\n4. **Warmup steps** — 1-5% of training.\r\n\r\n### The 2025 default recipe\r\n\r\n| Setting | Value |\r\n|---------|-------|\r\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\r\n| Weight decay | 0.1 |\r\n| LR schedule | Cosine decay or WSD |\r\n| Peak LR | 3e-4 (scale down for larger models) |\r\n| Precision | bf16 |\r\n| Grad clipping | max_norm=1.0 |\r\n| Normalization | RMSNorm (pre-norm) |\r\n| Activation | SwiGLU |\r\n| Position encoding | RoPE |\r\n| Attention | Flash Attention, optionally GQA |\r\n\r\n---\r\n\r\n## Debugging Checklist\r\n\r\n### Karpathy's recipe (still canonical)\r\n\r\n1. **Become one with the data** — visualize, check distributions, verify labels\r\n2. **Get end-to-end running first** — verify on a trivial case\r\n3. **Overfit one batch** — if you can't, you have a bug\r\n4. **Then regularize** — add regularization only after overfitting works\r\n5. **Tune hyperparameters** — start with known defaults\r\n\r\n### Loss exploding / NaN\r\n\r\n1. Reduce LR (3-10× smaller)\r\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\r\n3. Check for inf/nan in inputs\r\n4. Add logit soft capping: `softcap * tanh(logits / softcap)`\r\n5. Add QK-norm in attention\r\n6. Verify weight init (zero-init output projections?)\r\n7. Check loss reduction with gradient accumulation (`loss / grad_accum_steps`)\r\n\r\n### Slow training / Low MFU\r\n\r\n1. Verify `torch.compile` is active\r\n2. Check `torch.set_float32_matmul_precision(\"high\")`\r\n3. Pin memory + non_blocking transfers\r\n4. Profile with `torch.profiler`\r\n5. GC stalls? `gc.freeze(); gc.disable()`\r\n6. Tensor Core alignment: dims multiples of 8/64\r\n\r\n### Loss plateau / Slow convergence\r\n\r\n1. LR too low — try 2-5× larger\r\n2. Warmup too long\r\n3. Weight decay too high\r\n4. Verify LR schedule is actually applied (print each step)\r\n5. Model too small for task\r\n\r\n### Silent failures\r\n\r\n1. **Data leakage** between train/val\r\n2. **Wrong preprocessing at inference** — augmentation mismatch\r\n3. **Label errors** — use cleanlab to detect\r\n4. **Shuffling bugs** — correlated batches\r\n5. **Tokenizer mismatch** with pretrained model\r\n\r\n### What to monitor\r\n\r\n- **Gradient norms** — spike precedes loss spike\r\n- **Per-layer activation stats** — reveals exploding/vanishing\r\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\r\n- **Learning rate** — verify schedule applied (common silent bug)\r\n\r\n---\r\n\r\n## Experiment Management\r\n\r\nTrack experiments in TSV for easy comparison:\r\n\r\n```\r\ncommit  val_bpb  memory_gb  status   description\r\na1b2c3d 0.9979   44.0       keep     baseline\r\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\r\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\r\n```\r\n\r\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\r\nresults is a great outcome. For systematic agent-driven experimentation, see `references/experiment-loop.md`.\r\n\r\n### Evaluation metrics by domain\r\n\r\n| Domain | Primary Metric | Notes |\r\n|--------|---------------|-------|\r\n| LLM | BPB (bits per byte) | Vocab-size-independent |\r\n| Classification | Accuracy / F1 | Macro-F1 for imbalanced |\r\n| Segmentation | mIoU / Dice | Per-class IoU reveals weak spots |\r\n| Generation | FID | Needs >10k samples |\r\n| Regression | RMSE / MAE | Log-transform skewed targets |","author":"@OpenRaiser","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/OpenRaiser/NanoResearch/tree/main/skills/vendor-ai-research/ml-training-recipes","license":"MIT","category":null,"lang":"en","tokens":2884,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/architecture.md","size":10523,"sha256":"ac384a12b750c49ed461ba2a884653ff8b3591e74b89c30af3096f350f301ce8"},{"path":"references/biomedical.md","size":21275,"sha256":"0b70d206f2818ac4b62b42b86570fe82be9570de31f9cc8727b83414b8f2654e"},{"path":"references/domain-specific.md","size":19643,"sha256":"ca27f6d6b57d3c8ed08f1f047def6dc0891c02b42cbda5f3f96e3580a0508eb8"},{"path":"references/experiment-loop.md","size":4574,"sha256":"c508e70f173464347a0ec4de36a80541f17b429ceeedbf65aec53ae9afc4f12a"},{"path":"references/optimizers.md","size":10893,"sha256":"9f06866c66fb2626b61988c615d822cac6efd10668a825c7b6ce22252d630d90"},{"path":"references/scaling-and-selection.md","size":17259,"sha256":"62eb5df8a9aef21c74f262570cdf21967d7e791d9b363118ed1028ecbc3d9aa2"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[{"code":"code.eval","kind":"dangerous-code","where":"references/domain-specific.md:364","excerpt":"eval(","message":"evaluates code at runtime","severity":"warn"}],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}