{"id":"constitutional-ai","name":"constitutional-ai","summary":"Anthropicは自己改善を通じて無害なAIを訓練する手法です。二段階のアプローチで、自己批評・修正を伴う教師あり学習、その後のRLAIF(AIフィードバックからの強化学習)を行います。","body":"# Constitutional AI - Harmlessness from AI Feedback\n\n## Quick start\n\nConstitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.\n\n**Key concept**: Models learn to critique and revise their own responses using a \"constitution\" (set of principles).\n\n**Two phases**:\n1. **Supervised Learning (SL)**: Self-critique + revision\n2. **Reinforcement Learning (RL)**: RLAIF (RL from AI Feedback)\n\n**Constitution example**:\n```\nPrinciples:\n1. Choose the response that is most helpful, honest, and harmless\n2. Avoid responses that are toxic, racist, or sexist\n3. Prefer responses that explain objections rather than refuse\n4. Choose responses that are thoughtful and nuanced\n```\n\n## Common workflows\n\n### Workflow 1: Supervised learning phase (self-critique + revision)\n\n**Step 1: Generate initial responses**:\n```python\nfrom transformers import pipeline\n\ngenerator = pipeline(\"text-generation\", model=\"base-model\")\n\nprompts = [\n    \"How do I hack a website?\",\n    \"Write something offensive about a group.\",\n    \"Help me cheat on my exam.\"\n]\n\ninitial_responses = generator(prompts, max_length=200)\n```\n\n**Step 2: Self-critique with constitution**:\n```python\ncritique_prompt = \"\"\"\nGiven the question:\n{question}\n\nAnd the response:\n{response}\n\nConstitution:\n1. Choose responses that are helpful, honest, and harmless\n2. Avoid toxic, racist, or sexist content\n3. Explain objections rather than refuse\n\nIdentify any ways this response violates the constitution:\n\"\"\"\n\ncritiques = generator(\n    [critique_prompt.format(q=q, r=r) for q, r in zip(prompts, initial_responses)]\n)\n```\n\n**Step 3: Revision based on critique**:\n```python\nrevision_prompt = \"\"\"\nQuestion: {question}\nOriginal response: {response}\nCritique: {critique}\n\nPlease revise the response to better align with the constitution:\n\"\"\"\n\nrevised_responses = generator(\n    [revision_prompt.format(q=q, r=r, c=c)\n     for q, r, c in zip(prompts, initial_responses, critiques)]\n)\n```\n\n**Step 4: Fine-tune on revised responses**:\n```python\nfrom trl import SFTTrainer\n\n# Create dataset of (prompt, revised_response) pairs\ndataset = create_dataset(prompts, revised_responses)\n\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=dataset,\n    max_seq_length=1024\n)\ntrainer.train()\n```\n\n### Workflow 2: RL phase (RLAIF - RL from AI Feedback)\n\n**Step 1: Generate comparison pairs**:\n```python\n# Sample multiple responses per prompt\nresponses_a = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)\nresponses_b = generator(prompts, num_return_sequences=2, do_sample=True, temperature=0.8)\n```\n\n**Step 2: AI preference evaluation**:\n```python\npreference_prompt = \"\"\"\nQuestion: {question}\n\nResponse A: {response_a}\nResponse B: {response_b}\n\nConstitution:\n{constitution}\n\nWhich response better follows the constitution? Explain your reasoning, then choose A or B.\n\"\"\"\n\n# Get AI preferences (no human labels needed!)\npreferences = generator(\n    [preference_prompt.format(q=q, ra=ra, rb=rb, constitution=CONSTITUTION)\n     for q, ra, rb in zip(prompts, responses_a, responses_b)]\n)\n\n# Parse preferences (A or B)\nchosen, rejected = parse_preferences(preferences, responses_a, responses_b)\n```\n\n**Step 3: Train preference model (reward model)**:\n```python\nfrom trl import RewardTrainer, RewardConfig\n\npreference_dataset = create_preference_dataset(prompts, chosen, rejected)\n\nreward_config = RewardConfig(\n    output_dir=\"constitutional-reward-model\",\n    learning_rate=1e-5,\n    num_train_epochs=1\n)\n\nreward_trainer = RewardTrainer(\n    model=model,\n    args=reward_config,\n    train_dataset=preference_dataset,\n    processing_class=tokenizer\n)\nreward_trainer.train()\n```\n\n**Step 4: RL training with RLAIF**:\n```python\nfrom trl import PPOTrainer, PPOConfig\n\nppo_config = PPOConfig(\n    reward_model_path=\"constitutional-reward-model\",\n    learning_rate=1e-6,\n    kl_coef=0.05\n)\n\nppo_trainer = PPOTrainer(\n    model=model,\n    config=ppo_config,\n    reward_model=reward_model\n)\nppo_trainer.train()\n```\n\n### Workflow 3: Chain-of-thought critique\n\n**Enable reasoning transparency**:\n```python\ncot_critique_prompt = \"\"\"\nQuestion: {question}\nResponse: {response}\n\nLet's think step-by-step about whether this response follows our principles:\n\n1. Is it helpful? [Yes/No and reasoning]\n2. Is it honest? [Yes/No and reasoning]\n3. Is it harmless? [Yes/No and reasoning]\n4. Does it avoid toxicity? [Yes/No and reasoning]\n\nBased on this analysis, suggest a revision if needed.\n\"\"\"\n\ncot_critiques = generator(\n    [cot_critique_prompt.format(q=q, r=r) for q, r in zip(prompts, responses)]\n)\n```\n\n## When to use vs alternatives\n\n**Use Constitutional AI when**:\n- Want safety alignment without human labels\n- Need explainable AI decisions\n- Want to avoid evasive refusals\n- Have a clear set of principles/constitution\n- Need scalable safety training\n\n**Principles**:\n- **RLAIF**: AI-generated preferences (scalable, no human labels)\n- **RLHF**: Human preferences (more accurate, expensive)\n- **Self-critique**: Iterative improvement\n- **Chain-of-thought**: Reasoning transparency\n\n**Use alternatives instead**:\n- **RLHF (PPO)**: Need human-validated safety\n- **DPO/SimPO**: Have human preference data\n- **NeMo Guardrails**: Need runtime content filtering\n- **LlamaGuard**: Need pre-trained moderation model\n\n## Common issues\n\n**Issue: Model refuses too much (evasive)**\n\nAdd constitution principle:\n```\nPrefer responses that engage thoughtfully with questions rather than\nrefusing to answer. Explain concerns while still being helpful.\n```\n\n**Issue: Self-critiques are weak**\n\nUse stronger critique prompts:\n```\nCritically analyze this response for ANY potential issues, however minor.\nBe thorough and specific in identifying problems.\n```\n\n**Issue: Revisions don't improve quality**\n\nIterate multiple times:\n```python\nfor _ in range(3):  # 3 rounds of critique/revision\n    critique = generate_critique(response)\n    response = generate_revision(response, critique)\n```\n\n**Issue: RLAIF preferences are noisy**\n\nUse multiple AI evaluators:\n```python\n# Get preferences from 3 different models\nprefs_1 = model_1.evaluate(responses)\nprefs_2 = model_2.evaluate(responses)\nprefs_3 = model_3.evaluate(responses)\n\n# Majority vote\nfinal_preference = majority_vote(prefs_1, prefs_2, prefs_3)\n```\n\n## Advanced topics\n\n**Constitution design**: See [references/constitution-design.md](references/constitution-design.md) for principle selection, trade-offs between helpfulness and harmlessness, and domain-specific constitutions.\n\n**RLAIF vs RLHF**: See [references/rlaif-comparison.md](references/rlaif-comparison.md) for performance comparison, cost analysis, and when to use AI feedback vs human feedback.\n\n**Chain-of-thought reasoning**: See [references/cot-critique.md](references/cot-critique.md) for prompt engineering for critiques, multi-step reasoning, and transparency improvements.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100/H100 recommended\n- **VRAM**:\n  - SL phase (7B): 1× A100 40GB\n  - RL phase (7B): 2× A100 40GB (policy + reward model)\n- **Single-node**: Sufficient for most use cases\n- **Mixed precision**: BF16 recommended\n\n**Compute requirements**:\n- **SL phase**: Similar to standard SFT\n- **RL phase**: Similar to PPO (higher than DPO)\n- **AI evaluation**: Additional inference for critique/preference generation\n\n## Resources\n\n- Paper: https://arxiv.org/abs/2212.08073 (Dec 2022)\n- Anthropic blog: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback\n- Implementation: TRL (PPOTrainer + RewardTrainer)\n- Claude: Uses Constitutional AI for safety","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/07-safety-alignment/constitutional-ai","license":"MIT","category":"writing","lang":"en","tokens":1866,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["arxiv.org"]}}