{"id":"ab-test-planner","name":"ab-test-planner","summary":"製品機能、UIの変更、オンボーディングフロー、価格設定実験に対して統計的に厳密なA/Bテストを設計します。","body":"# A/B Test Planner Skill\n\nDesign experiments that produce trustworthy results — not just directional signals. Every test output includes hypothesis, success metrics, sample size, duration, and a results interpretation guide.\n\n## Required Inputs\n\nAsk the user for these if not provided:\n- **What is being tested** (feature, UI change, copy, pricing, onboarding step)\n- **Hypothesis** (or ask to help formulate one)\n- **Primary metric** (conversion rate, click-through, completion rate, etc.)\n- **Baseline rate** and **minimum detectable effect** (MDE)\n- **Daily eligible users** (to calculate duration)\n\n## Experiment Design Checklist\n\nBefore running any test, confirm:\n- [ ] Clear hypothesis with predicted direction\n- [ ] Single primary metric (plus up to 2 guardrail metrics)\n- [ ] Minimum detectable effect (MDE) defined\n- [ ] Sample size calculated\n- [ ] Test duration estimated\n- [ ] Segment isolated (no overlap with other running tests)\n- [ ] Rollback plan defined\n\n## Hypothesis Template\n\n> \"We believe that [change] will cause [primary metric] to [increase/decrease] by [X%] for [user segment], because [rationale based on data or insight].\"\n\nNever run a test without a directional hypothesis. \"Let's just see what happens\" is not a hypothesis.\n\n## Sample Size Calculator Logic\n\nUse this formula (provide the output, not the formula, to the user):\n\n- **Baseline conversion rate:** Current rate of primary metric\n- **MDE:** Smallest change worth detecting (recommend 10–20% relative lift for most features)\n- **Statistical power:** 80% (standard)\n- **Significance level:** 95% (p < 0.05)\n\nFor common scenarios, provide pre-calculated estimates:\n\n| Baseline Rate | MDE (Relative) | Required Sample per Variant |\n|---|---|---|\n| 5% | 20% | ~19,000 |\n| 10% | 15% | ~14,000 |\n| 20% | 10% | ~15,000 |\n| 40% | 10% | ~9,500 |\n| 60% | 5% | ~42,000 |\n\nAlways warn: \"These are estimates. Use a tool like Evan Miller's calculator or Statsig for precision.\"\n\n## Test Duration Guidance\n\nMinimum: 2 full weeks (to capture weekly seasonality)\nMaximum: 4 weeks (novelty effect distorts results beyond this)\n\n`Duration = Required sample ÷ (Daily traffic × % exposed)`\n\nFlag if traffic is too low to reach significance in under 8 weeks — recommend a different approach (e.g., holdout test, qualitative research).\n\n## Output Format\n\n### A/B Test Plan — [Test Name] — [Date]\n\n**Hypothesis:**\n> [Filled hypothesis template]\n\n**Variants:**\n- Control (A): [Current experience]\n- Treatment (B): [Changed experience — be specific]\n\n**Primary Metric:** [Metric name + how measured]\n**Guardrail Metrics:** [Metrics that must not degrade]\n\n**Target Segment:** [Who sees the test — % of traffic, user type]\n**Traffic Split:** [50/50 recommended unless ramp-up needed]\n\n**Sample Size Required:** ~[N] users per variant\n**Estimated Duration:** [X] weeks (based on [Y] daily eligible users)\n**Significance Threshold:** 95% confidence, 80% power\n\n**Exclusions:** [Any user segments to exclude and why]\n\n**Rollback Trigger:** If [guardrail metric] degrades by [X%], stop the test immediately.\n\n**Results Interpretation Guide:**\n- ✅ Ship if: Treatment shows [X%]+ lift on primary metric at 95% confidence AND guardrail metrics are stable\n- 🔄 Iterate if: Direction is positive but not significant — consider extending or redesigning\n- ❌ Reject if: No lift or negative direction at significance\n- ⚠️ Inconclusive: Do not ship. Do not call it a win.\n\n---\n\n## Guidelines\n\n- Always recommend against peeking at results before the test reaches planned sample size — explain p-hacking risk\n- If user wants to test multiple variants, explain the multiple comparisons problem and recommend a Bonferroni correction or a Bayesian approach\n- If traffic is very low (<1,000 users/day), recommend qualitative alternatives: moderated testing, 5-second tests, or user interviews\n- Never approve a test with no guardrail metrics — always protect revenue, retention, or core engagement\n\n## Anti-Patterns\n\n- [ ] Do not run a test without a directional hypothesis — \"let's see what happens\" produces uninterpretable results\n- [ ] Do not declare a winner before reaching the pre-planned sample size — peeking at results inflates false positive rates\n- [ ] Do not test multiple independent changes in a single variant — you won't know which change caused the result\n- [ ] Do not use engagement metrics (clicks, time-on-page) as the primary metric when the goal is revenue or retention — proxy metrics mislead\n- [ ] Do not ignore guardrail metrics — a conversion lift that causes a support ticket spike is not a win\n\n## Scoring Rubric (0–40)\n\nScore any output of this skill before handing it over; 32+ is ship-quality.\n\n| Dimension | 0 | 5 | 10 |\n|---|---|---|---|\n| Statistical rigour | No sample size, or a number with no stated baseline/MDE behind it | Sample size present but MDE is guessed or copied from the lookup table without checking the actual baseline; power/significance unstated | Sample size derived from the stated baseline and MDE at 80% power / 95% confidence, duration checked against real daily traffic and the 2–4 week window, and the low-traffic escape hatch invoked if it doesn't fit |\n| Hypothesis discipline | \"Let's see what happens\" — no direction, no magnitude, or multiple changes bundled into one variant | Directional hypothesis but missing magnitude, segment, or the evidence-based *because*; variant purity not confirmed | Full template filled (change, metric, direction, magnitude, segment, rationale citing data), and the treatment isolates exactly one change with excluded ideas named as follow-up tests |\n| Guardrails & rollback | No guardrail metrics, or a rollback line with no threshold | Guardrails named but denominators/definitions ambiguous; rollback trigger vague (\"if things look bad\") | 1–2 guardrails protecting revenue or core engagement with pre-agreed definitions, concrete rollback thresholds, and the peeking-vs-harm-monitoring distinction handled explicitly |\n| Decision readiness | No interpretation guide; results will be argued about after the fact | Ship/iterate/reject listed but thresholds fuzzy; inconclusive outcome missing or treated as a soft win | All four outcomes (ship / iterate / reject / inconclusive) mapped to pre-committed thresholds, including what an inconclusive result costs and what each outcome changes next |\n\n## Quality Checks\n\n- [ ] Hypothesis is directional (predicts a specific direction and magnitude, not \"let's see\")\n- [ ] Primary metric is singular (guardrail metrics are secondary)\n- [ ] Sample size is calculated from actual MDE and baseline (not guessed)\n- [ ] Test duration accounts for weekly seasonality (minimum 2 weeks)\n- [ ] Guardrail metrics are defined (at least one to protect revenue or core engagement)\n- [ ] Rollback trigger is specified with a concrete threshold","author":"@mohitagw15856","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/mohitagw15856/pm-claude-skills/tree/main/exports/openclaw/ab-test-planner","license":"MIT","category":"writing","lang":"en","tokens":1566,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}