{"id":"braintrust-tracing","name":"braintrust-tracing","summary":"ClaudeコードのBraintrustトレーシング - フックアーキテクチャ、サブエージェントの相関、デバッグ","body":"# Braintrust Tracing for Claude Code\n\nComprehensive guide to tracing Claude Code sessions in Braintrust, including sub-agent correlation.\n\n## Architecture Overview\n\n```\n                         PARENT SESSION\n                    +---------------------+\n                    |  SessionStart       |\n                    |  (creates root)     |\n                    +----------+----------+\n                               |\n                    +----------v----------+\n                    |  UserPromptSubmit   |\n                    |  (creates Turn)     |\n                    +----------+----------+\n                               |\n          +--------------------+--------------------+\n          |                    |                    |\n+---------v--------+  +--------v--------+  +--------v--------+\n| PostToolUse      |  | PostToolUse     |  | PreToolUse      |\n| (Read span)      |  | (Edit span)     |  | (Task - inject) |\n+------------------+  +-----------------+  +--------+--------+\n                                                    |\n                                         +----------v----------+\n                                         |   SUB-AGENT         |\n                                         |   SessionStart      |\n                                         |   (NEW root_span_id)|\n                                         +----------+----------+\n                                                    |\n                                         +----------v----------+\n                                         |   SubagentStop      |\n                                         |   (has session_id)  |\n                                         +---------------------+\n```\n\n## Hook Event Flow\n\n| Hook | Trigger | Creates | Key Fields |\n|------|---------|---------|------------|\n| **SessionStart** | Session begins | Root span | `session_id`, `root_span_id` |\n| **UserPromptSubmit** | User sends prompt | Turn span | `prompt`, `turn_number` |\n| **PreToolUse** | Before tool runs | (modifies Task prompts) | `tool_input.prompt` |\n| **PostToolUse** | After tool runs | Tool span | `tool_name`, `input`, `output` |\n| **Stop** | Turn completes | LLM spans | `model`, `tokens`, `tool_calls` |\n| **SubagentStop** | Sub-agent finishes | (no span) | `session_id` of sub-agent |\n| **SessionEnd** | Session ends | (finalizes root) | `turn_count`, `tool_count` |\n\n## Trace Hierarchy\n\n```\nSession (task span) - root_span_id = session_id\n|\n+-- Turn 1 (task span)\n|   |\n|   +-- claude-sonnet (llm span) - model call with tool_use\n|   +-- Read (tool span)\n|   +-- Edit (tool span)\n|   +-- claude-sonnet (llm span) - response after tools\n|\n+-- Turn 2 (task span)\n|   |\n|   +-- claude-sonnet (llm span)\n|   +-- Task (tool span) -----> [Sub-agent session - SEPARATE trace]\n|   +-- claude-sonnet (llm span)\n|\n+-- Turn 3 ...\n```\n\n## Sub-Agent Tracing: What Works and What Doesn't\n\n### What Doesn't Work\n\n**SessionStart doesn't receive the Task prompt.**\n\nWe tried injecting trace context into Task prompts via PreToolUse:\n\n```bash\n# PreToolUse hook injects:\n[BRAINTRUST_TRACE_CONTEXT]\n{\"root_span_id\": \"abc\", \"parent_span_id\": \"xyz\", \"project_id\": \"123\"}\n[/BRAINTRUST_TRACE_CONTEXT]\n```\n\nBut SessionStart only receives session metadata, not the modified prompt. The injected context is lost.\n\n### What DOES Work\n\n**Task spans in parent session contain everything:**\n- `agentId` - identifier for the sub-agent run\n- `totalTokens`, `totalToolUseCount` - metrics\n- `content` - full agent response/summary\n- `tool_input.prompt` - original task prompt\n- `tool_input.subagent_type` - agent type (e.g., \"oracle\")\n\n**SubagentStop hook receives the sub-agent's `session_id`:**\n- This equals the sub-agent's orphaned trace `root_span_id`\n- Allows correlation between parent Task span and child trace\n\n### The Correlation Pattern\n\n**Current state:** Sub-agents create orphaned traces (new `root_span_id`).\n\n**Correlation method:**\n1. Query parent session's Task spans for agent metadata\n2. Match `agentId` or timing with orphaned traces\n3. Sub-agent's `session_id` = its trace's `root_span_id`\n\n**Future solution (not yet implemented):**\n```\nSubagentStop fires -> writes session_id to temp file\nPostToolUse (Task) -> reads temp file -> adds child_session_id to Task span metadata\n```\n\nThis would link: `Task.agentId` + `Task.child_session_id` -> orphaned trace `root_span_id`\n\n## State Management\n\n### Per-Session State Files\n\n```\n~/.claude/state/braintrust_sessions/\n  {session_id}.json       # Per-session state\n```\n\nEach session file contains:\n```json\n{\n  \"root_span_id\": \"abc-123\",\n  \"project_id\": \"proj-456\",\n  \"turn_count\": 5,\n  \"tool_count\": 23,\n  \"current_turn_span_id\": \"turn-789\",\n  \"current_turn_start\": 1703456789,\n  \"started\": \"2025-12-24T10:00:00.000Z\",\n  \"is_subagent\": false\n}\n```\n\n### Global State\n```\n~/.claude/state/braintrust_global.json   # Cached project_id\n~/.claude/state/braintrust_hook.log      # Debug log\n```\n\n## Debugging Commands\n\n### Check if Tracing is Active\n```bash\n# View hook logs in real-time\ntail -f ~/.claude/state/braintrust_hook.log\n\n# Check if session has state\ncat ~/.claude/state/braintrust_sessions/*.json | jq -s '.'\n\n# Verify environment\necho \"TRACE_TO_BRAINTRUST=$TRACE_TO_BRAINTRUST\"\necho \"BRAINTRUST_API_KEY=${BRAINTRUST_API_KEY:+set}\"\n```\n\n### Query Braintrust Directly\n```bash\n# List recent sessions\nuv run python -m runtime.harness scripts/braintrust_analyze.py --sessions 5\n\n# Analyze last session\nuv run python -m runtime.harness scripts/braintrust_analyze.py --last-session\n\n# Replay specific session\nuv run python -m runtime.harness scripts/braintrust_analyze.py --replay <session-id>\n\n# Find sub-agent traces (orphaned roots)\nuv run python -m runtime.harness scripts/braintrust_analyze.py --agent-stats\n```\n\n### Debug Hook Execution\n```bash\n# Enable verbose logging\nexport BRAINTRUST_CC_DEBUG=true\n\n# Test hooks manually\necho '{\"session_id\":\"test-123\",\"type\":\"resume\"}' | \\\n  bash \"$CLAUDE_PROJECT_DIR/.claude/plugins/braintrust-tracing/hooks/session_start.sh\"\n\n# Test PreToolUse (Task injection)\necho '{\"session_id\":\"test-123\",\"tool_name\":\"Task\",\"tool_input\":{\"prompt\":\"test\"}}' | \\\n  bash \"$CLAUDE_PROJECT_DIR/.claude/plugins/braintrust-tracing/hooks/pre_tool_use.sh\"\n```\n\n### Troubleshooting Checklist\n\n1. **No traces appearing:**\n   - Check `TRACE_TO_BRAINTRUST=true` in `.claude/settings.local.json`\n   - Verify API key: `echo $BRAINTRUST_API_KEY`\n   - Check logs: `tail -20 ~/.claude/state/braintrust_hook.log`\n\n2. **Sub-agents not linking:**\n   - This is expected - sub-agents create orphaned traces\n   - Use `--agent-stats` to find agent activity\n   - Correlate via timing or `agentId` in parent Task span\n\n3. **Missing spans:**\n   - Check `current_turn_span_id` in session state\n   - Ensure Stop hook runs (turn finalization)\n   - Look for \"Failed to create\" errors in log\n\n4. **State corruption:**\n   - Remove session state: `rm ~/.claude/state/braintrust_sessions/*.json`\n   - Clear global cache: `rm ~/.claude/state/braintrust_global.json`\n\n## Key Files\n\n| File | Purpose |\n|------|---------|\n| `.claude/plugins/braintrust-tracing/hooks/common.sh` | Shared utilities, API, state management |\n| `.claude/plugins/braintrust-tracing/hooks/session_start.sh` | Creates root span, handles sub-agent context |\n| `.claude/plugins/braintrust-tracing/hooks/user_prompt_submit.sh` | Creates Turn spans per user message |\n| `.claude/plugins/braintrust-tracing/hooks/pre_tool_use.sh` | Injects trace context into Task prompts |\n| `.claude/plugins/braintrust-tracing/hooks/post_tool_use.sh` | Creates tool spans, captures agent/skill metadata |\n| `.claude/plugins/braintrust-tracing/hooks/stop_hook.sh` | Creates LLM spans, finalizes Turns |\n| `.claude/plugins/braintrust-tracing/hooks/session_end.sh` | Finalizes session, triggers learning extraction |\n| `scripts/braintrust_analyze.py` | Query and analyze traced sessions |\n| `~/.claude/state/braintrust_sessions/` | Per-session state files |\n| `~/.claude/state/braintrust_hook.log` | Debug log |\n\n## Environment Variables\n\n| Variable | Required | Default | Description |\n|----------|----------|---------|-------------|\n| `TRACE_TO_BRAINTRUST` | Yes | - | Set to `\"true\"` to enable |\n| `BRAINTRUST_API_KEY` | Yes | - | API key for Braintrust |\n| `BRAINTRUST_CC_PROJECT` | No | `claude-code` | Project name |\n| `BRAINTRUST_CC_DEBUG` | No | `false` | Verbose logging |\n| `BRAINTRUST_API_URL` | No | `https://api.braintrust.dev` | API endpoint |\n\n## Session Learnings\n\n### What We Learned About Sub-Agent Tracing (Dec 2025)\n\n**Attempted:** Inject trace context via PreToolUse into Task prompts.\n\n**Result:** Failed - SessionStart only receives session metadata, not the prompt.\n\n**Discovery:** Task spans already contain rich sub-agent data:\n- `metadata.agent_type` - agent type from `subagent_type`\n- `metadata.skill_name` - skill from Skill tool\n- `tool_input` - full prompt sent to agent\n- `tool_output` - agent response\n\n**Current correlation path:**\n1. Parent session Task span has `agentId` and timing\n2. Sub-agent creates orphaned trace with `root_span_id = session_id`\n3. SubagentStop provides the sub-agent's `session_id`\n4. Manual correlation: match timing or use `session_id` link\n\n**Future work:** Write `child_session_id` to Task span metadata from PostToolUse after SubagentStop.\n\n## What We Learned About Sub-Agent Correlation\n\n### The Problem\n\n- Sub-agents spawned via Task tool create orphaned Braintrust traces\n- Parent session has Task spans with `agentId`, sub-agent has separate `session_id`\n- No built-in link between them\n\n### What DOESN'T Work\n\n**1. Prompt injection via PreToolUse**\n\nSessionStart hook only receives session metadata (`session_id`, `type`, `cwd`), NOT the prompt. Injected trace context is never seen.\n\nThe hook receives:\n```json\n{\n  \"session_id\": \"...\",\n  \"type\": \"start|resume|compact|clear\",\n  \"cwd\": \"...\",\n  \"env\": {...}\n}\n```\n\nNo prompt field exists - context injection is impossible at SessionStart.\n\n**2. SubagentStop → PostToolUse file handoff**\n\nRace condition. These are independent async hooks with no timing guarantees:\n- SubagentStop fires when sub-agent session ends\n- PostToolUse (Task) fires when Task tool completes\n- No ordering guarantee between them\n- Writing to a correlation file creates a race\n\n**3. PreToolUse correlation files**\n\nSessionStart can't access the `task_span_id` because it has no context about which Task spawned it. PreToolUse modifies prompts but doesn't create a reliably accessible state file that SessionStart can find.\n\n### What DOES Work\n\n**Post-hoc matching for dataset building:**\n\nParent session Task spans contain:\n- `agentId` - identifier for the sub-agent run\n- `totalTokens`, `totalToolUseCount` - aggregated metrics\n- `content` - full agent response/summary\n- `tool_input.prompt` - original task prompt\n- `tool_input.subagent_type` - agent type (e.g., \"oracle\")\n- Start/end timestamps\n\nSub-agent sessions contain:\n- `session_id` (equals orphaned trace `root_span_id`)\n- Start/end timestamps\n- All internal spans and tool calls\n\n**Correlation strategy:**\n1. Export parent session traces (query parent `root_span_id`)\n2. Export sub-agent traces (query all sessions created within parent's time window)\n3. Match by:\n   - Timing: Task span end ≈ sub-agent session end\n   - Metadata: `subagent_type` from Task prompt\n   - IDs: SubagentStop hook provides `session_id` (can be captured and logged)\n\n### Architecture Insight\n\nSessionStart input is intentionally minimal - it contains no prompt or tool context:\n\n```typescript\ninterface SessionStartInput {\n  session_id: string;\n  type: \"start\" | \"resume\" | \"compact\" | \"clear\";\n  cwd: string;\n  env: { [key: string]: string };\n  // NO: prompt, tool_context, task_span_id, parent_span_id\n}\n```\n\nThis design boundary prevents real-time correlation at hook time.\n\n### Recommendation\n\nFor building agent run datasets with sub-agent correlation:\n\n1. **In-session logging:** Capture SubagentStop `session_id` in logs or state\n2. **Post-session export:** Query Braintrust API for parent and sub-agent traces\n3. **Offline correlation:** Match traces by timing and metadata in a script\n4. **Don't try real-time linking:** Hooks don't have necessary context\n\nExample script pattern:\n```bash\n# 1. Export parent session\nbraintrust_analyze.py --replay <parent-session-id> > parent_traces.json\n\n# 2. Query for orphaned sub-agent traces (those created during parent's time window)\nbraintrust_analyze.py --agent-stats > all_agent_traces.json\n\n# 3. Correlate in Python:\n#    - Parent Task spans -> agentId, timestamps, subagent_type\n#    - Orphaned traces -> root_span_id, timestamps\n#    - Match by timing and type\n```\n\nThis approach is reliable, testable, and doesn't require hooks to maintain implicit state.","author":"@parcadei","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/parcadei/Continuous-Claude-v3/tree/main/.claude/skills/braintrust-tracing","license":"MIT","category":"productivity","lang":"en","tokens":3102,"stars":0,"calls30d":1,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["api.braintrust.dev"]}}