{"id":"paper-writing-bench","name":"paper-writing-bench","summary":"既存のAI研究論文から原材料(Sparseアイデア、Denseアイデア、実験ログ)をリバースエンジニアリングし、論文作成パイプラインを評価するためのベンチマークケースを構築します。","body":"# PaperWritingBench (§3)\n\nFaithful implementation of the PaperWritingBench dataset construction\nprocedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and\nApp. C, F.2).\n\nThe original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025).\nFor each paper, the authors reverse-engineer the (I, E) tuple by stripping\nnarrative flow from the original PDF using the three prompts in App. F.2.\nYou can use this skill to reverse-engineer your own benchmark cases from\nany paper PDF.\n\n## What this skill does\n\nGiven an existing AI research paper (PDF or markdown extract), produce:\n\n- `idea.md` (Sparse variant) — high-level concept note, no math, no\n  experimental results\n- `idea.md` (Dense variant) — detailed technical proposal with LaTeX\n  equations and variable definitions, but still no experimental results\n- `experimental_log.md` — exhaustive raw experimental setup, numeric data,\n  and qualitative observations, with all narrative references stripped\n\nThese three files form a complete (I, E) input pair for the\npaper-orchestra pipeline. You can then run the pipeline and compare its\noutput to the original paper using `paper-autoraters`.\n\n## Inputs\n\n- A paper PDF or extracted markdown text. The paper uses MinerU\n  (Wang et al., 2024) for PDF→markdown extraction; you (the host agent)\n  should use whatever PDF extractor your environment provides.\n- For controlled experiments, you may also extract figures separately\n  (PDFFigures 2.0 in the paper).\n\n## Outputs\n\n- `bench/<paper_id>/idea_sparse.md` — Sparse variant\n- `bench/<paper_id>/idea_dense.md` — Dense variant\n- `bench/<paper_id>/experimental_log.md` — Experimental log\n\n## Workflow\n\nFor each paper, run three independent LLM calls using the verbatim prompts\nbelow:\n\n### 1. Sparse idea generation\n\nLoad `references/sparse-idea-prompt.md`. Pass the paper text (or\nmarkdown extract) as `{paper_content}`. The prompt instructs the model to:\n\n- Stop extracting at empirical verification (no Experiments / Results / Comparisons)\n- Use first-person future tense (\"We propose to explore...\")\n- Avoid LaTeX math; describe components by function\n- Anonymize authors and titles\n\nOutput: `idea_sparse.md` with the four sections (Problem Statement, Core\nHypothesis, Proposed Methodology high-level, Expected Contribution).\n\n### 2. Dense idea generation\n\nLoad `references/dense-idea-prompt.md`. Same input. The prompt instructs\nthe model to:\n\n- Preserve mathematical formulations using LaTeX\n- Define every variable used in equations\n- Include specific architectural choices and dimensions\n- Same exclusion zone (no experiments)\n\nOutput: `idea_dense.md` with the four sections (Problem Statement, Core\nHypothesis, Proposed Methodology detailed, Expected Contribution).\n\n### 3. Experimental log generation\n\nLoad `references/experimental-log-prompt.md`. Same input. The prompt\ninstructs the model to:\n\n- Use past-tense persona (\"We ran...\", \"The results were...\")\n- Strip all references to figure/table numbers\n- Deconstruct tables into raw numeric data\n- Log figure findings as factual observations\n- Anonymize authors\n\nOutput: `experimental_log.md` with sections for Setup, Raw Numeric Data,\nand Qualitative Observations.\n\n## Critical rules from the prompts\n\nThese are excerpted from App. F.2. The host agent MUST honor them:\n\n- **No citations.** None of the three outputs may contain `\\cite`,\n  reference numbers, or author names from the source paper.\n- **No URLs.** Strip all hyperlinks.\n- **Anonymize.** Author identities, affiliations, acknowledgements all\n  removed.\n- **Self-contained.** Each file must make sense without the original paper.\n- **No experimental leakage in idea files.** The Sparse and Dense ideas\n  must stop where empirical verification begins. They describe what will\n  be done, not what was done.\n- **No table/figure references in experimental log.** No \"as shown in\n  Table 1\", \"see Fig. 5\". The downstream paper-orchestra pipeline will\n  generate its own figures and tables — the log must not assume any\n  particular ones exist.\n- **100% numeric accuracy in experimental log.** This becomes the ground\n  truth for the section-writing-agent and content-refinement-agent's\n  hallucination check.\n\n## How the bench is used\n\nAfter producing `(idea_sparse.md, idea_dense.md, experimental_log.md)` for\na paper:\n\n1. Pick a variant (Sparse or Dense) — the paper ablates both, with Dense\n   producing more rigorous methodology and Sparse exercising the system's\n   robustness on under-specified inputs.\n2. Drop the chosen `idea.md`, plus `experimental_log.md`, plus a\n   `template.tex` for the target conference, plus a\n   `conference_guidelines.md`, into a paper-orchestra workspace.\n3. Run the pipeline.\n4. Compare the generated paper against the original using\n   `paper-autoraters` (citation F1, lit review quality, SxS paper quality).\n\n## Resources\n\n- `references/bench-overview.md` — the 200-paper bench, venue cutoffs, sizes\n- `references/sparse-idea-prompt.md` — verbatim from App. F.2\n- `references/dense-idea-prompt.md` — verbatim from App. F.2\n- `references/experimental-log-prompt.md` — verbatim from App. F.2","author":"@Ar9av","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Ar9av/PaperOrchestra/tree/main/skills/paper-writing-bench","license":"MIT","category":"writing","lang":"en","tokens":1202,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/bench-overview.md","size":4316,"sha256":"9777265d6bf68915378b6fc0b7c52f5bd373dd628cbbb8f37f42c78fa35b8b5a"},{"path":"references/dense-idea-prompt.md","size":3287,"sha256":"c1008d0508ea1f95a4b2e4d71bcbf27331b5e3ba960c656387107a679909967b"},{"path":"references/experimental-log-prompt.md","size":3288,"sha256":"7a65fdce90163efa7f50527be051af5418f3a8a5192809a0b7b5073a9d0b49b9"},{"path":"references/sparse-idea-prompt.md","size":2796,"sha256":"57829303c284ec5b562ff5f055795640f58f71cca423b2875f7e07f1c4888557"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}