domains / research / autonomous-research-pipeline / research-pipeline

research-pipeline

Skill card

combined

Both copies describe the same end-to-end research pipeline, but they disagree on stage numbering, gate behavior, and which sub-skills are invoked. The wanshuiyin copy adds a heartbeat/watchdog cadence, a Lean formal-proof route, and many extra constants (CODE_REVIEW, BASE_REPO, COMPACT, AUTO_WRITE, VENUE, RENDER_HTML, RESUMABLE) and a resume mechanism. The brycewang copy adds an explicit 10-second auto-proceed wait, a RESEARCH_BRIEF.md input, and a different stage layout (implementation and deployment as separate stages, auto-review as Stage 4). Crowning one would hide the other's real operational steps and constraints.

Repositories

Other copies

Combined skill

---
name: research-pipeline
description: "Full end-to-end research pipeline: from a broad research direction through idea discovery, experiments, and review all the way to a polished paper PDF. Use when user says \"全流程\", \"full pipeline\", \"从找idea到投稿\", \"end-to-end research\", or wants the complete autonomous research lifecycle."
argument-hint: "[research-direction] [— resume <run_id>]"
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, WebSearch, WebFetch, Skill, mcp__codex__codex, mcp__codex__codex-reply
---

# Full Research Pipeline: Idea → Experiments → Submission

> ⏱ **External cadence: non-judgmental heartbeat only.** An overnight `/loop` / `CronCreate` heartbeat may wake, detect a **stalled** phase (no progress, dead process, blocked on a freed resource) and **nudge** it forward — it may NEVER decide the work is good (paper good enough, proof holds, claim supported). Every such verdict stays on its own skill's internal cadence and terminates in the cross-model jury. A heartbeat may say "keep going," never "good enough." See `shared-references/external-cadence.md` (overnight-pipeline rule + stall detection & forced structural pivot). At heartbeat startup, touch the run state first each tick and register this run with the watchdog `loop` type (so a silent death surfaces as STALE); unregister on completion. The watchdog only detects — it never acquits. Each tick also record the new-finding count via the `iteration_log.py` helper (resolve through the canonical `.aris/tools → tools → $ARIS_REPO/tools → $ARIS_REPO/tools via ~/.aris/repo` chain, integration-contract §2; warn-and-skip if unresolved): `python3 "$ITER_LOG" note <root> <run_id> <phase> <n>`. On the returned `pivot=structural` (stale ≥ 2) the nudge must change a STRUCTURAL constraint and pick an untried direction; on `pivot=human` (stale ≥ 4) flag for attention. Counting only — never a quality verdict.

End-to-end autonomous research workflow for: **$ARGUMENTS**

## Optional formal proof route

Before creating an experimental run, route theorem-centered requests for Lean to `/lean-formalize` and report its actual proof result. Run the experimental stages below when the task includes empirical research. In a mixed project, use Lean for a user-requested formal proof or a specific mathematical obligation whose formal verification would materially help; return its checked scope and remaining obligations to the research summary and applicable proof audit. Ordinary mathematics stays in the existing theory skills. Difficulty alone does not make Lean mandatory, and a formal sublemma does not establish an empirical claim.

## Constants

- **AUTO_PROCEED = true** — When `true`, every selection checkpoint is informational: report the choice and continue in the same turn. When `false`, ask for explicit user confirmation and end the turn at the checkpoint.
- **ARXIV_DOWNLOAD = false** — When `true`, `/research-lit` downloads the top relevant arXiv PDFs during literature survey. When `false` (default), only fetches metadata via arXiv API. Passed through to `/idea-discovery` → `/research-lit`.
- **HUMAN_CHECKPOINT = false** — When `true`, the auto-review loops (Stage 3) pause after each round's review to let you see the score and provide custom modification instructions before fixes are implemented. When `false` (default), loops run fully autonomously. Passed through to `/auto-review-loop`.
- **REVIEWER_DIFFICULTY = medium** — How adversarial the reviewer is. `medium` (default): standard MCP review. `hard`: adds reviewer memory + debate protocol. `nightmare`: GPT reads repo directly via `codex exec` + memory + debate. Passed through to `/auto-review-loop`.
- **CODE_REVIEW = true** — GPT-6-Astra xhigh reviews experiment code before deployment. Catches logic bugs before wasting GPU hours. Set `false` to skip. Passed through to `/experiment-bridge`.
- **BASE_REPO = false** — GitHub repo URL to use as base codebase. When set, `/experiment-bridge` clones the repo first and implements experiments on top of it. When `false` (default), writes code from scratch or reuses existing project files. Passed through to `/experiment-bridge`.
- **COMPACT = false** — When `true`, generates compact summary files for short-context models and session recovery. Passed through to `/idea-discovery` and `/experiment-bridge`.
- **AUTO_WRITE = false** — When `true`, automatically invoke Workflow 3 (`/paper-writing`) after Stage 4. `VENUE` is needed only when Stage 5 begins — a missing venue defers paper writing; it never blocks Stages 1-4. When `false` (default), Stage 4 generates `NARRATIVE_REPORT.md` and stops — user invokes `/paper-writing` manually.
- **VENUE = (unset)** — Target venue for paper writing; bound only when Stage 5 begins. Options: `ICLR`, `NeurIPS`, `ICML`, `CVPR`, `ACL`, `AAAI`, `ACM`, `IEEE_CONF`, `IEEE_JOURNAL`. No default: a missing venue defers paper writing — it never blocks Stages 1-4 and is never guessed.
- **RENDER_HTML = true** — When `true` (default), auto-render `NARRATIVE_REPORT.md` to HTML at Stage 4 completion via `/render-html`. Uses `--no-review` (this is an internal handoff doc to `/paper-writing`, not a reviewer-facing final artifact — the upstream Stage 3 auto-review loop already cross-model-reviewed the claims). Set `false` to skip, or pass `— render html: false`. **Non-blocking**: if `/render-html` fails or Codex MCP is unavailable, log the failure and continue — the HTML view is a nice-to-have, not a Stage 4 prerequisite.
- **RESUMABLE = true** — When `true` (default), the pipeline records per-stage state to `.aris/runs/<run_id>.json` so a crashed/interrupted run can resume via `/research-pipeline — resume <run_id>` instead of restarting. Stage status splits `done` (executor finished writing) from `accepted` (the stage's cross-model gate / deterministic verifier passed); resume re-validates any `done`-but-unaccepted stage.

> 💡 Override via argument, e.g., `/research-pipeline "topic" — AUTO_PROCEED: false, human checkpoint: true`.

## Overview

This skill chains the entire research lifecycle into a single pipeline:

```
/idea-discovery → implement → /run-experiment → /auto-review-loop → submission-ready
├── Workflow 1 ──┤            ├────────── Workflow 2 ──────────────┤
```

It orchestrates two major workflows plus the implementation bridge between them.

## Pipeline

### Stage 1: Idea Discovery (Workflow 1)

If `RESEARCH_BRIEF.md` exists in the project root, it will be automatically loaded as detailed context (replaces one-line prompt). See `templates/RESEARCH_BRIEF_TEMPLATE.md`.

Invoke the idea discovery pipeline:

```
/idea-discovery "$ARGUMENTS"
```

This internally runs: `/research-lit` → `/idea-creator` → `/novelty-check` → `/research-review`

**Output:** `IDEA_REPORT.md` with ranked, validated, pilot-tested ideas.

**🚦 Gate 1 — Human Checkpoint:**

After `IDEA_REPORT.md` is generated, **pause and present the top ideas to the user**:

```
📋 Idea Discovery complete. Top ideas:

1. [Idea 1 title] — Pilot: POSITIVE (+X%), Novelty: CONFIRMED
2. [Idea 2 title] — Pilot: WEAK POSITIVE (+Y%), Novelty: CONFIRMED
3. [Idea 3 title] — Pilot: NEGATIVE, eliminated

Recommended: Idea 1. Shall I proceed with implementation?
```

**If AUTO_PROCEED=false:** Wait for user confirmation before continuing. The user may:
- **Approve an idea** → proceed to Stage 2.
- **Pick a different idea** → proceed with their choice.
- **Request changes** (e.g., "combine Idea 1 and 3", "focus more on X") → update the idea prompt with user feedback, re-run `/idea-discovery` with refined constraints, and present again.
- **Reject all ideas** → collect feedback on what's missing, re-run Stage 1 with adjusted research direction. Repeat until the user commits to an idea.
- **Stop here** → save current state to `IDEA_REPORT.md` for future reference.

**If AUTO_PROCEED=true:** Present the top ideas, wait 10 seconds for user input. If no response, auto-select the #1 ranked idea (highest pilot signal + novelty confirmed) and proceed to Stage 2. Log: `"AUTO_PROCEED: selected Idea 1 — [title]"`.

> ⚠️ **This gate waits for user confirmation when AUTO_PROCEED=false.** When `true`, it auto-selects the top idea after presenting results. The rest of the pipeline (Stages 2-4) is expensive (GPU time + multiple review rounds), so set `AUTO_PROCEED=false` if you want to manually choose which idea to pursue.

### Stage 2: Implementation

Once the user confirms which idea to pursue:

1. **Read the idea details** from `IDEA_REPORT.md` (hypothesis, experimental design, pilot code)

2. **Implement the full experiment**:
   - Extend pilot code to full scale (multi-seed, full dataset, proper baselines)
   - Add proper evaluation metrics and logging (wandb if configured)
   - Write clean, reproducible experiment scripts
   - Follow existing codebase conventions

3. **Code review**: Before deploying, do a self-review:
   - Are all hyperparameters configurable via argparse?
   - Is the random seed fixed and controllable?
   - Are results saved to JSON/CSV for later analysis?
   - Is there proper logging for debugging?

### Stage 3: Deploy Experiments (Workflow 2 — Part 1)

Deploy the full-scale experiments:

```
/run-experiment [experiment command]
```

**What this does:**
- Check GPU availability on configured servers
- Sync code to remote server
- Launch experiments in screen sessions with proper CUDA_VISIBLE_DEVICES
- Verify experiments started successfully

**Monitor progress:**

```
/monitor-experiment [server]
```

Wait for experiments to complete. Collect results.

### Stage 4: Auto Review Loop (Workflow 2 — Part 2)

Once initial results are in, start the autonomous improvement loop:

```
/auto-review-loop "$ARGUMENTS — [chosen idea title]"
```

**What this does (up to 4 rounds):**
1. GPT-5.4 xhigh reviews the work (score, weaknesses, minimum fixes)
2. Claude Code implements fixes (code changes, new experiments, reframing)
3. Deploy fixes, collect new results
4. Re-review → repeat until score ≥ 6/10 or 4 rounds reached

**Output:** `AUTO_REVIEW.md` with full review history and final assessment.

### Stage 5: Final Summary

After the auto-review loop completes, write a final status report:

```markdown
# Research Pipeline Report

**Direction**: $ARGUMENTS
**Chosen Idea**: [title]
**Date**: [start] → [end]
**Pipeline**: idea-discovery → implement → run-experiment → auto-review-loop

## Journey Summary
- Ideas generated: X → filtered to [final count]
- Experiments run: [count]
- Review rounds: [count]
- Final score: [score]/10
```

## Conflicts

- **wanshuiyin/Auto-claude-code-research-in-sleep::skills/research-pipeline/SKILL.md** places the auto-review loop at Stage 3 and treats selection checkpoints as informational when AUTO_PROCEED=true; **brycewang-stanford/Auto-Empirical-Research-Skills::skills/42-wanshuiyin-ARIS/skills/research-pipeline/SKILL.md** places the auto-review loop at Stage 4 and, when AUTO_PROCEED=true, waits 10 seconds for user input before auto-selecting the top idea.
- **wanshuiyin/Auto-claude-code-research-in-sleep::skills/research-pipeline/SKILL.md** includes an external heartbeat/watchdog cadence, a Lean formal-proof route, and a RESUMABLE run-state mechanism; **brycewang-stanford/Auto-Empirical-Research-Skills::skills/42-wanshuiyin-ARIS/skills/research-pipeline/SKILL.md** omits all of these and instead loads RESEARCH_BRIEF.md and uses /run-experiment and /monitor-experiment as explicit stages.
- **wanshuiyin/Auto-claude-code-research-in-sleep::skills/research-pipeline/SKILL.md** lists REVIEWER_DIFFICULTY, CODE_REVIEW, BASE_REPO, COMPACT, AUTO_WRITE, VENUE, RENDER_HTML, and RESUMABLE constants; **brycewang-stanford/Auto-Empirical-Research-Skills::skills/42-wanshuiyin-ARIS/skills/research-pipeline/SKILL.md** does not.