domains / documents / url-to-markdown / baoyu-url-to-markdown

baoyu-url-to-markdown

Skill card

combined

The three copies describe the same task but disagree on the underlying tooling, CLI surface, and setup behavior. JimLiu/baoyu-skills is the most complete and appears to be the upstream procedure (baoyu-fetch CLI, site-specific adapters, media download, first-time setup). anbeime/skill is a simplified wrapper around a local main.ts script with a different flag set and no first-time setup. LeoYeAI/openclaw-master-skills is a mirror of the upstream with a different version, additional fallbacks, and a different runtime resolution strategy. Crowning one would hide the real alternatives and conflicting parameters.

Repositories

Other copies

Combined skill

---
name: baoyu-url-to-markdown
description: Fetch any URL and convert to markdown using Chrome CDP with site-specific adapters or a local script. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown.
---

# URL to Markdown

Fetches any URL via Chrome CDP and converts it to clean markdown.

## Script Directory

**Important**: All scripts are located in the `scripts/` subdirectory of this skill.

**Agent Execution Instructions**:
1. Determine this SKILL.md file's directory path as `{baseDir}`
2. Resolve `${BUN_X}` runtime: if `bun` installed → `bun`; if `npx` available → `npx -y bun`; else suggest installing bun
3. If `{baseDir}/scripts/node_modules` does not exist, run `${BUN_X} install --cwd {baseDir}/scripts`
4. `${READER}` = `{baseDir}/scripts/baoyu-fetch` (or `{baseDir}/scripts/main.ts` if using the local script variant)
5. Replace all `${READER}` and `${BUN_X}` in this document with the resolved values

## Preferences (EXTEND.md)

Check EXTEND.md in priority order — the first one found wins:

| Priority | Path | Scope |
|----------|------|-------|
| 1 | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Project |
| 2 | `${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | XDG |
| 3 | `$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | User home |

| Result | Action |
|--------|--------|
| Found | Read, parse, apply settings |
| Not found | **MUST** run first-time setup (see below) — do NOT silently create defaults |

**EXTEND.md supports**: download media by default, default output directory, default capture mode, timeout settings.

### First-Time Setup ⛔ BLOCKING

When EXTEND.md is not found, you **MUST** use `AskUserQuestion` to gather preferences before creating EXTEND.md. **NEVER** create EXTEND.md with silent defaults. Generation is BLOCKED until setup completes. Batch all three questions into a single call:

- **Q1 — Media** (header "Media"): "How to handle images and videos in pages?"
  - "Ask each time (Recommended)" — Prompt after each save
  - "Always download" — Download to local `imgs/` and `videos/`
  - "Never download" — Keep remote URLs
- **Q2 — Output** (header "Output"): "Default output directory?"
  - "url-to-markdown (Recommended)" — Save to `./url-to-markdown/{domain}/{slug}.md`
  - User may pick "Other" and type a custom path
- **Q3 — Save** (header "Save"): "Where to save preferences?"
  - "User (Recommended)" — `~/.baoyu-skills/` (all projects)
  - "Project" — `.baoyu-skills/` (this project only)

After answers, write EXTEND.md, confirm "Preferences saved to [path]", then continue.

### Supported Keys

| Key | Default | Values | Description |
|-----|---------|--------|-------------|
| `download_media` | `ask` | `ask` / `1` / `0` | `ask` = prompt each time, `1` = always, `0` = never |
| `default_output_dir` | empty | path or empty | Default output directory (empty = `./url-to-markdown/`) |

**EXTEND.md → CLI mapping**:

| EXTEND.md key | CLI argument | Notes |
|---------------|-------------|-------|
| `download_media: 1` | `--download-media` | Requires `--output` to be set |
| `default_output_dir: ./posts/` | Agent constructs `--output ./posts/{domain}/{slug}.md` | Agent generates path, not a direct flag |

**Value priority**: CLI arguments → EXTEND.md → skill defaults.

## Usage

```bash
# Default: headless capture, markdown to stdout
${READER} <url>

# Save to file
${READER} <url> --output article.md

# Save with media download
${READER} <url> --output article.md --download-media

# Wait for interaction (login/CAPTCHA) — auto-detect and continue
${READER} <url> --wait-for interaction --output article.md

# Wait for interaction — manual control (Enter to continue)
${READER} <url> --wait-for force --output article.md

# JSON output
${READER} <url> --format json --output article.json

# Force specific adapter
${READER} <url> --adapter youtube --output transcript.md
```

## Options

| Option | Description |
|--------|-------------|
| `<url>` | URL to fetch |
| `--output <path>` | Output file path (default: stdout) |
| `--format <type>` | Output format: `markdown` (default) or `json` |
| `--json` | Shorthand for `--format json` |
| `--adapter <name>` | Force adapter: `x`, `youtube`, `hn`, or `generic` (default: auto-detect) |
| `--headless` | Force headless Chrome (no visible window) |
| `--wait-for <mode>` | Interaction wait mode: `none` (default), `interaction`, or `force` |
| `--wait-for-interaction` | Alias for `--wait-for interaction` |
| `--wait-for-login` | Alias for `--wait-for interaction` |
| `--timeout <ms>` | Page load timeout (default: 30000) |
| `--interaction-timeout <ms>` | Login/CAPTCHA wait timeout (default: 600000 = 10 min) |
| `--interaction-poll-interval <ms>` | Poll interval for interaction checks (default: 1500) |
| `--download-media` | Download images/videos to local `imgs/` and `videos/`, rewrite markdown links. Requires `--output` |
| `--media-dir <dir>` | Base directory for media downloads |

## Capture Modes

| Mode | Behavior | Use When |
|------|----------|----------|
| Auto (default) | Capture on network idle | Public pages, static content |
| Wait (`--wait-for interaction`) | Auto-detect and continue | Login-required, lazy loading, paywalls |
| Force (`--wait-for force`) | User signals when ready | Complex pages requiring manual interaction |

**Wait mode workflow**:
1. Run with `--wait-for interaction` → script outputs "Press Enter when ready"
2. Ask user to confirm page is ready
3. Send newline to stdin to trigger capture

## Output Format

YAML front matter with `url`, `title`, `description`, `author`, `published`, `captured_at` fields, followed by converted markdown content.

## Output Directory

```
url-to-markdown/<domain>/<slug>.md
```

- `<slug>`: From page title or URL path (kebab-case, 2-6 words)
- Conflict resolution: Append timestamp `<slug>-YYYYMMDD-HHMMSS.md`

## Environment Variables

| Variable | Description |
|----------|-------------|
| `URL_CHROME_PATH` | Custom Chrome executable path |
| `URL_DATA_DIR` | Custom data directory |
| `URL_CHROME_PROFILE_DIR` | Custom Chrome profile directory |

**Troubleshooting**: Chrome not found → set `URL_CHROME_PATH`. Timeout → increase `--timeout`. Complex pages → try `--wait-for interaction` mode.

## Extension Support

Custom configurations via EXTEND.md. See **Preferences** section for paths and supported options.

## Conflicts

- **JimLiu/baoyu-skills** uses the `baoyu-fetch` CLI with site-specific adapters and a `--download-media` flag; **anbeime/skill** uses a local `main.ts` script with only `--wait` and `-o` flags and no media download; **LeoYeAI/openclaw-master-skills** uses a local `main.ts` script with a Defuddle pipeline and fallback to a hosted API, and a different runtime resolution (`bun` or `npx -y bun`).
- **JimLiu/baoyu-skills** and **LeoYeAI/openclaw-master-skills** require a blocking first-time setup via `AskUserQuestion` when EXTEND.md is not found; **anbeime/skill** says to use defaults when not found.
- **JimLiu/baoyu-skills** resolves the runtime as `bun` only; **LeoYeAI/openclaw-master-skills** resolves as `bun` or `npx -y bun`; **anbeime/skill** uses `npx -y bun` directly.
- **JimLiu/baoyu-skills** and **LeoYeAI/openclaw-master-skills** include an XDG config path in the EXTEND.md search order; **anbeime/skill** does not.