domains / data / web-scraping-extraction / scrapling

scrapling

Skill card

combined

The three copies overlap on installing Scrapling and using the CLI to extract content, but each contains material steps the others lack. The Vibe-Skills copy is CLI-first and documents the optional MCP server and a Windows PowerShell wrapper. The daymade copy is a diagnostic/troubleshooting workflow that insists on verifying the install, validating output files, handling TLS trust-store failures, and using a specific WeChat selector. The LeoYeAI copy is Python-library-first and adds Spider crawling, adaptive parsing, stealthy-fetch, and API reverse-engineering patterns. Crowning one would hide real alternatives (CLI vs Python API, MCP vs no MCP, diagnostic vs library-first).

Repositories

Other copies

Combined skill

# Scrapling Skill

## Overview
Scrapling is a Python web scraping toolkit with a CLI, optional MCP server, and Python library API. Use it to extract HTML, Markdown, or text from web pages, crawl multiple pages, and bypass basic anti-bot protections.

## When to Use
- Extract specific parts of a web page using CSS selectors or XPath.
- Run repeatable scraping jobs over a list of URLs.
- Crawl multiple pages with a Spider.
- Reduce token usage by extracting only the relevant DOM region before passing to an LLM.
- Provide a local MCP endpoint for structured scraping tools.

Do not use for:
- X/Twitter (use a dedicated tweet-fetcher skill).
- Login-protected sites unless credentials are provided.
- Paywalled content or sites that prohibit scraping in their TOS.

## Prerequisite Check
1. Verify Python version (>= 3.10):
   ```bash
   python --version
   ```
2. Verify Scrapling CLI availability:
   ```bash
   scrapling --help
   ```
3. If a diagnostic script is available, run it:
   ```bash
   python3 scripts/diagnose_scrapling.py
   ```

## Installation
Choose the extras you need:

- CLI + fetchers (recommended for most CLI workflows):
  ```bash
  pip install "scrapling[fetchers]"
  ```
- CLI shell tools:
  ```bash
  pip install "scrapling[shell]"
  ```
- MCP server support:
  ```bash
  pip install "scrapling[ai]"
  ```
- Everything:
  ```bash
  pip install "scrapling[all]"
  ```

If using browser-backed fetchers, install browser binaries:
```bash
scrapling install
# or
python -m playwright install
```

## CLI Extraction
Always quote URLs in shell commands, especially when they contain `?`, `&`, or other special characters.

### Static Fetch (default)
```bash
scrapling extract get 'https://example.com' page.html
scrapling extract get 'https://example.com' article.md --css-selector 'main'
```

### Browser-Backed Fetch (for JS-rendered pages)
```bash
scrapling extract fetch 'https://example.com' page.html --timeout 20000
```

### Stealthy Fetch (for anti-bot or challenge pages)
```bash
scrapling extract stealthy-fetch 'https://example.com' page.html
```

### WeChat Public Articles
Use `#js_content` as the default selector for article body on `mp.weixin.qq.com`:
```bash
scrapling extract get 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' article.md --css-selector '#js_content'
```

## Python Library API

### Basic Fetch
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')
title = page.css('h1::text').get()
paragraphs = page.css('p::text').getall()
```

### Stealthy Fetch
```python
from scrapling.fetchers import StealthyFetcher
StealthyFetcher.adaptive = True
page = StealthyFetcher.fetch('https://example.com', headless=True, solve_cloudflare=True)
```

### Dynamic Fetch
```python
from scrapling.fetchers import DynamicFetcher
page = DynamicFetcher.fetch('https://example.com', headless=True, network_idle=True)
```

### Adaptive Parsing
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')
items = page.css('.product', auto_save=True)
# Later, if the site changes:
items = page.css('.product', adaptive=True)
```

### Spider (Multiple Pages)
```python
from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com"]
    concurrent_requests = 3

    async def parse(self, response: Response):
        for item in response.css('.item'):
            yield {"item": item.css('h2::text').get()}
        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

MySpider().start()
```

## MCP Server (Optional)
Start the MCP server with stdio transport:
```bash
scrapling mcp
```
Or with HTTP transport:
```bash
scrapling mcp --http --host 127.0.0.1 --port 8765
```

Example MCP server config:
```json
{
  "servers": {
    "scrapling": {
      "mode": "stdio",
      "command": "scrapling",
      "args": ["mcp"],
      "required": false
    }
  }
}
```

## Validation
After every extraction, verify the output file:
```bash
wc -c article.md
sed -n '1,40p' article.md
```
For HTML output, check that the expected container or selector target is present:
```bash
rg -n '<title>|js_content|rich_media_title|main' page.html
```
If the file is tiny, empty, or missing the expected container, switch fetchers or selectors.

## Troubleshooting
- If `extract get` fails with `curl: (60) SSL certificate problem`, treat it as a local trust-store issue. Retry with `--no-verify` only after confirming the error pattern.
- If browser-backed fetch fails, re-run the diagnostic script, confirm Chromium and Chrome Headless Shell are present, retry with a longer timeout, and escalate to `stealthy-fetch` only if justified.
- Do not claim success from exit code alone. Inspect the saved file.

## Guardrails
- Prefer selector-based extraction to minimize data volume.
- Treat scraping as an external dependency: handle timeouts, retries, and failures explicitly.
- Do not default to browser-backed fetching unless the static result is missing the real content.
- Do not hardcode user-specific absolute paths into outputs or docs.