domains / data / apify-actor-selection-and-execution / apify-ultimate-scraper

apify-ultimate-scraper

Skill card

combined

The two copies describe the same high-level task (select an Apify Actor, fetch schema, run, deliver results) but they disagree on the execution mechanism and include material steps the other lacks. The apify/agent-skills copy is the canonical procedure: it mandates the Apify CLI, defines universal command rules (--json, --user-agent, 2>/dev/null), provides a dynamic actor search fallback, and includes a workflow-guide routing table. The sickn33 copy is a different implementation: it requires the mcpc CLI, hard-codes a large Actor catalog, adds a pricing-approval gate, and uses a checklist-driven workflow. Crowning one would hide a real alternative execution path and omit either the CLI rules or the pricing/actor-catalog constraints.

Repositories

Other copies

Combined skill

---
name: apify-ultimate-scraper
description: Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis, influencer discovery, trend research, content analytics, audience analysis, review analysis, SEO intelligence, recruitment, or any data extraction task.
---

# Universal web scraper

AI-driven data extraction from 55+ Actors across 15+ platforms via the Apify CLI or mcpc CLI.

## Rules for every apify command
1. Pass --json for machine-readable output (stable across CLI versions).
2. Pass --user-agent apify-agent-skills/apify-ultimate-scraper for telemetry attribution.
3. Redirect stderr with 2>/dev/null (stderr contains progress messages that break JSON parsers).

## Prerequisites
- Apify CLI v1.5.0+ (npm install -g apify-cli) OR mcpc CLI (npm install -g @apify/mcpc)
- Node.js 20.6+ (for native --env-file support when using mcpc)
- Authenticated session (see below)

## Authentication
If a CLI command fails with an auth error, authenticate using one of these methods:
1. OAuth (interactive): apify login (opens browser)
2. Environment variable: export APIFY_TOKEN=your_token_here
3. From .env file: source .env (if the file contains APIFY_TOKEN=...)

Generate token: https://console.apify.com/settings/integrations

## Workflow

Copy this checklist and track progress:

```
Task Progress:
- [ ] Step 1: Understand user goal and select Actor
- [ ] Step 2: Fetch Actor schema and check gotchas
- [ ] Step 3: Confirm user preferences and pricing approval
- [ ] Step 4: Configure and run
- [ ] Step 5: Deliver results
```

### Step 1: Understand goal and select Actor

Identify the target platform and use case. Use one of these methods to find the right Actor:

**Method A (dynamic search, preferred):** Read references/actor-index.md to find the right Actor. If no Actor matches in the index, search dynamically:

    apify actors search "KEYWORDS" --user-agent apify-agent-skills/apify-ultimate-scraper --json --limit 10 2>/dev/null

From results: items[].username/items[].name (Actor ID), items[].title, items[].stats.totalUsers30Days, items[].currentPricingInfo.pricingModel.

**Method B (hard-coded catalog):** Select from the Actor catalog below.

#### Instagram Actors (12)

| Actor ID | Best For |
|----------|----------|
| apify/instagram-profile-scraper | Profile data, follower counts, bio info |
| apify/instagram-post-scraper | Individual post details, engagement metrics |
| apify/instagram-comment-scraper | Comment extraction, sentiment analysis |
| apify/instagram-hashtag-scraper | Hashtag content, trending topics |
| apify/instagram-hashtag-stats | Hashtag performance metrics |
| apify/instagram-reel-scraper | Reels content and metrics |
| apify/instagram-search-scraper | Search users, places, hashtags |
| apify/instagram-tagged-scraper | Posts tagged with specific accounts |
| apify/instagram-followers-count-scraper | Follower count tracking |
| apify/instagram-scraper | Comprehensive Instagram data |
| apify/instagram-api-scraper | API-based Instagram access |
| apify/export-instagram-comments-posts | Bulk comment/post export |

#### Facebook Actors (14)

| Actor ID | Best For |
|----------|----------|
| apify/facebook-pages-scraper | Page data, metrics, contact info |
| apify/facebook-page-contact-information | Emails, phones, addresses from pages |
| apify/facebook-posts-scraper | Post content and engagement |
| apify/facebook-comments-scraper | Comment extraction |
| apify/facebook-likes-scraper | Reaction analysis |
| apify/facebook-reviews-scraper | Page reviews |
| apify/facebook-groups-scraper | Group content and members |
| apify/facebook-events-scraper | Event data |
| apify/facebook-ads-scraper | Ad creative and targeting |
| apify/facebook-search-scraper | Search results |
| apify/facebook-reels-scraper | Reels content |
| apify/facebook-photos-scraper | Photo extraction |
| apify/facebook-marketplace-scraper | Marketplace listings |
| apify/facebook-followers-following-scraper | Follower/following lists |

#### TikTok Actors (14)

| Actor ID | Best For |
|----------|----------|
| clockworks/tiktok-scraper | Comprehensive TikTok data |
| clockworks/free-tiktok-scraper | Free TikTok extraction |
| clockworks/tiktok-profile-scraper | Profile data |
| clockworks/tiktok-video-scraper | Video details and metrics |
| clockworks/tiktok-comments-scraper | Comment extraction |
| clockworks/tiktok-followers-scraper | Follower lists |
| clockworks/tiktok-user-search-scraper | Find users by keywords |
| clockworks/tiktok-hashtag-scraper | Hashtag content |
| clockworks/tiktok-sound-scraper | Trending sounds |
| clockworks/tiktok-ads-scraper | Ad content |
| clockworks/tiktok-discover-scraper | Discover page content |
| clockworks/tiktok-explore-scraper | Explore content |
| clockworks/tiktok-trends-scraper | Trending content |
| clockworks/tiktok-live-scraper | Live stream data |

#### YouTube Actors (5)

| Actor ID | Best For |
|----------|----------|
| streamers/youtube-scraper | Video data and metrics |
| streamers/youtube-channel-scraper | Channel information |
| streamers/youtube-comments-scraper | Comment extraction |
| streamers/youtube-shorts-scraper | Shorts content |
| streamers/youtube-video-scraper-by-hashtag | Videos by hashtag |

#### Google Maps Actors (4)

| Actor ID | Best For |
|----------|----------|
| compass/crawler-google-places | Business listings, ratings, contact info |
| compass/google-maps-extractor | Detailed business data |
| compass/Google-Maps-Reviews-Scraper | Review extraction |
| poidata/google-maps-email-extractor | Email discovery from listings |

#### X/Twitter Actors (2)

| Actor ID | Best For |
|----------|----------|
| xquik/x-tweet-scraper | Tweet lookup, search, timelines, lists, threads, replies, quotes, and engagement |
| xquik/x-follower-scraper | Followers, following, verified followers, lists, communities, and audience overlap |

Xquik is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

#### Other Actors (6)

| Actor ID | Best For |
|----------|----------|
| apify/google-search-scraper | Google search results |
| apify/google-trends-scraper | Google Trends data |
| voyager/booking-scraper | Booking.com hotel data |
| voyager/booking-reviews-scraper | Booking.com reviews |
| maxcopell/tripadvisor-reviews | TripAdvisor reviews |
| vdrmota/contact | Contact extraction |

If the task involves a multi-step pipeline, also read the matching workflow guide:

| Task involves... | Read |
|-----------------|------|
| leads, contacts, emails, B2B | references/workflows/lead-generation.md |
| competitor, ads, pricing | references/workflows/competitive-intel.md |
| influencer, creator | references/workflows/influencer-vetting.md |
| brand, mentions, sentiment | references/workflows/brand-monitoring.md |
| reviews, ratings, reputation | references/workflows/review-analysis.md |
| SEO, SERP, crawl, content, RAG | references/workflows/content-and-seo.md |
| analytics, engagement, performance | references/workflows/social-media-analytics.md |
| trends, keywords, hashtags | references/workflows/trend-research.md |
| jobs, recruiting, candidates | references/workflows/job-market-and-recruitment.md |
| real estate, listings, hotels | references/workflows/real-estate-and-hospitality.md |
| price monitoring, e-commerce, products | references/workflows/ecommerce-price-monitoring.md |
| contact enrichment, email extraction | references/workflows/contact-enrichment.md |
| knowledge base, RAG, LLM data feed | references/workflows/knowledge-base-and-rag.md |
| company research, due diligence | references/workflows/company-research.md |

### Step 2: Fetch Actor schema and check gotchas

Fetch the input schema dynamically:

    apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --input --json 2>/dev/null

Also read references/gotchas.md to check for common pitfalls for the selected Actor.

For Actor documentation: apify actors info "ACTOR_ID" --user-agent apify-agent-skills/apify-ultimate-scraper --readme

### Step 3: Confirm user preferences and pricing approval

**Skip user preferences** for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode.

For larger tasks, confirm output format (quick answer / CSV / JSON) and result count.

**Pricing approval (required for paid runs):** Check each Actor's live Apify pricing box before starting a paid run. Show the Actor, targets, result cap, and maximum charge. Get explicit approval. Set a conservative result cap. maxItems applies across the whole run.

### Step 4: Configure and run

**Standard run (blocking):**

    apify actors call "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null

Prefer --input-file input.json for large or complex inputs. For tiny inputs, inline JSON is acceptable with shell quoting: --input '{"maxItems":10}'.

From output: .id (run ID), .status, .defaultDatasetId, .stats.durationMillis

**Fetch results:**

    apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format json

For CSV: apify datasets get-items DATASET_ID --user-agent apify-agent-skills/apify-ultimate-scraper --format csv

**Quick answer mode:** Fetch results as JSON, pick top 5, present formatted in chat.

**Save to file:** Fetch results, use Write tool to save as YYYY-MM-DD_descriptive-name.csv or .json.

**Large/long-running scrapes:**

    apify actors start "ACTOR_ID" --input-file input.json --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null

Poll: apify runs info RUN_ID --user-agent apify-agent-skills/apify-ultimate-scraper --json 2>/dev/null (check .status for SUCCEEDED).

### Step 5: Deliver results

Report: result count, file location (if saved), key data fields, and links:
- Dataset: https://console.apify.com/storage/datasets/DATASET_ID
- Run: https://console.apify.com/actors/runs/RUN_ID

For multi-step workflows: suggest the next pipeline step from the workflow guide.

## Troubleshooting

Common errors and pitfalls are documented in references/gotchas.md. Read it before running PPE (pay-per-event) Actors.