domains / families / data / crawlee

crawlee

separate

Expert guide for building web scrapers and crawlers using Crawlee (JavaScript/TypeScript and Python). Use this skill whenever the user wants to: scrape a website, build a web crawler, extract data from web pages, automate browser navigation, handle anti-bot blocking, manage proxies or sessions for scraping, use Playwright/Puppeteer/Cheerio/BeautifulSoup for web data extraction, crawl sitemaps, download files from URLs, or deploy a scraper to the cloud. Trigger even for loosely related phrases like "get data from a website", "automate browser", "scrape prices", "extract links", "crawl URLs", or "bypass bot detection". Covers CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler, HttpCrawler, JSDOMCrawler (JS), and BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler (Python).

When to use

• scrape a website
• build a web crawler
• extract data from web pages
• automate browser navigation
• handle anti-bot blocking
• manage proxies or sessions for scraping
• crawl sitemaps
• deploy a scraper to the cloud

Skill page

Source