Crawlee

Crawlee is an open source web scraping and browser automation library for JavaScript and Python, built and maintained by Apify. It handles the parts of crawling that usually break scrapers, including blocking, request queues, proxies and headless browsers, so developers can focus on the extraction logic for each page. It is aimed at developers and data teams who write their own crawlers, whether for collecting data or feeding web content into AI and retrieval pipelines. Crawlers can drive headless browsers through Playwright, enqueue links automatically as they discover them, and push results into datasets stored as JSON that can be exported to a single CSV file or read directly in code. A CLI scaffolds a new project from templates with npx crawlee create for JavaScript or the equivalent uvx command for Python, and code written with Crawlee can be run on the Apify platform when you want hosted execution. Crawlee itself is free and open source under the Apache 2.0 license, described by the maintainers as forever free. Running crawlers on Apify's cloud is billed separately under Apify's own pricing, which is not published on the Crawlee site. The project is candid that it will not repair broken selectors for you; site changes still need manual updates to your extraction code.