Scraperai

ScraperAI is an open source Python tool that uses large language models to simplify web scraping. Instead of the user inspecting pages in developer consoles and hand writing selectors, the tool analyzes a web page, detects the data to collect and generates a reusable scraping configuration that can be saved and shared. Its documented features include serializable and reusable scraper configs, automatic data detection, automatic XPath detection, automatic pagination and page type detection, and HTML minification before content is sent to the model. It supports ChatGPT as well as custom language models, works with Selenium, and allows custom crawlers. Collected data can be output in structured formats such as JSON and CSV. The documentation positions it for scientists, analysts, entrepreneurs and SEO specialists who want to scrape data without code, particularly people unfamiliar with scraping techniques or looking to save time. Installation is through pip, and a command line application runs against a target address, for example a Y Combinator company listing used in the documentation's own example. Example scrapers are published alongside the docs, and help is handled through GitHub issues. The tool is free to install and use. No paid plans are published, though use of a hosted model such as ChatGPT would carry that provider's own charges.