Reader

Reader is an open source web scraping engine that converts web pages into clean markdown for language models and AI agents. It is distributed as a TypeScript package and a command line tool, and it exposes two primitives: a scrape call for a list of URLs and a crawl call for whole websites. Under the hood it is built on Ulixee Hero, with TLS fingerprinting and stealth browsing to reach pages that resist simple fetching. It extracts the main content automatically and returns markdown or HTML. A managed browser pool recycles instances, monitors their health and queues work. Crawling uses breadth first link discovery with depth and page limits, and proxy support covers datacenter and residential proxies with rotation strategies. Developers who want output without installing anything can use a free markdown endpoint by prepending the md.reader.dev address to any URL, with no API key required. The primary audience is engineers building retrieval pipelines, research agents and other applications that need web content in a model friendly format. The software is free to install from npm and the source is on GitHub. The site publishes no paid plans or prices.