Crawlee review

Crawlee is a library, not a hosted scraping API or SaaS crawler service. It sits in your application code: your crawler defines what to request, how to process pages and where to put results, while Crawlee provides tools for crawling, browser use, proxies and handling blocks. The homepage presents JavaScript and Python versions.

The JavaScript example uses PlaywrightCrawler, with a requestHandler that reads a page title, queues links, logs information and saves data. It starts a crawl with crawler.run(), then exports a dataset to CSV or reads the items directly. The example writes data under a local storage directory. A CLI can generate a project from a template; Crawlee itself does not provide the machines or proxy infrastructure. Developers on Reddit describe the library as usable standalone, with Apify as an optional ecosystem for running crawls and accessing related services.

Crawlee exposes crawler libraries for JavaScript and Python, rather than a remote REST API. The homepage’s JavaScript example configures a PlaywrightCrawler with a requestHandler, then uses crawler.run() to start work. Within the handler, it shows enqueueLinks() for adding pages to the queue and pushData() for saving results. The example also shows exportData() for a CSV file and getData() for accessing items directly.

The CLI starts a project from a template: JavaScript uses npx crawlee create my-crawler; Python uses uvx 'crawlee[cli]' create my-crawler. The browser example is built around Playwright. The homepage also warns that Crawlee will not fix broken selectors, so page-specific extraction remains your responsibility. The homepage links to getting-started paths for JavaScript and Python; check the vendor’s docs for current language-specific APIs, installation details and integrations before choosing a version.

The pricing information lists one plan: Free / Open-Source at $0 per month, described as forever free and licensed under Apache 2.0. It includes JavaScript and Python and has no published usage limits. There are no paid plans, annual prices, add-ons or published overage charges in the listed pricing. That makes the library’s price predictable, but it does not mean a production crawler has no operating cost.

Crawlee runs as code in an environment you provide. Your compute, storage and network bills depend on where and how you run it; proxy costs depend on the provider and usage. One Reddit commenter notes that proxies and servers require infrastructure and can cost money, whether managed yourself or through a service such as Apify. The library pricing does not give a per-request or per-seat rate to model those costs. Budget runtime and proxy spend separately, and confirm provider pricing and any service limits before putting a high-volume crawler into production.

Crawlee is described as forever free and open source under the Apache 2.0 licence. Because it is a library, developers can run crawler code in their own environment rather than needing a hosted Crawlee service. A Reddit discussion also describes the library as standalone, with Apify’s platform as an optional ecosystem rather than a requirement.

The homepage example saves results to a local dataset directory and shows exporting the dataset to a CSV file. That gives teams a straightforward route to move extracted records into other tools. Crawlee-specific crawler code is a different matter: request handling, queues and browser setup may need rewriting or adapting if you switch libraries. The published material does not establish a vendor-controlled data format or an export from a hosted service. Keep crawl logic and output schemas separate if portability matters, and review the licence and current repository terms before redistributing modified code.

The pricing listing gives no published usage limits, and the vendor's site names no rate limits or production capacity figures. That is not a guarantee of unlimited throughput: your crawler still depends on the target site, your runtime and any proxy service. The vendor’s pricing does not list hosted uptime, security or compliance commitments, so teams with those requirements should ask the vendor and their infrastructure providers before depending on them.

Reddit users report mixed results on blocking. A Crawlee for Python developer says the library can detect blocking pages and rotate sessions or proxies automatically through retry_on_blocked. Another commenter says some Cloudflare configurations still block new sessions because of Playwright detection. These comments concern the Python version and are not a reliability guarantee for every target. Test your own targets and failure paths, and plan for retries, blocked requests and selector changes. The homepage itself notes that Crawlee will not fix broken selectors.