Crawl4AI review

Crawl4AI is both an open-source Python crawler and a hosted scraping API. In a self-managed deployment, the Python library runs browser-based crawling in your process; the vendor also documents a Docker server and a command-line interface. Crawl4AI Cloud runs the browser and exposes scraping and data services over HTTP.

The crawler converts pages into Markdown for RAG, AI agents and data pipelines. Its extraction options include CSS, XPath and LLM-based approaches, and the cloud service adds web search, Google SERP data and answers with sources. The cloud endpoints include /scrape, /extract, /search, /serp and /answer, plus batch and job workflows for scraping multiple URLs. MCP is also available for connecting an agent.

In a stack, Crawl4AI sits between the web and the system that consumes page content: an ingestion pipeline, search index or agent. Choose the library or Docker route when you want to operate the browser infrastructure; choose Cloud when you want the vendor to run it.

For self-hosting, the documented integration starts with pip install -U crawl4ai, followed by crawl4ai-setup to install the browser. The Python example uses AsyncWebCrawler and its arun() method. The docs also cover a CLI, Docker server, multi-URL crawling and API reference topics such as arun_many() and CrawlResult.

The cloud example uses HTTP requests with a bearer token and JSON body: curl posts a URL to /scrape, then reads the Markdown response. The same key works with /extract, /search, /serp, /answer, /scrape/batch and /scrape/jobs. MCP uses HTTP transport at /mcp; the homepage gives setup commands for Claude Code and names Codex, Cursor and OpenCode. Confirm SDK support for your preferred language in the vendor's docs; the documented surfaces here are Python, HTTP and MCP.

The docs cover browser configuration, Markdown generation, extraction strategies, hooks, proxies, sessions and crawling strategies. That gives developers several ways to tune collection, but the range of options means teams should test their target sites and chosen extraction path.

The self-hosted library is free under Apache 2.0 with unmetered use; you pay for infrastructure, proxies and LLM costs. Crawl4AI Cloud has a free tier with 1,000 credits monthly, 10 requests per minute and two concurrent requests. The pricing information also lists a separate 20-credit trial with no signup.

Paid monthly plans start with Starter at $10 for 10,000 credits, 120 requests per minute and 10 concurrent requests. Builder is $99 monthly for 125,000 credits, 600 requests per minute and 20 concurrent requests. Scale is $499 monthly for 750,000 credits, 1,500 requests per minute and 50 concurrent requests. Annual billing lowers the effective monthly prices to $8.33, $82.50 and $415.83, respectively; the Starter annual charge is $100.

Overages cost $1 per 1,000 credits on Starter, $0.79 on Builder and $0.67 on Scale. Enterprise has custom volume, regions and SLA, with pricing by sales contact. Because credit consumption by endpoint is not stated in the pricing information, ask for the credit schedule before estimating production spend.

The Crawl4AI library is available under the Apache 2.0 license, and the vendor describes it as free forever. Developers can install it with pip, run its browser locally, or use the documented Docker server. Self-hosting removes the cloud credit meter, but infrastructure, proxy and LLM costs remain yours.

The hosted API returns content as Markdown or structured data, which can be stored in your own pipeline rather than tied to an agent interface. The vendor doesn't publish a cloud data-export guarantee, so ask how to retrieve stored job results and what retention rules apply before depending on cloud workflows.

Migration effort depends on how much of your application is built around Crawl4AI-specific extraction settings, job handling or MCP integration. Keeping a boundary around the crawler and storing its outputs in your own systems should make replacing the collection layer more manageable.

Cloud capacity is bounded by plan-level request rates and concurrent requests: the free tier allows 10 requests per minute and two at a time, while Scale lists 1,500 per minute and 50 concurrent requests. For larger or region-specific workloads, Enterprise offers custom volume and regions. The vendor lists a custom SLA for Enterprise; confirm its terms and any uptime commitment before making the service a dependency.

The homepage says Cloud handles JavaScript-heavy pages and bot walls automatically. The self-hosted route leaves browser settings and proxies to your team. The feature set includes hooks, proxy and security controls, stealth modes and session reuse, but configure and validate these against your target sites. The vendor's MCP integration uses a bearer key, so protect credentials and check your own data-handling requirements.

Reddit discussion is limited and mixed in scope: one r/AI_Agents thread raises concerns about complex browser-agent tasks, while an r/n8n post shares a SERP-analysis workflow involving Crawl4AI. Neither establishes production reliability. Test representative sites, monitor credit use and verify retry behavior before scaling.