Scrapy review

Scrapy is an open-source Python framework for crawling websites and extracting structured data. It runs as code in your project, rather than as a hosted API or a managed scraping service. Its asynchronous engine coordinates six pluggable parts: scheduler, downloader, spider, items, pipelines, and feed exports.

You write spiders to handle responses, select fields with CSS or XPath, and follow links to discover more pages. Pipelines can validate, clean, and store items; feed exports send structured results to JSON, CSV, S3, and other destinations. The Scrapy command-line interface scaffolds projects and runs an interactive shell for testing selectors against pages.

Scrapy sits in the collection layer of a data stack: it fetches pages and produces records for downstream storage or processing. You can run the framework on your own infrastructure or deploy crawls to the separate Scrapy Cloud service. Browser rendering, monitoring, and other crawl tasks are added through extensions.

Scrapy is a Python framework, not a hosted API with a client SDK. Its main integration surface is Python code: spiders receive responses, use CSS or XPath selectors, yield items, and follow links. The homepage provides code examples and links to the documentation for spiders, the architecture, and the interactive shell.

The CLI scaffolds a project with scrapy startproject, opens a selector-testing shell with scrapy shell, and supports running crawls. The extension model adds integrations without replacing the core workflow: scrapy-playwright renders JavaScript-heavy pages in a real browser, while spidermon monitors data and can send alerts to Slack, Discord, or email.

For agent workflows, the Scrapy MCP Server connects through MCP. The homepage documents installing it with uvx --from scrapy-mcp-official scrapy-mcp; it can inspect jobs, check status, and execute Python inside a crawl. Confirm current compatibility and setup details in the relevant extension and Scrapy docs.

The Scrapy framework costs $0 and is licensed under BSD-3-Clause. The pricing information lists no seat or usage limits for the open-source option. You pay to run your own infrastructure, but no infrastructure rates are published in Scrapy's pricing information.

Scrapy Cloud Free costs $0 per month and includes one low-resource unit, jobs of up to one hour, data retention of up to seven days, and one concurrent crawl. Scrapy Cloud Units are listed at $9 per unit per month. Each unit includes 1 GB of RAM and one concurrent crawl; paid jobs have unlimited run time and data retention of up to 120 days.

The unit model ties hosted capacity to the number of units purchased: additional units add RAM and concurrent-crawl capacity at the listed per-unit price. The pricing information lists no add-ons. It gives monthly prices only; annual billing rates and overage charges are not published. Check the current Cloud terms before estimating costs for production workloads.

Scrapy is open source under the BSD-3-Clause license, and the vendor links to its GitHub project and release history. You can run the framework on your own machines or infrastructure; Scrapy Cloud is an optional hosted deployment service, not a requirement for using Scrapy.

The framework exports structured data to JSON, CSV, S3, and other feeds, which gives teams routes into their own storage and downstream systems. Scrapy spiders and pipelines are code in your project, so moving the crawler itself does not depend on keeping a Scrapy Cloud account. Hosted deployment, scheduling, retained job data, and Cloud-specific operations may need replacements if you leave the service. The practical migration effort depends on how much your workflow relies on those hosted features and extensions.

Scrapy's asynchronous engine and pluggable downloader are built to coordinate concurrent requests. The vendor describes it as a framework used for public web data collection at scale, but that does not replace workload-specific checks for target-site behavior, crawl politeness, or operational monitoring. Scrapy Cloud Free limits jobs to one hour and one concurrent crawl; paid units include one concurrent crawl each and unlimited job run time.

Browser rendering, monitoring, and anti-ban functions are extensions rather than defaults in the core. Before relying on them, check maintenance, compatibility, and any separate service costs. The vendor does not publish reliability guarantees in the pricing information. Teams should also review access controls for live crawl inspection, data retention, and applicable security and compliance requirements.

Reddit users report mixed results: one thread describes Scrapy as fast on static sites but says dynamic content takes extra work. Another user describes a 403 error when crawling a site. These reports are anecdotes, not guarantees about how a particular target will behave.