Diffbot review

Diffbot is a hosted web-data API suite, not a scraping library or command-line tool. Its APIs extract structured JSON from a URL, crawl a site from a seed URL, process natural language, and query a Knowledge Graph covering public-web entities such as companies, people and news. That puts it between public websites and the application or data pipeline that needs normalized web information.

The crawl API is aimed at collecting many pages into one structured dataset; extraction handles individual pages. The natural-language API resolves entities, relationships, facts and sentiment from text. For search, Diffbot offers API access and describes a self-hostable 4TB web index for local and offline use. That local index is a distinct deployment option; the homepage does not say that the extraction, crawl or Knowledge Graph APIs can run locally. Treat Diffbot as a managed data dependency unless the search index specifically fits your architecture.

The integration surface is Diffbot’s APIs: call them to extract a page, crawl from a seed URL, process text or query the Knowledge Graph. The homepage describes structured JSON extraction and says its demo agent makes real API calls that can be inspected. It does not name a wire protocol, programming language, SDK, CLI, webhook or plugin. Confirm supported SDKs, authentication, request formats, pagination and error behavior in Diffbot’s developer docs before choosing a stack.

A typical integration puts Diffbot between a URL or text source and your application’s storage or search layer. The crawl API can collect pages into a dataset, but the homepage does not describe scheduling, incremental recrawls or delivery into a specific database. Those details matter if you need repeatable ingestion rather than one-off extraction. For web search, choose between API access and the vendor’s self-hosted index; confirm deployment requirements and update procedures for the local option before planning operations around it.

The Free plan costs $0 per month and includes 10,000 credits per month, all four APIs and a 5 requests/minute limit, with no credit card required. That is an evaluation tier, not a high-throughput production allowance. Startup costs $299 per month for 250,000 credits and 5 requests/second. The pricing page lists overage at $0.001 per credit.

Plus costs $899 per month for 1,000,000 credits, 25 active crawls, three user seats and 25 requests/second; overage is $0.0009 per credit. Enterprise has custom credit allotments and seats, 100+ active crawls, dedicated support and 25+ requests/second, with price not published. The vendor also lists free Startup-tier access for students and academic researchers through Diffbot for Students.

Subscriptions are monthly and can be canceled anytime; the FAQ does not list annual billing. Model the bill using your expected credit use and overages, and verify how each API operation consumes credits. The published plan details make the $299 Startup tier the production entry point.

Diffbot’s search offer includes a self-hostable 4TB web index designed for local and offline use. That is a meaningful option if your search workload must run on your hardware, but it does not establish that the extraction, crawling, natural-language or Knowledge Graph APIs are self-hostable. The vendor does not publish a source-code availability or open-source licence on the vendor's site; ask Diffbot for the index’s licence, deployment requirements and update terms before adopting it.

Diffbot describes structured JSON extraction and crawl datasets, but does not publish data-export controls or a migration path in the published product details. Keep your own copies of extracted data and avoid making Diffbot-specific fields your only canonical schema. Moving away would mean replacing API calls and adapting downstream code to another provider’s outputs; Knowledge Graph queries may also depend on Diffbot’s data model. Confirm export formats and retention terms with the vendor.

The published rate limits range from 5 requests/minute on Free to 5 requests/second on Startup and 25 requests/second on Plus. Plus includes 25 active crawls; Enterprise lists 100+ active crawls and 25+ requests/second. Plan capacity around those limits and credit consumption, and ask the vendor how retries, crawl failures and service availability are handled. Diffbot’s homepage claims Web Search API responses under 300ms P90; treat that as a vendor claim, not a service-level guarantee.

A thread on r/selfhosted reports Diffbot crawling a site despite a robots.txt disallow. That is one user’s report, not proof of a general policy, but site owners should verify crawler behavior and use server-side controls where access must be blocked. The thread also includes a reminder that robots.txt is not an access-control mechanism. The published product details do not state security certifications or compliance terms, so check those, data retention and support commitments before sending sensitive URLs or text.