Firecrawl
Turn websites into LLM-ready data, with search built in.
Overall score
74.1/100
firecrawl-rest-v1
Measured Oct 5, 2026
Status
Ranked
Complete benchmark. `overall` is a weighted average of the applicable criteria, comparable to other surfaces in the same category and the same rubric version.
Coverage
78% of the rubric
Surfaces
evaluated.
We score a surface, not a product. One product often exposes several, and they are not worth the same.
firecrawl-rest-v1restprimary
78% coverage
74.1/100
Criterion
by criterion.
Measured on firecrawl-rest-v1, rubric v0.2.0. A criterion with no score is not counted as zero: it is excluded and the weights are renormalised.
Offload Valuejudged · weight 15 · Oct 5, 2026
90/100
This is genuine external work the model cannot perform by reasoning: live web search across the open index, geo-targeted/time-filtered retrieval, fetching and rendering pages, converting them to markdown/summary, query-relevant highlighting, and tool-contract discovery. It supplies fresh data the model does not have and would otherwise need many scrape-parse-clean steps to obtain. It falls just short of the top band because it is retrieval and transformation rather than an irreversible real-world side effect or truly proprietary/regulated data; index-based search is also approximable by other providers or raw HTTP fetching plus in-context cleanup, though at substantial token cost.
- https://docs.firecrawl.dev/api-reference/endpoint/search
The search endpoint combines web search with Firecrawl’s scraping capabilities to return full page content for any query.
- https://docs.firecrawl.dev/api-reference/endpoint/search
The search endpoint combines web search with Firecrawl’s scraping capabilities to return full page content for any query.
Interaction Costmeasured · weight 15 · Oct 5, 2026
75/100
Median over 2 successful run(s): 15480 tokens, 2 calls. Ratio to the best in "web-search" (3 surfaces measured, v0.2 scale). Cold, without documentation: 1/1 succeeded.
Error Recoverabilityprobed · weight 15 · Oct 5, 2026
55/100
unknown_endpoint → HTTP 502, clarity 25/100 (field named: no, allowed values: no, RFC 9457: no) · wrong_type → HTTP 400, clarity 75/100 (field named: yes, allowed values: yes, RFC 9457: no) · hallucinated_enum → HTTP 400, clarity 50/100 (field named: no, allowed values: yes, RFC 9457: no) · unknown_field → HTTP 400, clarity 70/100 (field named: yes, allowed values: yes, RFC 9457: no)
- https://api.firecrawl.dev/v1/search
HTTP 403: {"success":false,"error":"Unfortunately, your IP address looks suspicious, so Firecrawl can't be used without an API key from here. Sign up for a free API key at https://firecrawl.dev/k/ffa7pwcbfrt6 f
- https://api.firecrawl.dev/v1/nonexistent_resource_xyz
HTTP 502: Bad Gateway
- https://api.firecrawl.dev/v1/search
HTTP 400: {"success":false,"error":"Invalid request body","details":[{"expected":"number","code":"invalid_type","path":["limit"],"message":"Invalid input: expected number, received string"}]}
- https://api.firecrawl.dev/v1/search
HTTP 400: {"success":false,"error":"Invalid request body","details":[{"code":"invalid_value","values":["markdown","html","rawHtml","links","screenshot","screenshot@fullPage","extract","json","product","menu"],"
Doc Legibilitymeasured · weight 12 · Oct 5, 2026
48/100
text/markup ratio 4.0% (weight 30) · renders without JavaScript: yes (weight 25) · machine-readable spec: none found (weight 25) · llms.txt: https://docs.firecrawl.dev/llms.txt (weight 10) · 15 runnable example(s) (weight 10). Weighted average over 100 points of available sub-criteria.
Auth Frictionmeasured · weight 10 · Oct 5, 2026
95/100
Authorization: Bearer <key>, verified by a successful call. One header, no ceremony.
- https://api.firecrawl.dev/v1/search
HTTP 403: {"success":false,"error":"Unfortunately, your IP address looks suspicious, so Firecrawl can't be used without an API key from here. Sign up for a free API key at https://firecrawl.dev/k/ffa7pwcbfrt6 f
- https://api.firecrawl.dev/v1/nonexistent_resource_xyz
HTTP 502: Bad Gateway
- https://api.firecrawl.dev/v1/search
HTTP 400: {"success":false,"error":"Invalid request body","details":[{"expected":"number","code":"invalid_type","path":["limit"],"message":"Invalid input: expected number, received string"}]}
- https://api.firecrawl.dev/v1/search
HTTP 400: {"success":false,"error":"Invalid request body","details":[{"code":"invalid_value","values":["markdown","html","rawHtml","links","screenshot","screenshot@fullPage","extract","json","product","menu"],"
Safety & Reversibilityprobed · weight 10
—/100
If I retry after a timeout, do I charge the customer twice?
Payload Efficiencyprobed · weight 8
—/100
Can I ask for only the fields I need, or do I have to swallow 200 attributes to extract one?
Task Atomicitymeasured · weight 6 · Oct 5, 2026
75/100
Median over 2 successful run(s): 15480 tokens, 2 calls. Ratio to the best in "web-search" (3 surfaces measured, v0.2 scale). Cold, without documentation: 1/1 succeeded.
Prior Knowledge Coveragemeasured · weight 5 · Oct 5, 2026
100/100
Median over 2 successful run(s): 15480 tokens, 2 calls. Ratio to the best in "web-search" (3 surfaces measured, v0.2 scale). Cold, without documentation: 1/1 succeeded.
Async Compatibilityprobed · weight 2
—/100
If the operation is asynchronous, can I get the result without hosting a web server?
Tool-Native (MCP)binary · weight 2
—/100
Does the vendor publish an official, maintained MCP server?
Runs, verified
never on the agent’s word.
Every row is a real attempt at the category’s canonical task. Success is verified by reading the created resource back.
| Task | Tokens | Calls | Errors | Docs | Result |
|---|---|---|---|---|---|
| find-official-source | 6,691 | 1 | 0 | without | verified |
| find-official-source | 15,491 | 2 | 0 | with | verified |
| find-official-source | 15,469 | 2 | 0 | with | verified |
The same entry,
without the markup.
You publish this API and dispute a score? The methodology is public and every score above points to the evidence that produced it.