Summary: Firecrawl’s /scrape endpoint renders a URL in a server-side browser and returns selected formats such as markdown, HTML, screenshots, links, summaries, and JSON. Base scraping costs one credit, JSON extraction raises the total to five, and PDF parsing adds one credit per page. The guide recommends prompt-only extraction for exploration and schema-locked extraction for production.
The Headline: Scraping Is Now a Single API Call
Pulling usable data off the web keeps getting harder. Pages assemble themselves through JavaScript, automated visitors get throttled, and the same markup shifts from region to region. Even once the HTML lands, it seldom has the shape you need—so most projects staple a cleanup pipeline behind their fetcher to arrive at clean text, a structured record, a transcript, or a screenshot.
The /scrape endpoint from Firecrawl folds that whole chain into a single API call. Hand it a URL along with the formats you want, and it renders the page in a real browser on its own servers—sorting out JavaScript, proxies, redirects, and caching—then returns the result in the shape you asked for. This piece walks the endpoint from end to end: its formats, its parameters, what it costs, structured extraction, batch runs, and pages that demand interaction.
Under the Hood: Cost, Caching, and Output Formats
A scrape starts from a base of 1 credit. A handful of options push that number up:
| Capability | Credit Cost |
|---|---|
| Plain scrape (any one format) | 1 |
JSON extraction (formats: [{type: "json"}]) |
+4 (5 overall) |
Enhanced proxy (proxy: "enhanced") |
still 1 credit — nothing extra |
PDF parsing (parsers: ["pdf"]) |
+1 for each PDF page |
Audio extraction (formats: ["audio"], REST only) |
+4 (5 overall) |
| Cached result | 1 (caching cuts the wait, never the credits) |
As the official documentation points out, caching trims latency while leaving the bill untouched. The max_age parameter sets the cache window in milliseconds—0 forces a fresh fetch, while 86400000 buys a one-day window. A cache_state field in the response reports whether you got a hit or a miss.
Asking for Several Formats at Once
Multiple formats can ride along in one request. The Python snippet below pulls markdown, HTML, raw HTML, links, screenshot, summary, and images from arXiv.org:
from firecrawl import Firecrawl
from dotenv import load_dotenv
load_dotenv()
app = Firecrawl()
url = "https://arxiv.org"
data = app.scrape(
url,
formats=['html', 'rawHtml', 'links', 'screenshot', 'summary', 'images']
)Each format fills its own niche:
markdown: The page’s main content as tidy Markdown—a natural fit for LLM pipelines.html: HTML after cleaning, with scripts and styles stripped out.rawHtml: The server’s HTML, exactly as delivered.links: Every hyperlink on the page, listed flat.screenshot: A signed PNG URL of the rendered page (fullPagesupported).summary: A 1–3 paragraph summary generated by AI.images: Every image URL found.json: Structured data extracted by an LLM (more on this below).audio: An MP3 pulled from supported video URLs (REST only).
Structured Extraction: Prompt or Schema?
With the json format, an LLM reads the page and hands back a structured dict. For product builders chasing specific fields—product names, prices, reviews—this is where /scrape earns its keep, no custom parsers required.
Letting the Prompt Decide
Supply a natural language prompt and let the LLM pick the output shape. Writing one takes seconds, though field names may wander between runs. Here it is pulling the top stories from Hacker News:
url = "https://news.ycombinator.com"
data = app.scrape(
url,
formats=[
"markdown",
{
"type": "json",
"prompt": "Extract the top 3 stories on the page. For each story, include its title, the URL it links to, the points score, and the author who submitted it."
}
]
)
print(data.json)Expect output along these lines:
{
"stories": [
{
"title": "BYOMesh – New LoRa mesh radio offers 100x the bandwidth",
"url": "https://example.com/story1",
"points": 209,
"author": "nullagent"
}
]
}Schema-Locked Mode
In production, pass a Pydantic model instead, and field names plus types are locked in—every run then returns exactly the same shape. Nested schemas get demonstrated on the Firecrawl blog:
from pydantic import BaseModel, Field
class IndividualArticle(BaseModel):
title: str = Field(description="The title of the news article")
subtitle: str = Field(description="The subtitle")
url: str = Field(description="The URL")
author: str = Field(description="The author")
date: str = Field(description="The publication date")
read_duration: int = Field(description="Estimated reading time in minutes")
topics: list[str] = Field(description="List of topics")
class NewsArticlesSchema(BaseModel):
news_articles: list[IndividualArticle] = Field(description="Extracted articles")
data = app.scrape(
url,
formats=[
{
"type": "json",
"schema": NewsArticlesSchema,
"prompt": "Extract the top 5 stories as news articles."
}
]
)
print(data.json)Either mode runs 5 credits (1 base + 4 for JSON). Reach for prompt-only while exploring; switch to schema the moment downstream code depends on field names.
Where It Pays Off: Batches, Actions, and Niche Formats
Running Scrapes in Bulk
URL lists in the hundreds or thousands call for batch_scrape() (sync) or start_batch_scrape() (async): submit the URLs in bulk, then poll or attach a callback to collect results. SEO audits, price monitoring, and content aggregation are natural fits.
Pages That Need Actions
Some pages hold their content back until a visitor interacts. The actions parameter takes a sequence of steps—click, type, wait, screenshot—that Firecrawl performs on the rendered page before scraping. The official tutorial walks through a login flow: open a test site, enter credentials, click login, capture the post-login screenshot.
Niche Formats
- Audio: Pull an MP3 out of YouTube links (REST only, +4 credits).
- PDF Parsing: Pass
parsers=["pdf"]to parse PDF pages; each page adds +1 credit. - Screenshot: Set
fullPage: trueto capture an entire page. - Branding: Read out a page’s brand colors, fonts, and style.
- Question and Highlights: Ask specific questions or get back the relevant passages—token consumption can drop by up to 100x.
Country Targeting and Proxy Choice
The location parameter pins a request to a specific country (for instance, "country": "DE" for German results). proxy comes in three modes: basic (the default and cheapest), enhanced (harder to block), and auto (try basic first, escalate to enhanced only if needed). The metadata’s proxy_used field shows which backend actually ran.
Limits Worth Knowing
Every tool has edges. The trade-offs that matter:
- Cached responses still bill: A cache hit costs the full 1 credit—latency drops, spend doesn’t.
max_agelets you weigh freshness against speed, but every hit gets charged. - Prompt drift: With no schema in place, field names can move between runs. Define a Pydantic model for any production extraction.
- Audio is REST-only: SDKs can’t request the audio format; calling the REST API directly is the only route.
- Interactive complexity:
actionscover simple flows (click, type, wait), but longer multi-step journeys—say, a sign-in guarded by CAPTCHA—may need session handling you build outside Firecrawl. - Lockdown Mode: Security-sensitive environments can force cache-only scraping with
lockdown=True. An uncached URL errors out instead of triggering a live fetch—useful when outbound traffic is tightly restricted. - PDF page costs: Every PDF page takes one more credit, so extracting a 50-page PDF runs 51 credits (1 base + 50 pages).
The Takeaway: Where to Start
Brand new to Firecrawl? The gentlest on-ramp is the REST API through cURL:
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'Inside application code, take the Python or Node SDK as shown above. If your work happens in an MCP-compatible client such as Claude Desktop or Cursor, install the Firecrawl MCP server and the model can call firecrawl_scrape directly—no code to write.
The decisions product builders face:
- One-off exploration → prompt-only JSON over cURL.
- Production data pipeline → schema-driven extraction through an SDK.
- High volume → batch scraping driven by
start_batch_scrape(). - Interactive pages → lean on the
actionsparameter. - Cost-sensitive → skip JSON extraction when it isn’t needed (it runs 5x the base cost) and let caching lower latency, even though credits are still spent.
Firecrawl’s /scrape endpoint is no silver bullet—highly complex flows and strict compliance demands will still need custom logic—but for the bulk of “give me clean data from this URL” jobs, the barrier has dropped a long way. Start with a plain scrape, add formats as they become useful, and move up to schemas once your pipeline needs consistency.