AI models know a lot, but they don’t know what a competitor’s pricing page said last week, and they can’t read pages that need JavaScript, a login, or pagination. Firecrawl’s May 18 post, Firecrawl 101, names this the web context gap. The answer it proposes is not a stronger crawler but a family of six endpoints positioned as a context API for the web: Search, Scrape, Parse, Crawl, Map, and Interact. Each works independently and chains with the others, and the post is explicit that most production workflows combine only two or three of them. That sentence is itself the selection advice.
The gap comes from static snapshots meeting a live web
The root cause is not model capability but the shape of training data. AI models learn from static snapshots of the internet: they don’t see last week’s changes, can’t read pages they’ve never encountered, and can’t fill in forms or log in. A standard SERP API doesn’t fix this — it returns a list of links, and the agent still has to fetch every page itself and deal with JavaScript rendering. Firecrawl’s bet is to turn “getting web content an LLM can consume” into a service: rendering, cleaning, and structuring all happen inside the API, and the output is a format the model can reason over and cite.
Search and Scrape: the common two-step, in one call
/search differs from a typical search API in one design decision: the same call returns not just URLs but the cleaned full content of each page, such as Markdown. The wiring between search and fetch simply disappears, which is an obvious saving for research-style tasks.
/scrape covers the half where you already know the URL. It takes a single address and can return clean Markdown, structured JSON extracted with a prompt, a screenshot, or any combination. The docs’ example shows the point: against a pricing page, a prompt like “extract every plan name and monthly price” returns something like {name: "Starter", price: "$19/mo"} — a structure you can write straight to a database. JavaScript rendering is handled automatically, so the target site’s tech stack stops being your problem.
Crawl, Map, and Parse: site breadth and non-HTML documents
When a task scales from one page to a whole site, /crawl follows links across the entire site and returns the content of every page — the right tool for building a knowledge base or anything that needs full coverage. /map is its lightweight sibling: it discovers all the URLs without fetching content, useful for understanding a site’s structure and estimating scope before committing to a full crawl. Used together, they keep cost control in your hands.
/parse targets a different kind of dirty data: PDFs and Word documents. Multi-column layouts, mid-sentence footnotes, and scanned pages are the classic killers when an LLM reads documents; Firecrawl converts them into the same output as Scrape — clean Markdown or structured JSON — so document content and web content flow through one pipeline.
Interact: data that only appears after an action
Some data isn’t read, it’s operated into existence: products behind a Load more button, results after a search-box filter, information that only shows up once a multi-step form is completed. /interact keeps a live browser session open so the agent can click, type, and scroll to reach data that doesn’t exist before the action.
There are two levels of control. Natural language covers most tasks: in the CLI, firecrawl interact "Click the Load more button..." is enough; switch to the SDK when you need fine-grained control. Sessions can also be reused from Scrape: take result.metadata.scrape_id to get the session, chain any number of interactions with app.interact(scrape_id, prompt=...), and call stop_interaction when done. The one hard limit is that a session lasts at most 10 minutes — enough for most read-oriented tasks, but stuffing a long workflow into a single session is the wrong design direction.
It is also the endpoint that demands the most deliberate safety design. Entering credentials, submitting forms, or performing actions that change external state should sit behind permission boundaries and human confirmation; just because an agent can click something doesn’t mean every action should be automated.
Two production patterns you can copy directly
The post’s two patterns are close to copy-paste ready. The first is a research agent: use /search to find sources, /scrape to pull the full Markdown of the top results, then hand the corpus to your LLM for synthesis — a fit for competitive monitoring and market research. The second is an enrichment pipeline: take a list of company URLs, use batch_scrape with a json prompt to extract fields like company name and founding year, and write them into a CRM — zero custom code per site, which is its most attractive property. The split between them reflects a trade-off: research fetches fresh data on every task and optimizes for freshness; enrichment runs a list once and optimizes for batch economics.
Start with the smallest composition
The entry cost is deliberately low: signing up at firecrawl.dev comes with 1,000 free credits, pip install firecrawl-py or npm install -g firecrawl-cli both get you started, and npx -y firecrawl-cli@latest init --all --browser registers the CLI as an agent skill. If you already work inside Claude, Cursor, or Windsurf, Firecrawl also ships an MCP server that exposes Search, Scrape, Crawl, and Map directly to the host, with the remote URL https://mcp.firecrawl.dev/{FIRECRAWL_API_KEY}/v2/mcp.
The selection principle matters more than the install commands: start from the minimal endpoint set the task needs. Price monitoring may need nothing beyond Search plus Scrape; documentation Q&A is Map plus Crawl plus Parse; reach for Interact only when login or genuine interaction is required. Before scaling, measure three numbers: source success rate, data freshness, and average pages consumed per task. The value of web context is not scraping as much as possible — it’s handing the agent sufficiently clean, sufficiently fresh evidence at a predictable cost.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.