Summary: Firecrawl 101, published May 18, frames the web context gap and offers six endpoints: Search, Scrape, Parse, Crawl, Map, and Interact. The post says most production workflows combine only two or three, so teams should start from the minimum combination a task requires. Interact can operate a live browser session for up to 10 minutes and needs explicit permission boundaries.
However much an AI model has absorbed, it still can’t tell you what a competitor did to its pricing page last week, and it still can’t open pages that demand JavaScript, a login, or endless pagination. On May 18, Firecrawl published Firecrawl 101 and gave the problem a name: the web context gap. The proposed fix is not a brawnier crawler; it is six endpoints that together amount to a context API for the web — Search, Scrape, Parse, Crawl, Map, and Interact. The six can go it alone or be wired into a chain, and the post is upfront that production workflows, most of them, combine just two or three of the six. That one sentence is, by itself, the selection advice.
The root cause: frozen snapshots, a web that keeps moving
Dig past model capability and the cause is the shape of the training data. What a model knows of the internet is a static snapshot frozen at some moment: last week’s changes are unknown to it, unseen pages stay unreadable, and filling in forms or logging in is out of reach. The everyday SERP API doesn’t help — it hands back a list of links, and fetching every page plus taming JavaScript rendering still lands on the agent. Firecrawl’s angle is to wrap the whole job of “getting web content an LLM can consume” up as a service — rendering, cleaning, and structuring happen inside the API, and what comes back is a format models can reason with and cite.
The familiar two steps, Search then Scrape, collapsed into one call
One design decision separates /search from a typical search API: a single call brings back more than URLs — each page’s cleaned full content, Markdown for instance, comes back with it. The wiring between searching and fetching vanishes — a clear saving on research-style work.
/scrape takes the other half: the URL you already know. Give it a single address and it can hand back clean Markdown, a screenshot, structured JSON extracted via a prompt — or any mix of the three. The docs’ example captures the use case well: feed a pricing page a prompt such as “extract every plan name and monthly price” and back comes something like {name: "Starter", price: "$19/mo"} — data shaped for a straight write into a database. JavaScript rendering gets handled by the service, so whatever tech stack the target site runs is no longer the user’s problem.
Site-wide reach and non-HTML documents: Crawl, Map, and Parse
Once a task grows from one page to an entire site, /crawl is what you reach for: it walks the links across the whole site and brings back every page’s content — exactly what putting a knowledge base together, or any demand for complete coverage, calls for. /map is the lighter take on the same idea: it explores every URL without fetching content, the sensible first move for reading how a site is laid out and sizing up scope before you commit to a full crawl. Pair the two and the cost lever stays in your own hands.
/parse takes on a different mess: PDFs and Word files. Multi-column layouts, footnotes landing mid-sentence, scanned pages — these are what usually kills an LLM’s document reading. Firecrawl’s answer is to convert such files into the same output Scrape produces, clean Markdown or structured JSON, so documents and web pages move through one pipeline with identical treatment.
Interact: the data you get only by operating
Some data can’t be read into view — it has to be operated into view: the filtered results of a search box, the products hiding behind a Load more button, details that appear only after a multi-step form is finished. /interact runs a live browser session within which the agent can type, click, and scroll its way to data that simply isn’t there until the action happens.
Control comes in two layers. For most tasks natural language is enough: in the CLI, one line like firecrawl interact "Click the Load more button..." settles it; move up to the SDK when fine-grained control is needed. A session can also be carried over from Scrape: grab result.metadata.scrape_id to claim the session, then run app.interact(scrape_id, prompt=...) as many times as the job demands, finishing with a call to stop_interaction. The only hard limit: a session lives for at most 10 minutes — sufficient for the typical read-oriented job, but forcing a long workflow through one session means the design has gone the wrong way.
Of the six, this is also the endpoint that most needs deliberate safety design. Typing in credentials, sending forms, or carrying out any action that might change external state belongs behind permission boundaries and needs a human’s sign-off; being able to click is no reason to automate every action.
Two production patterns ready to run as-is
The post’s two patterns can nearly be transplanted wholesale. One is a research agent: /search locates sources, /scrape pulls complete Markdown for the leading hits, and your LLM does the synthesizing — suited to competitive monitoring as well as market research. The other is an enrichment pipeline: start from a list of company URLs, run batch_scrape together with a json prompt that pulls out fields like the company name or founding year, and land them in a CRM — zero per-site custom code, its most appealing trait. The split carries a trade-off on each side: research refetches on every task and lets freshness rule; enrichment clears a list once into the database and lets batch economics rule.
Getting in, and choosing well: begin from the smallest combination
Getting in barely costs anything by design: an account created at firecrawl.dev arrives bundled with 1,000 free credits, npm install -g firecrawl-cli or pip install firecrawl-py either one starts you off, and npx -y firecrawl-cli@latest init --all --browser sets the CLI up as an agent skill. If your daily work already lives inside Claude, Cursor, or Windsurf — Firecrawl additionally offers an MCP server that puts Search, Scrape, Crawl, and Map in front of the host directly; the remote URL is https://mcp.firecrawl.dev/{FIRECRAWL_API_KEY}/v2/mcp.
But the selection principle outweighs the install commands: begin from the minimal endpoint combination the task actually calls for. Price monitoring may get away with Search plus Scrape alone; for documentation Q&A, the trio is Map, Crawl, and Parse; Interact stays on the bench until a real login or actual interaction is called for. Before going live, put three numbers on record: how often source fetches succeed, how fresh the data stays, and how many pages an average task burns. Web context earns its keep not by scraping more — it hands the agent evidence that is clean enough and fresh enough, at a cost you can predict.