Summary: Firecrawl’s crawl endpoint follows links across a site and returns markdown plus metadata for each retained page, with synchronous runs for small jobs and polling, streaming, or webhooks for larger asynchronous jobs. Each crawled page costs one credit, JSON extraction adds four, and PDF parsing adds one per PDF page. The guide recommends starting narrow with limits and path filters before widening the crawl.
Plenty of jobs start bigger than a single page: docs sites, product catalogs, blog archives. If what you need is thousands of pages of clean markdown for a training set or knowledge base, /v2/crawl is built for exactly that — one starting URL in, markdown for every page worth keeping out. The official guide covers all of it; this piece narrows it to three: parameters, delivery modes, billing.
Before touching the API, separate two labels often treated as one. Web scraping pulls content from a single page at a known URL; web crawling follows links across a site, discovering pages along the way, where navigation and URL discovery are the point. A chatbot answering questions about Stripe’s documentation needs both: crawling locates and covers every docs page, scraping pulls out each page’s content. Firecrawl’s crawl wraps the sequence in one job: URL analysis (sitemap or page traversal), recursive traversal, per-page extraction, structured compilation. Entry points: REST, the Python/Node SDKs (Go and Rust also exist), the MCP server, the CLI.
The Synchronous Route: One Line, One Site
On the practice site books.toscrape.com, the Python SDK needs two lines: crawl(url=base_url, limit=20). What comes back carries status, total, credits_used, completed, and a data list whose elements each pair one page’s markdown with its metadata (title, description, url, language, robots). The same job in beautifulsoup4 or lxml means dozens of lines of parsing and pagination logic before any content lands; here a single call does it, and the official walkthrough clocked roughly 8 seconds for 3 pages (the figure moves with network and target site). That per-page metadata doubles as the provenance layer for retrieval: hold onto it, and every chunk in your vector store can cite the page it came from. Synchronous crawl() fits small jobs where waiting is acceptable.
The Parameters That Matter: limit, Path Filters, Sitemap Mode
limitputs a ceiling on pages scraped. Take it away and the crawler can chase link after link endlessly, spending credits throughout — riskiest on large sites or with external links enabled. It decides your invoice, too.include_paths/exclude_pathsdraw the crawl’s boundaries with path patterns — just/docs/, no/blog/— saving credits and time versus filtering after the fetch.crawl_entire_domainlets the crawl roam across subdomains to cover a whole domain — for large sites with structure spread thin.sitemapmoves URL discovery into sitemap mode, which beats link traversal on speed and coverage alike, documentation sites above all.scrape_optionscarries per-page scrape settings (formats, wait conditions) into every page, so one job behaves uniformly.
Combine them “narrow first, then widen”: cut a small slice with limit + include_paths, inspect the output, and expand only after format and quality check out — rather than letting a full-domain crawl loose on a big site and meeting the credit bill afterwards.
When Jobs Get Big: Three Ways to Receive Async Results
Waiting synchronously stops scaling once jobs get large. After start_crawl() kicks off an async job, the output comes back three ways, each matching a situation: poll via get_crawl_status() — least code, fine for scripted batches; stream over WebSocket with watcher() — pages land the moment they finish, for pipelines that process incrementally; or webhook push, where Firecrawl posts events to your URL, for existing event-driven architectures. Match the downstream: polling for batch ingestion, streaming for live indexing or progress reporting, webhooks for enterprise pipelines. One operational note worth keeping: the status fields (status, total, completed, credits_used) appear on both sync and async surfaces, so cost telemetry is independent of mode — during a large streaming job you can watch credits_used climb as it happens.
How Billing Works: Baking Cost Into the Parameters
The pricing is straightforward: each crawled page costs 1 credit; JSON extraction adds 4 credits per page; PDF parsing adds 1 credit per PDF page. In plain terms, total cost ≈ pages × (1 + optional extras). That elevates limit from a safety rail into a first-class cost lever: size up the target site’s page count first (the map endpoint lists URLs without scraping any content), then set the limit and decide whether JSON extraction stays on. Turn JSON extraction loose across a few thousand pages and the bill runs five times a markdown-only pass — do that arithmetic before pressing go.
Feeding RAG: Straight Into LangChain
Finally, the guide pipes crawl results directly into LangChain’s document loaders — markdown arrives, gets chunked, embedded, and landed in a vector store, one pipeline end to end. Three practical pointers for builders: match include_paths to the knowledge base’s topic boundary, since crawling an entire site drags in many pages with nothing to do with the Q&A; retain metadata (url, language) so retrieval results trace back to their sources; and on recurring refreshes, diff the old and new URL lists in sitemap mode and re-crawl only what changed, so update costs track the delta rather than the whole site.