Mastering Firecrawl's Crawl Endpoint: From Single Page Scrape to Full Site Crawl

Most real workloads need more than one page: docs sites, product catalogs, blog archives — and when the goal is thousands of pages of clean markdown for a training set or knowledge base, the /v2/crawl endpoint is the tool. Give it a starting URL and it returns scraped markdown for every page worth keeping. The official guide covers it end to end; this article distills it into three practical dimensions: parameters, delivery modes, and billing.

First, untangle two terms that get used interchangeably. Web scraping extracts content from individual pages at known URLs; web crawling walks a site by following links, discovering pages as it goes, with the emphasis on navigation and URL discovery. A chatbot that answers questions about Stripe’s documentation needs both: crawling to discover and traverse every docs page, scraping to extract content from each one. Firecrawl’s crawl fuses them into a single job — URL analysis (sitemap or page traversal), recursive traversal, per-page extraction, structured compilation. You can call it via REST, the Python/Node SDKs (Go and Rust also exist), the MCP server, or the CLI.

Synchronous Start: Crawl a Site in One Line

Using the practice site books.toscrape.com, the Python SDK needs two lines: crawl(url=base_url, limit=20). The result carries status, total, credits_used, completed, and a data list — each element holding that page’s markdown plus metadata (title, description, url, language, robots). The traditional approach with beautifulsoup4 or lxml means dozens of lines of parsing and pagination logic before any content lands; here it is one line, and the official walkthrough reports roughly 8 seconds for 3 pages (varying with network and target site). The returned element also carries per-page metadata — title, description, url, language, robots — which later becomes the provenance layer for retrieval: keep it, and every chunk in your vector store can cite the page it came from. Synchronous crawl() suits small jobs where waiting is fine.

Core Parameters: limit, Path Filters, and Sitemap Mode

• limit caps the number of pages scraped. Without it, the crawler can follow an endless chain of links and burn credits along the way — most critical on large sites or when external links are enabled. It also directly shapes the bill.
• include_paths / exclude_paths scope the crawl with path patterns — only /docs/, exclude /blog/ — which saves credits and time compared to filtering after the fact.
• crawl_entire_domain traverses subdomains across the whole domain, for large sites with scattered structure.
• sitemap switches URL discovery to sitemap mode — faster and more complete than link traversal, especially for documentation sites.
• scrape_options passes per-page scrape options (formats, wait conditions) down through the crawl, unifying behavior in one job.

The combination strategy is “narrow first, then widen”: validate output on a small slice with limit + include_paths, confirm format and quality, then broaden scope — rather than opening a full-domain crawl on a big site and discovering the credit bill afterwards.

Large Jobs: Three Async Delivery Modes

Synchronous waiting doesn’t scale to big jobs. After start_crawl() launches an async job, three retrieval patterns fit different scenarios: poll with get_crawl_status() — simplest, fine for scripted batches; stream over WebSocket with watcher() — pages arrive as they finish, suiting pipelines that process incrementally; or webhook push, where Firecrawl posts events to your URL, fitting existing event-driven architectures. The choice follows your downstream: polling for batch ingestion, streaming for live indexing or progress reporting, webhooks for enterprise pipelines. One operational note from the guide worth keeping: status fields (status, total, completed, credits_used) exist on both the sync and async surfaces, so cost telemetry does not depend on which mode you pick — you can watch credits_used climb while a big job streams.

The Credit Formula: Designing Cost into Parameters

Billing is clean: 1 credit per crawled page; JSON extraction adds 4 credits per page; PDF parsing adds 1 credit per PDF page. Total cost is roughly pages × (1 + optional extras). That makes limit a first-class cost control, not just a safety rail: estimate the target site’s page count first (the map endpoint lists URLs without scraping content), then decide the limit and whether JSON extraction is on. A few-thousand-page site with JSON extraction costs five times the markdown-only run — a number worth computing before you press go.

Into RAG: The LangChain Handoff

The guide closes by feeding crawl results straight into LangChain’s document loaders — markdown in, chunked, embedded, and landed in a vector store, one pipeline end to end. Practical advice for builders: align include_paths with your knowledge base’s topic boundary, because crawling an entire site often pulls in swaths of pages irrelevant to the Q&A; keep metadata (url, language) so retrieval results can link back to sources; and for recurring refreshes, use sitemap mode to diff old and new URL lists and re-crawl only what changed, keeping update costs proportional to the delta rather than the whole site.

Sources

• Mastering the Crawl endpoint in Firecrawl

AI-assisted summary compiled from the sources above, reviewed by a human before publishing.