Summary: A July 15 Firecrawl tutorial uses schema-conforming JSON per page and stops pagination only when a page returns an empty array, so completion is verifiable without knowing the total page count. The author applied it to all 61 Hacker News pages and collected just under 1,040 stories. The guide reserves /crawl for whole-site traversal and Interact for interfaces where the URL does not change.
Give a paginated site to an agent and the failure you will meet most often is not a broken parser — it comes back as a tidy batch of the thirty items on page one, capped with a confident “done.” Beneath that failure sit two problems: how results from every page merge into one consistent shape, and how you can tell for certain that more pages are waiting. A July 15 tutorial from Firecrawl settles both with one bare-bones loop: convert every page into structured JSON that conforms to the schema, advance the page number, and quit the moment a page answers with an empty array. Sweeping the whole of Hacker News — all 61 pages — the author gathered just under 1,040 stories. The total page count was never known in advance.
Ending at Page One: the Verifiable Stop Condition That Was Missing
At its core this failure is missing proof that the job is finished. Handed only the natural-language instruction “scrape all the pages,” the agent has no way of knowing that dozens more pages may be queued; the list in front of it has both content and length; nothing visible separates “all finished” from “page one only.” Hacker News shows the trap plainly: each page carries exactly 30 stories, so a single scrape against the same URL hands back the same 30 items every time. Whether to stop is a process design question, not a parsing one — short of a rule a program can verify, page one is where the agent ends.
The rule Firecrawl offers is simple to the point of triviality: whichever page returns zero results is, by definition, the final page. What makes it valuable: the total page count is never needed beforehand — it comes out of the finished loop as an output rather than going in as an input.
Structured JSON Per Page Before Anything Else, Not a Markdown Wall
Only when each page’s output carries a fixed shape can the stop condition compress into a one-line check. Leave out formats and scrape hands you markdown by default — easy on the human eye, yet the page melts into one text wall — comment counts, titles, and scores all run together — leaving no stable field for a program to grab. The more fundamental distinction: Firecrawl’s response is structured JSON, not raw HTML, doing away with the site-specific parser every website would otherwise need — the ground the claim “one loop works on any paginated site” stands on.
The json format has two grades: prompt only, with the model inventing the fields, or a prompt backed by schema, with the shape guaranteed every time. A loop calls for the schema; prompt-only output cannot support a programmatic stop judgment such as “check the array length.”
With the Shape Locked by a Schema, Stopping Becomes a One-Line Test
Under the Node SDK the schema is written in Zod and translated into JSON Schema — the wire format — before the request leaves. For Hacker News that shape amounts to a stories array whose every entry holds title, url, points, comments. What comes back in result.json conforms precisely — field names and types alike, not roughly.
Given that guarantee, collecting the data and deciding when to stop merge into a single loop:
while (true) {
const rows = (await scrapePage(page)).json.stories;
if (rows.length === 0) break;
stories.push(...rows.map((row) => ({ ...row, page })));
page++;
}Before any row is pushed it receives a stamp naming the page it came from. The habit pays for itself in auditability: every record traces back to its source page, exactly what reconciling or deduplicating later calls for. When the loop finishes, stories goes into stories.json, and the task ends as one file.
Running It Against Hacker News: 61 Pages and Just Under 1,040 Stories
Pagination on Hacker News could hardly be simpler: the ?p= URL parameter climbs with each page number, while a fixed 30 stories fill every page. Tie the loop from the last section onto that rule, and the tutorial’s implementation stands complete. In the author’s own testing, the loop covered all 61 pages and brought in just under 1,040 stories; nothing ever set 61 in advance. The loop kept running until some page finally answered with an empty array — 61 is just the last page with content completed before that stop.
What deserves a pause: the page count is discovered, not presumed. A run that glances at the opening pages and concludes “that is probably everything” is gambling; on a paginated site, the outcome of guessing wrong cannot be told apart from a truncated dataset — a whole stretch of records nobody sees. Delegate completion to a rule, and the agent finally holds evidence of “done” that survives a challenge.
Offset Pagination: All That Changes Is the Variable Being Incremented
Page numbers are not the only scheme. Older forums and directories favor offset pagination — a start parameter in the URL advancing one page size at a time: start=0, start=25, start=50. The adaptation is minimal: swap page++ for start += PAGE_SIZE; the stop condition itself — empty results mean stop — stays exactly as it was.
That surfaces the tutorial’s broader lesson: the same loop fits any paginated site; the only things that ever move are the way the URL gets built and the fields the schema declares. The stop condition belongs to every paginated site at once; site differences stay quarantined inside those two parameters, which is why the pattern earns a direct move into production code.
Quotas, Limits, and When Not to Use This Loop
Be clear about the quota rules first. The scrape endpoint works keyless: called through an official client (SDK, CLI, or MCP server), it needs no API key, with a per-IP rate limit as the only brake. crawl, extract, and map do require an API key; a dashboard registration unlocks those endpoints and lifts the limits. This post’s single-page loop can therefore run anonymously; anything site-wide makes registration a prerequisite.
Choosing the correct entrance matters even more. The tutorial divides the labor without ambiguity: when the goal is whole-site traversal, /crawl is the tool; link discovery and traversal ship built-in and are no job for a manual loop. When pagination leaves the URL untouched — click-to-load or infinite scroll — interact is the match: it keeps a real browser session open, and the data surfaces only once you act. The loop in this piece is for list pages where the URL shifts with the page number or with an offset. Three situations, three entrances — stretching this loop across the other two is a design error.