What is Depth-First Crawling?

Depth-first crawling implements the DFS graph algorithm using a LIFO stack. When a page is fetched and links are extracted, those links are pushed on top of the stack — so the next fetch goes deeper, not sideways. The crawler follows one path until it hits a dead end (no new links, or max depth reached), then pops back up the stack to try the next unvisited sibling. The visited set — typically a bloom filter — prevents revisiting URLs already fetched and stops the crawler from looping on cycles in the link graph.

Seed URL goes onto the stack. Worker pops it, fetches the page, pushes all discovered links. Worker pops the top — which is the first link from the page just fetched — and goes deeper. This continues until the stack's top link leads to a page with no new links, triggering a backtrack. The crawler then pops back to the previous frame and tries that frame's next unvisited link. From an observer's perspective, DFS looks like a crawler that disappears deep into one section of a site, completes it fully, then resurfaces and moves to the next section.

DFS's biggest operational risk is getting trapped in a branch that never terminates. This happens on sites with calendar archives (infinite date-based URLs), filtered search pages (every combination of filters generates a new URL), or faceted navigation (size × colour × brand = exponential combinations). A hard D_max cap stops the stack from growing beyond a known depth. URL normalisation (stripping redundant query parameters before the bloom filter check) reduces the effective branch width before it compounds.

We use DFS for archive and thread pipelines where a single entity spans many pages — product review threads, forum discussions, long-form article series. Before any DFS crawl, we manually inspect the target's URL structure and set D_max to data-depth + 2. We run URL normalisation through a configurable rule set per target before bloom filter insertion. Every DFS pipeline has a branch-time alert: if the crawler spends more than N minutes in a single branch, it's flagged for review and the branch is capped.

Given unlimited budget, BFS and DFS visit exactly the same set of pages — they're both complete graph traversals. DFS is not less thorough; it's differently ordered. The practical difference is which pages you see first. DFS reaches deep pages faster; BFS reaches all shallow pages first. "DFS misses things" is usually a symptom of a missing visited set or a premature depth cap, not a property of DFS itself.