What is Link Extraction?

Link extraction is a four-stage pipeline that runs on every successfully fetched page:

• Parse — read the HTML or evaluated DOM and collect raw URL candidates from <a href>, <link rel=canonical>, JSON-LD @id fields, and any other configured sources
• Normalize — strip fragments, sort query params, canonicalize trailing slashes, remove tracking tokens; convert relative URLs to absolute
• Filter — apply scope rules: must match allowed patterns, must not match blocked patterns, must be within the configured host set
• Deduplicate — check against the Bloom filter; URLs already seen or already in the frontier are dropped

What survives gets enqueued into the URL frontier with a priority score. On a typical retail page this reduces 80–150 raw links to 5–20 frontier-worthy URLs.

The crawler's fetch worker returns HTML to the extraction pipeline immediately after a successful response. For static HTML, an lxml or BeautifulSoup parse takes ~5–20ms per page. For JavaScript-rendered pages, the Playwright instance that fetched the page evaluates document.querySelectorAll('a[href]') on the final DOM state — no second browser launch needed since the page is already loaded. Extracted URLs flow into the normalizer synchronously, then into the scope filter, then into the Bloom check. The Bloom check is the bottleneck at scale: at 100M URLs, a well-sized filter still runs in ~1µs per lookup, making it negligible compared to parse time.

The gap between static and rendered extraction is significant on modern retail sites. A product grid page on a React SPA may return raw HTML with zero product hrefs — the links are injected after DOMContentLoaded by JavaScript. In those cases:

• Static extraction yields: navigation links, footer links, and script tags — none of which are product URLs
• Playwright extraction yields: the full rendered product grid, lazy-loaded images with data-href attributes, and pagination controls

The cost is ~2–4 seconds of render overhead per page. For catalog-wide link discovery, that compounds — which is why we route JS-rendered sites through a dedicated Playwright pool and static sites through a lightweight HTTP client pool, never mixing them.

We configure extraction per-target, not globally. Each pipeline specifies: which DOM sources to extract from, which URL patterns to allow and block, and whether to use the static or browser extraction path. Scope filters are compiled to regex automata at pipeline startup — not evaluated per-URL as strings, which would be 100x slower at volume. We track extraction precision as an SLO metric: if the 7-day rolling precision for any pipeline drops below 0.50 (more than half of enqueued URLs producing no useful data), it triggers a filter rule review. Most precision drops trace to new site sections the block-list doesn't cover yet.

One of the most expensive extraction mistakes is failing to strip session tokens from extracted URLs before the Bloom check. A site that appends ?sid=abc123 to every internal link will cause the Bloom filter to treat each session variant as a unique URL — you'll crawl the same product page hundreds of times, once per unique session token, before the dedup layer catches on. The fix is a URL normalization rule that strips known session parameter names before the Bloom check. We maintain a shared blocklist of ~80 common session and tracking parameters that are stripped unconditionally across all pipelines.