What is Seed URL?

A seed URL is any URL explicitly provided to the crawler before it has fetched anything. Seeds are:

• Static — maintained in config, not discovered at runtime
• Scoped — typically chosen to cover distinct sections of a target site with minimal overlap
• Versioned — should be treated as infrastructure: committed, reviewed, and updated when site structure changes

Once the crawler starts fetching, seeds are indistinguishable from other frontier entries — the distinction is only at initialization time. What makes seeds important is that they determine reachability: any page that is only reachable through a path that starts from a seed will only be found if the right seed is in the initial set.

At crawl initialization, seeds are loaded directly into the URL frontier — bypassing the deduplication and priority-scoring stages that incoming discovered URLs go through. They're the first entries in every host's back-queue. Workers start fetching seeds immediately, and the link extractor begins populating the frontier from the first response. On a well-seeded crawl, the frontier grows faster than workers can drain it within the first few minutes — seeds are the spark, not the fuel.

There are two philosophically different seed strategies:

• Sitemap seeding — load the sitemap XML and extract all listed URLs as seeds. Produces a known, bounded frontier immediately. Misses pages the site owner chose not to list. Best for product detail pages.
• Discovery seeding — seed a small number of structural entry points (homepage, category roots) and let link extraction do the work. Broader coverage but slower to converge and more likely to hit crawl traps. Best for news sites or blogs where content is not enumerated in a sitemap.

Most production e-commerce crawls combine both: sitemap seeds for known products, discovery seeds for new category pages the sitemap doesn't yet list.

Every new target goes through a seed audit before the first crawl run. We check for sitemap availability, probe known API endpoints, map category taxonomy, and verify robots.txt scope. Seed configs are stored in YAML, version-controlled in Git, and linked to each pipeline config. We run a seed health check at the start of every scheduled crawl — any seed returning non-200 for two consecutive runs triggers a Slack alert and pauses that seed pending review. This prevents stale seeds from silently degrading crawl coverage.

The most common seed mistake in scraping projects is using only the homepage. It feels intuitive — the homepage links to everything, right? In practice, large retail sites have product pages 5–8 hops from the homepage. With a crawl depth cap of 4 (which most pipelines need to avoid crawl traps), homepage-only seeding will miss entire catalog sections. We've seen clients with homepage-only seeds achieve 30–40% product coverage on the same targets where sitemap seeds yield 90%+ — on identical infrastructure.