What is Surface Web?

The surface web is the subset of the internet that is:

• publicly accessible — no login, no session token, no form submission required
• indexable — a search engine crawler following links or reading a sitemap can discover and fetch the page
• URL-addressable — the content lives at a stable URL, not generated behind a POST request or JavaScript event

The canonical test: if curl https://example.com/page returns the full target content with a 200, it's surface web. If you need to authenticate, submit a form, or execute JavaScript to materialise the content, it's deep web — even if the URL is technically public.

A surface web pipeline starts with URL discovery: fetching the sitemap index, processing sub-sitemaps, and building a priority queue ordered by last-modified timestamp. The crawler fetches each URL through a residential proxy with a consistent browser fingerprint, extracts the target fields, and writes structured records to the delivery sink. No authentication is involved — the crawler presents itself as any other HTTP client. The main operational concerns are rate compliance (respecting Crawl-delay and anti-bot thresholds) and fingerprint quality (ensuring the TLS and HTTP/2 signatures match a real browser, not a Go HTTP client).

robots.txt is an informal convention (standardised in RFC 9309) that publishers use to communicate crawl permissions and rate limits. The two most operationally significant directives are Disallow (paths to exclude) and Crawl-delay (minimum seconds between requests). Ignoring Disallow paths is a ToS violation on most platforms. Ignoring Crawl-delay is the single fastest way to get an IP range banned — automated rate-limit enforcement is almost always watching the same paths the robots.txt covers. Compliant crawlers read robots.txt on the first request to any new domain and cache it for the duration of the crawl.

Every DataFlirt surface web pipeline reads and caches robots.txt before the first production request. Crawl-delay is used as a floor, not a ceiling — we set our concurrency budget so the aggregate rate across all workers stays below the directive. Disallowed paths are excluded from the URL queue at ingestion time, not filtered after fetching. For targets without a Crawl-delay directive, we run a calibration crawl at increasing rates until we identify the soft-block threshold, then operate at 60% of that ceiling. Pipeline documentation records the robots.txt state at pipeline creation and alerts on any directive changes.

Estimates from the University of California and BrightPlanet put the surface web at 4–10% of total web content by page count. The majority is deep web — not because it's hidden, but because it's dynamically generated on demand (search results, personalised feeds, filtered views) and therefore has no stable URL for a search engine to index. This is why sitemap-based crawling is so valuable: the publisher voluntarily indexes their own deep content for you, surfacing URLs that would otherwise require form submissions to discover.