What is HTML Scraping?

HTML scraping is a three-step operation: fetch the HTTP response, parse the HTML into a navigable tree, extract target nodes using selectors or traversal. The fetch returns bytes. The parser turns bytes into a DOM. The extractor queries the DOM for specific values.

The parser is almost never the failure point. Libraries like cheerio, lxml, BeautifulSoup, and html.parser handle malformed markup well. The failure points are: getting a response that contains the actual data (not a bot wall), and keeping selectors aligned as the site's HTML changes.

The server sends raw HTML over the wire. A browser then executes JavaScript, which may rewrite the DOM entirely before the user sees anything. For SPAs built on React, Vue, or Next.js, the raw HTML is often just a loader skeleton — all meaningful content is injected client-side.

Test for this before choosing your approach: fetch the URL with curl and check whether your target data is present. If it is, you don't need a browser. If it isn't, you need a JS-executing environment. Using a headless browser on static HTML wastes 5–10× the resources for no gain.

Selector stability is the primary long-term cost of HTML scraping. Prefer selectors in this order of stability:

• Semantic IDs and data attributes — #productTitle, data-testid="price" — intentional, rarely renamed
• Structural semantic tags — h1, main, article — stable across redesigns
• Class names with semantic intent — .product-price — moderately stable
• Generated class names — .a3B7_x — break on every build deploy
• Positional XPath — //div[3]/span[2] — breaks on any layout change

Always write a fallback selector. The first selector is optimistic; the fallback is what keeps the pipeline alive at 3am.

Every pipeline we operate runs a continuous schema fingerprint check. On each run, extracted field yield is compared against the last 10 runs. A drop triggers a Slack alert to the extraction team within minutes — not hours.

We maintain primary and fallback selector pairs for every field, auto-tested on every deploy. When a site ships a breaking change, we're usually patching before the client's dashboard shows a gap. Mean time to restore yield after a structural change across our active pipelines: under 4 hours.

They validate the scraper, not the output. A scraper that returns something for every record looks healthy in monitoring. A scraper that silently returns stale, wrong, or incomplete data because a selector drifted looks identical.

The correct monitoring target is field yield — the fraction of expected fields populated per record — not request success rate. A 200 OK with a bot-wall HTML body and a 200 OK with real product data are indistinguishable at the HTTP level. You have to validate what came back.