What is XPath?

XPath is a standard syntax for addressing parts of an XML or HTML document. It treats the document as a tree of nodes (element nodes, attribute nodes, text nodes) and provides a path-like syntax to navigate through them.

An XPath expression consists of steps separated by slashes. Each step defines an axis (direction of traversal), a node test (what kind of node to match), and optional predicates (filters in square brackets). For example, //div[@class='product']/h2 finds all <h2> elements that are direct children of a <div> with the class "product", anywhere in the document.

An absolute XPath starts from the root node (e.g., /html/body/div[2]/ul/li[3]). These are generated by browser "Copy XPath" tools and are universally considered an anti-pattern in scraping. If the site adds a single banner div, the path breaks.

A relative XPath starts with //, which searches the entire document for a matching node, regardless of depth (e.g., //ul[@id='nav']/li[3]). Relative paths are much more resilient because they ignore irrelevant structural changes above the target element.

The true power of XPath lies in predicates and axes. You can filter nodes by their text content using [text()='Exact Match'] or [contains(text(), 'Partial')]. Once you find a stable anchor node, you can use axes to navigate to the data you actually want.

For example, to extract a price next to a label, you might use: //span[text()='Price:']/following-sibling::span[1]. This works even if the container classes change daily, because the semantic relationship between the label and the value remains constant.

We mandate semantic anchoring for all extraction logic. Our engineers never write XPaths that rely on presentation classes (like .text-gray-500) or strict index positions. We target data-* attributes, ARIA roles, or stable text labels.

For high-throughput pipelines, we pre-compile XPath expressions using lxml to eliminate parsing overhead during the extraction loop. When a target site undergoes a major redesign, our schema validation catches the null fields, and our engineers update the XPath definitions in a central registry, deploying the fix to the fleet without pipeline downtime.

The most common beginner mistake with XPath involves HTML tables. Browsers automatically insert a <tbody> element into tables if it is missing from the raw HTML. If you copy an XPath from Chrome DevTools, it will often look like //table/tbody/tr/td.

When your scraper fetches the raw HTML, the <tbody> tag isn't there, and the XPath returns null. The fix is to write paths that account for optional elements, such as //table//tr/td, which uses the descendant axis to skip the missing tbody entirely.