Self-Healing Scrapers in Python: Where the LLM Actually Belongs

A price selector slides one node to the left. The scraper does not crash. It returns the struck-through MRP instead of the sale price. Forty thousand rows land in the warehouse looking perfectly valid. Your pricing team finds it four days later, in a deck. Self-healing starts with a contract that knows what wrong looks like. The model doing the repair costs less than most teams budget for it.

The break that never pages anyone

Every scraper fails in three ways. Exactly one of them is loud.

The loud one throws. The selector matches nothing. The parse step raises, and the run exits non-zero. Alerting catches it inside a single cycle. Mean time to detection is minutes, and the fix is a normal engineering task.

The other two are the expensive ones.

A soft break looks like a shirt priced at 2,499 that actually sells for 1,299. The type is right. The currency is right. Nothing raises.

A semantic break is subtler. The price field starts including VAT after a checkout redesign. Availability starts reflecting a nearby store, not national stock. The selector never moved.

Both classes share one root cause. Presentation markup is not an interface. Every selector is a bet that it will not move. Utility-class frameworks and build-time hashing worsen that bet each year. Dynamic class name obfuscation means class="css-1x9dk3f" changes on every deploy. Selector rot is the slow version of the same problem.

Retry logic does nothing here. A soft break retries perfectly. It returns the same wrong number every time.

A schema contract is what notices

Detection has to be a property of the data, not the process. Every record is validated against a contract before it counts as extracted.

A type check alone is not enough. price: Decimal accepts 2,499 as happily as 1,299. What catches a soft break is comparison against what you already know about that identifier.

# contract.py
# Defines what a valid product record looks like for one target.
# A parse that "succeeds" but violates this contract is treated as a break.

from decimal import Decimal
from pydantic import BaseModel, Field, model_validator


class ProductRecord(BaseModel):
    sku: str = Field(min_length=3)                     # the site's own identifier
    title: str = Field(min_length=5, max_length=300)
    price: Decimal = Field(gt=0, lt=Decimal("1000000"))
    currency: str = Field(pattern=r"^[A-Z]{3}$")       # ISO 4217, never a symbol
    in_stock: bool
    prior_price: Decimal | None = None                 # last known good for this SKU

    @model_validator(mode="after")
    def price_within_band(self):
        # A selector sliding to the MRP node still returns a valid number.
        # Only a comparison against last known good catches that.
        if self.prior_price and self.prior_price > 0:
            drift = abs(self.price - self.prior_price) / self.prior_price
            if drift > Decimal("0.60"):                # 60% move: suspect, not fatal
                raise ValueError(f"price_drift {drift:.2%} exceeds band")
        return self

Sample input, the raw fields pulled by the current selector set:

{"sku": "MYN-88213", "title": "Slim Fit Cotton Shirt",
 "price": "2499.00", "currency": "INR", "in_stock": True,
 "prior_price": "1299.00"}

Sample output, written to the repair queue instead of the warehouse:

{
  "sku": "MYN-88213",
  "target": "fashion_marketplace_a",
  "status": "contract_violation",
  "field": "price",
  "error": "price_drift 92.38% exceeds band",
  "observed": "2499.00",
  "last_known_good": "1299.00",
  "action": "queued_for_repair"
}

That record is the foundation for everything after it. Without it there is nothing to repair, because nothing knows anything is wrong.

Tune the band per target. A fashion marketplace running flash sales needs a wide band. A real estate listing site needs a narrow one, since asking prices move slowly. Cross-field rules matter as much as range rules. A record with in_stock: true and a null price is broken, and neither field says so alone. This is schema drift detection applied at record level rather than table level.

A scraper without a contract does not have a success rate. It has an exit code.

Where the LLM earns its place

The instinct here is to hand the whole page to a model. Let it read the price itself. That works, and it is the wrong default for a production feed.

Build extraction as a ladder instead. Cheapest deterministic path first, model last.

# resolve.py
# Extraction ladder. The model is never the first attempt, and never the second.

import json
from selectolax.parser import HTMLParser


def from_jsonld(html: str) -> dict | None:
    """Prefer the site's own Schema.org markup. It is published for search
    engines, so it drifts far less often than presentation markup."""
    tree = HTMLParser(html)
    for node in tree.css('script[type="application/ld+json"]'):
        try:
            blob = json.loads(node.text())
        except json.JSONDecodeError:
            continue                                   # partial JSON-LD is common
        for item in blob if isinstance(blob, list) else [blob]:
            if item.get("@type") != "Product":
                continue
            offers = item.get("offers") or {}
            if isinstance(offers, list):               # multi-seller listing pages
                offers = offers[0]
            return {
                "sku": item.get("sku"),
                "title": item.get("name"),
                "price": offers.get("price"),
                "currency": offers.get("priceCurrency"),
                "in_stock": "InStock" in str(offers.get("availability", "")),
            }
    return None


def from_selectors(html: str, selectors: dict[str, str]) -> dict:
    """Versioned CSS selectors, stored per target. Microseconds of CPU, zero cost."""
    tree = HTMLParser(html)
    return {
        field: (n.text(strip=True) if (n := tree.css_first(css)) else None)
        for field, css in selectors.items()
    }

On a large marketplace or most job boards, the first rung covers most fields. Marginal cost is zero. Stored selectors handle the rest. The model handles only what both rungs dropped.

When the contract fails, a repair job fires. It asks the model for selectors, never for data.

# repair.py
# Runs only on contract violation. Proposes selectors; a validator decides.

import json
from google import genai
from selectolax.parser import HTMLParser

PROMPT = """Repair a broken CSS selector on a product page.
Field: {field}
Expected: {expected}
Selector that stopped working: {old_selector}
Value it now returns: {bad_value}
Return JSON only, at most 4 candidates, most likely first:
{{"candidates": [{{"css": "...", "why": "..."}}]}}
HTML:
{html}"""


def prune(html: str, budget_chars: int = 24_000) -> str:
    """Strip script, style, svg, nav, footer before the model sees anything.
    Cuts a heavy retail page from tens of thousands of tokens to a few thousand."""
    tree = HTMLParser(html)
    tree.strip_tags(["script", "style", "svg", "noscript", "nav", "footer"])
    return tree.body.html[:budget_chars]


def propose(html, field, expected, old_selector, bad_value, client) -> list[dict]:
    resp = client.models.generate_content(
        model="gemini-2.5-flash-lite",                 # cheapest tier; repair is rare
        contents=PROMPT.format(
            field=field, expected=expected, old_selector=old_selector,
            bad_value=bad_value, html=prune(html)),
        config={"response_mime_type": "application/json"},
    )
    return json.loads(resp.text)["candidates"]


def validate(html, candidates, contract, base_record) -> dict | None:
    """Every candidate is tested against the contract. None are trusted."""
    tree = HTMLParser(html)
    for cand in candidates:
        node = tree.css_first(cand["css"])
        if node is None:
            cand["verdict"] = "rejected: no match"
            continue
        raw = node.text(strip=True).lstrip("INR₹$ ").replace(",", "")
        try:
            contract(**{**base_record, "price": raw})  # raises on violation
        except Exception as exc:
            cand["verdict"] = f"rejected: {exc.__class__.__name__}"
            continue
        cand["verdict"] = "passed"
        return cand
    return None

Sample output from one repair job:

{
  "target": "fashion_marketplace_a",
  "field": "price",
  "old_selector": "div.pdp-price > span",
  "candidates": [
    {"css": "div.pdp-price strong", "why": "sale price, sibling of struck MRP",
     "verdict": "passed"},
    {"css": "div.pdp-price s", "why": "a price-shaped node",
     "verdict": "rejected: ValidationError"},
    {"css": "meta[itemprop=price]", "why": "microdata attribute",
     "verdict": "rejected: no match"}
  ],
  "promoted": false,
  "gate": "awaiting_holdout_run",
  "tokens_in": 7412,
  "tokens_out": 186
}

The model proposed. The contract decided. That split is the entire design, and it is what makes the loop safe to run unattended.

Classify the failure before you route it

A null price and a Cloudflare challenge page produce the same contract violation. Only one of them is a selector problem.

This distinction is where naive repair loops burn money. Hand a block page to a model and ask for the price selector. It will find something. Block pages contain nodes. A model asked for four candidates returns four. Every one of them is nonsense, and the loop will promote the least-wrong nonsense if the gates are loose.

# classify.py
# A contract violation is not automatically a selector problem.

BLOCK_MARKERS = ("cf-chl", "Just a moment", "Access Denied", "Pardon Our Interruption")


def classify(status: int, html: str, violation: str | None) -> str:
    if status in (401, 403, 407, 429):
        return "transport"                    # proxy, session, or rate-limit problem
    if any(m in html[:4000] for m in BLOCK_MARKERS):
        return "transport"                    # 200 OK, but it is a challenge page
    if len(html) < 2_048:
        return "transport"                    # truncated or soft-blocked response
    if violation is None:
        return "ok"
    return "parse"                            # only this class reaches the model

Sample output across two records in the same run:

{"url": "/p/88213", "status": 200, "bytes": 184320,
 "violation": "price_drift", "class": "parse", "route": "repair_queue"}
{"url": "/p/91104", "status": 403, "bytes": 1207,
 "violation": "price_missing", "class": "transport", "route": "fetch_retry"}

Deduplication matters as much as classification. When 400 SKUs on one target fail identically in one run, that is one template change. Group violations by target, page type, and field before enqueuing. That grouping turns 400 model calls into one.

The cost math that picks the architecture

Take a feed of one million product pages a month. That is a normal volume for a mid-market eCommerce intelligence pipeline.

Token counts above are illustrative, sized from typical retail product pages. The rates come from Google’s published list prices for Gemini 2.5 Flash-Lite. That is $0.10 per million input tokens. Output runs $0.40 per million. Substitute your own model and your own page weights before quoting a number internally.

The spend gap is the less interesting half of that table. The determinism gap is the half that ends arguments. A stored selector returns the same value from the same HTML every run. A model can return 1,299 on Tuesday and 1,299.00 on Wednesday. On an ambiguous layout it will disagree with itself. For a feed that reconciles against yesterday’s snapshot, that creates diffs nobody can explain.

Latency compounds the same way. Selector extraction is CPU time you cannot measure without a profiler. A model call is a network round trip per page. That reshapes concurrency, scheduling, and cost per record at once.

One survey number is worth sitting with. The Apify and Web Scraping Club State of Web Scraping Report 2026 surveyed practitioners in December 2025. It found 45.8% use AI in their scraping workflows. Among those who do not, the reasons given were hallucinations, cost, and unreliable performance on specific sites. Every one of those is an objection to a model in the parse path. None of them apply to a model in the repair path. There, the output is a hypothesis, and it gets tested before anything reaches production.

Model spend is the cheapest line in this architecture. Target knowledge is the expensive one.

What a healed selector must prove before it ships

A candidate that passes the contract on one page has proven almost nothing. It might be matching a coincidence.

Five gates, in order:

1. It satisfies the contract on the failing page. The cheap check, already done inside validate().
2. It satisfies the contract on held-out pages from the same template. Pull 20 to 50 recently fetched pages of the same page type. A selector that passes one and fails nine is a coincidence.
3. It agrees with last known good on fields that should not have moved. If a repaired price selector also shifts the title, it is matching the wrong container.
4. It is written to a versioned selector store, not to code. A repair that needs a deploy is not healing anything. Store selectors as data, keyed by target, page type, field, and version, with the prior version kept for rollback.
5. It runs in shadow for one cycle before becoming primary. Both selectors extract. Only the incumbent’s output ships. The diff gets logged.

Blast radius decides how much of that runs unattended.

Cosmetic fields such as image URL, breadcrumb, or description can auto-promote after gate three. Structural fields such as pagination links and listing containers should clear a shadow run first. Decision-bearing fields such as price, stock status, SKU, and rating count get human sign-off. Model confidence does not change that.

Auto-promotion without a held-out test is not self-healing. It is automated, confident wrongness at scale.

The five numbers that tell you the loop is working

Parse success rate is the metric most teams already track. It is also the one that lies most. A run can report 100% parse success and 100% wrong values, because parsing succeeded on every page.

Promotion rate is the sharpest of the five. A rate near 100% means the gates are too weak to reject a bad proposal. A rate near zero points at the pruning step. It is cutting the answer out before the model ever sees it. Both are fixable in an afternoon, and neither is visible without the metric.

Silent-break share is the one that surprises people. It should climb as the contracts improve. A rising number means band rules are catching failures that exceptions never would. Wire all five into whatever monitoring and alerting stack already watches your pipelines, next to feed latency.

Running this across a fleet, not a script

One target with one contract is a weekend project. Thirty targets with per-page-type contracts is an operations problem, and that is where most in-house builds stall.

Repair needs a daily budget cap per target. One badly broken template will otherwise queue ten thousand jobs overnight. Records that fail repair need somewhere to go. A dead letter queue holds them for replay after a human fix. Dropping them is how gaps enter a dataset quietly. Prompts need versioning alongside selectors, because a prompt edit changes extraction behaviour exactly the way a code change does.

The layer nobody budgets for is the per-target contract itself. A food delivery platform prices by store and by time window. A JavaScript-rendered listing site hydrates half its fields after first paint. That is target knowledge, and no model infers it from raw HTML.

DataFlirt writes that contract per target during the crawler build. A repaired selector then meets rules matching how that site behaves. Not a generic product schema. The same holds on job board feeds. A posting that silently loses its salary field looks identical to one that never carried it. Delivery stays deliberately boring. Clean JSON, CSV, or a live API. Contract violations surface as a metric, not as a line in a run log.

Every record travels the cheap deterministic path. Only classified parse failures reach the model. Only gated candidates reach the selector store the next run reads from.

Public product and listing data carries fewer restrictions than personal data, though site terms and jurisdiction still apply. The legal orientation on web crawling covers where those lines sit.

Next Steps

Choose DataFlirt if:

• Your targets are JavaScript-heavy or hash their class names on every deploy, and selector maintenance is eating capacity you would rather spend on product.
• You need per-target validation contracts written by engineers who have watched that specific site’s markup change, not one generic schema applied everywhere.
• Delivery has to land as clean JSON, CSV, or a live API into an existing warehouse, with contract violations surfaced as a metric.
• You want the repair loop operated for you, including classification, promotion gates, and human sign-off on decision-bearing fields.

Look elsewhere if:

• DataFlirt runs lean, augmented with freelance engineering as projects need. That works well under roughly 100 target sites or 30 million pages a month. Beyond that, running and maintaining 1,000+ site crawlers in parallel is a heavier lift than this model is built for.

Send your target list, the exact fields, and the validation rules you already know matter. A senior engineer reviews every enquiry directly. You get a scope and a fixed quote within 2 business hours. A structured sample dataset comes with it, to check before anything is built.