eCommerce Web Scraping Use Cases: The Data That Actually Moves a Pricing Decision

Amazon changes prices more than 2.5 million times a day, per Profitero. A once-a-day price check is stale before lunch. Most scraped price feeds still run on a nightly job. This post shows the extraction pattern that closes the gap. Real code, real output, no theory.

A Single Price Point Isn’t a Feed

A pricing analyst doesn’t need today’s price. They need today’s price, set against yesterday’s price and the promotion behind it. A basic scraper that grabs one number misses both. So does a spreadsheet someone updates by hand once a week.

Availability tells the same kind of story. A competitor selling out of a key SKU is a demand signal, not a footnote. Caught as a one-off spot check, that signal arrives too late to use. Caught as a feed, it’s an opportunity window, not a surprise.

Why Competitor Prices Move Before You Notice

Buy Box ownership on Amazon can shift between sellers within the same day. Price is the single largest lever behind who wins it. Repricing software adjusts a listing dozens of times without a human involved. Shipping speed and seller rating factor in too. Price still moves first.

A promotional window can open at 6 AM and close by noon. A pricing team relying on a morning check misses it entirely. A team relying on last week’s export misses it by a wider margin. This is the mechanism a recurring feed exists to catch.

Why JavaScript-Rendered Storefronts Need a Second Extraction Path

Not every product page hands over its price in the raw HTML. Many storefronts built on React or Next.js load price and stock through client-side JavaScript instead. A plain HTTP request never sees that data. Those pages need a browser, not a request.

# extract_price_js.py
# For storefronts that render price and stock client-side.
# Waits for the price element to mount, then reads the rendered DOM.
# Real library: playwright (sync API)

from playwright.sync_api import sync_playwright


def extract_price_rendered(url: str) -> dict:
    """Load a JS-rendered product page and read price after render."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent="Mozilla/5.0 (compatible; DataFlirtBot/1.0)"
        )
        page.goto(url, wait_until="networkidle", timeout=20000)

        # Wait for the price node to actually mount before reading it
        page.wait_for_selector("[data-testid='price']", timeout=10000)

        price_text = page.locator("[data-testid='price']").inner_text()
        stock_text = page.locator("[data-testid='stock-status']").inner_text()
        browser.close()

    return {
        "url": url,
        "price": float(price_text.replace("$", "").replace(",", "")),
        "availability": stock_text.strip().lower(),
        "source": "rendered-dom",
    }

Sample output:

{
  "url": "https://storefront.example.com/products/desk-lamp-x3",
  "price": 39.0,
  "availability": "in stock",
  "source": "rendered-dom"
}

A headless browser is heavier than an HTTP request. Running it across 200,000 SKUs on a schedule is a different problem entirely. It also needs rate limiting tuned per target, to keep the crawl from tripping a bot check. DataFlirt tunes both paths per site: request-only where it works, a browser where it doesn’t.

Extracting Price and Stock Without Breaking on a Redesign

Most price scrapers rely on CSS selectors tied to one page layout. A retailer redesign breaks that overnight. Parsing a page’s schema.org JSON-LD data first is more durable. Selectors become the fallback, not the primary path.

# extract_price.py
# Parses schema.org Product/Offer JSON-LD first, most stable against redesigns.
# Falls back to CSS selectors when a page has no structured data.
# Real libraries: httpx, BeautifulSoup4

import json
import httpx
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; DataFlirtBot/1.0)"}


def extract_price(url: str) -> dict:
    """Fetch a product page and return price, currency, and stock status."""
    response = httpx.get(url, headers=HEADERS, timeout=15, follow_redirects=True)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    # 1. Try structured data first
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        offer = _find_offer(data)
        if offer:
            return {
                "url": url,
                "price": float(offer.get("price", 0)),
                "currency": offer.get("priceCurrency", "USD"),
                "availability": offer.get("availability", "").split("/")[-1],
                "source": "json-ld",
            }

    # 2. Fall back to CSS selectors, the page has no structured data
    price_tag = soup.select_one("[data-price], .price-current, .product-price")
    stock_tag = soup.select_one("[data-availability], .stock-status")
    return {
        "url": url,
        "price": _parse_price(price_tag.get_text() if price_tag else ""),
        "currency": "USD",
        "availability": stock_tag.get_text().strip().lower() if stock_tag else "unknown",
        "source": "css-fallback",
    }


def _find_offer(data) -> dict | None:
    """Walk a JSON-LD block for an Offer. Handles both dict and list shapes."""
    if isinstance(data, list):
        data = data[0] if data else {}
    offers = data.get("offers")
    return offers if isinstance(offers, dict) else None


def _parse_price(text: str) -> float:
    """Strip currency symbols and separators from a raw price string."""
    digits = "".join(c for c in text if c.isdigit() or c == ".")
    return float(digits) if digits else 0.0

Sample input: https://example-retailer.com/products/wireless-mouse-m2. Sample output:

{
  "url": "https://example-retailer.com/products/wireless-mouse-m2",
  "price": 24.99,
  "currency": "USD",
  "availability": "instock",
  "source": "json-ld"
}

DataFlirt runs this exact pattern: JSON-LD first, selectors second, a headless browser only where the site actually requires one. It’s built across hundreds of named retail and marketplace targets, from Target and Best Buy to Shopify storefronts with no structured data at all. Each crawler is tuned to that site’s rendering approach and anti-bot layer, not run off one generic template.

Handling the Anti-Bot Layer Most Scripts Skip

None of the extraction code above survives a real anti-bot layer alone. Cloudflare and similar systems fingerprint request patterns, not just IP addresses. A script making 5,000 requests from one IP gets blocked fast.

Residential proxies spread requests across real consumer IP ranges. Randomized delays and rotating headers make the pattern look human. None of this is exotic. It’s table-stakes engineering for any crawler running against a hardened retail target.

Storing Price History So It’s Actually Useful

A price feed that overwrites its last value can’t answer “how has this changed.” Every extraction run should insert a new row, never overwrite the last one. That’s what makes trend detection and MAP-violation history possible later.

-- price_history: append-only time-series table
-- One row per extraction run, never updated in place

CREATE TABLE price_history (
    id SERIAL PRIMARY KEY,
    sku TEXT NOT NULL,
    seller TEXT NOT NULL,
    price NUMERIC(12, 2) NOT NULL,
    map_floor NUMERIC(12, 2),
    availability TEXT NOT NULL,
    scraped_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Index for the query pricing teams actually run: "what changed for this SKU"
CREATE INDEX idx_price_history_sku_time ON price_history (sku, scraped_at DESC);

This is a small schema, on purpose. A 30-day price trend, a MAP-violation history, and a stockout timeline all come from this same table. Each is just a different query. Nothing about it needs a specialized time-series database at this scale.

Turning Raw Price Pulls into a MAP and Buy Box Signal

A single seller’s price means little alone. The same SKU across five sellers, compared to a MAP floor, means everything. That’s what turns a price pull into a compliance signal. The same crawler also feeds catalog and review-sentiment pipelines, a different team’s workflow entirely.

Sample multi-seller pull for one illustrative SKU.

# flag_map_violations.py
# Takes raw per-seller price records for one SKU.
# Flags any seller priced below the MAP floor.
# Also flags the cheapest in-stock seller as the likely Buy Box winner.

def flag_sku_records(sku: str, map_floor: float, records: list[dict]) -> list[dict]:
    """Annotate seller records with MAP violation and buy-box-eligible flags."""
    in_stock = [r for r in records if r["availability"] == "instock"]
    cheapest_in_stock = min(in_stock, key=lambda r: r["price"]) if in_stock else None

    flagged = []
    for r in records:
        flagged.append({
            "sku": sku,
            "seller": r["seller"],
            "price": r["price"],
            "map_violation": r["price"] < map_floor,
            "buy_box_eligible": r is cheapest_in_stock,
        })
    return flagged

Sample output, for the table above:

[
  {"sku": "wm-2201", "seller": "Seller A", "price": 24.99, "map_violation": true, "buy_box_eligible": true},
  {"sku": "wm-2201", "seller": "Seller B", "price": 27.50, "map_violation": false, "buy_box_eligible": false},
  {"sku": "wm-2201", "seller": "Seller C", "price": 26.80, "map_violation": false, "buy_box_eligible": false}
]

The same records export just as easily as CSV, for a team whose tools expect a flat file instead of JSON:

sku,seller,price,map_violation,buy_box_eligible
wm-2201,Seller A,24.99,true,true
wm-2201,Seller B,27.50,false,false
wm-2201,Seller C,26.80,false,false

Seller A undercuts the MAP floor by $2.01, flagged the moment it lands in the feed. Seller C looks cheaper. It’s also out of stock. That’s a stronger signal for a demand model than for a pricing dashboard.

What a Finished Record Actually Looks Like

Everything above lands in one place: a single record per SKU, per seller, per crawl. That’s what a pricing team actually consumes, not raw HTML or a half-filled spreadsheet.

{
  "sku": "wm-2201",
  "seller": "Seller A",
  "price": 24.99,
  "map_floor": 27.00,
  "map_violation": true,
  "buy_box_eligible": true,
  "availability": "instock",
  "source": "json-ld",
  "scraped_at": "2026-07-18T09:02:11Z"
}

Nine fields, one row, one crawl. That’s the entire output a pricing decision needs. Everything before this point exists to produce exactly this record, reliably, on schedule.

Deciding What Actually Needs a Recurring Crawl

Not every signal needs the same refresh rate. MAP compliance works fine checked daily. Violations tend to run for days before anyone notices anyway. Buy Box-relevant pricing needs hourly checks on high-velocity categories.

Match the crawl schedule to the decision it feeds. Running everything hourly wastes crawl budget on signals that don’t need it. A promotional calendar checked hourly returns far less value than the compute it costs.

What Building This In-House Actually Costs

A data engineer can build the JSON-LD pattern above in a day. Keeping it running against 50 sites, through redesigns and rate limits, is the actual job. That’s where most in-house attempts stall, not at the first version.

A scraping API solves access, not the finished feed. You still parse, normalize, and flag MAP violations yourself. DataFlirt builds the crawler, the normalization, and the flagging as one delivered pipeline. It’s priced for a team that doesn’t want to hire for this.

Watching the Feed So Nobody Else Has To

A recurring crawl fails quietly more often than it fails loudly. A selector stops matching and returns null instead of throwing an error. Without monitoring, a team can run on stale data for days and never know it.

# freshness_check.py
# Alerts when a SKU's latest record is older than the freshness SLA allows.
# Runs on its own schedule, separate from the crawl itself.

from datetime import datetime, timezone, timedelta

FRESHNESS_SLA = timedelta(hours=2)


def check_freshness(sku: str, last_scraped_at: datetime) -> bool:
    """Return True if the SKU's latest record is within the SLA window."""
    age = datetime.now(timezone.utc) - last_scraped_at
    is_fresh = age <= FRESHNESS_SLA
    if not is_fresh:
        # In production this pages someone, it doesn't just print
        print(f"ALERT: {sku} is {age} stale, breaches the {FRESHNESS_SLA} SLA")
    return is_fresh

Sample output for a stale SKU:

ALERT: wm-2201 is 3:15:00 stale, breaches the 2:00:00 SLA

The fix is alerting on the signal, not the symptom. A freshness-SLA breach, a null-field spike, or a CAPTCHA-rate jump should each page someone. That’s before a pricing decision gets made on bad data. It’s what separates a script from a pipeline built to run unattended, whether it’s checked once a quarter or every hour.

From a Spot Check to a Recurring Feed

A one-off price audit answers one question: what the market looks like today. A recurring feed answers a different one: what changed, and what to do about it. That shift changes the infrastructure need. Deduplication and schema-drift handling move from nice-to-have to mandatory.

A scheduled crawl needs deduplication. A re-run should never double-count a SKU. Schema-drift handling matters just as much, keeping a template change from failing silently. DataFlirt builds pipelines for exactly this shift: output as CSV, JSON, a live API, or straight into S3 or MongoDB.

From product page to pricing feed: JSON-LD extraction first, a headless browser second, then dedup, MAP flagging, and delivery as CSV, JSON, or API.

Scraping publicly listed prices and stock status is generally permissible. ToS terms, MAP agreements, and regional rules still apply. See DataFlirt’s guide on is web crawling legal.

Next Steps

Choose DataFlirt if:

• Competitor pricing or MAP compliance needs tracking at SKU level, daily or faster.
• Target sites are JavaScript-heavy, React, or Shopify storefronts that break generic scrapers.
• Stockout and availability signals need to feed a pricing pipeline or demand model, not sit in a spreadsheet.
• Data needs to land as CSV, JSON, a live API, or directly in an existing pipeline.

Look elsewhere if:

• DataFlirt runs on cloud infrastructure and third-party proxy vendors rather than owning infrastructure in-house. That’s true of nearly every provider in this space, so it isn’t a differentiator against alternatives, but it’s worth knowing if in-house infrastructure ownership is a hard requirement.

Most projects are scoped within 48 hours. Talk to DataFlirt’s team about a price and inventory feed built for your SKU list.