Web Data for Business Intelligence: From Scraped Page to BI-Ready Table

A pricing dashboard built on a broken scraper still loads. It just shows the wrong numbers.

Most BI teams get web data from a script someone wrote once. It runs fine for a while. Then a target site renames a CSS class. The scraper keeps returning HTTP 200. Nobody gets alerted, because nothing technically failed. The dashboard keeps refreshing. It just refreshes with empty or stale fields.

This post shows a real extraction pattern. It includes a real schema. It includes the monitoring layer most one-off scrapers skip.

Why a Pricing Dashboard Is Only as Good as the Table Behind It

A dashboard doesn’t know when its source data is wrong. It only renders whatever lands in the table behind it. A null field doesn’t throw an error. Most BI tools show a zero instead. Or a blank. Or last week’s cached value.

That’s the real failure mode with scraped data. It’s not “the scraper stopped working.” It’s “the scraper kept working and quietly stopped being right.” A page redesign can cause this. A lazy-loaded price block can too. So can a new anti-bot check. An availability field stuck on “InStock” long after a competitor actually sold out is a common version of the same problem. The page moved on. The extraction didn’t catch up. None of these trip an alert a basic monitoring tool would catch. By the time someone notices a flat pricing chart, the decisions built on it already happened.

The Spreadsheet Version of This Always Breaks First

Most competitor pricing tracking starts as a spreadsheet. Someone copies prices in by hand once a week. It works, for a handful of SKUs and a handful of competitors.

It stops working the moment either number grows. A hundred SKUs across five competitors is 500 manual checks. Nobody sustains that cadence for long. Excel wasn’t built for this kind of recurring data project. Most teams know they should track this. The real gap is automation, not interest. Getting to an organization that actually runs on data usually comes down to this one step.

A Scraping API Gets You Access, Not This Table

A scraping API solves a different problem. It hands back raw HTML or JSON per request, fast, at volume. What lands in that response is still just a page. It isn’t a schema.

Turning that raw response into price_fact and product_dim is still your job. Parsing, deduping, normalizing currency, and watching for drift all sit on your team’s plate either way. For a BI team, a scraping API is one layer in the stack, not the finished table. The pattern this post walks through is the layer that actually sits behind a dashboard.

Two Ways a Feed Breaks Without Throwing an Error

Both common failure modes hide the same way: a normal-looking response, quietly wrong content.

Selector rot. A target site renames a class. It restructures a div. It moves a price into a new container during a routine redesign. A brittle CSS-selector scraper returns None for that field. The request still succeeds. The extraction doesn’t. See selector rot for the full mechanism.

Soft blocks. An anti-bot system flags a scraper’s traffic pattern. Instead of blocking outright, it serves a cached or generic version of the page. The HTTP status still reads 200. The content is real, just wrong for that moment. That’s what makes a soft block harder to catch than an outright block.

Neither shows up in a basic uptime check. Both show up in field extraction completeness tracking instead, which is why it matters more than raw scrape success rate. A production pipeline pairs that tracking with self-healing selector logic that attempts a repair before paging anyone. Most drift resolves on its own, leaving only the genuine breaks for a human to check.

From a Product Page to a BI-Ready Record

CSS selectors break the moment a site changes a class name. A more durable pattern exists. It extracts from a page’s own schema.org Product markup, shipped as JSON-LD. Most eCommerce sites already publish this for search engines. They rarely restructure it on a whim. A production crawler also respects rate limiting and each site’s robots.txt crawl delay, spacing requests so a daily pull never looks like an attack.

# Extract price, availability, and SKU from a product page's
# schema.org Product JSON-LD block, not from CSS selectors.
import httpx
from bs4 import BeautifulSoup
import json
from datetime import datetime, timezone

def extract_product(url: str) -> dict:
    """Pull structured fields from a product page's JSON-LD.
    Returns null fields on a miss instead of raising, so one bad
    page doesn't take down a whole batch run."""
    response = httpx.get(url, timeout=20, follow_redirects=True)  # 1. fetch page
    soup = BeautifulSoup(response.text, "html.parser")            # 2. parse HTML

    record = {
        "url": url, "sku": None, "price": None,
        "currency": None, "availability": None,
        "captured_at": datetime.now(timezone.utc).isoformat(),
    }

    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue  # malformed JSON-LD, skip this block
        if isinstance(data, dict) and data.get("@type") == "Product":
            offer = data.get("offers", {})
            record["sku"] = data.get("sku")
            record["price"] = offer.get("price")
            record["currency"] = offer.get("priceCurrency")
            record["availability"] = str(offer.get("availability", "")).split("/")[-1]
            break

    return record

Sample input: extract_product("https://example-retailer.test/product/wireless-mouse-x200") (illustrative URL, not a real target)

Sample output:

{
  "url": "https://example-retailer.test/product/wireless-mouse-x200",
  "sku": "WM-X200-BLK",
  "price": "24.99",
  "currency": "USD",
  "availability": "InStock",
  "captured_at": "2026-07-18T06:00:00+00:00"
}

That single record is a start, not a deliverable. A BI team can’t query 50,000 individual JSON files sitting in a folder. These records need a schema built for comparison over time, not a pile of snapshots. Where a page skips structured markup entirely, AI-assisted extraction can pull the same fields out of unstructured HTML instead.

What Else Typically Rides Along in the Same Pull

Price and availability aren’t the only fields worth capturing while a crawler is already on the page. Product images, review counts, and star ratings usually sit in the same JSON-LD block, at close to zero extra cost per request.

That’s useful beyond a pricing dashboard. The same pull can feed catalog enrichment for a PIM platform, not just a competitive intelligence table. Add one more column to product_dim. Don’t stand up a second crawler for it.

Normalizing Into a Schema Your BI Tool Can Query

A BI tool doesn’t want a stream of product records. It wants a fact table it can join against dimension tables. That’s the star schema pattern behind most warehouse-backed dashboards.

Getting a raw record into that fact table isn’t one simple insert. Repeat pulls of the same product need deduplication logic. An upsert pattern stops a re-run from creating duplicate rows for a day that already has data. The payoff shows up fast. A pricing analyst can now answer “who undercut us yesterday” with one query. That query joins price_fact to product_dim directly, not a manual comparison across a dozen browser tabs.

-- Upsert a daily price observation without creating a duplicate row
-- if this pipeline re-runs for the same product/competitor/date.
INSERT INTO price_fact (date_key, product_key, competitor_key, price, currency, availability)
VALUES ('2026-07-18', 4471, 2, 24.99, 'USD', 'InStock')
ON CONFLICT (date_key, product_key, competitor_key)
DO UPDATE SET
    price = EXCLUDED.price,
    availability = EXCLUDED.availability;

Sample price_fact rows after two days of loads:

A feed that still returns 200 but stops updating is worse than a feed that fails outright. A failure gets fixed fast. A quiet drift gets trusted for weeks.

What a BI Team Actually Does With That Table

A table like this feeds a handful of concrete decisions, not a generic “insights” dashboard. Daily undercut alerts, weekly assortment reports, and price-elasticity models all read from the same price_fact table. They just aggregate it differently.

This is where price intelligence and competitive monitoring earns its place in a BI stack. The value isn’t the scrape itself. It’s a table still queryable six months later. A one-time snapshot can’t answer how pricing moved last holiday season. A recurring pipeline can. Freshness here drives a decision that repeats.

The Same Table Design Extends Beyond Pricing

Pricing is the most common use of a table shaped this way. It isn’t the only one. The same product_dim and competitor_dim structure supports:

• Assortment gap analysis: which SKUs a competitor carries that you don’t.
• Out-of-stock tracking as a demand signal, not just a pricing one.
• Bestseller-rank data feeding a product-launch positioning model.
• Variant-level pricing, where size or color carries its own premium over the base SKU.

Swap the fact table’s measure column. Keep the dimensions the same. The same pipeline design now answers a different business question, without a rebuild from scratch.

What Bad Data Costs When Nobody’s Watching

Bad inputs here carry a real cost, not just an inconvenience. Gartner puts the average annual cost of poor data quality at $12.9 million. That figure comes from its 2020 Magic Quadrant for Data Quality Solutions research. It spans every kind of bad data, not just scraped feeds. A silently stale pricing feed fits the pattern exactly.

It keeps refreshing. It keeps being wrong. Nobody flags it as broken, because nothing threw an error, so nobody budgets time to fix it. The fix costs less than the silence does. Catching it early is usually a monitoring problem, not a bigger-scraper problem.

How Fresh This Table Actually Needs to Be

Refresh cadence isn’t a technical decision first. It’s a business one. Match it to how fast the underlying decision changes. Daily refresh covers most standard competitive pricing. A pricing analyst checking undercuts once a morning doesn’t need hourly numbers.

Fast-moving categories need faster refresh. Flash sales, event tickets, and holiday promotions can shift within hours. That pushes cadence toward hourly pulls and tighter data freshness targets. Too slow, and a dashboard shows yesterday’s world. Too frequent for no reason, and a pipeline burns compute chasing a number that wasn’t moving. Neither extreme is kind to the data latency budget.

From a Weekly Script to a Pipeline a BI Team Can Trust

A script that runs once a week, checked by whoever wrote it, doesn’t need much. A pipeline fifteen people pull dashboards from needs more. It needs schema-drift detection: an alert when a field’s null rate jumps, not just when a request fails outright.

That’s the real difference between a one-off scrape and a production pipeline. Recurring-pipeline design treats monitoring, retries, and delivery cadence as first-class requirements from the start. Deduplication, schema consistency, and validation happen before delivery, not after. A BI team receives a table it can trust, without re-checking it first.

Checking the Table Before a Dashboard Ever Sees It

A schema-drift alert catches extraction problems early, but not every problem. A field can extract cleanly and still be wrong: a price of zero, a duplicate SKU, a currency mismatch across regions nobody normalized.

A validation step before delivery closes that gap. Data completeness checks flag a table falling below an expected fill rate on key fields, before anything downstream consumes it. The same check that catches a broken scraper also catches a bad batch. See assessing data quality and data quality for the fuller checklist this pattern draws from.

What Changes as the Target List Grow

Delivery format matters more as scale grows. Output can land as a scheduled CSV or JSON drop. It can be a live API a dashboard tool polls directly. It can go straight into S3, Snowflake, or Postgres, alongside the rest of a warehouse. Which one fits depends on how the BI stack already pulls data. It’s not about what’s easiest to generate for one target.

A real BI stack rarely tracks just one competitor. Most named retailers a BI team would want already have a dedicated crawler built for them, from Target to Flipkart. That shortens the path from “we need this one added” to a new row in competitor_dim. Widening from ten SKUs to ten thousand, or adding a competitor mid-project, is where an elastic engineering bench matters more than fixed headcount. Some teams also want the calculation logic and dashboard built on top of the table, not just the raw feed underneath it. That’s worth scoping upfront, not discovered after launch.

Schema-drift alerting runs alongside the pipeline, watching field-level null rates, catching a broken feed before a dashboard ever renders it.

Scraping publicly accessible pricing pages is broadly permitted, though the legal detail varies by jurisdiction. See the full breakdown in is web crawling legal.

Next Steps

Choose DataFlirt if:

• Structured CSV, JSON, or API delivery matters more than raw HTML access.
• The need is a recurring, scheduled feed, not a one-time pull.
• Data needs to land directly in an existing S3, Snowflake, or Postgres warehouse.
• An analytics layer and schema design are part of what’s needed, not just raw extraction.

Look elsewhere if: DataFlirt runs on cloud infrastructure and third-party proxy vendors rather than owning infrastructure in-house. That’s true of most providers in this space, so it isn’t a differentiator against alternatives. It’s worth knowing if in-house infra ownership is a hard requirement.

Most projects are scoped within 48 hours. Start a scoping conversation with the engineers who would actually build the pipeline, not an account manager relaying it secondhand.