3 Datasets That Actually Power Competitive Intelligence

A press release describes where a competitor wants to be. A job board describes where they’re actually going. Most competitive intelligence programs read the press release. Few read the job board, the pricing page, or the review feed underneath it. Those three sources update weekly or daily. A quarterly earnings call doesn’t. This post covers the three datasets that actually move a competitive intelligence program, and a working pattern for turning one of them into a signal.

Hiring data as an expansion signal

A hiring page is a forward-looking dataset. A company posting ten new sales roles in a new region signals expansion before any press release exists. Nasdaq’s 2025 State of Alternative Data report puts hiring and workforce signals among the categories driving alternative data adoption, now used by 78% of hedge funds surveyed.

Two raw hiring records, scraped and structured, look like this:

LinkedIn, Indeed, and Glassdoor job pages carry this signal in structured form already. A rising count of engineering postings in a new city says more than a quarter of press coverage. The full investment-decision pattern applies just as well to a competitor as it does to a portfolio company.

Pricing and catalog data as a product signal

A pricing page updates before a press release does. A new SKU, a dropped feature tier, or a discount pattern shows up in the catalog first. What a competitor’s catalog reveals is more concrete than a strategy memo, because it’s what they actually shipped.

Promotional patterns reveal strategy too, not just price. A competitor discounting every Tuesday signals a demand problem on slow days. A MAP violation pattern across resellers signals where enforcement is weak.

Review data as a product-gap signal

A competitor’s product page lists features. A competitor’s reviews list complaints. Sentiment analysis on review text surfaces the gap between the two. Aspect-based sentiment breaks that further, tying a complaint to a specific feature instead of one star rating.

A repeated complaint about onboarding on G2 or Capterra points at a real product gap, not a guess. Mining competitor reviews for product development turns that complaint pattern into a roadmap input instead of an anecdote.

Why not just buy a pre-built alternative-data feed

Off-the-shelf alternative data feeds exist for public companies with a stock ticker. Most competitors worth tracking don’t have one. A private robotics company, a regional retailer, or a newly funded startup rarely shows up in a vendor’s coverage list.

A custom scrape covers exactly the competitor set that matters to one business, not the subset a data vendor decided to cover. A packaged feed is faster to start. A custom crawler covers the company that actually competes with you, ticker or not, down to a single regional player a vendor’s coverage list never included.

Where the data actually hides

Not every source publishes this data in plain HTML. A modern career page often loads roles through a JavaScript call after the page finishes loading, not in the initial response. A review site paginates results behind infinite scroll instead of numbered pages.

A headless browser renders the JavaScript first, then hands off the finished page. Scroll emulation handles the pagination problem the same way, triggering the same scroll events a real visitor would. Neither trick is exotic on its own, but both need to run reliably across ten or more target sites at once. That’s where a dynamic-site scraping approach earns its keep.

One career page, several possible platforms

Not every competitor hosts job postings the same way. One runs on Greenhouse. Another runs on Lever, Ashby, or Workday. Each platform structures its job listing HTML, and often its API, differently.

def detect_ats(page_url: str, html: str) -> str:
    """Identify which applicant tracking system served this career page."""
    if "boards.greenhouse.io" in page_url or "greenhouse" in html.lower():
        return "greenhouse"
    if "jobs.lever.co" in page_url:
        return "lever"
    if "jobs.ashbyhq.com" in page_url:
        return "ashby"
    return "generic"

A generic HTML scraper still works as a fallback for the rest, catching whatever career page doesn’t match a known platform. Most of the real engineering effort in a hiring-data pipeline goes into this platform-detection step, not the parsing itself. Parsing a known ATS structure is comparatively simple once it’s identified, since the field names and layout stay consistent across every company that uses it.

Turning job postings into a structured signal

Hiring data only becomes a signal once it’s aggregated over time. One posting means nothing. A trend across weeks does. Here’s a working pattern for turning raw scraped postings into a spike flag.

# Turn raw scraped job postings into a week-over-week hiring signal
from collections import defaultdict
from datetime import date, timedelta

def weekly_posting_counts(postings: list[dict]) -> dict:
    """Group postings by week-start and department. Count each group."""
    counts = defaultdict(int)
    for job in postings:
        posted = date.fromisoformat(job["posted_date"])
        week_start = posted - timedelta(days=posted.weekday())  # Monday
        key = f"{week_start.isoformat()}_{job['department']}"
        counts[key] += 1
    return dict(counts)

def flag_hiring_spike(this_week: int, last_week: int, threshold: float = 0.5) -> bool:
    """Flag a department growing postings faster than the threshold, week over week."""
    if last_week == 0:
        return this_week >= 3  # a brand-new department counts as a spike on its own
    return (this_week - last_week) / last_week > threshold

Sample input, three raw postings scraped off one career page:

postings = [
    {"department": "engineering", "posted_date": "2026-07-06"},
    {"department": "engineering", "posted_date": "2026-07-08"},
    {"department": "sales", "posted_date": "2026-07-07"},
]

weekly_posting_counts(postings) returns:

{
  "2026-07-06_engineering": 2,
  "2026-07-06_sales": 1
}

flag_hiring_spike(2, 0) returns True. Two new engineering postings against zero the week before reads as expansion, not noise. Run weekly, this turns a scraped job board into an early flag on DataFlirt’s company-data pipelines, not a manual page-by-page check.

Filtering out noise before it becomes signal

Not every scraped record is real signal. A company reposts the same open role every 30 days to keep it visible on a job board. Counted twice, that looks like a second hire instead of one unfilled seat.

def dedupe_postings(postings: list[dict]) -> list[dict]:
    """Collapse reposted listings for the same role, department, and location."""
    seen = {}
    for job in postings:
        key = (job["department"], job["location"], job["role_title"])
        if key not in seen or job["posted_date"] > seen[key]["posted_date"]:
            seen[key] = job
    return list(seen.values())

The same discipline applies to reviews. Anomaly detection on posting timing catches a burst of five-star reviews within a single hour. That pattern reads as incentivized activity, not organic sentiment, and it should get flagged before it enters an average.

One competitor record, three signals merged

Hiring, pricing, and review data answer different questions on their own. Merged into one record, they answer a bigger one: is this competitor expanding, under pressure, or standing still.

def build_competitor_signal(hiring: dict, pricing: list[dict], reviews: list[dict]) -> dict:
    """Merge three independently scraped datasets into one competitor record."""
    avg_discount = sum(p["discount_pct"] for p in pricing) / len(pricing)
    avg_sentiment = sum(r["sentiment_score"] for r in reviews) / len(reviews)
    spike = flag_hiring_spike(hiring["this_week"], hiring["last_week"])
    flag = "expanding_but_quality_slipping" if spike and avg_sentiment < 0 else "steady"
    return {
        "competitor": hiring["competitor"],
        "hiring_spike": spike,
        "avg_discount_pct": round(avg_discount, 1),
        "review_sentiment_avg": round(avg_sentiment, 2),
        "flag": flag,
    }

That last field, flag, is the actual output a competitive intelligence team wants. It reads as a sentence, not three disconnected charts. Getting there needs all three datasets on the same schema and the same time window, not three separate spreadsheets updated on three separate schedules by three different people.

What a hiring spike doesn’t tell you

A hiring spike isn’t always growth. A company backfilling roles after a layoff round posts just as aggressively as one expanding into a new market. Posting count alone doesn’t distinguish the two.

Pairing hiring data with pricing discipline and review sentiment is what separates them. A hiring spike with rising discounts and falling sentiment reads as backfill under pressure. A hiring spike with stable pricing and flat sentiment reads as real expansion. Neither reading is certain from hiring data alone, which is exactly why it isn’t the only dataset on this list.

Scaling from one competitor to a portfolio

One competitor record is a data point. Ten competitors, refreshed on the same schedule, is a portfolio view. The same build_competitor_signal function runs once per competitor, on the same weekly cadence.

Read as a table, three competitors sort themselves by what actually matters. One is expanding while its product quality slips. One is discounting hard with stable sentiment. One is expanding cleanly. That’s a weekly review meeting’s worth of signal, built from data any team could scrape on its own.

What a complete competitor record extends into

Hiring, pricing, and reviews cover product and people. They don’t cover footprint. A retailer or restaurant chain’s physical expansion shows up in store locator pages before it shows up anywhere else.

Store network analysis extends the same pattern to physical location data. The schema changes, store count, new market entries, closures, but the underlying idea doesn’t. Scrape the page the competitor actually updates, not the one written for the press.

Where this fits an existing workflow

This data doesn’t replace a competitive intelligence analyst. It replaces the manual page-by-page check that ate an analyst’s week. Checking four competitors’ career pages, catalogs, and review feeds by hand takes a full afternoon, every week, for as long as the role exists. A weekly portfolio table like the one above becomes the agenda for a recurring review instead, not a research project every time.

Delivered as structured CSV, JSON, or a live API, the same data plugs into a BI dashboard or a CRM without a second transformation step. That’s the difference between data you receive and data you still have to build.

Where the merged output actually lands

A weekly competitor table only helps if it reaches the tool a team already uses. Loading it by hand into a spreadsheet defeats the point of automating the scrape in the first place.

Output can land straight in S3, MongoDB, or a BI tool’s own ingestion endpoint, refreshed on the same schedule as the crawl itself. The dashboard updates on its own instead of waiting on someone to paste in last week’s numbers.

How often each dataset needs to refresh

Not all three datasets move at the same speed. Pricing changes daily on many retail and industrial sites. Reviews accumulate over weeks. Job postings turn over on a company’s own hiring cycle.

Data freshness requirements should set the crawl schedule, not the other way around. A daily pricing crawl paired with a weekly hiring crawl costs less to run and still catches every signal that actually matters.

What breaks at scale

Scraping ten competitors’ career pages daily looks nothing like scraping one page once. Job boards and review sites rate-limit aggressively once request volume climbs. An IP ban on a shared proxy can quietly stall a whole competitor’s feed for days.

Proxy rotation across residential IPs spreads that risk out across many addresses instead of one. Anti-bot handling becomes a standing job at this volume, not a one-time fix, especially once a target site adds a new block rule mid-quarter.

Three independently scraped datasets, hiring, pricing, and reviews, merge into one composite competitor signal, refreshed on a schedule matched to how fast each source actually changes.

Scraping public job boards, catalogs, and review pages for competitive research is generally low-risk, though exposure varies by site terms and jurisdiction. See DataFlirt’s guide to web crawling legality for specifics.

Next Steps

Choose DataFlirt if:

• The target sites span job boards, review platforms, and retail catalogs, not just one source type.
• Anti-bot resilience against rate limiting, IP blocking, and JavaScript rendering is a real requirement at this volume.
• The three datasets need to land on one schema and one refresh schedule, not three disconnected spreadsheets.
• The need is a recurring, scheduled feed, not a one-time competitor snapshot.

Look elsewhere if:

• DataFlirt runs on cloud infrastructure and third-party proxy vendors rather than owning infrastructure in-house. That’s true of nearly every provider in this space, so it isn’t a differentiator against alternatives, but it’s worth knowing if in-house infra ownership is a hard requirement.

Most one-time competitive intelligence snapshots ship in 3-4 days. Scope a project and get a sample record back before committing to a recurring feed.