Data Scraping Myths: 5 Claims Checked Against Real Rulings and Real Code

A pricing lead cancels a competitor-tracking project. The reason: “scraping is illegal.” Nobody checks that claim before killing the project. A competitor starts re-pricing daily off scraped data instead. The myth ends up costing more than the risk it was meant to avoid.

Why These Myths Keep Winning

Most data scraping myths contain a real fact, stretched past its actual boundary. Scraping can be illegal. It usually isn’t. Not for the reason most people assume. Scraping can violate privacy law, but public data and personal data aren’t treated the same way under that law.

This post checks five specific claims. Not against opinion or a competitor’s marketing copy. Against actual Ninth Circuit rulings, actual regulatory fines, and working extraction code.

What the Ninth Circuit Actually Decided

hiQ Labs v. LinkedIn answered this question twice. The Ninth Circuit ruled in 2019, then again on appeal in April 2022. Its holding: scraping publicly accessible, non-login pages doesn’t violate the Computer Fraud and Abuse Act. The court leaned on the Supreme Court’s 2021 ruling in Van Buren v. United States, which reads the CFAA as a “gates up or down” test. If there’s no gate to begin with, there’s nothing to bypass.

That’s not where the story ends. In December 2022, hiQ settled with LinkedIn anyway. It agreed to stop scraping, delete its data, and pay LinkedIn $500,000 in damages. That penalty came from a contract-breach claim, not the CFAA. hiQ had used fake accounts to reach data sitting behind LinkedIn’s login wall. Meta v. Bright Data, decided in January 2024, drew the same line from the other side. Judge Chen ruled that logged-out scraping of public Facebook and Instagram data didn’t breach Meta’s terms of service. Those terms bind logged-in users only.

What courts and regulators actually look at, case after case:

• Whether the page sits behind a login wall.
• Whether access required deception, like fake accounts or shared credentials.
• What robots.txt actually signals, which is intent, not a binding legal wall.
• What kind of data is being collected: product and listing data, or personal and biometric data.
• Whether the request volume itself caused measurable harm, the narrower trespass-to-chattels theory that sometimes runs alongside CFAA and contract claims.

Public Doesn’t Mean Unregulated

Clearview AI scraped more than 30 billion facial images from public web pages. No login walls involved, no fake accounts, no deception in how the pages were reached. In September 2024, the Dutch Data Protection Authority fined the company. The amount: 30.5 million euros ($33.7 million). The reason: building an illegal facial-recognition database out of that data. Regulators in France, Italy, Greece, and Austria issued similar rulings against the same company, over the same practice. The pages were public. The fines happened anyway.

GDPR treats biometric identifiers as a protected category of personal data, regardless of where they were posted. Scraping a competitor’s product catalog and scraping faces for an identity database aren’t the same act under that law. Product prices, stock counts, and star ratings don’t identify a person. Names, photos, and biometric markers do. A similar line runs through CCPA in the US. The rule tracks what’s actually being collected. It also tracks how that data gets used, not simply whether the source page required a login.

Legal Isn’t the Same as Ethical

A page can be perfectly legal to scrape and still get hammered by a badly built script. Hitting a small site with hundreds of requests a second isn’t illegal on most public pages. It’s still a bad way to collect data. It’s the kind of thing that turns a defensible legal position into an actual server load problem for the target.

Rate limiting and polite crawling exist for this reason. Respect a site’s stated crawl delay. Space out requests. Scrape during off-peak hours where possible. None of that costs much. It also removes most of the goodwill argument a target site could otherwise raise. DataFlirt’s own ethics and best-practices guide covers this in more detail, separate from the legal exposure question above.

You Don’t Need a Developer for Every Target

Most product and listing pages already carry structured data, embedded for search engines to read. It’s called schema.org markup, delivered as a JSON-LD block sitting inside the page’s HTML. An Amazon listing usually has one. So does a real estate listing, a job posting, and most review pages. Reading that block is a parsing problem. It isn’t an engineering project.

# Pull a Product record straight from a page's JSON-LD block.
# No CSS selectors and no site-specific scraping logic required.
import json
import httpx
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; DataResearchBot/1.0)"}

def extract_product_json_ld(url: str) -> dict | None:
    """Return the first schema.org Product record found on the page."""
    response = httpx.get(url, headers=HEADERS, timeout=15, follow_redirects=True)  # 1. fetch the page
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")  # 2. parse the HTML

    for tag in soup.find_all("script", type="application/ld+json"):  # 3. scan JSON-LD blocks
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        for entry in data if isinstance(data, list) else [data]:
            if entry.get("@type") == "Product":  # 4. match the Product schema
                offers = entry.get("offers", {})
                return {
                    "name": entry.get("name"),
                    "price": offers.get("price"),
                    "currency": offers.get("priceCurrency"),
                    "availability": offers.get("availability"),
                    "rating": entry.get("aggregateRating", {}).get("ratingValue"),
                }
    return None  # 5. no Product block on this page

Sample input: extract_product_json_ld("https://example-retailer.com/products/wireless-earbuds")

{
  "name": "AeroBuds Pro Wireless Earbuds",
  "price": "89.99",
  "currency": "USD",
  "availability": "https://schema.org/InStock",
  "rating": "4.6"
}

That covers the easy half of the web. It doesn’t cover a Cloudflare challenge page. It doesn’t cover a catalog that never renders that markup. It doesn’t cover a site fingerprinting your browser before serving anything back. That’s where AI-assisted extraction earns its place. A fixed selector breaks the day a layout changes. A self-healing selector pulls the same field by meaning instead, not by a specific tag path. It keeps working through a redesign that would break a rigid script outright. That price, availability, and rating record is also already shaped for the next step. Drop it straight into a pricing dashboard or a repricing rule, with no reformatting in between.

Messy Data Is a Process Failure, Not a Scraping Problem

Raw scraped data can absolutely be messy. Inconsistent date formats, duplicate listings, missing fields, the usual complaints. None of that is a property of scraping itself. It’s a property of skipping the validation step that should run before delivery, not after a client finds the gaps.

Sites also redesign their pages without warning. That’s its own quality risk if nobody’s watching for it. A pipeline that tracks schema drift catches a broken field the same day a layout changes. Without that check, the same problem surfaces a week later, once someone notices the numbers look wrong.

Automated Doesn’t Mean Unsupervised

Automated scraping and manual review aren’t opposites competing for the same job. A script pulls 50,000 listing pages overnight, faster than any team of people could. It doesn’t know that a page’s layout changed halfway through the run. It doesn’t know a field it just extracted is nonsense. A promotion banner got parsed as the price instead. The script has no way to tell on its own.

A human check still earns its place on the edge cases automation can’t judge for itself. A one-off due-diligence pull is one example, where every record needs eyes on it before it ships. A compliance-sensitive dataset is another, where a person confirms what actually got collected before delivery. The two aren’t a tradeoff. A production pipeline runs the automated extraction first. It then routes flagged, low-confidence records to a person, instead of shipping them blind.

Build It, Rent Access, or Buy the Finished Dataset

For a founder or ops lead deciding whether to scrape at all, the myths usually mask the real question. It isn’t “is this legal, or is it too technical.” It’s “who owns the parsing, the anti-bot fight, and the maintenance, once this is actually running.”

Speed and volume favor renting access if a team already has data engineers to build on top of it. Building in-house only pays off with the time and headcount to treat it as a real, ongoing engineering commitment, not a side project someone picks up between other work. A defined, structured feed on a schedule, without hiring for it, is what a partner like DataFlirt is built to deliver instead.

Anti-Bot Systems Made Amateur Scraping Obsolete, Not Scraping

Automated traffic passed human traffic for the first time in a decade in 2024. It made up 51% of all web traffic that year, according to Imperva’s 2025 Bad Bot Report. Bad bots alone accounted for 37% of it, up from 32% the year before. That surge is exactly why Cloudflare, Akamai, and Imperva keep tightening browser fingerprint checks and behavioral traps.

A plain requests script with a static header dies against that within minutes. A crawler engineered specifically against those systems, tuned to the target’s own defenses, doesn’t. The myth isn’t wrong about the trend. It’s wrong about the conclusion. Rising anti-bot sophistication didn’t kill scraping. It killed the version of scraping that treats anti-bot handling as an afterthought, which is exactly the gap DataFlirt’s crawlers are built to close.

What Changes at Production Scale

A one-off script pulling 200 product pages is a weekend task. A recurring feed pulling millions of pages a day, on a schedule, with anti-bot resilience already built in, is a different problem entirely. That gap is where most in-house scraping efforts stall out.

Scaling a scraping operation usually means horizontal scaling across distributed workers, not one script running longer. It also means someone re-tuning selectors every time a target site redesigns a page, and delivering the result somewhere useful: straight into S3, MongoDB, or a live API endpoint, not a folder of CSVs nobody imports. Monitoring matters just as much as the crawl itself. A feed that silently starts returning empty fields is worse than one that fails loudly. DataFlirt builds that maintenance and monitoring into the pipeline itself, whether the target is a single ecommerce catalog or a hundred of them running on the same schedule.

From a target page to a decision-ready record: extraction, validation, and structured delivery, all before the actual business call gets made.

This isn’t legal advice for your specific situation. For a full breakdown of what’s settled and what’s still contested, see DataFlirt’s guide on whether web crawling is legal.

Next Steps

Choose DataFlirt if:

• Anti-bot resilience against Cloudflare, Akamai, or Imperva is a real requirement, not a nice-to-have.
• There’s no in-house data engineering team to build and maintain this.
• Structured CSV, JSON, or API delivery matters more than raw HTML access.
• The need is a recurring, scheduled feed, not a one-off script somebody has to remember to re-run.

Look elsewhere if:

• DataFlirt runs on cloud infrastructure and third-party proxy vendors rather than owning infrastructure in-house. That’s true of nearly every provider in this space, so it isn’t a differentiator against alternatives. It’s worth knowing if in-house infrastructure ownership is a hard requirement for your team.

Get in touch with the target site and the fields you need. One-time projects are typically scoped and delivered in 3-4 days.