Scraping beauty and cosmetics product data — fields, sources and quirks

The online beauty and personal care market is hitting a total global market value of 70.3 billion USD in 2025 (source). Tracking competitor inventory in this space requires more than just scraping prices and titles. You must extract deep metadata like ingredient lists, shade variations, and compliance flags. Missing these details means your market analysis falls completely apart.

Key takeaways

• Beauty product data requires extracting unstructured INCI ingredient lists from text, HTML, and images.
• MoCRA enforcement makes compliance data tracking a mandatory requirement for retailers.
• Shopify standard endpoints fail to capture complex cosmetic metadata without targeting specific Metafields.
• Amazon’s Selling Partner API now enforces strict JSON schemas for beauty ingredients.
• Modern OCR pipelines resolve image-based ingredient lists using fuzzy matching against canonical dictionaries.

The critical fields in beauty and cosmetics data

You must map highly specialized attributes to build a complete cosmetic catalog. Standard pricing algorithms fail if they cannot differentiate between product sizes, skin type suitability, and compliance flags.

Ingredient lists and compliance markers

The INCI list is the most critical text block on a cosmetic product page. The Modernization of Cosmetics Regulation Act imposes strict facility registration and ingredient listing mandates. As of January 1, 2025, there are 589,762 unique, active cosmetic product listings actively registered with the FDA due to new MoCRA mandates (source). This volume highlights the immense scale of required compliance data.

You must also capture specific badges and formulation flags. Vegan, cruelty-free, and paraben-free claims dictate consumer purchasing behavior. SPF and PA ratings appear inconsistently across product titles or hidden specification tabs. Extracting these flags requires precise targeting.

Shelf life is another vital data point. The Period After Opening symbol often exists as a small icon on the packaging. Scraping this requires pulling text equivalents from the compliance tab or specification block. Country of manufacture also holds significant regulatory weight for imported goods.

Shade variants and volume metrics

Beauty products possess complex parent-child variant structures. A single foundation might feature forty different shades. You have to capture the shade name, the SKU, and the specific hex color code assigned to that variant.

Volume and size metrics directly impact unit pricing calculations. A standard cleanser comes in travel, standard, and value sizes. Your scraper must bind these volume metrics directly to the child variant to calculate accurate price-per-ounce metrics.

Where to get the data and platform notes

Beauty data lives in highly variable structures across different retail platforms. You will find ingredients neatly packed in JSON on one site and buried in an image on another.

Structured markup on major retailers

Extracting data from major aggregators requires targeting specific page elements. When you scrape Sephora, you will find the INCI information located in a dedicated ingredients tab. This tab typically contains a well-structured HTML list. This makes extraction relatively straightforward using a standard CSS selector.

Other platforms use dynamic elements to hide dense text. Scraping Nykaa involves interacting with collapsible accordions. These accordions are JavaScript rendered. You need a headless browser to simulate the click event, expand the accordion, and parse the resulting text.

Department stores present similar hurdles. Retailers like Macy’s and Nordstrom embed beauty specifications within deeply nested JSON objects. You must intercept the backend network requests to grab the raw data before the frontend formats it.

Major retailers often embed the entire product catalog in a JSON blob within the HTML document. You can extract this directly without simulating a full browser instance.

const cheerio = require('cheerio');

// Parse the static HTML to find the Next.js data blob
const $ = cheerio.load(htmlContent);
const nextData = $('#__NEXT_DATA__').html();
const productJson = JSON.parse(nextData);

// Extract the deeply nested ingredient string
console.log(productJson.props.pageProps.product.ingredients);

Direct-to-consumer brand sites and Shopify

Direct-to-consumer beauty brands usually build their storefronts on Shopify. These brand sites often dump the INCI list directly into the product description as a single paragraph. They provide no structural markup.

Standard Shopify CSV imports are variant-centric. They do not natively handle unstructured metadata like complex INCI ingredients out of the box. To properly store and scrape ingredient data in Shopify stores today, merchants utilize Metafields.

These Metafields map to custom namespaces and keys. Extracting this requires targeting the specific Metafield JSON output rather than standard product API queries. DataFlirt routinely isolates these custom Metafields to ensure no compliance data is left behind during a catalog migration.

Handling the image extraction problem

INCI ingredient lists are long, inconsistently formatted, and sometimes embedded entirely in images. Can they be extracted reliably? Yes.

Extracting INCI ingredient lists from cosmetic labels via OCR is highly dependent on image quality. Flatness, lighting, and non-cylindrical angles dictate the raw read accuracy. A distorted image yields fragmented text. DataFlirt engineers handle this by preprocessing the image to correct contrast and warp before data extraction.

Modern OCR pipelines make this reliable by combining raw text extraction with post-extraction normalization. Tools like Tesseract pull the raw characters. The pipeline then uses Levenshtein-distance fuzzy matching against canonical INCI dictionaries. This corrects misread characters and automatically resolves synonyms or chemical fragments into standardized cosmetic ingredient names.

Extraction quirks specific to this vertical

Scraping cosmetics data involves deep normalization and joining metadata across disparate review structures. You have to clean the text aggressively to make it useful for analysis.

Review volume and shade-specific sentiment

Consumer reviews dictate beauty product success. There is a 4.6% increase in conversion rate for products that secure at least 50 customer reviews, compared to products with fewer than 10 reviews (source).

This makes reviews scraping a core component of beauty intelligence. Generic reviews are completely useless for color cosmetics. Reviewers filter their feedback by specific shades. DataFlirt extracts these micro-segments to give analysts a clear picture of product performance.

You must extract the shade variant directly from the review metadata. This allows you to understand if a foundation formulation works perfectly for fair skin but oxidizes terribly on deeper shades. Without this variant connection, the sentiment analysis is flawed.

Skin type filters on platforms like Sephora and Target offer another layer of insight. Capturing this segment-specific sentiment reveals exactly how a product performs for oily, dry, or combination skin.

INCI normalization and dictionary mapping

Raw ingredient text is rarely clean. Brands misspell chemical names; they use colloquial terms alongside scientific nomenclature. You have to map these raw strings to INCI standard names using databases like the Cosmetic Ingredient Review.

You have to map these raw strings to INCI standard names. A simple fuzzy matching script handles the basic corrections before database insertion.

import fuzzywuzzy.process as fw_process

canonical_inci = ["Water", "Glycerin", "Niacinamide", "Titanium Dioxide"]
raw_extracted_text = "Glycerine"

# Match raw OCR text against the canonical INCI dictionary
best_match, score = fw_process.extractOne(raw_extracted_text, canonical_inci)
print(f"Mapped to: {best_match} with score {score}")

This normalization is critical for marketplace syndication. Amazon legacy XML and flat-file feeds were officially deprecated in July 2025 in favor of the JSON-based Listings Items API. DataFlirt maps all extracted ingredients to comply with this new standard.

For the beauty and personal care category, ingredients is a strictly enforced field. The formatting must map to Amazon’s specific schema. Incomplete or improperly formatted ingredient strings are currently a leading cause of ASIN suppression. Tracking this ensures your Amazon scraping and listing efforts remain viable.

Executing compliance tracking and competitor intelligence

You must choose between building a custom crawling infrastructure or outsourcing the data collection entirely. The technical overhead dictates this critical business decision.

The cost of in-house extraction

Building your own scraper requires significant engineering resources. You have to handle aggressive rate limiting and deploy residential proxies to access localized pricing. DataFlirt eliminates this friction by providing a managed infrastructure that scales instantly.

Regulatory monitoring adds another layer of stress. A staggering 48% of imported cosmetics sold at America’s top 25 retailers were not compliant with MoCRA registration and listing requirements as of November 2024 (source).

Tracking compliance gaps across competitors demands daily scraping. If a site changes its DOM structure, your in-house data extraction script breaks immediately. You then spend days repairing selectors instead of analyzing the data. DataFlirt monitors these DOM changes and automatically repairs broken selectors to ensure uninterrupted data delivery.

Scaling across global marketplaces

Global beauty brands sell across dozens of platforms simultaneously. You might track inventory on Walmart in the US, ASOS in the UK, and Temu or Shein globally. Each catalog requires a dedicated DataFlirt crawler configuration.

Each of these marketplaces employs unique browser fingerprinting checks. Writing custom bypass logic for every regional site drains your development budget rapidly. You will constantly battle new CAPTCHA challenges and behavioral analysis scripts.

If you want to read more about how these systems function, check out our guide on how web scraping works. It covers the foundational elements of bypassing basic protections. DataFlirt integrates these bypass methods directly into its core architecture.

DataFlirt for beauty and cosmetics catalog extraction

Relying on a managed partner removes the infrastructure burden from your internal team. You get clean data without managing the proxy rotations or decoding complex site logic.

Specialized pipelines for complex data

A freelancer on a gig platform can handle a simple flat-catalog export at low cost. Assuming the site has no JavaScript rendering and no bot protection, you might get a decent CSV. Once you add OCR requirements for ingredient lists or hit a Cloudflare-protected site, the job gets technically heavier.

The quality gap between a cheap gig and a managed extraction widens fast when dealing with complex vertical data. That is the exact range where DataFlirt’s QA layer and anti-bot engineering start paying for themselves. We provide a specialized extraction pipeline built specifically for beauty and cosmetics.

Consider a catalogue manager tracking 40,000 SKUs across six marketplaces. Every Monday, she needs last week’s compliance flags; not yesterday’s, not a live feed. A one-time weekly extraction via DataFlirt costs her a fraction of a live API subscription and delivers perfectly mapped INCI lists.

Our systems handle vertical-specific field normalization automatically. We parse the JavaScript accordions, run the OCR against image labels, and deliver import-ready files. DataFlirt guarantees that the shade variants map perfectly to their parent products. If you want to understand the pricing behind these managed systems, read our breakdown on understanding scraping cost factors.

We also ensure your compliance flags match strict marketplace schemas. Maintaining data quality is our primary objective for every catalog extraction. When DataFlirt processes a beauty catalog, we validate every PAO icon and SPF rating against your required output schema.

FAQ

Is scraping beauty product data legal?

Product data, including prices and ingredient lists, is generally publicly available information. Scraping this public data is common practice for market research. However, you must avoid extracting personal identifiable information from product reviews. Always review the terms of service for the target website. We recommend consulting qualified legal counsel for your specific situation.

Can OCR reliably extract cosmetic ingredients from images?

Yes. Modern OCR pipelines combine raw text extraction tools like Tesseract with fuzzy matching against canonical INCI dictionaries. This resolves misread characters and standardizes the output accurately.

How do I scrape ingredient data from a Shopify store?

Standard Shopify CSV endpoints are variant-centric and often omit unstructured text. You must target the specific Metafields mapped to custom namespaces and keys to extract the full ingredient lists reliably.

If you would rather not scope this complex extraction yourself, DataFlirt’s ecommerce scraping service handles the extraction, QA, and delivery. DataFlirt manages the proxies and parses the dynamic accordions so you can focus on market analysis. Reach out for a free scoping call today to discuss your catalog requirements.