We extract loose gem listings, per-carat pricing, origin data, treatments, and certification details from Gemselect. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Gemstone Listings objects from gemselect.com. All fields typed and schema-versioned.
"item_id": "684932", "gem_type": "Sapphire", "weight_carat": 2.45, "shape": "Oval", "colour": "Blue", "treatment": "Heated", "origin": "Madagascar", "price": 1250.0
| # | item_id | gem_type | weight_carat | dimensions | shape | cut |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Gemological Specs objects from gemselect.com. All fields typed and schema-versioned.
"item_id": "684932", "refractive_index": "1.762 - 1.778", "specific_gravity": "3.95 - 4.03", "hardness_mohs": "9.0", "luster": "Vitreous", "certification_lab": "AIG"
| # | item_id | refractive_index | specific_gravity | hardness_mohs | crystal_structure | cleavage |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Pricing & Wholesale objects from gemselect.com. All fields typed and schema-versioned.
"item_id": "684932", "final_price": 1250.0, "price_per_carat": 510.2, "currency": "USD", "wholesale_lot": false, "piece_count": 1, "price_timestamp": "2023-10-24T08:12:00Z"
| # | item_id | base_price | discount_pct | final_price | price_per_carat | currency |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Media & Certificates objects from gemselect.com. All fields typed and schema-versioned.
"item_id": "684932", "primary_image_url": "https://www.gemselect.com/photos/sapphire/sapphire-684932a.jpg", "video_url": "https://www.gemselect.com/video/sapphire/sapphire-684932.mp4", "certificate_image_url": "https://www.gemselect.com/cert/aig/684932.jpg", "report_number": "AIG-1928374", "validation_url": "https://aiglabs.com/verify/1928374"
| # | item_id | primary_image_url | video_url | gallery_urls | certificate_image_url | report_number |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Search & Taxonomy objects from gemselect.com. All fields typed and schema-versioned.
"keyword": "blue sapphire", "category_path": "Sapphire > Blue Sapphire", "position": 12, "item_id": "684932", "title": "2.45 ct Oval Blue Sapphire", "facet_colour": "Blue", "scraped_at": "2023-10-24T08:12:05Z"
| # | keyword | category_path | position | item_id | title | facet_colour |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our Gemselect scraper captures structured data for unique, single-stock items. We normalise treatment codes, calculate per-carat math, and map origins across the entire inventory.
Capture carat weight, exact dimensions, shape, cut style, colour grading, and clarity for every unique stone.
Extract and normalise treatment disclosures (e.g., heated, beryllium treated, unheated) and geographic origins.
Calculate and track exact per-carat pricing alongside total item cost to normalise market comparisons.
Extract certification laboratory names, report numbers, and verification URLs for graded stones.
Distinguish between single stones, matched pairs, and wholesale parcels, capturing piece counts and lot pricing.
Capture direct URLs for primary images, gallery angles, and 360-degree videos for visual analysis.
Extract refractive index, specific gravity, and Mohs hardness data provided in the detailed specs.
Track when unique items sell out and when new stock arrives, identifying high-velocity gem categories.
Extract pricing in USD, EUR, GBP, or other supported currencies based on configured crawl parameters.
Brief in. Clean data out.
Provide gem types, origins, or size ranges. We design the extraction schema together.
We configure Scrapy crawlers, facet traversal logic, and schema normalisation for gemselect.com.
Schema validation, unit standardisation (mm/carats), and null-rate checks before full launch.
JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
Gemselect's inventory consists of unique, single-quantity items. Here is how we manage the high churn and complex taxonomy.
Unlike standard retail, loose gems are unique. When an item sells, the URL often 404s or redirects. We maintain a stateful index of all known item IDs, emitting 'out_of_stock' events rather than simply dropping the record from the dataset.
Gemselect uses deep category trees (e.g., Sapphire -> Blue -> Unheated -> >2 Carats). We traverse the facet matrix systematically to ensure 100% catalogue coverage without missing edge-case inventory.
Gem dimensions are often presented as unstructured strings (e.g., '8.1 x 6.2 x 4.5 mm'). Our pipeline parses these into distinct length, width, and depth floats for direct database insertion.
We bypass CDN obfuscation to extract the raw, highest-resolution image and video URLs, ensuring ML models or catalogue systems receive optimal visual inputs.
To minimise load and reduce processing costs, we monitor the 'New Arrivals' feeds and category counts, triggering deep scrapes only when new inventory is detected or prices shift.
Wholesalers aggregate per-carat pricing across gem types, origins, and treatments to establish market baselines.
Custom jewellers monitor specific parameters (e.g., unheated Montana sapphires over 2 carats) for client commissions.
ML teams train automated valuation algorithms using historical price-to-spec ratios from the catalogue.
Analysts track inventory velocity to identify rising consumer interest in specific stones, like teal sapphires or spinel.
Rival gemstone retailers track Gemselect's markup strategies, discount cadences, and shipping tiers.
Buyers identify underpriced wholesale lots or matched pairs for individual resale at retail margins.
"Gemselect holds the market's most transparent pricing for loose coloured stones — but extracting normalised per-carat data requires a dedicated pipeline."
Most teams underestimate the complexity of scraping unique, single-stock items. Gemselect's inventory turns over rapidly, and parsing unstructured treatment codes, origins, and laboratory certificates requires strict schema enforcement. DataFlirt handles the extraction, normalisation, and bot mitigation so your analysts can focus on market pricing — not maintaining selectors.
Everything supported by our gemselect.com scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles crawl orchestration, deduplication, and retry logic. Playwright handles JavaScript rendering, cookie sessions, and interaction flows. Combined via scrapy-playwright middleware.
We maintain pools of residential ISP proxies across IN/US/UK/DE regions. Rotation happens per-request with sticky sessions where required. IP score monitoring prevents blacklisted pool contamination.
Pipelines run on AWS Lambda (burst) and ECS (sustained). Airflow handles scheduling, dependency management, and SLA alerting. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About gemselect.com scraping, legality, and pipeline operations.
Ask us directly →Scraping publicly available pricing and specification data is generally permissible. DataFlirt targets only public, non-authenticated inventory data. We do not circumvent authentication walls or extract user data. Clients should consult legal counsel for specific use cases.
Every loose gem has a unique item ID. We use this as the primary key in our database, tracking its state from 'in_stock' to 'sold_out'. This prevents duplicate entries and ensures accurate historical pricing.
We extract the certification laboratory name, report number, and direct URLs to the certificate images or verification portals. We do not perform OCR on the certificate images themselves.
We can configure pipelines to run daily or hourly. For high-velocity categories, we monitor the 'New Arrivals' feeds to capture new stock within minutes of listing.
Yes. We distinguish between single stones, matched pairs, and wholesale parcels, extracting the piece count and calculating the per-carat price accordingly.
Yes. Gemselect lists treatments in unstructured text. We use regex and mapping dictionaries to normalise these into standard categories (e.g., Unheated, Heated, Beryllium Treated, Fracture Filled).
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a daily catalogue dump or real-time alerts for unheated sapphires — we scope, build, and operate the pipeline. Tell us what you need.
