We extract shipping manifests, supplier networks, HS code classifications, and bill of lading records from ImportGenius. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Shipment Records objects from importgenius.com. All fields typed and schema-versioned.
"bill_of_lading_number": "MEDU123456789", "arrival_date": "2023-10-14", "shipper_name": "Shenzhen Electronics Co Ltd", "consignee_name": "Acme Corp Logistics", "weight_kg": 14500.5, "hs_code": "851712", "port_of_loading": "Yantian, China", "vessel_name": "MSC ISABELLA"
| # | bill_of_lading_number | arrival_date | shipper_name | consignee_name | product_description | weight_kg |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Shipper Profiles objects from importgenius.com. All fields typed and schema-versioned.
"company_name": "Shenzhen Electronics Co Ltd", "country": "China", "total_shipments": 1432, "total_weight_kg": 850000.0, "top_hs_codes": "['851712', '850440']", "recent_activity_date": "2023-10-14", "address": "Baoan District, Shenzhen, Guangdong"
| # | company_name | country | total_shipments | total_weight_kg | top_consignees | top_hs_codes |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Consignee Data objects from importgenius.com. All fields typed and schema-versioned.
"company_name": "Acme Corp Logistics", "country": "United States", "total_imports": 894, "total_teu": 1250, "top_suppliers": "['Shenzhen Electronics Co Ltd', 'Taiwan Semi Mfg']", "port_of_discharge": "Los Angeles, CA", "industry_category": "Consumer Electronics"
| # | company_name | country | total_imports | total_teu | top_suppliers | top_hs_codes |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for HS Code Volumes objects from importgenius.com. All fields typed and schema-versioned.
"hs_code": "851712", "description": "Smartphones and related communication devices", "total_volume_kg": 45000000.0, "total_shipments": 28400, "top_origin_countries": "['China', 'Vietnam', 'India']", "average_weight_per_shipment": 1584.5, "duty_rates_extracted": false
| # | hs_code | description | total_volume_kg | total_shipments | top_origin_countries | top_destination_countries |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Logistics & Routing objects from importgenius.com. All fields typed and schema-versioned.
"vessel_name": "MSC ISABELLA", "voyage_number": "IN342A", "carrier_name": "Mediterranean Shipping Company", "port_of_loading": "Yantian, China", "port_of_discharge": "Los Angeles, CA", "transit_time_days": 18, "container_type": "40HC", "teu_count": 2.0
| # | vessel_name | voyage_number | carrier_name | port_of_loading | port_of_discharge | transit_time_days |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our ImportGenius scraper navigates complex search pagination, extracts nested bill of lading tables, and maps multi-tier supplier relationships with full anti-bot circumvention built in.
Extract raw manifest data including shipper, consignee, product description, weight, and arrival dates for every shipment.
Aggregate shipment histories to build comprehensive profiles of overseas manufacturers and their export volumes.
Track import volumes, supplier dependencies, and preferred ports for any US or international importer.
Analyse trade volumes and routing patterns for specific Harmonized System commodity codes.
Capture shipping routes, carrier market share, and transit times between global port pairs.
Standardise weight and container metrics across different manifest formats for accurate volume aggregation.
Automated traversal of ImportGenius search results, bypassing UI limits to extract complete historical datasets.
Extract records across US, Latin America, India, and other available customs jurisdictions in the database.
Run continuous pipelines to capture new shipments as they clear customs, delivering only the net-new records.
Brief in. Clean data out.
Provide target companies, HS codes, or port pairs. We design the extraction schema together.
We configure Scrapy crawlers, proxy rotation, session management, and authentication handling for importgenius.com.
Schema validation, null-rate checks, volume verification, and sample manifests before full launch.
JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
Extracting trade data at scale requires bypassing strict rate limits and parsing complex, nested tables. Here is how we maintain pipeline stability.
ImportGenius restricts deep data access behind authentication walls. We manage pools of authenticated sessions, rotating cookies and headers to distribute load and prevent account-level rate limiting.
Search interfaces often cap results at 10,000 records. Our crawlers dynamically slice date ranges and filter parameters to ensure complete data extraction without hitting pagination walls.
Bill of lading descriptions are notoriously unstructured. We apply regex patterns and NLP to standardise weights, clean company names, and extract valid HS codes from raw text blocks.
We route requests through ISP-grade residential proxies, matching the geographic location of the session to prevent IP bans and CAPTCHA triggers.
Customs data updates continuously. We maintain state on the last extracted arrival dates per target, only querying and extracting new shipments to minimise compute and API calls.
Procurement teams identify reliable overseas manufacturers by analysing their historical export volumes and current client lists.
Brands track competitor import volumes, discover their exact factory partners, and monitor their inventory buildup ahead of peak seasons.
Financial institutions verify physical trade flows against stated revenues to detect fraud and assess credit risk.
Analysts aggregate total TEU volumes per HS code to calculate market share shifts between major importers.
Freight forwarders analyse carrier routing, port congestion, and transit times to optimise their own shipping networks.
Quantitative funds ingest daily shipment volumes to forecast quarterly earnings for publicly traded retail and manufacturing companies.
"ImportGenius holds the definitive record of global physical trade, but mapping multi-tier supply chains requires structured, queryable databases."
Extracting millions of shipping manifests requires precise session management, handling complex search pagination, and parsing nested bill of lading tables. DataFlirt manages the proxy rotation, authentication states, and schema normalisation so your analysts can focus on trade flows, not scraping infrastructure.
Everything supported by our importgenius.com scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles high-concurrency search traversal and deduplication. Playwright manages complex authentication flows and dynamic table rendering.
We maintain pools of residential ISP proxies to distribute request load, preventing account flags and IP bans during high-volume manifest extraction.
Pipelines run on AWS ECS with Airflow managing schedule dependencies and delta extraction logic. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About importgenius.com scraping, legality, and pipeline operations.
Ask us directly →Scraping public customs data is generally permissible, but ImportGenius aggregates and structures this data behind terms of service and authentication walls. DataFlirt builds the extraction infrastructure. Clients must provide their own authenticated accounts and ensure their use case complies with the platform's terms.
When search queries return more records than the UI displays, our crawlers automatically slice the query into smaller date ranges or filter permutations until all records are extracted.
Yes. We can target US customs, Latin American manifests, Indian import/export data, and any other jurisdiction available within your ImportGenius access tier.
Yes. Due to the gated nature of the platform, clients must provide credentials with the appropriate access tier for the data they wish to extract. We manage the session pooling and rotation.
Customs data inherently has a lag. Our pipelines can run daily or weekly to capture new manifests as soon as ImportGenius indexes them.
We apply basic regex and string cleaning to raw manifest text. For advanced normalisation, we can integrate custom NLP parsing steps into the pipeline before delivery.
Our minimum engagement typically starts at a defined list of 500 companies or specific HS code sets. Contact us with your target volume for a scoped quote.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off historical export of a competitor's supply chain or continuous monitoring of an HS code category, we scope, build, and operate the pipeline. Tell us what you need.
