We extract article text, metadata, author intelligence, and historical archives from the South China Morning Post. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Article Metadata objects from scmp.com. All fields typed and schema-versioned.
"article_id": "3214567", "url": "https://www.scmp.com/tech/big-tech/article/3214567/example", "headline": "China tech giants increase AI investment", "author_name": "Jane Doe", "publish_date": "2026-05-12T09:14:00Z", "category": "Tech", "paywall_status": "free", "word_count": 842
| # | article_id | url | headline | author_name | author_url | publish_date |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Article Content objects from scmp.com. All fields typed and schema-versioned.
"article_id": "3214567", "headline": "China tech giants increase AI investment", "subheadline": "Major firms allocate new budgets for research.", "body_text": "Leading technology companies in Shenzhen have announced...", "image_urls": "['https://cdn.i.scmp.com/sites/default/files/example.jpg']", "quotes": "['This investment marks a turning point.']", "summary_bullets": "['New funding exceeds previous estimates.']"
| # | article_id | headline | subheadline | body_text | image_urls | video_urls |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Author Profiles objects from scmp.com. All fields typed and schema-versioned.
"author_id": "auth_8912", "name": "Jane Doe", "role": "Senior Tech Reporter", "location": "Hong Kong", "bio": "Jane covers the intersection of technology and policy.", "article_count": 412, "twitter_handle": "@janedoe_scmp", "profile_image_url": "https://cdn.i.scmp.com/author/jane.jpg"
| # | author_id | name | role | location | bio | article_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Category Feeds objects from scmp.com. All fields typed and schema-versioned.
"category_name": "Economy", "sub_category": "China Economy", "url": "https://www.scmp.com/economy/china-economy", "top_stories": "['3214568', '3214569']", "trending_articles": "['3214570']", "scrape_timestamp": "2026-05-12T10:00:00Z", "total_articles_indexed": 154
| # | feed_id | category_name | sub_category | url | top_stories | trending_articles |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Comments objects from scmp.com. All fields typed and schema-versioned.
"comment_id": "cmt_99213", "article_id": "3214567", "user_name": "HK_Reader_99", "comment_text": "This analysis misses the key regulatory changes from last week.", "timestamp": "2026-05-12T11:22:00Z", "upvotes": 45, "replies_count": 3
| # | comment_id | article_id | user_name | user_id | comment_text | timestamp |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our scmp.com scraper captures every layer of the publication: breaking news, historical archives, author intelligence, and multimedia content.
Extract headlines, subheadlines, body text, quotes, and embedded media from any public SCMP article.
Capture categories, tags, publication dates, and update timestamps to categorise news flow accurately.
Track journalist output, location, bio changes, and social handles across the entire SCMP contributor base.
Traverse sitemaps and pagination to extract decades of historical reporting and opinion pieces.
Monitor category pages and RSS endpoints for breaking news with sub-minute extraction latency.
Extract public user comments, upvotes, and reply threads for sentiment and engagement analysis.
Automate keyword searches on scmp.com and extract the ranked article results and metadata.
Extract high-resolution image URLs, infographic links, and embedded video sources from article bodies.
Continuously monitor high-value articles and emit diffs when headlines or body text are updated post-publication.
Brief in. Clean data out.
Provide target categories, author profiles, or keyword sets. We design the extraction schema together.
We configure Scrapy crawlers, proxy rotation, session management, and CDN bypass for scmp.com.
Schema validation, null-rate checks, and text completeness verification before full launch.
JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
News sites deploy strict caching, paywalls, and bot protection. Here is how we maintain reliable data flow.
Media sites use aggressive edge caching and bot mitigation. We route requests through residential proxies with realistic TLS fingerprints to prevent IP bans and ensure consistent access to article pages.
Images, embedded tweets, and comment sections often require client-side execution. We use Playwright to render the full DOM, ensuring no text or media links are missing from the final payload.
We identify and tag articles protected by SCMP Pro or standard paywalls. For gated content, we extract the available public teaser text and metadata without attempting unauthorised access.
Feature articles, opinion pieces, and standard news reports use different HTML templates. Our extraction logic uses fallback chains and JSON-LD parsing to ensure structural consistency across all formats.
Extracting historical data requires navigating complex pagination and sitemap structures. We maintain strict state tracking to prevent duplicate extraction and ensure complete historical coverage.
Think tanks and analysts monitor reporting on China-US relations, regional policies, and diplomatic shifts.
Quantitative funds extract breaking news on Asian markets, corporate earnings, and regulatory changes to inform trading algorithms.
Machine learning teams use the vast SCMP English corpus to train language models on Asian geopolitical terminology and sentiment.
PR firms and corporate communications teams track brand mentions, executive quotes, and industry coverage across the publication.
Universities analyse decades of historical archives to study media narratives and economic development in the APAC region.
Other media organisations track SCMP publishing volume, author output, and category focus to benchmark their own editorial strategy.
"South China Morning Post holds the most comprehensive English language reporting on Asian markets and geopolitical shifts. We make it queryable."
Most teams underestimate the complexity of scraping global news publishers. Extracting SCMP requires handling strict rate limits, dynamic content loading, CDN blocks, and paywall variations. DataFlirt absorbs that infrastructure burden so your analysts can focus on the data.
Everything supported by our scmp.com scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles crawl orchestration, deduplication, and retry logic. Playwright handles JavaScript rendering and interaction flows. Combined via scrapy-playwright middleware.
We maintain pools of residential ISP proxies. Rotation happens per-request with sticky sessions where required. IP score monitoring prevents blacklisted pool contamination.
Pipelines run on AWS Lambda and ECS. Airflow handles scheduling, dependency management, and SLA alerting. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About scmp.com scraping, legality, and pipeline operations.
Ask us directly →Scraping publicly available news articles is generally permissible for factual data extraction. DataFlirt targets only public, non-authenticated content and metadata. We do not bypass paywalls to steal premium content or extract personal user data. Clients should consult legal counsel regarding copyright and fair use for their specific application of the text.
We detect paywalled articles and extract only the publicly available teaser text, headline, and metadata. We do not use compromised credentials or exploits to access SCMP Pro or subscriber-only content.
Yes. We can traverse SCMP sitemaps and search pagination to extract decades of historical reporting. Volume and timeframe parameters are defined during the scoping phase.
For continuous monitoring, we poll category feeds and RSS endpoints at high frequency. Breaking news can be extracted and delivered via Webhook within minutes of publication.
We extract the high-resolution URLs for images, infographics, and embedded videos. We do not download and host the media files directly, but provide the source links in the structured data.
Yes. We can configure pipelines to revisit specific URLs at defined intervals, hashing the body text and headline to detect stealth edits or updates, delivering only the diffs.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a full historical archive export or a continuous live news feed, we scope, build, and operate the pipeline. Tell us what you need.
