We extract news articles, editorial metadata, author profiles, and category archives from smh.com.au. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Article Metadata objects from smh.com.au. All fields typed and schema-versioned.
"article_id": "p5ey12", "url": "https://www.smh.com.au/politics/federal/example-article.html", "headline": "Federal budget targets inflation with structural reforms", "author": "Jane Doe", "published_date": "2023-10-24T14:30:00Z", "category": "Politics", "is_premium": true
| # | article_id | url | headline | subheadline | author | published_date |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Article Content objects from smh.com.au. All fields typed and schema-versioned.
"article_id": "p5ey12", "word_count": 842, "body_text": "The federal government has announced a series of structural reforms designed to curb inflation...", "image_urls": "['https://static.ffx.io/images/example1.jpg']", "pull_quotes": "['Inflation remains the primary economic challenge of this decade.']", "scraped_at": "2023-10-25T08:15:00Z"
| # | article_id | body_text | word_count | image_urls | video_urls | pull_quotes |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Author Profiles objects from smh.com.au. All fields typed and schema-versioned.
"name": "Jane Doe", "role": "Chief Political Correspondent", "twitter_handle": "@janedoe_smh", "bio": "Jane Doe covers federal politics and economic policy.", "article_count": 412, "last_active_date": "2023-10-24T14:30:00Z"
| # | author_id | name | role | twitter_handle | bio | article_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Category Archives objects from smh.com.au. All fields typed and schema-versioned.
"category_slug": "business/markets", "page_number": 1, "article_url": "https://www.smh.com.au/business/markets/asx-update.html", "headline": "ASX drops 50 points following Wall Street lead", "published_date": "2023-10-24T10:00:00Z", "scraped_at": "2023-10-25T08:16:00Z"
| # | category_slug | page_number | article_url | headline | author | published_date |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Comments & Engagement objects from smh.com.au. All fields typed and schema-versioned.
"article_url": "https://www.smh.com.au/politics/federal/example-article.html", "comment_count": 124, "top_comment_author": "John Citizen", "engagement_score": 85, "share_count": 312, "scraped_at": "2023-10-25T08:17:00Z"
| # | article_url | comment_count | top_comment_text | top_comment_author | engagement_score | share_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our SMH scraper captures structured text, metadata, and author profiles while handling paywall detection, dynamic layouts, and pagination.
Extract clean body text, headlines, subheadlines, and pull quotes without boilerplate navigation or advertising markup.
Capture bylines, author roles, social handles, and historical publication frequency to map editorial focus.
Traverse sitemaps and category pagination to extract years of historical articles for long-term trend analysis.
Identify and flag premium subscriber-only content versus freely accessible articles during the extraction process.
Extract high-resolution image URLs, video embed links, and caption text associated with each article.
Capture primary categories, sub-categories, and editorial tags to categorise news volume by topic.
Poll category pages and RSS feeds at high frequency to detect breaking news and article updates within minutes.
Capture total comment counts and engagement metrics to measure audience response to specific topics.
Receive normalised JSON or Parquet files pushed directly to your cloud storage environment.
Brief in. Clean data out.
Provide target categories, author lists, or date ranges. We design the extraction schema together.
We configure Scrapy crawlers, proxy rotation, and layout parsing rules for smh.com.au.
Schema validation, null-rate checks, and text formatting review before full launch.
JSON, CSV, or Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
News sites deploy strict anti-bot measures and complex DOM structures. Here is how we maintain data quality.
Media sites use aggressive WAF rules to block scrapers. We use Australian residential proxies and realistic request headers to ensure uninterrupted access to public article pages.
SMH uses different templates for standard news, live blogs, and interactive long-form features. Our selector strategy uses multiple fallback chains to extract core text regardless of the presentation layer.
Standard category pagination often caps at recent articles. We parse XML sitemaps and search indices to bypass pagination limits and extract historical content spanning decades.
We identify paywall states dynamically. For hard-gated premium content, we extract all available metadata, headlines, and summaries while correctly flagging the payload as restricted.
News articles are frequently updated after initial publication. We hash article content and emit diffs when headlines or body text change, providing a full revision history.
PR firms and corporate communications teams track brand mentions, sentiment, and crisis coverage in real time.
Machine learning teams use clean, high-quality Australian English journalism to train language models and text classifiers.
Quantitative funds parse business and market news to extract macroeconomic signals and company-specific events.
Other publishers analyse SMH publication velocity, author output, and topic coverage to benchmark their own editorial strategy.
Universities study political bias, media framing, and historical reporting trends across decades of structured news archives.
Analysts track public sentiment on policy changes by correlating article tone with comment volume and engagement metrics.
"High-quality journalism provides the cleanest NLP training data available. We structure the archives so your models can read them."
Extracting text from modern news sites requires navigating dynamic layouts, live blog updates, and aggressive rate limiting. DataFlirt manages the proxy rotation and selector maintenance so you receive clean text files ready for analysis.
Everything supported by our smh.com.au scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles sitemap traversal and deduplication. Playwright handles JavaScript rendering for interactive feature articles.
We maintain pools of AU residential proxies to bypass WAF rules and ensure consistent access to public article pages.
Pipelines run on AWS Lambda. Airflow handles scheduling for real-time polling and historical backfills.
Data delivered to where your team already works — no new tooling required.
About smh.com.au scraping, legality, and pipeline operations.
Ask us directly →Scraping publicly available headlines, metadata, and non-paywalled text is generally permissible for analysis purposes. DataFlirt does not bypass hard paywalls using stolen credentials. Clients should review copyright laws regarding the reproduction of full-text articles for commercial use.
We detect the paywall state of each article. For metered content, we use proxy rotation to access the public view. For hard-gated premium content, we extract the headline, summary, and metadata while marking the body text as restricted.
Yes. We use sitemap parsing and search index traversal to extract articles dating back over a decade, providing comprehensive datasets for longitudinal research.
For real-time monitoring, we poll specific category pages and RSS feeds every 5 to 15 minutes, delivering new articles via webhook immediately upon publication.
Yes. News articles are frequently edited after publication. We monitor target URLs and emit a new record with a diff flag whenever the headline or body text changes.
Our minimum engagement typically starts with a defined historical backfill or a continuous monitoring pipeline for specific categories. Contact us to scope your exact requirements.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a full historical archive export or a real-time news monitoring feed, we build and operate the pipeline. Tell us what you need.
