We extract breaking news, editorial content, author profiles, and public comment threads from news.com.au. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Article Data objects from news.com.au. All fields typed and schema-versioned.
"article_id": "nca-123456789", "headline": "RBA holds interest rates steady at 4.35pc", "author": "Finance Reporter", "publish_date": "2026-05-12T14:30:00Z", "category": "Finance", "tags": "['RBA', 'Interest Rates', 'Economy']"
| # | article_id | url | headline | subheadline | author | publish_date |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Author Profiles objects from news.com.au. All fields typed and schema-versioned.
"author_id": "auth-9876", "name": "Jane Doe", "role": "Senior Political Reporter", "twitter_handle": "@janedoe_news", "article_count": 342, "profile_url": "https://www.news.com.au/author/jane-doe"
| # | author_id | name | role | bio | twitter_handle | article_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Comment Threads objects from news.com.au. All fields typed and schema-versioned.
"comment_id": "cmt-55432", "article_id": "nca-123456789", "user_name": "AusReader99", "comment_text": "Expected outcome given the inflation data.", "upvotes": 45, "timestamp": "2026-05-12T15:01:22Z"
| # | comment_id | article_id | user_name | comment_text | timestamp | upvotes |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Category Feeds objects from news.com.au. All fields typed and schema-versioned.
"category_name": "Sport", "sub_category": "AFL", "feed_url": "https://www.news.com.au/sport/afl", "article_count": 50, "last_updated": "2026-05-12T16:00:00Z", "trending_topics": "['Collingwood', 'Trade Draft']"
| # | category_name | sub_category | top_stories | trending_topics | feed_url | article_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Multimedia Metadata objects from news.com.au. All fields typed and schema-versioned.
"media_id": "vid-88776", "media_type": "video", "url": "https://video.news.com.au/v/123.mp4", "caption": "Press conference following RBA decision", "duration": 124, "format": "mp4"
| # | media_id | article_id | media_type | url | caption | credit |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our pipeline handles the News Corp network architecture: dynamic paywall detection, JavaScript rendered comment widgets, and infinite scroll category feeds.
Clean text extraction from article bodies, stripping out inline advertisements, related story widgets, and newsletter signup forms.
Map articles to specific journalists. Extract author bios, social handles, and historical publication counts.
Extract user generated content from third party commenting widgets. Capture upvotes, downvotes, and nested reply structures.
Convert relative publish times into absolute UTC timestamps. Track both initial publication and subsequent update times.
Extract primary categories, subcategories, and keyword tags assigned to each article for precise content classification.
Capture image URLs, video embed links, captions, and photographer credits embedded within the article body.
High frequency polling on homepage and category feeds to capture breaking news alerts within minutes of publication.
Automatically detect and flag articles locked behind the News+ premium subscription tier.
Monitor live blogs and developing stories. Emit diffs when headlines change or new paragraphs are added to existing URLs.
Brief in. Clean data out.
Provide target categories, author profiles, or keyword sets. We design the extraction schema together.
We configure Scrapy crawlers, JavaScript rendering for comment widgets, and proxy rotation for news.com.au.
Schema validation, null rate checks, and text normalisation before full launch.
JSON, CSV, or Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
News publishers utilise aggressive caching and dynamic rendering. Here is how we maintain data integrity.
Comments and embedded social media posts on news.com.au rely heavily on client side rendering. We use Playwright to execute JavaScript and wait for network idle states, ensuring we capture content that basic HTTP clients miss.
Article bodies are littered with inline ads, related link blocks, and newsletter embeds. Our parsers isolate the core editorial text, delivering clean paragraphs without the layout noise.
Category feeds and search results use infinite scroll mechanisms. We simulate browser scroll events and intercept background API calls to extract the full historical feed.
News Corp employs edge protection to block aggressive scraping. We route requests through Australian residential proxies with randomised delays to maintain high throughput without triggering blocks.
Media layouts change frequently for special events or breaking news. We use fallback selector chains targeting semantic HTML and JSON LD blocks to ensure continuous data flow during layout shifts.
PR firms and corporate communications teams track brand mentions, sentiment, and share of voice across the News Corp network.
Machine learning teams ingest clean, locally contextualised Australian English text corpora to train regional LLMs and NLP models.
Financial analysts process article tone and comment thread reactions to gauge public sentiment on economic policies and market events.
Rival publishers monitor publication velocity, author output, and trending topics to optimise their own editorial strategies.
Researchers aggregate keyword frequencies and tag usage over time to identify emerging social and political trends.
Investigative teams map relationships between journalists, topics, and cited sources to understand editorial bias and influence.
"News.com.au drives the Australian daily narrative, but transforming unstructured editorial content into queryable time series data requires dedicated infrastructure."
Media scraping involves bypassing aggressive caching layers, handling dynamic paywalls, and normalising inconsistent editorial formats. DataFlirt manages the proxy rotation, JavaScript execution, and schema validation so your data science team receives clean text corpora instead of broken HTML.
Everything supported by our news.com.au scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles crawl orchestration and deduplication. Playwright handles JavaScript rendering for dynamic comment widgets and infinite scroll feeds.
We route requests through Australian residential IPs to prevent edge blocking and ensure accurate regional content delivery.
Pipelines run on AWS Lambda and ECS. Airflow handles scheduling for high frequency breaking news polling. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About news.com.au scraping, legality, and pipeline operations.
Ask us directly →No. DataFlirt only extracts publicly available information. We detect and flag articles locked behind the News+ premium tier, but we do not bypass authentication walls or use subscriber credentials to access gated content.
For monitored category feeds and the homepage, we can configure polling intervals as low as 5 minutes. New articles are extracted and pushed via Webhook or S3 immediately upon detection.
Yes. We use Playwright to execute the JavaScript required to load third party commenting widgets. We extract the commenter name, text, timestamp, and vote counts.
Our change detection system maintains a hash of the article body. When polling a live blog URL, we compare the current state against the previous run and only emit a new record if the content has changed.
Yes. Our parsing logic specifically targets the editorial content blocks. We strip out inline advertisements, related story links, newsletter signups, and navigation elements, delivering clean paragraphs of text.
We can execute backfills by crawling category archives and sitemaps. The depth of historical data depends on the publisher's site structure and archive availability.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one off historical text corpus or a continuous feed of breaking news and comments we scope, build, and operate the pipeline. Tell us what you need.
