We extract full-text articles, author metadata, publication timestamps, and category tags from Independent Online. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Article Content objects from iol.co.za. All fields typed and schema-versioned.
"article_id": "8f7d9a2b-1c3e-4d5f-8a9b-0c1d2e3f4a5b", "url": "https://www.iol.co.za/business-report/economy/inflation-target-shift", "headline": "Reserve Bank maintains repo rate amid inflation concerns", "author": "Sizwe Dlamini", "published_date": "2023-11-23T14:30:00Z", "category": "Business Report > Economy", "tags": "['SARB', 'Interest Rates', 'Inflation', 'Economy']"
| # | article_id | url | headline | subheadline | author | published_date |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Author Profiles objects from iol.co.za. All fields typed and schema-versioned.
"author_id": "auth_4921", "name": "Sizwe Dlamini", "role": "Senior Financial Journalist", "twitter_handle": "@SizweD_Finance", "article_count": 342, "latest_article_date": "2023-11-23T14:30:00Z"
| # | author_id | name | role | bio | twitter_handle | article_count |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Category Feeds objects from iol.co.za. All fields typed and schema-versioned.
"category_name": "News", "section": "South Africa", "article_url": "https://www.iol.co.za/news/south-africa/gauteng/power-grid-update", "headline": "Eskom announces stage 4 load shedding for the weekend", "position": 1, "scraped_at": "2023-11-24T08:15:22Z"
| # | category_name | section | article_url | headline | snippet | position |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Business Report objects from iol.co.za. All fields typed and schema-versioned.
"company_names": "['Naspers', 'Prosus', 'Tencent']", "ticker_mentions": "['NPN', 'PRX']", "financial_sentiment": "neutral", "headline": "Naspers shares hold steady despite Tencent tech curbs", "market_category": "Companies", "published_date": "2023-11-24T09:00:00Z"
| # | ticker_mentions | company_names | financial_sentiment | article_url | headline | author |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Multimedia Assets objects from iol.co.za. All fields typed and schema-versioned.
"asset_type": "image", "source_url": "https://image.iol.co.za/image/1/process/1200x800", "caption": "SARB Governor Lesetja Kganyago announces the latest interest rate decision.", "credit": "African News Agency (ANA)", "dimensions": "1200x800", "article_url": "https://www.iol.co.za/business-report/economy/inflation-target-shift"
| # | asset_id | article_url | asset_type | source_url | caption | alt_text |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our IOL scraper handles news sections, dynamic feeds, and varying article templates — with JavaScript rendering, session management, and anti-bot circumvention built in.
Extract complete article bodies, subheadlines, and embedded media across all IOL sections including News, Sport, and Entertainment.
Capture published and updated timestamps, normalised to UTC for accurate time-series analysis and media monitoring.
Extract hierarchical category data and article tags to classify content and track topic frequency over time.
Targeted extraction for IOL's Business Report section, capturing company mentions, financial data, and market-specific tags.
Compile author profiles, track publication volume per journalist, and map internal contributor networks.
Extract high-resolution image URLs, alt text, captions, and photo credits embedded within the article body.
Automatically flag IOL Premium articles and truncate datasets cleanly without breaking pipeline schemas.
Monitor category index pages at high frequency to capture new publications within minutes of going live.
Traverse sitemaps and paginated archives to build comprehensive historical NLP datasets spanning years of publications.
Brief in. Clean data out.
Provide target sections, author lists, or historical date ranges. We design the extraction schema together.
We configure Scrapy crawlers, proxy rotation, and template-specific parsing logic for different IOL sections.
Schema validation, null-rate checks, and text-encoding verification before full launch.
JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
News sites deploy varied templates and rate limits. Here is how we maintain data integrity and pipeline uptime.
IOL uses different DOM structures for Business Report, IOL Sport, and standard news. Our selector strategy uses fallback chains tailored to each section, ensuring consistent output regardless of the source template.
For real-time media monitoring, scraping category pages is inefficient. We monitor IOL's sitemaps and RSS feeds to detect new publications instantly, triggering targeted article extraction without excessive crawling.
High-volume extraction triggers rate limits. We use residential and data centre proxies rotated per request, adhering to polite crawl delays while maintaining throughput across distributed nodes.
News articles often contain irregular whitespace, embedded ads, and mixed character encodings. Our pipeline strips boilerplate HTML, removes inline advertisements, and normalises all text to UTF-8.
Every run emits structured logs to our observability stack. We alert on null-rate spikes, missing body text, schema drift, and coverage drops — and respond before you notice.
PR agencies and corporate communications teams track brand mentions, sentiment, and share of voice across South African media.
AI researchers use clean, structured South African news corpora to train regional language models and sentiment classifiers.
Quant funds parse Business Report articles to gauge market sentiment regarding JSE-listed companies and macroeconomic policy.
Media organisations analyse IOL's publication velocity, category focus, and author output to benchmark their own editorial strategies.
Risk consultancies monitor news output to track political developments, policy changes, and social unrest indicators.
Institutions maintain searchable archives of historical news coverage for legal, compliance, and academic research purposes.
"IOL publishes thousands of articles weekly across diverse sections. Extracting this at scale requires resilient infrastructure, not basic HTML parsing."
News layouts change constantly. IOL Sport looks different from Business Report. We maintain the selectors, handle the paywall logic, and manage the proxy rotation. DataFlirt absorbs the complexity so your NLP and media monitoring teams can focus on analysis — not the infrastructure.
Everything supported by our iol.co.za scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles crawl orchestration, deduplication, and retry logic. Playwright handles JavaScript rendering for dynamic content feeds. Combined via scrapy-playwright middleware.
We maintain pools of proxies rotated per-request to manage rate limits and prevent IP bans during high-volume historical archive extraction.
Pipelines run on AWS Lambda (burst) and ECS (sustained). Airflow handles scheduling, dependency management, and SLA alerting. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About iol.co.za scraping, legality, and pipeline operations.
Ask us directly →Scraping publicly available news articles is generally permissible for analysis and monitoring, provided it does not violate copyright law via mass republication. DataFlirt targets only public, non-authenticated metadata and text. We do not bypass paywalls or extract personal reader data. Clients should consult legal counsel regarding copyright and fair use for their specific applications.
We detect paywall flags in the DOM or metadata. For Premium articles, we extract the headline, author, timestamp, and available snippet, but we do not circumvent the authentication system to retrieve the gated body text. The record is flagged as paywalled in your delivery payload.
Yes. We maintain a hash index of previously scraped articles. If an article is updated with a new headline or modified text, we capture the new version and emit a diff record.
For continuous pipelines, we monitor sitemaps and category feeds at high frequency, typically achieving sub-15-minute latency from publication to delivery in your warehouse.
We extract the URLs, captions, alt text, and credits for embedded multimedia assets. We do not download the physical media files, but provide the source links for your systems to ingest.
Yes. We can traverse IOL's historical archives and sitemaps to build comprehensive back-catalogues for NLP training or historical sentiment analysis.
Absolutely. We provide a sample run of up to 500 articles as part of the pre-engagement scoping process — so you can validate schema fit and text cleanliness before signing any contract.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a historical archive dump or a continuous news-monitoring feed — we scope, build, and operate the pipeline. Tell us what you need.
