SYSTEM all green source economictimes.indiatimes.com queue 12,943 articles p99 latency 214ms dataflirt.com · scraper/economictimes-indiatimes
RUN - 84 active pipelines - economictimes.indiatimes.com live

Financial news data,
at warehouse scale.

We extract breaking news, corporate announcements, market tickers, and editorial metadata from The Economic Times. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.

Articles extracted
42.1K /day
Market updates
3.8M /24h
Author profiles
1,240 /run
Active pipelines
84
Uptime
99.98%
Data Dictionary

Every field we extract from economictimes.indiatimes.com

Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.

Complete list of extractable fields for News Articles objects from economictimes.indiatimes.com. All fields typed and schema-versioned.

article_idurlheadlinesubheadlineauthorpublished_dateupdated_datecontent_bodytagssectoret_prime_flag
news_articles
● 200 OK
"article_id": "art_98472615",
"url": "https://economictimes.indiatimes.com/markets/stocks/news/...",
"headline": "Sensex rallies 500 points on global cues",
"author": "ET Markets Web Team",
"published_date": "2026-05-12T08:30:00Z",
"tags": "['Sensex', 'Nifty', 'Stock Market']",
"et_prime_flag": false,
"sector": "Markets"
# article_idurlheadlinesubheadlineauthorpublished_date
1
2
3

Complete list of extractable fields for Market Data objects from economictimes.indiatimes.com. All fields typed and schema-versioned.

symbolcompany_nameltpchange_abschange_pctvolumemarket_cappe_ratiosectortimestamp
market_data
● 200 OK
"symbol": "RELIANCE",
"company_name": "Reliance Industries Ltd.",
"ltp": 2845.5,
"change_abs": 32.1,
"change_pct": 1.14,
"volume": 4582910,
"pe_ratio": 28.4,
"timestamp": "2026-05-12T10:15:00Z"
# symbolcompany_nameltpchange_abschange_pctvolume
1
2
3

Complete list of extractable fields for Corporate Announcements objects from economictimes.indiatimes.com. All fields typed and schema-versioned.

announcement_idcompany_namesymboldatecategorysubjectpdf_urlexchange
corporate_announcements
● 200 OK
"announcement_id": "ann_738291",
"company_name": "Infosys Ltd.",
"symbol": "INFY",
"date": "2026-05-11T16:45:00Z",
"category": "Board Meeting",
"subject": "Intimation of Board Meeting for Dividend",
"exchange": "NSE"
# announcement_idcompany_namesymboldatecategorysubject
1
2
3

Complete list of extractable fields for Mutual Funds objects from economictimes.indiatimes.com. All fields typed and schema-versioned.

scheme_nameamccategorynavnav_dateone_year_returnthree_year_returnfive_year_returnfund_sizeexpense_ratio
mutual_funds
● 200 OK
"scheme_name": "HDFC Mid-Cap Opportunities Fund",
"amc": "HDFC Mutual Fund",
"category": "Equity - Mid Cap",
"nav": 142.85,
"nav_date": "2026-05-11",
"one_year_return": 24.5,
"expense_ratio": 0.85
# scheme_nameamccategorynavnav_dateone_year_return
1
2
3

Complete list of extractable fields for Authors & Editorials objects from economictimes.indiatimes.com. All fields typed and schema-versioned.

author_nameauthor_urlbiotwitter_handlearticle_countrecent_articlestopics_coveredpublication_frequency
authors_& editorials
● 200 OK
"author_name": "Sugata Ghosh",
"author_url": "https://economictimes.indiatimes.com/sugata-ghosh",
"bio": "Associate Editor at The Economic Times covering banking and finance.",
"article_count": 842,
"topics_covered": "['Banking', 'RBI', 'Macroeconomics']",
"publication_frequency": "Weekly"
# author_nameauthor_urlbiotwitter_handlearticle_countrecent_articles
1
2
3

Capabilities

Everything you need from ET - nothing you don't

Our Economic Times scraper handles every layer of the publication: breaking news feeds, dynamic stock tickers, corporate filings, and mutual fund NAVs, with automated ad-stripping and paywall detection built in.

Full Article Extraction

Headlines, subheadlines, clean body text, publication timestamps, and author metadata - scraped with inline ads and promotional widgets stripped out.

Market Ticker Tracking

Capture Last Traded Price, volume, absolute change, and percentage change for equities, indices, and commodities across NSE and BSE.

Corporate Announcements

Extract board meeting notices, dividend declarations, and quarterly earnings reports mirrored on ET from primary exchanges.

ET Prime Flagging

Identify and flag paywalled ET Prime content automatically, ensuring your downstream NLP pipelines only ingest fully accessible text.

Clean Text Parsing

Heuristic DOM parsing removes newsletter signups, read-more links, and sponsored content blocks from the main article body.

Author & Byline Mining

Track journalist output, topics covered, and editorial bias by extracting author profiles and their complete article histories.

Mutual Fund Data

Extract daily NAVs, historical return percentages, fund sizes, and expense ratios across all tracked mutual fund schemes.

Sector & Industry Categorisation

Map articles and market movements to specific industry verticals using ET's internal taxonomy and tagging system.

Scheduled + Streaming Modes

Run historical archive exports or configure continuous pipelines at 15-minute intervals for breaking market news.

// engagement pipeline

From URL list to warehouse record

Brief in. Clean data out.

Define Scope
d 0

Provide target sections, ticker symbols, or author URLs. We design the extraction schema together.

Pipeline Build
d 2–4

We configure Scrapy crawlers, ad-stripping heuristics, and JavaScript renderers for dynamic market data.

Validation & QA
d 4–6

Schema validation, null-rate checks, and text-cleanliness audits before full launch.

Delivery
ongoing

JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.

Under the hood

How our ET pipeline handles the hard parts

News publishers optimise for ad revenue, resulting in messy DOM structures. Here is how we extract clean data.

pipeline-monitor · economictimes.indiatimes.com · live ● active
// fingerprinting
Identity rotation
TLS fingerprintrandomised
User-agentrotated
IP poolresidential
Challenges blocked0
// pagination
Page coverage
48,291 pages queued running
// observability
Pipeline health
99.9%
uptime
142ms
p99 lat
0.3%
null rate
2
alerts
Text normalisation
Ad and widget stripping

The Economic Times embeds related article links, newsletter forms, and sponsored native ads directly within paragraph blocks. We use heuristic DOM parsing to identify and strip these non-editorial elements, delivering clean, contiguous strings ready for NLP ingestion.

JavaScript rendering
Dynamic market data hydration

Market tickers and live stock charts rely heavily on client-side JavaScript. We run Playwright browser sessions to execute scripts and hydrate these widgets, capturing real-time pricing data that static HTTP clients cannot see.

Paywall detection
Handling ET Prime content

Articles tagged as ET Prime restrict full text access to logged-in subscribers. Our pipeline detects paywall boundaries, flags the record as gated, and extracts the available preview text without breaking the crawl flow.

Anti-bot layer
Bypassing rate limits

High-frequency scraping of news sites triggers Akamai and Cloudflare blocks. We distribute requests across Indian residential IP pools with randomised intervals to maintain uninterrupted access during breaking news events.

Schema stability
Resilient selectors for layout variations

ET uses different templates for standard articles, live blogs, photo galleries, and market updates. We maintain separate fallback selector chains for each template type to ensure consistent extraction regardless of the editorial format.

Applications

Who uses ET data - and how

Teams across industries use economictimes.indiatimes.com data to build competitive products and smarter operations.

01
Algorithmic Trading Sentiment

Quant funds ingest clean article text to run sentiment analysis models, correlating news tone with short-term equity price movements.

02
Market Research & Sector Analysis

Analysts aggregate corporate announcements and sector-specific news to track industry trends and competitor activity.

03
Media Monitoring

PR agencies and corporate communications teams track brand mentions, executive quotes, and editorial coverage in real time.

04
NLP & LLM Training

AI researchers use historical financial news archives to train domain-specific language models for the Indian market.

05
Wealth Management Integration

Fintech platforms source mutual fund NAVs and market indices to populate client dashboards and portfolio trackers.

06
Event-Driven Investing

Traders monitor board meeting notices and earnings report publications to execute strategies around corporate actions.

Why DataFlirt

"The Economic Times holds the definitive record of Indian corporate activity and market sentiment, but extracting clean text from its ad-heavy DOM requires precision engineering."

News sites prioritise ad delivery and dynamic layout shifts over structured markup. Extracting reliable financial data and article text requires JavaScript rendering to hydrate market widgets, heuristic text extraction to strip inline promotions, and proxy rotation to bypass rate limits. DataFlirt manages this pipeline end-to-end so your quantitative models receive clean, normalised text and pricing signals.

Technical Spec

Economic Times scraper - technical capabilities

Everything supported by our economictimes.indiatimes.com scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.

JavaScript rendering
Playwright sessions required for live market tickers and dynamic charts
Supported
CAPTCHA bypass
Automated solver integration for WAF challenges during high-volume crawls
Supported
Residential proxy rotation
ISP-grade residential IPs from Indian pools to avoid regional blocks
Supported
Ad and widget stripping
Heuristic removal of inline promotions and newsletter forms from article text
Supported
Historical archive extraction
Pagination through date-based archives for backtesting datasets
Supported
Change detection (diffs)
Hash-based diff to track headline updates and article revisions
Supported
Webhook delivery
HTTP POST per article for real-time news monitoring
Supported
ET Prime full text
Gated premium articles require paid account credentials
Partial
User portfolio data
Personalised watchlists and portfolio tracking require user authentication
Partial
Infrastructure

Infrastructure powering the ET pipeline

Open-source tooling on proven cloud infra — no vendor lock-in, full observability.

ScrapyPlaywrightPython 3.12RedisPostgreSQLApache AirflowAWS LambdaS3CloudWatch2CaptchaCapSolverResidential ProxiesDockerKubernetesGrafanaPrometheus
Scrapy + Playwright Stack

Scrapy handles crawl orchestration and deduplication. Playwright handles JavaScript rendering for dynamic market widgets and lazy-loaded article content.

Residential Proxy Infrastructure

We maintain pools of residential ISP proxies across Indian regions. Rotation happens per-request to bypass rate limits imposed by CDNs and WAFs.

Cloud-Native Orchestration

Pipelines run on AWS Lambda and ECS. Airflow handles scheduling, dependency management, and SLA alerting. All state stored in managed Postgres.

Output & Delivery

Your data, your destination

Data delivered to where your team already works — no new tooling required.

JSON
Newline-delimited or nested - schema versioned per run
CSV
Flat file with typed columns - Excel/Sheets compatible
XLS
Excel format for business analysts
Parquet
Columnar format for BigQuery, Snowflake, Athena
AWS S3
Direct bucket delivery - compatible with any data lake
Webhook
HTTP POST per record for real-time downstream processing
API
REST endpoint to query extracted records
Postgres
Upsert into your existing schema with conflict resolution
BigQuery
Streamed directly into your dataset with schema auto-detect
Snowflake
Stage + COPY INTO workflow - incremental or full-replace
S3
Direct bucket delivery — compatible with any data lake
// faq

Common questions.

About economictimes.indiatimes.com scraping, legality, and pipeline operations.

Ask us directly →
Is scraping The Economic Times legal?

Scraping publicly available factual data, such as market prices and headlines, is generally permissible. DataFlirt targets only public, non-authenticated content. We do not extract personal data or bypass ET Prime paywalls using stolen credentials. Clients must ensure their downstream use cases, such as republishing full text, comply with copyright laws.

Do you extract full text from ET Prime articles?

No. ET Prime articles are paywalled. Our pipeline identifies these articles, flags them with a boolean field, and extracts only the publicly visible preview text and metadata.

How accurate is your ad and widget stripping?

We maintain strict heuristic rules to remove newsletter signups, 'Read Also' blocks, and sponsored native ads from the article body. Our QA process tracks null rates and string continuity to ensure high-fidelity text output for NLP models.

How fresh is the market data?

For market tickers, we can configure real-time streaming pipelines that achieve sub-5-minute latency during trading hours. News feeds can be polled at 15-minute intervals.

Can I get historical news archives?

Yes. We can configure one-off backfills to extract historical articles based on specific date ranges, tags, or author profiles, subject to the site's pagination limits.

What is the minimum viable engagement?

Our smallest packages start at tracking specific sections or ticker lists with daily delivery. For historical backfills or real-time webhook feeds, we price based on compute volume and delivery frequency.

$ dataflirt scope --new-project --source=economictimes.indiatimes.com ready

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a historical archive of corporate announcements or a real-time feed of breaking market news, we scope, build, and operate the pipeline. Tell us what you need.

hello@dataflirt.com · Bengaluru · IST · typical reply < 4h
Related Scrapers

More in news and media

Services

Data Extraction for Every Industry

View All Services →