We extract articles, author metadata, comment threads, and topical categories from welt.de. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Articles objects from welt.de. All fields typed and schema-versioned.
"url": "https://www.welt.de/politik/article2451234/example-news.html", "headline": "Bundesregierung beschliesst neues Gesetz", "author": "Olaf Gersemann", "published_at": "2026-05-12T08:30:00Z", "category": "Politik", "is_weltplus": false, "comment_count": 342, "tags": "['Bundestag', 'Gesetz', 'Politik']"
| # | url | headline | subheadline | author | published_at | updated_at |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Authors objects from welt.de. All fields typed and schema-versioned.
"author_id": "A19384", "name": "Robin Alexander", "profile_url": "https://www.welt.de/autor/robin-alexander/", "role": "Stellvertretender Chefredakteur", "twitter_handle": "@robinalexander_", "article_count": 1420, "latest_article_date": "2026-05-11T18:15:00Z"
| # | author_id | name | profile_url | role | bio | twitter_handle |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Comments objects from welt.de. All fields typed and schema-versioned.
"comment_id": "C938471", "article_url": "https://www.welt.de/politik/article2451234/example-news.html", "user_name": "Max Mustermann", "timestamp": "2026-05-12T09:14:00Z", "text": "Das ist eine interessante Entwicklung.", "upvotes": 45, "replies_count": 3, "is_deleted": false
| # | comment_id | article_url | user_name | user_id | timestamp | text |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Frontpage objects from welt.de. All fields typed and schema-versioned.
"position": 1, "section": "Top-News", "headline": "Eilmeldung: Wichtige Entscheidung gefallen", "url": "https://www.welt.de/eilmeldung-123.html", "is_breaking": true, "is_weltplus": false, "scraped_at": "2026-05-12T09:15:00Z"
| # | position | section | headline | url | is_breaking | time_on_page |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Categories objects from welt.de. All fields typed and schema-versioned.
"category_name": "Wirtschaft", "subcategory": "Finanzen", "url": "https://www.welt.de/wirtschaft/", "article_count": 50, "trending_topics": "['DAX', 'Inflation', 'Zinsen']", "scraped_at": "2026-05-12T09:20:00Z"
| # | category_name | subcategory | url | top_article_url | article_count | last_updated |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our Welt.de scraper handles the complexities of modern news platforms: dynamic loading, GraphQL comment APIs, paywall detection, and author metadata extraction.
Extract headlines, subheadlines, publication dates, update timestamps, and full body text for all public articles.
Accurately flag paywalled articles (WELTplus) to filter out incomplete text and maintain dataset integrity.
Capture author names, roles, biographies, social media handles, and link articles to specific journalist profiles.
Extract paginated comment threads via underlying APIs, including user names, timestamps, upvotes, and reply hierarchies.
Extract the internal taxonomy of welt.de, mapping articles to their specific categories, subcategories, and topics.
Monitor the main homepage and section fronts to track article positioning, breaking news flags, and visibility duration.
Configure high-frequency pipelines to track breaking news, headline changes, and rapid comment accumulation.
Utilise German residential proxies to ensure accurate rendering of regional content and avoid geo-blocking.
Receive normalised JSON or Parquet files directly into your data warehouse, ready for NLP or sentiment analysis.
Brief in. Clean data out.
Provide target categories, author profiles, or specific article URLs. We design the extraction schema together.
We configure Scrapy / Playwright crawlers, proxy rotation, session management, and API interceptors for welt.de.
Schema validation, null-rate checks, paywall detection accuracy, and sample text extraction before full launch.
JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.
News sites deploy strict caching, dynamic APIs, and bot protection. Here is how we ensure reliable extraction from welt.de.
Welt.de loads comments dynamically using complex API requests. We intercept these network calls to extract structured comment data directly, bypassing fragile DOM parsing and capturing full pagination.
Paid articles return truncated text or obfuscated DOM elements. Our pipeline detects WELTplus markers at the network level, ensuring you do not ingest partial paragraphs disguised as full articles.
Media sites often serve different content or trigger aggressive bot protection for non-domestic IPs. We route all welt.de requests through German residential proxies to mirror genuine local reader behaviour.
News articles evolve. Headlines change, and text is updated. Our change detection system hashes article content per run, emitting diffs when editors update a live story.
Welt.de uses different layouts for videos, live blogs, and standard articles. Our selectors use multiple fallback chains to ensure consistent data extraction regardless of the article format.
Agencies track brand mentions, executive quotes, and crisis development across articles and comment sections.
Quant funds and researchers analyse comment threads to gauge public sentiment on political decisions or corporate news.
AI teams ingest high-quality German journalistic text to train language models and improve translation algorithms.
Publishers monitor welt.de publication frequency, trending topics, and author output to benchmark editorial strategies.
Analysts track tag frequency and category article counts to identify emerging political or economic narratives.
Universities archive political coverage and opinion pieces for longitudinal studies on media bias and framing.
"Welt.de publishes thousands of articles daily, forming a critical historical and cultural dataset but extracting it cleanly requires bypassing aggressive paywall overlays and dynamic rendering."
Most teams underestimate the complexity of news scraping. Reliable extraction from welt.de requires German residential proxies, handling complex GraphQL requests for comments, and accurately separating public text from WELTplus paywalls. DataFlirt absorbs that complexity so your engineers can focus on the analysis, not the infrastructure.
Everything supported by our welt.de scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Scrapy handles crawl orchestration, deduplication, and retry logic. Playwright handles JavaScript rendering, cookie sessions, and interaction flows. Combined via scrapy-playwright middleware.
We maintain pools of residential ISP proxies specifically for the DACH region. Rotation happens per-request with sticky sessions where required to maintain API access.
Pipelines run on AWS Lambda (burst) and ECS (sustained). Airflow handles scheduling, dependency management, and SLA alerting. All state stored in managed Postgres.
Data delivered to where your team already works — no new tooling required.
About welt.de scraping, legality, and pipeline operations.
Ask us directly →Scraping publicly available factual data and headlines is generally permissible. However, full article text is subject to copyright law. Clients typically rely on fair use exemptions for internal analysis, NLP training, or academic research. DataFlirt provides the extraction infrastructure; clients are responsible for ensuring their specific use case complies with German copyright law and welt.de terms of service.
No. We do not bypass authentication walls or steal paid content. Our pipeline detects WELTplus articles and flags them in the dataset, extracting only the publicly visible metadata, headline, and preview text.
Welt.de comments are loaded dynamically. We intercept the underlying API requests to extract structured comment data, including user details, timestamps, upvotes, and full pagination, ensuring complete thread capture.
For frontpage and RSS monitoring, we can configure pipelines to run at sub-5-minute intervals, delivering new URLs or headline changes via webhook instantly.
Yes. Our change detection system compares the current article state against previous runs. We capture the 'updated_at' timestamp and can emit diffs if the headline or body text is modified by editors.
We can crawl the site's public archives and sitemaps to extract historical articles. The volume and speed depend on the depth required and the site's rate limits for historical pagination.
20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a continuous feed of breaking news or a historical archive of author publications, we scope, build, and operate the pipeline. Tell us what you need.
