Scrapy Splash is a legacy integration that connects the Scrapy framework to Splash, a lightweight, scriptable browser based on Qt WebKit. It was historically the standard way to render JavaScript-heavy pages without breaking Scrapy's asynchronous event loop. Today, it is largely obsolete—replaced by modern CDP-based integrations like Scrapy-Playwright—but remains embedded in thousands of older enterprise data pipelines.
How Python developers historically solved the JavaScript rendering problem before headless Chrome became the universal standard.
Ask a DataFlirt engineer →Scrapy Splash routes Scrapy requests through a separate Splash HTTP API server. The Splash server renders the DOM, executes custom Lua scripts to interact with the page, and returns the final HTML back to the Scrapy spider. It's fast but runs an outdated WebKit engine that modern anti-bot systems flag instantly.
scrapy-splash) that routes requests to it. Instead of Scrapy fetching a URL directly, it asks the Splash server to fetch it, render the JavaScript, and return the final HTML.
SplashRequest, the middleware intercepts it and forwards it to the Splash server's /render.html or /execute endpoint. The Splash server loads the page in its Qt WebKit engine, waits for network idle or a specific timeout, and sends the rendered DOM back to Scrapy as a standard HtmlResponse, allowing the spider to parse it using normal CSS/XPath selectors.
/execute endpoint. This created massive developer friction: Python engineers had to maintain complex Lua strings embedded inside their Python spiders, making debugging and syntax highlighting nearly impossible.
scrapy-playwright for JavaScript rendering. Playwright allows us to write interaction logic natively in Python async coroutines, supports modern Chromium/WebKit engines, and integrates seamlessly with our anti-bot fingerprinting stack.
Splash used a shared Qt event loop rather than spawning isolated browser processes. This made it highly concurrent but vulnerable to memory leaks and modern fingerprinting.
A standard Splash request involves sending a Lua script to the Splash API, which controls the WebKit instance before returning the HTML.
The primary failure modes that drive engineering teams to migrate legacy Scrapy Splash pipelines to modern CDP-based alternatives.
upgrading to Playwright without rewriting the spider.
DataFlirt migrated its last internal Splash cluster in 2023. The transition to scrapy-playwright requires swapping the middleware and replacing Lua scripts with Python async coroutines. While Playwright consumes more RAM per concurrent page, the reduction in anti-bot blocks and the elimination of Lua-related technical debt yields a massive net positive for pipeline reliability.
Key configuration changes when moving a Scrapy project off Splash.
Anti-bot shifts, scraping infrastructure updates, dataset delivery patterns, and business outcomes from our pipelines. Short, technical, no fluff.
Common questions about Scrapy Splash, its legacy status, and how to modernize JavaScript-heavy Scrapy pipelines.
Ask us directly →scrapy-playwright for modern JavaScript rendering. Building a new pipeline on Splash today is strongly discouraged.Request yields to use playwright=True instead of meta={'splash': ...} and rewrite any custom Lua interaction scripts into Python async functions.20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a one-off catalogue dump or a continuous feed across millions of records — we scope, build, and operate the pipeline.