SYSTEM all green source libertymutual.com queue 14,293 pages p99 latency 312ms dataflirt.com · scraper/libertymutual-com
RUN · 31 active pipelines · libertymutual.com live

Liberty Mutual data,
at warehouse scale.

We extract agent directories, branch locations, coverage specifications, and quote engine outputs from Liberty Mutual. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.

Agents extracted
8,412 /run
Quote flows executed
12,941 /day
Branch locations
2,104 /run
Active pipelines
31
Uptime
99.94%
Data Dictionary

Every field we extract from libertymutual.com

Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.

Complete list of extractable fields for Agent Directory objects from libertymutual.com. All fields typed and schema-versioned.

agent_idfirst_namelast_nametitlephone_numberemail_addressstreet_addresscitystatezip_codelanguages_spokenspecialtiesprofile_url
agent_directory
● 200 OK
"agent_id": "LM-84921",
"first_name": "Sarah",
"last_name": "Jenkins",
"title": "Lead Sales Representative",
"phone_number": "617-555-0198",
"city": "Boston",
"state": "MA",
"languages_spoken": "['English', 'Spanish']"
# agent_idfirst_namelast_nametitlephone_numberemail_address
1
2
3

Complete list of extractable fields for Branch Locations objects from libertymutual.com. All fields typed and schema-versioned.

branch_idbranch_namestreet_addresscitystatezip_codephone_numberhours_mondayhours_tuesdaylatitudelongitudeservices_offered
branch_locations
● 200 OK
"branch_id": "BR-0421",
"branch_name": "Liberty Mutual - Downtown Boston",
"street_address": "175 Berkeley St",
"city": "Boston",
"state": "MA",
"hours_monday": "09:00-17:00",
"latitude": 42.3496,
"longitude": -71.0736
# branch_idbranch_namestreet_addresscitystatezip_code
1
2
3

Complete list of extractable fields for Coverage Details objects from libertymutual.com. All fields typed and schema-versioned.

coverage_typecategorystate_availabilitybase_featuresoptional_add_onsdiscount_eligibilityminimum_limitsmaximum_limitsdescriptionurl
coverage_details
● 200 OK
"coverage_type": "Better Car Replacement",
"category": "Auto",
"state_availability": "['MA', 'NY', 'CT', 'RI']",
"discount_eligibility": "['Multi-Policy', 'Safe Driver']",
"minimum_limits": "State Minimum",
"description": "Replaces your totaled car with a model that is one year newer."
# coverage_typecategorystate_availabilitybase_featuresoptional_add_onsdiscount_eligibility
1
2
3

Complete list of extractable fields for Quote Output objects from libertymutual.com. All fields typed and schema-versioned.

quote_idzip_codevehicle_yearvehicle_makevehicle_modeldriver_agequoted_premium_monthlyquoted_premium_6moliability_limitcomprehensive_deductiblecollision_deductibleapplied_discounts
quote_output
● 200 OK
"zip_code": "02116",
"vehicle_year": 2022,
"vehicle_make": "Toyota",
"vehicle_model": "Camry",
"driver_age": 34,
"quoted_premium_monthly": 142.5,
"liability_limit": "100/300/100",
"applied_discounts": "['Paperless', 'Anti-Theft']"
# quote_idzip_codevehicle_yearvehicle_makevehicle_modeldriver_age
1
2
3

Complete list of extractable fields for Career Listings objects from libertymutual.com. All fields typed and schema-versioned.

job_idtitledepartmentlocationremote_eligibleposted_datedescriptionrequirementssalary_minsalary_maxapply_url
career_listings
● 200 OK
"job_id": "REQ-99214",
"title": "Senior Actuarial Analyst",
"department": "Global Retail Markets",
"location": "Boston, MA",
"remote_eligible": true,
"posted_date": "2026-05-10",
"salary_min": 115000,
"salary_max": 145000
# job_idtitledepartmentlocationremote_eligibleposted_date
1
2
3

Capabilities

Extract Liberty Mutual data without the WAF friction

Our infrastructure handles the heavy lifting of insurance scraping: stateful multi-step form execution, WAF circumvention, and geo-targeted proxy routing to extract accurate, state-level pricing and agent data.

Agent Directory Extraction

Extract names, contact information, office locations, and licensed states for every registered Liberty Mutual agent nationwide.

Multi-Step Quote Execution

Navigate complex, JavaScript-heavy quote engines for auto and home insurance based on predefined input matrices.

Branch Location Mapping

Capture geocoded coordinates, operating hours, and service availability for all physical retail locations.

Coverage & Policy Specs

Extract coverage descriptions, limits, and optional add-ons, mapped by state availability and product category.

Discount & Bundle Tracking

Monitor advertised discounts, multi-policy bundle rates, and telematics program requirements across different zip codes.

State-Level Routing

Handle strict geo-IP redirects and zip code logic using state-specific residential proxy nodes to ensure accurate regional data.

Anti-Bot Circumvention

Bypass strict insurance WAFs and rate limits using realistic browser fingerprints and automated CAPTCHA solving.

Form Validation Handling

Manage dynamic DOM changes and JavaScript-based form validation errors during automated quote runs.

Scheduled + Streaming Modes

Run daily agent directory syncs or configure continuous pipeline runs for competitor rate monitoring.

// engagement pipeline

From target zip codes to warehouse tables

Brief in. Clean data out.

Define Scope
d 0

Provide zip code lists, vehicle matrices, or directory targets. We design the extraction schema together.

Pipeline Build
d 2–4

We configure Playwright scripts, state-level proxy rotation, and form-state management for libertymutual.com.

Validation & QA
d 4–6

Schema validation, WAF block-rate checks, and sample quote outputs before full launch.

Delivery
ongoing

JSON / CSV / Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage on agreed cadence.

Under the hood

How our pipeline handles insurance WAFs and SPAs

Liberty Mutual protects its quote engines and directories with strict bot mitigation. Here is how we maintain reliable extraction.

pipeline-monitor · libertymutual.com · live ● active
// fingerprinting
Identity rotation
TLS fingerprintrandomised
User-agentrotated
IP poolresidential
Challenges blocked0
// pagination
Page coverage
48,291 pages queued running
// observability
Pipeline health
99.9%
uptime
142ms
p99 lat
0.3%
null rate
2
alerts
WAF Mitigation
Residential proxies + TLS fingerprinting

Insurance carriers use aggressive WAFs (like Akamai or DataDome). We use residential ISP proxies with realistic TLS and browser fingerprints to blend in with legitimate consumer traffic.

Complex State
Playwright for multi-step forms

Quote engines are Single Page Applications (SPAs) requiring sequential data entry. We use full Playwright sessions to manage cookies, local storage, and dynamic DOM updates throughout the quote flow.

Geo-Targeting
State-specific IP routing

Insurance pricing and coverage availability are strictly regulated by state. Our proxy rotation engine binds requests to IPs matching the target zip code to prevent regional redirects.

Dynamic Validation
Adaptive DOM parsing

Form fields change based on previous inputs (e.g., adding a specific vehicle year triggers new questions). Our crawlers adapt to these dynamic DOM trees to ensure complete data entry.

Change Detection
Hash-based diffing for directories

Agent directories change slowly. We maintain a hash index of last-seen values per agent, pushing only diffs to reduce your storage bloat and downstream processing load.

Applications

Who uses Liberty Mutual data — and how

Teams across industries use libertymutual.com data to build competitive products and smarter operations.

01
Competitor Rate Monitoring

Actuarial and pricing teams track base quote changes across specific zip codes and demographic profiles to benchmark market positioning.

02
Distribution Network Mapping

Insurtechs and carriers analyze agent density, language capabilities, and branch footprints to identify underserved geographic regions.

03
Product & Coverage Benchmarking

Product managers compare optional add-ons, limits, and state-level availability against their own insurance offerings.

04
Discount Analysis

Marketing teams monitor promotional bundling offers, telematics program discounts, and affinity group rates.

05
Regulatory Compliance

Compliance officers audit public-facing coverage descriptions and state-specific disclosures for industry analysis.

06
Talent Acquisition

Recruiting teams track hiring velocity, remote role distribution, and salary bands via career portal scraping.

Why DataFlirt

"Insurance pricing is highly localised and hidden behind complex quote engines. Querying it at scale requires executing thousands of multi-step forms daily."

Extracting data from Liberty Mutual requires navigating strict WAFs, managing stateful multi-step JavaScript forms, and routing requests through state-specific residential IPs. DataFlirt handles the proxy rotation, session management, and DOM parsing so your analysts receive clean, normalised datasets without managing the infrastructure.

Technical Spec

Liberty Mutual scraper — technical capabilities

Everything supported by our libertymutual.com scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.

JavaScript rendering
Full Playwright sessions required for SPA quote engines and dynamic directories
Supported
Multi-step form execution
Sequential data entry for complex auto and home quote generation
Supported
State-specific proxy routing
Requests routed through IPs matching the target zip code
Supported
Agent directory pagination
Recursive crawling through state and city-level agent lists
Supported
CAPTCHA bypass
Automated solver integration for WAF challenges during high-volume runs
Supported
Quote engine extraction
Capture final premium numbers and applied discounts based on input matrices
Supported
Change detection (diffs)
Only emit records with changed fields since last run for directories
Supported
Customer policy details
Individual policy limits, billing history, and personal data
Partial
Claims history
Historical claims data gated behind the authenticated customer portal
Partial
Infrastructure

Infrastructure powering the Liberty Mutual pipeline

Open-source tooling on proven cloud infra — no vendor lock-in, full observability.

ScrapyPlaywrightPython 3.12RedisPostgreSQLApache AirflowAWS LambdaS3CloudWatch2CaptchaCapSolverResidential ProxiesDockerKubernetesGrafanaPrometheus
Stateful Form Execution

Playwright handles the complex, multi-step JavaScript forms required by quote engines, maintaining session state across dynamic DOM changes and validation checks.

Geo-Targeted Proxies

We maintain pools of residential ISP proxies mapped to specific US states, ensuring that region-locked content and pricing engines return accurate local data.

Cloud-Native Orchestration

Pipelines run on AWS ECS with Airflow handling scheduling, dependency management, and SLA alerting. All state and input matrices are stored in managed Postgres.

Output & Delivery

Your data, your destination

Data delivered to where your team already works — no new tooling required.

JSON
Nested output capturing complex coverage arrays and form states
CSV
Flat file with typed columns for quick analyst review
XLS
Excel compatible export for business stakeholders
Parquet
Columnar format optimized for BigQuery and Snowflake
AWS S3
Direct bucket delivery matching your internal partition scheme
Webhook
HTTP POST per quote record for real-time monitoring
API
REST endpoint to query the latest directory snapshots
PostgreSQL
Direct upsert into your existing relational schema
S3
Direct bucket delivery — compatible with any data lake
// faq

Common questions.

About libertymutual.com scraping, legality, and pipeline operations.

Ask us directly →
Is scraping Liberty Mutual legal?

Scraping publicly available information, such as agent directories, branch locations, and public quote engine outputs, is generally permissible. DataFlirt targets only public, non-authenticated data. We do not extract personal customer data, circumvent authentication walls, or scrape private claims history. Clients should review Liberty Mutual's ToS and consult legal counsel for specific use cases.

How do you handle the multi-step quote forms?

We use Playwright to execute full browser sessions, programmatically entering data into each step of the form just as a user would. We map your input matrix (e.g., specific vehicle models, driver ages, zip codes) to the required fields and capture the final premium output.

Can you scrape quotes for specific zip codes?

Yes. You provide the input matrix (zip codes, vehicle types, coverage limits), and our pipeline executes the quote flow for those specific parameters, routing the request through a proxy in the matching state.

How do you bypass WAF protections?

We use state-targeted residential ISP proxies combined with realistic browser fingerprints and request timing modelled on human behaviour. If a CAPTCHA is presented, our automated solvers clear the challenge to proceed with the extraction.

Can I get the agent directory for all 50 states?

Yes. We recursively crawl the state and city-level directories to build a complete, normalised dataset of all listed agents, including their contact details and licensed regions.

How fresh is the data?

Agent directories and branch locations are typically refreshed weekly. Quote monitoring pipelines can be scheduled daily or on-demand based on your input matrix volume.

Do you scrape customer portals?

No. We do not support scraping authenticated customer portals, personal policy documents, or private claims data. We only extract data accessible without a user login.

$ dataflirt scope --new-project --source=libertymutual.com ready

Tell us what
to extract.
We do the rest.

20-minute scoping call. Pilot dataset within the week. Production within two. Whether you need a full agent directory extraction or continuous quote monitoring across 5,000 zip codes — we scope, build, and operate the pipeline. Tell us what you need.

hello@dataflirt.com · Bengaluru · IST · typical reply < 4h
Services

Data Extraction for Every Industry

View All Services →