We extract institutional profiles, financial aid metrics, post-graduation earnings, and admission statistics from the US College Scorecard. Delivered as clean JSON, CSV, or Parquet to S3, BigQuery, or Snowflake on your cadence.
Structured, schema-consistent data across all major object types — delivered clean, typed, and ready to query.
Complete list of extractable fields for Institution Overview objects from collegescorecard.ed.gov. All fields typed and schema-versioned.
"unitid": "166027", "institution_name": "Harvard University", "city": "Cambridge", "state": "MA", "institution_type": "Private nonprofit", "student_population": 7240, "hbcu_status": false
| # | unitid | opeid | institution_name | city | state | zip_code |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Admissions & Test Scores objects from collegescorecard.ed.gov. All fields typed and schema-versioned.
"unitid": "166027", "admission_rate": 0.04, "sat_math_25th": 740, "sat_math_75th": 800, "sat_read_25th": 720, "sat_read_75th": 780, "open_admissions_policy": false
| # | unitid | admission_rate | sat_math_25th | sat_math_75th | sat_read_25th | sat_read_75th |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Cost & Financial Aid objects from collegescorecard.ed.gov. All fields typed and schema-versioned.
"unitid": "166027", "avg_net_price": 18030, "tuition_in_state": 54002, "tuition_out_state": 54002, "pct_pell_grant": 0.2, "pct_federal_loan": 0.03, "avg_grant_aid": 61000
| # | unitid | avg_net_price | tuition_in_state | tuition_out_state | pct_pell_grant | pct_federal_loan |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Graduation & Retention objects from collegescorecard.ed.gov. All fields typed and schema-versioned.
"unitid": "166027", "retention_rate_ft": 0.99, "grad_rate_150_pct": 0.98, "grad_rate_pell": 0.97, "grad_rate_non_pell": 0.98, "transfer_out_rate": 0.01, "completion_4yr": 0.86
| # | unitid | retention_rate_ft | retention_rate_pt | grad_rate_150_pct | grad_rate_pell | grad_rate_non_pell |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Complete list of extractable fields for Earnings & Debt objects from collegescorecard.ed.gov. All fields typed and schema-versioned.
"unitid": "166027", "median_earnings_10yr": 136700, "pct_earning_above_hs": 0.92, "median_debt_completers": 12000, "default_rate_3yr": 0.0, "monthly_loan_payment": 124, "repayment_rate_1yr": 0.95
| # | unitid | median_earnings_10yr | pct_earning_above_hs | median_debt_completers | median_debt_noncompleters | default_rate_3yr |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 |
Our pipeline handles the deeply nested JSON structures and historical data schemas of the College Scorecard, delivering flat, queryable tables directly to your warehouse.
Extract core identifiers (UNITID, OPEID), location data, institution type, and demographic distributions across all 6,400+ active institutions.
Capture published tuition rates, average net price by family income band, and overall cost of attendance metrics.
Extract median post-graduation earnings mapped to specific CIP (Classification of Instructional Programs) codes and degree levels.
Track median debt loads for completers vs non-completers, cohort default rates, and estimated monthly loan payments.
Capture admission rates, yield rates, and SAT/ACT percentile distributions for incoming freshman cohorts.
Extract student body composition by race, ethnicity, gender, and first-generation status.
Track first-year retention rates and completion rates at 150 percent of normal time, segmented by Pell Grant status.
Extract data across multiple academic years to track trends in tuition inflation and earnings growth.
Automated pipeline runs sync new data releases from the Department of Education directly into your database.
Brief in. Clean data out.
Specify the metrics, academic years, and institution types you need. We design the target schema.
We configure extraction logic to handle the Scorecard's nested JSON responses and normalise historical field changes.
Schema validation, null-rate checks, and data type enforcement before full launch.
JSON, CSV, or Parquet pushed to your S3 bucket, BigQuery dataset, or Snowflake stage.
The College Scorecard dataset is massive and deeply nested. Here is how we process it into usable formats.
The Scorecard API returns deeply nested JSON objects where metrics are buried under year, category, and sub-category keys. We flatten these structures into wide, columnar formats suitable for SQL querying.
The Department of Education frequently renames fields or changes metric definitions across academic years. Our pipeline maps legacy field names to a unified, version-controlled schema.
Government endpoints enforce strict rate limits. We implement distributed token bucket algorithms and exponential backoff to ensure reliable extraction without triggering firewall blocks.
We handle the API's pagination cursors to ensure all 6,400+ institutions and their historical records are extracted without missing pages or silent failures.
Government APIs often return mixed types (e.g., "PrivacySuppressed" strings in numeric fields). We cast data strictly, converting suppressed values to nulls and enforcing numeric types for downstream analytical use.
College search engines and application portals enrich their platforms with authoritative cost and outcome data.
Think tanks and researchers analyse the correlation between student debt loads, institution types, and long-term earnings.
Fintech lenders use institutional default rates and median earnings data to refine risk models for private student loans.
Universities track competitor tuition rates, yield rates, and graduation metrics to inform strategic planning.
Platforms calculate the ROI of specific degree programmes by comparing upfront costs against 10-year median earnings.
State agencies track graduate retention and earnings to assess the regional economic impact of higher education institutions.
"The College Scorecard contains the definitive dataset on higher education ROI in the US — but mapping its deeply nested structures into flat analytical tables requires serious engineering."
Most teams waste weeks untangling the Scorecard's nested JSON responses, handling rate limits from government endpoints, and normalising historical field changes. DataFlirt absorbs that complexity so your data engineers can focus on analysis, not pipeline maintenance.
Everything supported by our collegescorecard.ed.gov scraper — rendered SPA elements, auth walls, rate-limit evasion and beyond.
Open-source tooling on proven cloud infra — no vendor lock-in, full observability.
Custom Python 3.12 processing layers flatten complex JSON dictionaries and cast mixed-type fields into strict warehouse-ready formats.
Redis-backed distributed rate limiting ensures extraction stays within government API quotas while maximising throughput.
Airflow orchestrates extraction, transformation, and load sequences, pushing clean Parquet files directly to S3 or BigQuery.
Data delivered to where your team already works — no new tooling required.
About collegescorecard.ed.gov scraping, legality, and pipeline operations.
Ask us directly →Yes. The College Scorecard dataset is public government data provided by the US Department of Education. We extract only aggregate, publicly available metrics and respect API rate limits. No personal identifiable information (PII) is accessed or extracted.
The Department of Education typically releases major updates annually, with occasional minor revisions throughout the year. We can configure pipelines to poll for changes or run on a defined schedule to capture updates as they occur.
Yes. The dataset frequently uses 'PrivacySuppressed' strings in numeric fields to protect small cohort identities. Our pipeline automatically converts these to NULL values to maintain strict numeric typing for your warehouse.
Yes. We can target the most recent reporting year or extract historical time-series data spanning multiple academic years to support trend analysis.
We build custom transformation layers that flatten the nested dictionaries. We map fields like earnings by CIP code or net price by income band into wide relational tables or distinct dimension tables based on your schema preference.
Yes. The dataset includes earnings metrics mapped to specific fields of study (CIP codes) and credential levels (e.g., Bachelor's vs Master's). We extract these relationships intact.
Due to the width of the flattened dataset (often hundreds of columns), we strongly recommend Parquet format delivered to S3, BigQuery, or Snowflake for optimal query performance.
20-minute scoping call. Pilot dataset within the week. Production within two. Stop wrestling with nested JSON and rate limits. We build and manage the extraction infrastructure so you can query clean College Scorecard data immediately.
