MGNREGA Assets Scraper (Bihar)
A concurrent scraping + processing pipeline for MGNREGA asset records across Bihar's districts.
Overview
A resilient data-engineering pipeline for scraping and processing Bihar MGNREGA asset records at district scale, with checkpointed runs and retry logic for unstable endpoints.
Data and Sources
- Bhuvan and MGNREGA-linked public records assembled into district-level raw and processed directories.
- Creation-time sheets and checkpoint files used to support resumable long runs.
- Structured intermediate layers that separate raw capture from categorization and merged outputs.
Workflow
- Parallelized extraction with capped workers and controlled retry/backoff patterns.
- Checkpoint-aware execution to resume failed jobs and avoid expensive reruns.
- Merge and categorize district outputs into clean processed tables for analysis.
Outputs
- District-level datasets ready for policy analysis and infrastructure auditing.
- Smoke-test modes and reproducible run scripts for operational reliability.
- A maintainable package structure for extending beyond Bihar.