Project description
TranslatedWe need a production web-scraping system that on a daily schedule collects structured data from 5-8 websites and transforms it into a format suitable for analytics. This isn't a one-off script but a service that runs unattended every day, survives markup changes at the sources, respectfully reduces load when a website asks, and doesn't crash entirely if one of the target resources breaks. Some sources are plain static HTML, some are heavy JavaScript SPAs, so the crawler needs to handle both confidently.
Equally important is what happens after the page loads. Raw HTML needs to turn into verified, deduplicated records, and prices, dates, and phone numbers need to be normalized to uniform formats before hitting the database. It makes sense to store only new or changed data, so we need incremental logic. Beyond that, operational transparency matters to us: a dashboard showing run success rates and volumes, notifications for failures and data anomalies, and a reproducible deployment that can be spun up from scratch without tribal knowledge.
The specific target websites are confidential and shared under NDA after terms are agreed. In your response, write about what anti-bot protections you've encountered and how you kept the parser stable over time. Full scope of work below.
1. Scraper architecture
— Scrapy + proxy rotation and User-Agent pool
— Playwright/Selenium for JS sites
— CAPTCHA bypass (2captcha / anti-captcha)
— Rate limiting, robots.txt compliance
2. Data pipeline
— HTML to structured JSON
— Validation and deduplication
— Normalization of prices, dates, phone numbers
— PostgreSQL storage with schema design
— Incremental updates (only new/changed records)
3. Monitoring and operations
— Dashboard: run success, record counts, errors
— Slack/email alerts on failures
— Docker Compose deployment
— Cron scheduling with retry logic
Target websites
— Confidential (NDA) — details after agreement
— Mix of static and SPA sites
Deliverables
— Source code with tests
— Docker Compose configuration
— Database schema migrations
— Runbook and monitoring setup
Equally important is what happens after the page loads. Raw HTML needs to turn into verified, deduplicated records, and prices, dates, and phone numbers need to be normalized to uniform formats before hitting the database. It makes sense to store only new or changed data, so we need incremental logic. Beyond that, operational transparency matters to us: a dashboard showing run success rates and volumes, notifications for failures and data anomalies, and a reproducible deployment that can be spun up from scratch without tribal knowledge.
The specific target websites are confidential and shared under NDA after terms are agreed. In your response, write about what anti-bot protections you've encountered and how you kept the parser stable over time. Full scope of work below.
1. Scraper architecture
— Scrapy + proxy rotation and User-Agent pool
— Playwright/Selenium for JS sites
— CAPTCHA bypass (2captcha / anti-captcha)
— Rate limiting, robots.txt compliance
2. Data pipeline
— HTML to structured JSON
— Validation and deduplication
— Normalization of prices, dates, phone numbers
— PostgreSQL storage with schema design
— Incremental updates (only new/changed records)
3. Monitoring and operations
— Dashboard: run success, record counts, errors
— Slack/email alerts on failures
— Docker Compose deployment
— Cron scheduling with retry logic
Target websites
— Confidential (NDA) — details after agreement
— Mix of static and SPA sites
Deliverables
— Source code with tests
— Docker Compose configuration
— Database schema migrations
— Runbook and monitoring setup
Software development Cash, Bank transfer
