Python Web Scraper & Data Collection System
Employer
Mykola
Project parameters
Type of cooperationOne-time project
SectionSoftware development
Prepaymentwithout prepayment
Payment methodsCash, Bank transfer
Acceptance of requestsfrom today, 07:22 until Sep 2, 2026
Project description
We need a production-grade web scraping system that collects structured data from five to eight target websites on a daily schedule and stores it in a form our analysts can actually query. This is not a one-off script that grabs a few pages — it needs to run unattended every day, cope with sites changing their layout, back off politely when asked, and keep going when one target breaks without taking the rest down with it. Some of the targets are plain static HTML, others are JavaScript-heavy single-page applications, so the crawler has to handle both worlds cleanly.
Equally important is what happens after a page is fetched. Raw HTML has to be turned into validated, de-duplicated records, with prices, dates and phone numbers normalised to consistent formats before anything reaches the database. We only want to store what is new or changed, so incremental logic matters. On top of that we need real operational visibility: a dashboard that shows success rates and volumes, alerts when something goes wrong or the data looks anomalous, and a deployment we can stand up again from scratch without tribal knowledge.
The specific target sites are confidential and will be shared under an NDA once we agree terms. In your reply, please mention anti-bot situations you have dealt with before and how you kept a crawler stable over time. Full scope below.
1. Scraper Architecture
— Scrapy framework with rotating proxies and a user-agent pool
— Playwright/Selenium integration for JS-heavy pages
— CAPTCHA-bypass layer (2captcha or anti-captcha API)
— Rate limiting and polite crawling (robots.txt compliance)
2. Data Pipeline
— Raw HTML to structured JSON extraction
— Data validation and deduplication logic
— Normalization of prices, dates, phone numbers
— PostgreSQL storage with schema design
— Incremental updates (only new or changed records)
3. Monitoring & Ops
— Health dashboard: success rate, items per run, errors
— Slack/email alerts on failure or data anomalies
— Docker Compose deployment
— Cron-based scheduling with retry logic
Target Sites
— Confidential (NDA required) — details shared after agreement
— Mix of static HTML and dynamic SPA sites
Deliverables
— Full source code with tests
— Docker Compose production config
— Schema migrations
— Runbook and monitoring setup
Equally important is what happens after a page is fetched. Raw HTML has to be turned into validated, de-duplicated records, with prices, dates and phone numbers normalised to consistent formats before anything reaches the database. We only want to store what is new or changed, so incremental logic matters. On top of that we need real operational visibility: a dashboard that shows success rates and volumes, alerts when something goes wrong or the data looks anomalous, and a deployment we can stand up again from scratch without tribal knowledge.
The specific target sites are confidential and will be shared under an NDA once we agree terms. In your reply, please mention anti-bot situations you have dealt with before and how you kept a crawler stable over time. Full scope below.
1. Scraper Architecture
— Scrapy framework with rotating proxies and a user-agent pool
— Playwright/Selenium integration for JS-heavy pages
— CAPTCHA-bypass layer (2captcha or anti-captcha API)
— Rate limiting and polite crawling (robots.txt compliance)
2. Data Pipeline
— Raw HTML to structured JSON extraction
— Data validation and deduplication logic
— Normalization of prices, dates, phone numbers
— PostgreSQL storage with schema design
— Incremental updates (only new or changed records)
3. Monitoring & Ops
— Health dashboard: success rate, items per run, errors
— Slack/email alerts on failure or data anomalies
— Docker Compose deployment
— Cron-based scheduling with retry logic
Target Sites
— Confidential (NDA required) — details shared after agreement
— Mix of static HTML and dynamic SPA sites
Deliverables
— Full source code with tests
— Docker Compose production config
— Schema migrations
— Runbook and monitoring setup
