Build an ETL pipeline for data collection into warehouse
Employer
Mykola
Project parameters
Type of cooperationOne-time project
SectionAdministration
Prepaymentwithout prepayment
Payment methodsCash, Bank transfer
Acceptance of requestsfrom Aug 18, 2026 until Sep 8, 2026
Project description
Our company operates on several disconnected systems, and right now no one has a single reliable picture of what's happening across them all. Sales data lives in one database, billing information is available through a third-party API, and some departments still send us CSV dumps every week. I need someone to build a full ETL pipeline: extract data from all these sources, transform and clean raw records into a single consistent format, and load it all into central data warehouse. This is not a reporting dashboard and not a complete backend rewrite; the deliverable is exactly a data integration layer that works reliably in the background so downstream tools always have fresh and correct numbers.
What matters most to me is that the pipeline is predictable and doesn't corrupt anything silently. I want incremental syncs rather than full reloads wherever the source allows, so we only move changes and keep jobs fast as volume grows. Each run must include data quality checks that catch missing fields, broken types, duplicate keys, and rows violating business rules, and on such cases emit a clear alert rather than silently write garbage to the warehouse. I also care about idempotency: if a job fails halfway and reruns, it must not double data or leave the warehouse in a half-loaded state.
Please document source mappings and transformation logic so my team can maintain everything after handoff, and set up scheduled execution without my constant oversight. Tell us what tech stack you'd propose and why.
— Extract from multiple sources: SQL databases, REST APIs, and weekly CSVs
— Transform and clean records into a single schema without duplicates
— Load into central warehouse with staging and final layers
— Incremental syncs of changed rows, not full reloads
— Automatic data quality checks with error alerts
— Idempotent restartable jobs on schedule and documentation for handoff
What matters most to me is that the pipeline is predictable and doesn't corrupt anything silently. I want incremental syncs rather than full reloads wherever the source allows, so we only move changes and keep jobs fast as volume grows. Each run must include data quality checks that catch missing fields, broken types, duplicate keys, and rows violating business rules, and on such cases emit a clear alert rather than silently write garbage to the warehouse. I also care about idempotency: if a job fails halfway and reruns, it must not double data or leave the warehouse in a half-loaded state.
Please document source mappings and transformation logic so my team can maintain everything after handoff, and set up scheduled execution without my constant oversight. Tell us what tech stack you'd propose and why.
— Extract from multiple sources: SQL databases, REST APIs, and weekly CSVs
— Transform and clean records into a single schema without duplicates
— Load into central warehouse with staging and final layers
— Incremental syncs of changed rows, not full reloads
— Automatic data quality checks with error alerts
— Idempotent restartable jobs on schedule and documentation for handoff
