Freelance › Freelancers services › Software development › ETL & ELT pipelines on Apache Airflow — DAGs that run themselves
ETL & ELT pipelines on Apache Airflow — DAGs that run themselves
Service description
I build data pipelines on Apache Airflow that quietly do their job every day without anyone babysitting them. I cover the whole ETL/ELT lifecycle: I pull data from APIs, files, queues and operational databases, transform it into clean typed tables, and load it into your warehouse on a schedule you can trust. Every workflow is a DAG with explicit dependencies, so tasks run in the right order and never before their inputs are ready.
Reliability is where most of my work goes. I set sensible retries with backoff, idempotent tasks you can re-run without duplicating rows, and alerting that reaches you on Slack or email the moment a run fails or runs long. I add data-quality checks between stages so a bad source file is caught before it poisons a dashboard, and I wire in monitoring for run times, backlog and success rates. Backfills become one command instead of a weekend of manual scripts.
I integrate with what you already run — PostgreSQL, MySQL, ClickHouse, Snowflake, BigQuery, S3-compatible storage, dbt models and REST APIs — and I document every DAG so your team can extend it. You get a version-controlled repository, a clear deployment path, and orchestration that is observable rather than mysterious.
— DAG design, scheduling, dependencies, sensors and backfills
— Retries, idempotency, SLAs, alerting and data-quality checks
— Integrations: warehouses, databases, object storage, APIs and dbt
Reliability is where most of my work goes. I set sensible retries with backoff, idempotent tasks you can re-run without duplicating rows, and alerting that reaches you on Slack or email the moment a run fails or runs long. I add data-quality checks between stages so a bad source file is caught before it poisons a dashboard, and I wire in monitoring for run times, backlog and success rates. Backfills become one command instead of a weekend of manual scripts.
I integrate with what you already run — PostgreSQL, MySQL, ClickHouse, Snowflake, BigQuery, S3-compatible storage, dbt models and REST APIs — and I document every DAG so your team can extend it. You get a version-controlled repository, a clear deployment path, and orchestration that is observable rather than mysterious.
— DAG design, scheduling, dependencies, sensors and backfills
— Retries, idempotency, SLAs, alerting and data-quality checks
— Integrations: warehouses, databases, object storage, APIs and dbt
Contact the freelancer
Order the service or ask the freelancer a question.
Freelancer contacts
E-mailShow
