MediumData EngineeringSystem Designdbt & Transformation Airflow Snowflake

Apache Airflow — Designing a Daily Pipeline with External Dependencies

Design an Airflow pipeline that runs daily at 7am. It: (1) extracts yesterdays orders from Postgres, (2) waits for an upstream S3 file from a third-party provider usually arriving 6am-9am, (3) joins the two datasets and loads to Snowflake, (4) runs dbt transformations, (5) sends a Slack alert with row counts. Define each task and its operator type, draw the dependency graph, set retry and timeout configuration, and identify which task is not idempotent as written and how you would fix it.

Sign in to attempt this problem

Free account gives you full access to community problems with the complete solution reveal: golden answer, senior walkthrough, and score breakdown, after submission.

Start free →Already have an account? Sign in