A Data Engineer builds and operates the pipelines, storage, and infrastructure that move, transform, and serve data — so that analysts, data scientists, and ML engineers can trust and use it. You are a software engineer who treats data as infrastructure: your product is reliable, fresh, well-modeled tables, and your hardest problems are scale, reliability, schema change, and cost — not building dashboards or models.
Why this appears in interviews
Data engineering is one of the most misunderstood roles in tech, constantly conflated with data analysis and data science. Interviewers calibrate on it immediately, because the framing predicts everything else: a candidate who thinks the job is "writing SQL and dashboards" will design fragile pipelines, while one who thinks in terms of reliability, idempotency, and schema stability will design robust ones. Getting the distinction right before any technical question signals you understand the job.
The mental model — the plumbing behind the house
Think of a city's water system. Residents turn on a tap and water flows; they never think about the pipes, pumps, and treatment plants. When it works, nobody notices — when a main bursts, everything stops. Data engineers build that plumbing. Your success is measured in the absence of problems: data arrives on time, is correct, and doesn't break when a source changes. This is why reliability, not cleverness, is the defining value.
How the role differs
- Data Analyst — Consumes clean data to answer business questions (SQL, BI tools). Doesn't build pipelines. Output: insight.
- Data Scientist — Builds models on data engineers prepared. Relies on you for reliable, well-modeled data. Output: models/analysis.
- ML Engineer — Productionizes and serves models; overlaps with you on feature pipelines and orchestration. Output: model services.
- Data Engineer — Builds the systems that make all of the above possible: the pipeline from raw source to clean, queryable, well-modeled tables, and the platform they run on. Output: trustworthy data as a product. Cares about freshness, reliability, schema stability, cost, and query performance.
The honest summary: a data engineer is a strong software/distributed-systems engineer specialized in data movement and modeling. SQL is table stakes, but the differentiated skill is designing pipelines that stay correct under scale, failure, and change.
The mindset: pipelines are systems that must survive failure
If you're coming from app engineering, the key shift is that data pipelines fail constantly and silently — a source API changes its schema, a job dies at 3 a.m., a batch arrives late, a duplicate load double-counts revenue. None of these throw a user-visible error; they produce wrong data, which is worse. So the discipline centers on:
- Reliability by design — retries, dead-letter queues, alerting, and graceful handling of late/malformed data.
- Idempotency — a job you rerun (after a failure or for a backfill) must not corrupt or double-count. This one property underlies safe operations.
- Schema stability and evolution — contracts with upstream producers, and pipelines that tolerate new/renamed fields instead of breaking.
- Data quality as a monitored property — automated checks (nulls, ranges, volume, freshness) that fail loudly, because bad data is invisible until someone downstream trusts it.
Engineers who internalize "the pipeline must be correct when everything goes wrong" build data platforms people trust; those who don't ship pipelines that quietly poison the warehouse.
What data engineers actually build
- Ingestion — moving data from apps, databases (via CDC), APIs, and files into the platform, in batch or streaming.
- Transformation — turning raw data into clean, conformed, business-ready tables (dbtdbtData Build Tool — SQL-based transformation layer that makes data transformations version-controlled and modular./Spark/SQL), usually in layered stages (raw → cleaned → business).
- Modeling — designing schemas (dimensional/star models) that analysts and ML can query efficiently and correctly.
- Orchestration — expressing pipelines as scheduled DAGs with dependencies, retries, and backfills (Airflow/Dagster/Prefect).
- Storage and performance — choosing warehouse vs lake vs lakehouse, columnar formats (Parquet), partitioning, and tuning for cost and query speed.
- Reliability and quality — idempotent loads, DLQs, data tests, freshness SLAs, and lineage/observability.
Every one of these is a concept in this track.
The modern data stack
A common shape: Fivetran/Airbyte (ingestion) → Snowflake/BigQuery/Databricks (storage) → dbtdbtData Build Tool — SQL-based transformation layer that makes data transformations version-controlled and modular. (transformation) → Airflow/Dagster/Prefect (orchestration) → Great Expectations/Monte Carlo (quality/observability). Know the categories and their tradeoffs more than any single tool — interviews test whether you can reason about "which layer and why," not brand recall.
Common interview mistakes
Mistake 1: Describing analytics as data engineering. "I built Tableau dashboards" is analysis; data engineering is making the data behind them reliable.
Mistake 2: Ignoring reliability. The interesting part of a pipeline is how it behaves when a source is late, a schema changes, or a job fails at 3 a.m. — not the happy path.
Mistake 3: Treating SQL as the whole job. Production data engineering also needs distributed processing, orchestration, storage formats, modeling, and cost optimization.
Mistake 4: No idempotency/quality story. Designing pipelines with no plan for retries, duplicates, or bad data.
Key vocabulary
- Pipeline — A sequence of steps moving data from source to destination, transforming it along the way.
- ETL / ELTETL / ELTExtract-Transform-Load / Extract-Load-Transform — patterns for moving data from source systems to a data warehouse. — Extract-Transform-Load vs Extract-Load-Transform; ELTETL / ELTExtract-Transform-Load / Extract-Load-Transform — patterns for moving data from source systems to a data warehouse. (transform in the warehouse) dominates the modern stack.
- Idempotency — Rerunning a job produces the same result without double-counting; the basis of safe retries and backfills.
- Schema evolution — Handling upstream field changes without breaking downstream consumers.
- Data warehouse / lake / lakehouse — Structured analytics store vs cheap raw store vs ACID tables over object storage.
- Orchestration — Scheduling, sequencing, retrying, and backfilling pipeline steps as a DAG.