What questions are asked in a data engineer interview?

Five areas cover almost every loop: SQL (window functions, deduplication, joins and query performance), distributed processing with Spark (lazy evaluation, shuffles, joins and skew), streaming with Kafka (partitions, consumer groups, offsets and delivery guarantees), data modelling (star schemas, grain and slowly changing dimensions), and pipeline design (idempotency, backfills, late data and data quality). Early screens lean on SQL and short Python exercises; onsite rounds move to Spark internals, streaming semantics and an end-to-end pipeline design.

How much SQL do you need for a data engineer interview?

More than for most engineering roles, and at a higher level than basic joins. Expect window functions such as ROW_NUMBER, RANK and running totals, deduplicating to the latest record per key, anti-joins, NULL handling in comparisons and NOT IN, and reading a query plan well enough to explain why a query is slow. The SQL screen is the most common place data engineering candidates are filtered out, so it deserves the largest share of preparation time.

What is data skew in Spark and how do you fix it?

Data skew is when a few keys hold far more rows than the rest, so after a shuffle a few tasks get most of the data. A stage finishes only when its slowest task finishes, so one oversized partition can make a job take many times longer than the average task. Fixes include broadcasting the smaller side of a join so no shuffle is needed, enabling Adaptive Query Execution's skew-join handling, which splits oversized partitions at runtime, and salting the hot key by adding a random suffix and aggregating in two steps.

What does idempotent mean in a data pipeline?

A pipeline step is idempotent when running it twice for the same input produces the same result as running it once. In practice that means loads overwrite or merge into a well-defined target, such as one date partition or a set of primary keys, instead of blindly appending rows. Idempotency is what makes retries, reruns and backfills safe, and it is the property interviewers probe most in pipeline design questions.

Is data engineering a good career in 2026?

Demand has held up better than for many software roles because every AI initiative depends on clean, well-modelled, well-governed data, and someone has to build the pipelines that feed models, retrieval systems and analytics. The job has shifted from writing ETL scripts towards owning data platforms: table formats such as Iceberg and Delta, streaming, cost control on Snowflake or Databricks, and data quality contracts.

What is the difference between a data engineer and a data scientist?

A data engineer builds and runs the systems that collect, move, store and model data so it is reliable and available. A data scientist uses that data to answer questions and build models. Interviews reflect the split: data engineering loops focus on SQL, distributed processing, streaming and pipeline design, while data scientist loops focus on statistics, experimentation, product sense and machine learning, usually with a SQL screen in common.

Data Engineer Interview Questions: How to Prepare in 2026

Data engineer interview questions in 2026 look like a list of technologies: SQL, Spark, Kafka, a warehouse, an orchestrator. The interviewers asking them are mostly after one property, though, and it is not familiarity with any of those tools. It is whether your pipelines stay correct when something goes wrong: when a job reruns, when an event arrives two hours late, when one customer ID owns a third of the rows, when a consumer crashes after writing but before committing its offset.

Candidates who know the tools and have not thought about those failures do well in the screen and struggle in the onsite. So the useful preparation covers five areas, each studied through the question of what breaks:

SQL - window functions, deduplication, and the NULL semantics that produce wrong answers silently
Spark and distributed processing - lazy evaluation, shuffles, joins and skew
Kafka and streaming - partitions, consumer groups, offsets and what "exactly once" really means
Data modelling - grain, star schemas and history
Pipeline design - idempotency, backfills, late data and quality checks

Before reading any of it, take the quiz. It is quicker than guessing which sections you can skip.

Start here: a 6-question self-check

Six questions from squizzu's SQL, Spark and Kafka sets. You see the reasoning as soon as you answer, and a longer explanation if you want to dig in. There is no sign-up. Pay more attention to which ones you miss than to the total.

SQL • Window Functions

Question 1 / 6

You need to assign a unique sequential number - 1, 2, 3, and so on - to customer rows in order of signup_date, while keeping every row in the output (no grouping, no collapsing). Which approach does this directly?

The misses that matter most are in Spark and Kafka, because those are the ones interviewers turn into a scenario: a job that is slow for no visible reason, or a stream that processes some events twice after a restart. The Kafka version is often a consumer that the group declares dead while it is still working. Both come down to a mechanism that is easy to use without understanding. If either section cost you a question, start there.

What a data engineer interview looks like in 2026

Most loops run five to seven rounds over two or three weeks, and the shape is consistent across companies:

SQL screen. Live or take-home, medium difficulty, usually on a realistic schema. This round filters out more candidates than any other.
Coding. Python, sometimes PySpark: parsing, transforming, deduplicating. Rarely algorithm puzzles for their own sake.
Distributed processing and streaming. Spark internals, Kafka semantics, batch against streaming.
Pipeline or system design. Design the ingestion and modelling for some business process, end to end, under constraints on freshness, cost and correctness.
Behavioural. Often with a data-quality incident at its centre: what broke, how you found out, what you changed.

The emphasis moved this year, in two directions. Business context is weighted more heavily than framework trivia: interviewers want to hear why a table has the grain it has and who reads it, not just how you would partition it. And AI workloads are now part of the job, since the retrieval systems and models every company is building need data pipelines behind them, with the same questions of freshness, lineage and quality. Cost comes up too, because warehouse and lakehouse bills are large enough to have owners.

Area 1: SQL beyond joins

The SQL round assumes you can join and aggregate. What it tests is the next level: window functions, the correctness traps, and whether you can tell a fast query from a slow one.

Window functions compute a value for each row from a set of related rows without collapsing them the way GROUP BY does. The three ranking functions are the most common trap, because they differ only on ties:

Function	Scores 90, 80, 80, 70	Use it for
`ROW_NUMBER()`	1, 2, 3, 4	picking exactly one row per group
`RANK()`	1, 2, 2, 4	competition ranking with gaps after ties
`DENSE_RANK()`	1, 2, 2, 3	"top N distinct values" without gaps

The canonical data engineering task built on them is deduplication to the latest record per key: number the rows within each key by descending update time with ROW_NUMBER(), then keep row 1. Using RANK() there is a real bug, because two rows with the same timestamp both get rank 1 and the duplicate survives.

NULL is where SQL gives wrong answers without an error. Any comparison with NULL yields unknown, not true or false, and WHERE keeps only true. You will hit this constantly. COUNT(column) skips NULLs while COUNT(*) does not. And x NOT IN (subquery) returns no rows at all if the subquery contains a single NULL, which is why anti-joins are safer written with NOT EXISTS or a left join that keeps rows where the right side is missing.

Performance questions are about reading plans rather than memorising rules: whether a filter can use an index or partition, why wrapping an indexed column in a function prevents that, and when a join explodes because the join key is not unique on either side. In a columnar warehouse the equivalent questions are about partition pruning and clustering, since scanning less data is the whole game.

What the interviewer is testing: whether your query is correct on the awkward rows, not just the typical ones. Common follow-up: "What happens if two events have the same timestamp?" Have an answer, such as a deterministic tie-breaker column in the ORDER BY.

Area 2: Spark and distributed processing

Spark questions are rarely about API syntax. They are about the execution model, and almost all of them trace back to one expensive operation: the shuffle.

Start with who does what. The driver runs your program, turns it into a plan of stages and tasks, and schedules those tasks; the executors on the worker nodes run the tasks and hold cached and shuffled data. A collect() that pulls a large result back to the driver is therefore a classic way to run a job out of memory on the one machine that coordinates everything else.

Spark is lazy. Transformations such as filter, select and join only build a plan. Nothing runs until an action such as count, collect or a write asks for a result. That is what lets the optimiser rearrange the plan, and it is also why a job's error can surface at a line far away from the transformation that caused it.

Narrow and wide transformations are the distinction everything else rests on. A narrow transformation, such as a filter or a map, can compute each output partition from one input partition. A wide one, such as groupBy, a join on unaligned keys or repartition, needs rows with the same key from every input partition to meet in one place. That is a shuffle: data is written out, sent across the network and read back. Every shuffle is a stage boundary, and shuffles are where most of a slow job's time goes.

Joins are therefore a question of avoiding shuffles where possible. If one side is small, a broadcast join copies it to every executor so the large side never moves; Spark does this automatically below a size threshold and can be told to with a hint. If both sides are large, a sort-merge join shuffles both by the join key.

Skew is the scenario interviewers like best. When a few keys own most of the rows, the shuffle sends most of the data to a few tasks. A stage finishes only when its slowest task does, so one task can hold up a whole job while the cluster sits idle:

Bar chart of task durations in one Spark stage. Eleven tasks finish in about 1 minute each, while one task that received a skewed key runs for 12 minutes, so the whole stage takes 12 minutes.

The fixes, roughly in the order to try them: broadcast the small side so the hot key never shuffles; let Adaptive Query Execution, on by default in modern Spark, split oversized partitions in skewed joins at runtime; and salt the hot key by appending a random suffix, aggregating per salted key, then aggregating again without the salt.

A couple of distinctions get asked almost every time. DataFrames over RDDs: DataFrame and SQL operations go through the Catalyst optimiser and run on Tungsten, Spark's engine for compact binary memory layout and generated code, while RDD code is opaque to both, and Python UDFs add serialisation overhead that built-in functions avoid. A DataFrame is a Dataset of generic rows, so a misspelled column or a wrong type surfaces only when the job runs; typed Datasets in Scala and Java catch those mistakes at compile time. repartition against coalesce: repartition shuffles to reach any partition count, evenly, while coalesce only merges existing partitions to reduce the count without a full shuffle, which is cheaper and can leave partitions uneven.

What the interviewer is testing: reading the plan to explain a slow job, rather than throwing executors at it. Common follow-up: "Adding more executors did not make the job faster. Why?" Skew is the usual answer, because more executors do not help a single oversized task.

Area 3: Kafka and streaming

Kafka interview questions check whether you know what the log guarantees and, more usefully, what it does not.

A topic is split into partitions, and each partition is an append-only, ordered log. Ordering is guaranteed only within a partition. A producer that sends records with a key has them hashed to a partition, so every event for one key lands in the same place and stays in order; records without a key are spread across partitions and have no ordering relative to each other.

A consumer group shares a topic's partitions among its members, and each partition is assigned to exactly one member of the group at a time. So a group cannot use more consumers than there are partitions, because the extras have nothing to read. Separate groups, on the other hand, each receive every message independently, which is how several services consume the same stream without interfering.

Group membership is kept alive by the consumer itself. Calling poll() fetches records and also signals that the consumer is still working, and a consumer that goes longer than max.poll.interval.ms between polls, because processing one batch took too long, is removed from the group. Its partitions are reassigned, and whatever it had not committed is processed again by someone else.

Offsets are where delivery guarantees come from. A consumer records its position by committing offsets, and when it commits decides what a crash does:

Commit after processing - a crash between processing and committing means the records are processed again on restart. This is at-least-once, the usual choice.
Commit before processing - a crash after committing loses the records that had not been processed yet. This is at-most-once.

Exactly-once is where careful candidates slow down. Kafka offers idempotent producers, which stop retries from writing duplicates, and transactions, which let a consume-transform-produce loop commit its output and its input offsets atomically. That gives exactly-once within Kafka. The moment a consumer writes to an external database or calls an API, the guarantee stops at the boundary, and the consumer has to be idempotent on its own: upsert by a key, or record processed event IDs.

On the producer side, acks=all means a write is acknowledged only once every in-sync replica has it, and min.insync.replicas sets how many that must be before the broker accepts writes at all. Together they decide whether a broker failure can lose acknowledged data. Compression is set on the producer too, and it is end to end: batches are compressed once, stored and replicated as they are, and decompressed by the consumer, which keeps the broker's CPU out of it. On the operations side, Kafka 4.0 removed ZooKeeper entirely in favour of KRaft, so older answers about ZooKeeper managing the cluster are now dated.

What the interviewer is testing: whether you can reason about duplicates and ordering after a failure. Common follow-up: "A consumer restarted and some events were processed twice. Is that a bug?" Under at-least-once it is expected behaviour; the bug is a downstream write that is not idempotent.

Area 4: Data modelling

Modelling questions sound soft and are not. The one decision that matters most is the grain: what exactly one row of a table represents. "One row per order line per day" and "one row per order" produce different answers to the same question, and most wrong dashboards come from joining tables at different grains and double-counting.

The star schema is still the default for analytics. A fact table holds measurable events at a declared grain, such as sales lines, with foreign keys and numeric measures. Dimension tables describe the context, such as customer, product and date, with descriptive attributes. Analysts filter and group by dimensions and aggregate facts. The main alternative in modern warehouses is a wide denormalised table built for a specific use, which is faster to query and harder to keep consistent.

Slowly changing dimensions are how history is kept. A customer moves from Berlin to Munich: do last year's sales now show up under Munich?

Type 1 overwrites the attribute. History is lost, and last year's sales move to Munich.
Type 2 closes the old row and inserts a new one with validity dates and a current flag. Facts join to the version that was valid when they happened.

Type 2 is the common interview topic, and the follow-up is usually about the join: facts have to link to the dimension row that was current at the event's time, not just to the customer's natural key.

Storage has moved on as well. Open table formats such as Apache Iceberg and Delta Lake add ACID transactions, schema evolution and time travel to files in object storage. That is what lets a lakehouse do reliable MERGE operations, and it is increasingly assumed knowledge.

What the interviewer is testing: tables designed around the questions people will ask of them. Common follow-up: "What is the grain of this table?" If you cannot answer in one sentence, the model is not finished.

Area 5: Pipeline design, idempotency and late data

This is where the onsite design round lives, and one property organises the whole discussion.

Idempotency means running a step twice for the same input leaves the same result as running it once. Pipelines are rerun all the time: after failures, after bug fixes, for backfills. A job that appends its output without checking what is already there double-counts on every rerun. The standard patterns are to overwrite a whole partition (the job for 14 March replaces the 14 March partition) or to merge on a key (update matching rows, insert new ones). Either way, a rerun converges instead of accumulating.

Backfills follow from that. If every run is parameterised by the interval it processes, rather than reading "now", then reprocessing last month is simply thirty runs with different parameters. Pipelines that compute their own time window from the clock are the ones that cannot be backfilled safely.

Late data is the streaming half of the same problem. Every event has an event time, when it happened, and a processing time, when the pipeline saw it. Mobile clients go offline, and an event from 14:05 can arrive at 16:30. Aggregating by processing time is simple and wrong. Aggregating by event time is right and raises the question of how long to wait. A watermark is the answer: the pipeline's estimate that it has seen everything up to a given event time. Windows are closed when the watermark passes them, and anything arriving later is either dropped, sent to a side output, or used to update an already-published result.

Data quality is what senior answers spend the most time on. Checks belong in the pipeline, not in a dashboard someone might look at: row counts against expectations, uniqueness of keys, non-null required fields, referential integrity, freshness. A failing check should stop bad data from being published, not just raise an alert after consumers have read it. Data contracts, meaning agreed schemas and guarantees between producing and consuming teams, extend the same idea across team boundaries.

What the interviewer is testing: whether your design survives the rerun, the late event and the bad upstream file. Common follow-up: "Your job failed halfway through writing. What state is the table in, and what happens when it reruns?" Answer that and you have answered most of the round.

A week of breaking your own pipelines

If you already work with data, you have written most of this before. What is new is causing the failures on purpose.

Day 1 - Windows. Write deduplication to the latest record per key three ways, then add rows with equal timestamps and see which versions break.
Day 2 - NULLs and plans. Reproduce the NOT IN with NULL trap, rewrite it with NOT EXISTS, and read the query plan for a slow join in any database you have.
Day 3 - Shuffles. Run a Spark job locally and open the Spark UI. Find the stage boundaries, then force a broadcast join and compare the plans.
Day 4 - Skew. Generate data where one key holds half the rows, join it, and watch one task dominate the stage. Fix it with salting, then with AQE.
Day 5 - Kafka. Run a local broker, produce keyed messages, and run a consumer group with more members than partitions. Kill a consumer between processing and committing and count the duplicates.
Day 6 - Modelling. Design a star schema for a business you know, state the grain of every table in one sentence, and add a Type 2 dimension.
Day 7 - Design out loud. Take one pipeline end to end: source, ingestion, model, quality checks. Then answer three questions about it aloud: what happens on a rerun, on a late event, and on a bad input file.

Practise the rest

The quiz above is a sample. squizzu has a full set for each of the three tools, going deep on joins, Spark execution, and Kafka producers and consumers.

Work through the data engineering questions on squizzu. Scenario questions are where this pays off: the tempting wrong answer is usually the one you would have shipped, and the explanation says why it fails.

The SQL screen deserves separate practice, and the SQL quiz is the place to start; the Python quiz and Python interview questions for 2026 cover the coding round. If you are interviewing at a company that asks product-analytics SQL, 20 Meta data scientist interview questions works through the same kind of screen. For the architecture side of pipeline design, system design interview questions for 2026 covers queues, partitioning and idempotent retries from the service perspective.