Identify Common Performance Bottlenecks Such as Data Skew, Shuffling, and Disk Spilling by Interpreting Stage-Level Metrics in the Spark UI. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   In the Spark UI stage details, which combination of metrics is most indicative of data skew?

   • AHigh shuffle write but low shuffle read
   • BHigh disk spill but low memory usage
   • CUniform task durations and uniform input sizes
   • DWide variance in task durations and input sizes across tasks
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A data engineering team is running a daily batch job that joins a large fact table with a small dimension table. In the Spark UI, the stage for the join shows 200 tasks. Most tasks complete in under 30 seconds, but 5 tasks take over 10 minutes and read significantly more input data than the others. Which bottleneck is most clearly indicated by these stage-level metrics?

   • AData skew, because a few tasks process disproportionately more data and take much longer than the majority.
   • BExecutor failure, because only a few tasks are slow while the rest finish quickly.
   • CDisk spilling, because the long-running tasks are likely writing intermediate data to disk.
   • DExcessive shuffling, because the join operation requires data to be exchanged across the cluster.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A team is investigating a performance issue in a Spark job. In the Spark UI, they see a stage with 100 tasks. The 'Duration' column shows a wide range: 10 tasks take 5 minutes, 80 tasks take 30 seconds, and 10 tasks take 10 seconds. The 'Shuffle Read' column shows that the 10 slowest tasks read 10 GB each, while the others read less than 100 MB. Which bottleneck is most likely?

   • ADisk spilling, because the slow tasks are likely spilling to disk.
   • BData skew, because a few tasks are reading much more shuffle data and taking much longer.
   • CExcessive shuffling, because the total shuffle read is large.
   • DNetwork congestion, because the shuffle read is slow for some tasks.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A team is running a join between two large tables. In the Spark UI, the stage for the join shows 500 tasks. The 'Input Size' column shows that 450 tasks read about 10 MB each, while 50 tasks read about 1 GB each. The 'Duration' column shows the 50 tasks with 1 GB input take 15 minutes, while the others take 1 minute. What is the most appropriate first step to address this bottleneck?

   • AIncreasing the number of partitions to reduce the amount of data per task.
   • BSalting the join keys to distribute the skewed data more evenly across tasks.
   • CIncreasing the executor memory to prevent disk spilling.
   • DUsing a broadcast join to avoid shuffling the large table.
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A team is troubleshooting a slow nightly aggregation job. In the Spark UI, they see a stage with 'Shuffle Write' of 500 GB and the next stage with 'Shuffle Read' of 500 GB. The cluster has 20 executors with 8 cores each. The job runs much slower than expected. Which conclusion is best supported by these metrics?

   • AThe job is disk spilling, because the shuffle data is too large to fit in memory.
   • BThe job is performing excessive shuffling, moving a large volume of data across the network, which is a likely bottleneck.
   • CThe job is running efficiently, because the shuffle write and read sizes match perfectly.
   • DThe job is suffering from data skew, because the shuffle write and read sizes are equal.
   Reveal Solution Discussion Select an answer first