Identify Common Performance Bottlenecks Such as Data Skew, Shuffling, and Disk Spilling by Interpreting Stage-Level Metrics in the Spark UI. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A data engineer is reviewing the Spark UI for a job that performs a wide transformation (e.g., groupBy) on a large dataset. The stage details show 'Shuffle Write' of 200 GB and 'Shuffle Read' of 200 GB. Additionally, the 'Spill (Disk)' column shows values for many tasks. What is the most likely sequence of events causing the performance issue?

   • AThe job is failing due to an out-of-memory error, and the spill metrics are a side effect.
   • BThe job is reading a large amount of data from disk, causing both shuffle and spilling.
   • CThe shuffle operation is moving a large amount of data, and the memory pressure from the shuffle is causing disk spilling.
   • DThe job is experiencing data skew, with some tasks handling more data than others.
   Reveal Solution Discussion Select an answer first
2. 12expert · hard

   A data engineer is debugging a slow Spark job. The Spark UI shows the following for a stage: 'Shuffle Write' = 100 GB, 'Shuffle Read' = 100 GB, and 'Spill (Disk)' = 50 GB for many tasks. The job has 50 executors with 4 cores each. The engineer suspects the bottleneck is due to excessive shuffling. Which additional piece of evidence from the Spark UI would most strongly confirm this suspicion?

   • AThe 'Task Duration' metric shows a wide range, indicating data skew.
   • BThe 'Shuffle Read Size/Records' metric shows a high number of records being read, indicating a large amount of data movement.
   • CThe 'Spill (Disk)' metric is high, indicating memory pressure.
   • DThe 'Input Size' metric is high, indicating a large amount of data being read from source.
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   A team is running a groupBy operation on a large dataset. In the Spark UI, the stage shows 200 tasks. The 'Duration' column shows that 10 tasks take 20 minutes, while the rest take 2 minutes. The 'Spill (Disk)' column shows that the 10 slow tasks have spilled 10 GB each, while the others have spilled less than 100 MB. What is the most likely root cause of the performance issue?

   • AInsufficient executor memory, causing all tasks to spill equally.
   • BData skew, where a few keys have a disproportionately large number of records, causing those tasks to process more data and spill to disk.
   • CExcessive shuffling, where the total shuffle data is too large for the cluster.
   • DNetwork congestion during the shuffle phase, slowing down data transfer.
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A data engineer is tuning a Spark job that performs a large sort operation. In the Spark UI, the stage details show that many tasks have high values in the 'Spill (Memory)' and 'Spill (Disk)' columns. The job is running slower than expected. Which configuration change is most likely to reduce the spilling?

   • AEnable compression for shuffle data to reduce the amount of data written to disk.
   • BIncrease the executor memory (spark.executor.memory) to provide more memory for the sort operation.
   • CDecrease the number of partitions to reduce the number of tasks.
   • DIncrease the shuffle partitions (spark.sql.shuffle.partitions) to create more, smaller tasks.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   A data engineer is investigating a slow Spark job. In the Spark UI, they see a stage with a very large 'Shuffle Write' metric (e.g., 500 GB) and a subsequent stage with a correspondingly large 'Shuffle Read' metric. What does this indicate?

   • AMemory pressure
   • BExcessive shuffling
   • CData skew
   • DDisk spilling
   Reveal Solution Discussion Select an answer first