Identify Common Performance Bottlenecks Such as Data Skew, Shuffling, and Disk Spilling by Interpreting Stage-Level Metrics in the Spark UI. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   A data engineer is reviewing the Spark UI for a job that processes user events by country. In the stage details, they notice that most tasks complete in under 30 seconds, but a few tasks take over 5 minutes and process significantly more input data. Which condition is most likely indicated by these observations?

   • ANetwork congestion
   • BDisk spilling
   • CExcessive shuffling
   • DData skew
   Reveal Solution Discussion Select an answer first
2. 7expert · hard

   A data engineer is analyzing a Spark job that is running slowly. The Spark UI shows the following for a stage: 'Input Size' = 10 GB, 'Shuffle Write' = 100 GB, 'Shuffle Read' = 100 GB, and 'Spill (Disk)' = 20 GB. The job has 100 tasks. What is the most likely primary bottleneck?

   • AData skew, because the shuffle write is much larger than the input size.
   • BExcessive shuffling, because the shuffle write (100 GB) is 10 times larger than the input size (10 GB).
   • CDisk spilling, because the spill is 20 GB.
   • DInput read bottleneck, because the input size is only 10 GB.
   Reveal Solution Discussion Select an answer first
4. 8expert · hard

   A data engineer is optimizing a Spark job that performs a join between two large tables. The Spark UI shows a stage with 'Shuffle Write' of 300 GB and 'Shuffle Read' of 300 GB. The job is slow. The engineer is considering using a broadcast join, but the table is 200 GB. What is the most appropriate action?

   • AUse a broadcast join hint to force Spark to broadcast the table.
   • BDo not use a broadcast join, as the table is too large to fit in executor memory, and instead focus on optimizing the shuffle.
   • CIncrease the broadcast threshold to 300 GB so the table can be broadcast.
   • DReduce the shuffle partitions to decrease the amount of data shuffled.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A data engineer is reviewing a Spark job that performs a groupBy on a large dataset. In the Spark UI, they see a stage with 'Shuffle Write' of 150 GB. The job is slow. What is the most likely reason for the large shuffle write?

   • AThe job is experiencing data skew, causing some tasks to write more shuffle data.
   • BThe groupBy operation requires all records with the same key to be sent to the same executor, resulting in a large amount of data movement.
   • CThe job is disk spilling, causing the shuffle data to be written to disk.
   • DThe job is reading a large amount of data from the source, causing a large shuffle.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A data engineer is monitoring a running job in the Spark UI. In the stage details, they notice that the 'Spill (Memory)' and 'Spill (Disk)' columns show large values for many tasks. The job is running slower than expected. What does this metric pattern indicate?

   • AThe tasks are experiencing network congestion during the shuffle phase.
   • BThe tasks are processing a skewed dataset, with some tasks handling more data than others.
   • CThe tasks are reading more data from disk than from memory, indicating a storage bottleneck.
   • DThe tasks are running out of memory and writing intermediate data to disk, causing performance degradation.
   Reveal Solution Discussion Select an answer first