Identify Common Performance Bottlenecks Such as Data Skew, Shuffling, and Disk Spilling by Interpreting Stage-Level Metrics in the Spark UI. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   Which Spark UI metric is most directly used to detect excessive shuffling in a stage?

   • ASpill (Disk) Size
   • BTask Duration
   • CInput Size / Records
   • DShuffle Read Size / Records
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A data engineer is troubleshooting a slow Spark job. The Spark UI shows a stage with 100 tasks. The 'Duration' column shows that 90 tasks take 1 minute, while 10 tasks take 15 minutes. The 'Shuffle Read' column shows that the 10 slow tasks read 10 GB each, while the others read 100 MB. The 'Spill (Disk)' column shows that the 10 slow tasks spilled 5 GB each. What is the most likely sequence of events?

   • ADisk spilling causes the shuffle read to be slow, leading to longer task durations.
   • BExcessive shuffling causes all tasks to read a large amount of data, leading to disk spilling.
   • CData skew causes a few tasks to read a large amount of shuffle data, leading to memory pressure and disk spilling.
   • DNetwork congestion causes the shuffle read to be slow for some tasks, leading to spilling.
   Reveal Solution Discussion Select an answer first
4. 18expert · hard

   A team is running a complex ETL job. In the Spark UI, they see a stage with high 'Spill (Disk)' metrics. They increase the executor memory from 8 GB to 16 GB, but the spilling persists. What is the most likely reason the spilling continues?

   • AThe spilling is caused by a bug in the Spark version, and increasing memory will not help.
   • BThe spilling is caused by the shuffle operation, which cannot be fixed by increasing memory.
   • CThe executor memory setting is not being applied because the job is running in a notebook.
   • DThe data per task is still too large for the increased memory, or the spilling is caused by a different factor such as data skew.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A team is analyzing a Spark job that joins a large fact table with a small lookup table (10 MB). In the Spark UI, they see a stage with a 'Shuffle Write' of 50 GB. The job is slow. What is the most likely cause of the large shuffle?

   • AThe job is disk spilling, causing the shuffle data to be written to disk.
   • BThe lookup table is being broadcast, causing a large shuffle.
   • CThe fact table is skewed, causing some tasks to shuffle more data.
   • DThe join is not using a broadcast join, so the large fact table is being shuffled across the network.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   Which Spark UI metric is used to measure the amount of data read from the source (e.g., files, tables) by a stage?

   • AShuffle Read Size / Records
   • BInput Size / Records
   • CSpill (Disk) Size
   • DTask Duration
   Reveal Solution Discussion Select an answer first