Identify Common Performance Bottlenecks Such as Data Skew, Shuffling, and Disk Spilling by Interpreting Stage-Level Metrics in the Spark UI. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 5)

Questions 21–22

1. 21application · medium

   A team is running a Spark job that performs a join between two tables. In the Spark UI, they see a stage with 'Shuffle Write' of 50 GB and 'Shuffle Read' of 50 GB. The job is slow. Which additional metric would best help determine if the shuffle is the primary bottleneck?

   • AThe 'Task Duration' metric, to see if tasks are taking a long time.
   • BThe 'Spill (Disk)' metric, to see if the shuffle is causing memory pressure.
   • CThe 'Shuffle Read Size/Records' metric, to see if the number of records being shuffled is large.
   • DThe 'Input Size' metric, to see how much data is being read from the source.
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   In the Spark UI stage details, which metric indicates that data is being written to disk due to memory pressure?

   • AInput Size
   • BTask Duration
   • CSpill (Disk) Size
   • DShuffle Write Size
   Reveal Solution Discussion Select an answer first