Implement Performance Tuning Strategies & Optimize Cluster Utilization, Including Partitioning, Repartitioning, Coalescing, Identifying Data Skew, and Reducing Shuffling ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   What is a key consideration when optimizing cluster utilization in Spark?

   • ASetting spark.sql.shuffle.partitions to the highest possible value
   • BMaximizing the number of partitions to ensure every core is busy
   • CBalancing the number of partitions with the available cores to avoid idle resources and excessive overhead
   • DUsing as many executors as possible, regardless of data size
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A Spark job joins a large fact table with a small dimension table. The fact table has a heavily skewed join key: one key value appears in 80% of the rows. The job is slow because the single task processing that key becomes a bottleneck. Which strategy is most effective to mitigate this skew?

   • AIncrease spark.sql.shuffle.partitions to a higher value.
   • BUse a broadcast join by caching the dimension table and calling broadcast() on it.
   • CAdd a random salt to the join key in the fact table and a corresponding salted key in the dimension table.
   • DFilter out the skewed key before the join and process it separately.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A Spark job joins two large DataFrames on a key that is heavily skewed. The engineer applies salting by adding a random number to the key in the fact table and replicating the dimension table. After the join, the engineer needs to aggregate the results by the original key. Which additional step is necessary to ensure correct results?

   • AUse a broadcast join instead of salting.
   • BRemove the salt from the key after the join before aggregating.
   • CCache the joined DataFrame before aggregation.
   • DIncrease the number of shuffle partitions to accommodate the salted keys.
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   Which statement correctly compares coalesce() and repartition() in terms of data distribution?

   • Acoalesce() produces a more even distribution because it uses a hash partitioning strategy
   • BBoth methods produce identical data distribution when reducing partitions
   • Ccoalesce() distributes data evenly but repartition() does not
   • Drepartition() produces a more even distribution of data across partitions, while coalesce() may result in uneven partition sizes
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   What is the purpose of adding a random salt to a join key when mitigating data skew?

   • ATo reduce the number of partitions in the DataFrame
   • BTo sort the data within each partition for faster joins
   • CTo distribute the skewed key's data across multiple partitions, reducing the load on a single partition
   • DTo encrypt the data before shuffling to improve security
   Reveal Solution Discussion Select an answer first