Configure Spark Partitioning in Distributed Data Processing, Including Shuffles and Partitions ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A Spark job joins two large DataFrames on a key. The engineer wants to minimize shuffle overhead. The DataFrames are already partitioned by the same key using the same number of partitions. Which additional step would best ensure a co-partitioned join without an extra shuffle?

   • AIncrease spark.sql.shuffle.partitions to a high value.
   • BEnsure both DataFrames use the same partitioner (e.g., both repartitioned by the join key).
   • CUse a broadcast join on one of the DataFrames.
   • DUse coalesce to reduce the number of partitions on both DataFrames.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A Spark SQL query performs a join and then a groupBy. The engineer wants to reduce the number of shuffle partitions to improve performance. The cluster has 50 cores. Which value for spark.sql.shuffle.partitions is most appropriate?

   • A25
   • B100
   • C500
   • D50
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   Which technique is commonly used to mitigate partition skew caused by a few hot keys?

   • AIncreasing the number of partitions without changing the key
   • BDisabling shuffle to avoid data movement
   • CSalting the keys by adding a random suffix to spread data across more partitions
   • DUsing coalesce to reduce partitions
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A Spark job performs a wide transformation (groupByKey) on a DataFrame with 100 partitions. After the shuffle, the engineer observes that the number of tasks in the next stage is 200. What is the most likely reason for this change?

   • AThe DataFrame was repartitioned to 200 before the groupByKey.
   • BThe groupByKey operation repartitions the data to match the number of executors.
   • CThe shuffle operation automatically doubles the number of partitions.
   • DThe shuffle operation used the default shuffle partition count of 200.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Which method should you use to increase the number of partitions of a DataFrame from 10 to 100?

   • Adf.partitionBy(100)
   • Bdf.setPartitions(100)
   • Cdf.repartition(100)
   • Ddf.coalesce(100)
   Reveal Solution Discussion Select an answer first