Configure Spark Partitioning in Distributed Data Processing, Including Shuffles and Partitions ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16application · medium

   A Spark job performs a join between two DataFrames. The engineer wants to minimize shuffle overhead. The DataFrames are large, and the join key is not sorted. Which approach should the engineer use?

   • AUse a broadcast join if one DataFrame is small enough.
   • BUse repartition on both DataFrames by the join key.
   • CUse a sort-merge join with a higher number of shuffle partitions.
   • DUse coalesce on both DataFrames to reduce partitions.
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   What is partition skew in Spark?

   • AA condition where all partitions have equal data but tasks run slowly
   • BA condition where some partitions contain significantly more data than others
   • CA condition where partitions are stored on non-local nodes
   • DA condition where data is lost during shuffle
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   What does co-partitioning mean in the context of optimizing joins?

   • ACombining both datasets into a single partition
   • BSorting both datasets by the join key
   • CUsing the same executor for both datasets
   • DEnsuring both datasets have the same number of partitions and are partitioned by the join key
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   What is the main advantage of a broadcast join in Spark?

   • AIt shuffles both datasets to ensure all keys are co-located
   • BIt sorts both datasets before joining to improve speed
   • CIt increases the number of partitions to speed up the join
   • DIt avoids shuffling the large dataset by sending a copy of the small dataset to each executor
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A Spark job performs a reduceByKey on a large RDD. The engineer notices that the shuffle write size is very large, causing network congestion. The data has many unique keys, but each key has only a few values. Which change would most directly reduce the shuffle write size?

   • AUse combineByKey with a merge function that combines values locally before the shuffle.
   • BUse repartition to increase the number of partitions.
   • CIncrease spark.sql.shuffle.partitions.
   • DUse coalesce to reduce the number of partitions.
   Reveal Solution Discussion Select an answer first