Configure Spark Partitioning in Distributed Data Processing, Including Shuffles and Partitions ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   How does increasing spark.sql.shuffle.partitions affect shuffle performance?

   • AIt reduces the amount of data shuffled
   • BIt makes each task process more data, improving efficiency
   • CIt increases the number of tasks, potentially reducing the size of each shuffle block
   • DIt eliminates the need for shuffle entirely
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A Spark job aggregates user activity by user_id. A few popular users account for 90% of the records, causing severe partition skew during the groupByKey. The engineer needs to balance the load across tasks while preserving the final aggregated result per user. Which strategy should the engineer apply?

   • AUse repartition(user_id) to evenly distribute the data.
   • BUse coalesce to reduce the number of partitions.
   • CIncrease spark.sql.shuffle.partitions to a very high number.
   • DAdd a random salt to each user_id before grouping, then aggregate twice and combine results.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A Spark application repeatedly reads the same large Parquet table and filters it by a date column. The table is stored in cloud object storage. The engineer notices that each query reads all files, even though only a few dates are needed. Which action would most directly improve data locality and reduce I/O?

   • APartition the table by the date column and use partition pruning in queries.
   • BIncrease spark.sql.shuffle.partitions to speed up the query.
   • CUse repartition(1000) on the DataFrame before filtering.
   • DUse coalesce(1) to read the data in a single partition.
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   What is the main advantage of using coalesce over repartition when reducing the number of partitions?

   • Acoalesce can both increase and decrease partitions
   • Bcoalesce guarantees a balanced distribution of data across partitions
   • Ccoalesce avoids a full shuffle, which can reduce data movement
   • Dcoalesce always results in a more even data distribution
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   What is the default value of the Spark configuration spark.sql.shuffle.partitions?

   • A100
   • B400
   • CIt depends on the cluster size
   • D200
   Reveal Solution Discussion Select an answer first