Implement Performance Tuning Strategies & Optimize Cluster Utilization, Including Partitioning, Repartitioning, Coalescing, Identifying Data Skew, and Reducing Shuffling ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16application · medium

   A Spark application performs multiple aggregations on the same DataFrame keyed by a customer ID. The engineer notices that each aggregation triggers a full shuffle, causing high network I/O and slow performance. Which approach would most effectively reduce the number of shuffles?

   • ACache the DataFrame after the first aggregation and reuse it.
   • BUse broadcast variables to share the customer data across executors.
   • CRepartition the DataFrame by customer ID once before the aggregations.
   • DIncrease spark.sql.shuffle.partitions to reduce shuffle size.
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A Spark job performs a join between two DataFrames. The Spark UI shows that one task processes 10 GB of data while all other tasks process less than 100 MB. The join key has a few values that appear very frequently. Which technique is most appropriate to mitigate this skew?

   • AFilter out the skewed key before the join.
   • BIncrease spark.sql.shuffle.partitions to a higher value.
   • CAdd a random salt to the join key in both DataFrames.
   • DUse a broadcast join to avoid the shuffle.
   Reveal Solution Discussion Select an answer first
4. 18expert · medium

   A Spark job performs a join between a large fact table and a small dimension table. The dimension table is 50 MB, and the default broadcast threshold is 10 MB. The engineer wants to avoid a shuffle join but does not want to change the global configuration. Which approach is most appropriate?

   • AUse the broadcast() hint on the dimension table.
   • BIncrease spark.sql.autoBroadcastJoinThreshold to 60 MB.
   • CRepartition both DataFrames by the join key before the join.
   • DUse a sort-merge join with increased shuffle partitions.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A Spark cluster has 10 executors, each with 4 cores. A job reads a large dataset and performs a series of transformations. The Spark UI shows that only 20 tasks are running at a time, and the cluster utilization is low. The engineer wants to increase parallelism to better utilize the cluster. Which configuration change is most appropriate?

   • AIncrease spark.sql.shuffle.partitions to 200.
   • BSet spark.default.parallelism to 40.
   • CUse repartition(40) on the DataFrame before the transformations.
   • DIncrease spark.executor.cores to 8.
   Reveal Solution Discussion Select an answer first
6. 20application · easy

   A Spark DataFrame has 100 partitions. After a filter operation, the engineer wants to reduce the number of partitions to 20 to reduce the overhead of writing the output. The engineer is concerned about the cost of a full shuffle. Which method should be used?

   • Adf.coalesce(20)
   • Bdf.repartition(20)
   • Cdf.coalesce(100)
   • Ddf.repartition(100)
   Reveal Solution Discussion Select an answer first