Implement Performance Tuning Strategies & Optimize Cluster Utilization, Including Partitioning, Repartitioning, Coalescing, Identifying Data Skew, and Reducing Shuffling ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A data engineer notices that a Spark job reading a large Parquet table is taking much longer than expected. In the Spark UI, the job shows a few tasks taking significantly longer than the majority, and the input data size per task varies widely. The cluster has 16 executors, each with 4 cores. Which action is most likely to improve the job's performance?

   • AIncrease spark.sql.files.maxPartitionBytes to create larger partitions.
   • BCall repartition(64) on the DataFrame before the expensive operation.
   • CSet spark.sql.shuffle.partitions to 200.
   • DUse coalesce(16) to reduce the number of partitions.
   Reveal Solution Discussion Select an answer first
2. 7expert · hard

   A Spark cluster has 10 executors, each with 4 cores and 16 GB of memory. A job processes a large DataFrame and performs a groupBy operation. The Spark UI shows that the job is running with only 10 tasks, and the cluster is underutilized. The engineer suspects data skew because a few tasks are taking much longer than others. Which combination of actions would best address both the low parallelism and the skew?

   • AUse coalesce(10) to reduce partitions and rely on the existing shuffle partitions.
   • BUse repartition(40) on the DataFrame before the groupBy and increase spark.sql.shuffle.partitions to 200.
   • CIncrease spark.sql.shuffle.partitions to 40 and use salting on the groupBy key.
   • DIncrease spark.executor.cores to 8 and keep spark.sql.shuffle.partitions at 10.
   Reveal Solution Discussion Select an answer first
4. 8application · easy

   A Spark DataFrame has 10 partitions, and the engineer needs to increase the number of partitions to 100 to improve parallelism for a subsequent wide transformation. Which method should be used?

   • Adf.coalesce(100)
   • Bdf.repartition(10)
   • Cdf.repartition(100)
   • Ddf.coalesce(10)
   Reveal Solution Discussion Select an answer first
5. 9application · easy

   A Spark job processes a DataFrame with 200 partitions. After a filter operation, many partitions become empty. The engineer wants to reduce the number of partitions to 50 to improve the efficiency of subsequent operations, while minimizing the shuffle overhead. Which method should be used?

   • Adf.coalesce(200)
   • Bdf.repartition(200)
   • Cdf.repartition(50)
   • Ddf.coalesce(50)
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A Spark job performs multiple joins on the same key. The engineer notices that each join triggers a shuffle, causing significant network overhead. Which strategy would most effectively reduce the number of shuffles?

   • ACache the DataFrames after the first join.
   • BIncrease spark.sql.shuffle.partitions to reduce shuffle size.
   • CUse bucketing on the join key for both DataFrames.
   • DUse broadcast variables to share the join key data.
   Reveal Solution Discussion Select an answer first