Implement Performance Tuning Strategies & Optimize Cluster Utilization, Including Partitioning, Repartitioning, Coalescing, Identifying Data Skew, and Reducing Shuffling ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 6)

Questions 26–29

1. 26foundation · easy

   Which of the following operations typically causes a shuffle in Spark?

   • Aselect()
   • BgroupBy()
   • Cfilter()
   • Dmap()
   Reveal Solution Discussion Select an answer first
2. 27application · medium

   A Spark job reads a large Parquet file. The Spark UI shows that the job is running with only 4 tasks, even though the cluster has 16 cores available. The engineer wants to increase parallelism to utilize the cluster better. Which action is most appropriate?

   • AUse coalesce(4) to keep the current partition count.
   • BIncrease spark.sql.files.maxPartitionBytes to create larger partitions.
   • CUse repartition(16) on the DataFrame after reading.
   • DSet spark.sql.shuffle.partitions to 16.
   Reveal Solution Discussion Select an answer first
4. 28application · easy

   A Spark DataFrame has 50 partitions, and the engineer wants to increase the number of partitions to 200 to improve parallelism for a subsequent wide transformation. Which method should be used?

   • Adf.repartition(200)
   • Bdf.coalesce(200)
   • Cdf.coalesce(50)
   • Ddf.repartition(50)
   Reveal Solution Discussion Select an answer first
5. 29foundation · easy

   How can you detect data skew by analyzing partition sizes?

   • ABy counting the number of tasks; more tasks indicate skew
   • BBy comparing the sizes of partitions; skew is present when some partitions are much larger than others
   • CBy checking that all partitions have exactly the same number of rows
   • DBy looking at the total size of the dataset; larger datasets are always skewed
   Reveal Solution Discussion Select an answer first