Understand the Basic Tuning Parameters (spark.sql.shuffle.partitions:, Spark.default.parallelism, Spark.executor/driver.memory, spark.sql.autoBroadcastJoinThreshold) and Re-Measure the Performance. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A data engineer is tuning a join between a 2 GB table and a 1.5 GB table. The engineer sets spark.sql.autoBroadcastJoinThreshold to 2 GB, expecting a BroadcastHashJoin. However, the Spark UI shows a SortMergeJoin. The engineer has verified that ANALYZE TABLE was run on both tables. What is the most likely reason for the SortMergeJoin?

   • AThe estimated size of the 1.5 GB table exceeds 2 GB after serialization overhead is considered
   • BThe spark.default.parallelism is set too low
   • CThe spark.executor.memory is too low
   • DThe spark.sql.shuffle.partitions is set too high
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   Which Spark configuration parameter controls the number of partitions used when data is shuffled during SQL and DataFrame operations?

   • Aspark.sql.shuffle.partitions
   • Bspark.executor.memory
   • Cspark.default.parallelism
   • Dspark.sql.autoBroadcastJoinThreshold
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A data engineer is running a Spark job that performs a large join and then caches the result for reuse in multiple queries. The job is spilling data to disk during the join, causing slow performance. The cluster has 4 executors, each with 4 GB of memory. Which change is most likely to reduce disk spills?

   • AIncrease spark.executor.memory to 8g
   • BDecrease spark.sql.autoBroadcastJoinThreshold to 1m
   • CDecrease spark.sql.shuffle.partitions to 4
   • DIncrease spark.driver.memory to 8g
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   A data engineer wants to increase the memory available to each worker node process that executes tasks and stores data in memory. Which parameter should they adjust?

   • Aspark.sql.shuffle.partitions
   • Bspark.driver.memory
   • Cspark.executor.memory
   • Dspark.default.parallelism
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   After tuning a Spark job, a data engineer wants to confirm the improvement. Which of the following is the most reliable way to measure the impact?

   • ACheck the number of lines in the output data.
   • BCompare the job duration and resource usage from the Spark UI before and after the change.
   • CVerify that the job produces the same results as before.
   • DAsk another engineer to review the code.
   Reveal Solution Discussion Select an answer first