Understand the Basic Tuning Parameters (spark.sql.shuffle.partitions:, Spark.default.parallelism, Spark.executor/driver.memory, spark.sql.autoBroadcastJoinThreshold) and Re-Measure the Performance. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A data engineer is optimizing a join between a 5 GB fact table and a 300 MB dimension table. The current job uses a SortMergeJoin, which is slow. After setting spark.sql.autoBroadcastJoinThreshold to 500 MB, the engineer re-runs the job but notices it still uses SortMergeJoin. What is the most likely reason?

   • AThe spark.sql.shuffle.partitions setting is too low
   • BThe dimension table's size is estimated to be larger than 500 MB after statistics are computed
   • CThe spark.default.parallelism is set too high
   • DThe spark.executor.memory is too low
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   Which Spark configuration parameter sets the default number of partitions for RDD operations when none is specified?

   • Aspark.sql.autoBroadcastJoinThreshold
   • Bspark.sql.shuffle.partitions
   • Cspark.default.parallelism
   • Dspark.executor.memory
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   Which Spark configuration parameter controls the maximum size of a table that will be broadcast to all executors during a join?

   • Aspark.sql.autoBroadcastJoinThreshold
   • Bspark.default.parallelism
   • Cspark.executor.memory
   • Dspark.sql.shuffle.partitions
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A data engineer is working with an RDD-based pipeline that uses parallelize() to create an RDD from a list of 10,000 elements. The cluster has 8 cores. The engineer wants to ensure that the RDD is distributed across all cores for maximum parallelism. Which configuration is most appropriate?

   • ASet spark.default.parallelism to 8
   • BSet spark.sql.autoBroadcastJoinThreshold to 8m
   • CSet spark.executor.memory to 8g
   • DSet spark.sql.shuffle.partitions to 8
   Reveal Solution Discussion Select an answer first
6. 10expert · hard

   A data engineer is running a job that collects a large aggregated result to the driver for further processing. The job fails with an OutOfMemoryError on the driver. The engineer increases spark.driver.memory from 4 GB to 8 GB, but the job still fails. The driver node has 16 GB of RAM. What is the most likely reason the increase did not help?

   • AThe collected result is larger than 8 GB
   • BThe spark.sql.autoBroadcastJoinThreshold is set too low
   • CThe spark.executor.memory is too low
   • DThe spark.sql.shuffle.partitions is set too high
   Reveal Solution Discussion Select an answer first