Understand the Basic Tuning Parameters (spark.sql.shuffle.partitions:, Spark.default.parallelism, Spark.executor/driver.memory, spark.sql.autoBroadcastJoinThreshold) and Re-Measure the Performance. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A data engineer is migrating a legacy RDD-based ETL pipeline to a new Databricks cluster. The pipeline uses sc.textFile() and map/filter/reduceByKey transformations. The engineer notices that the initial stage reads the input file into only 4 partitions, causing poor parallelism across the 16-core cluster. Which parameter should be adjusted to increase the number of partitions for the initial RDD read?

   • AIncrease spark.default.parallelism to 16
   • BIncrease spark.sql.shuffle.partitions to 16
   • CIncrease spark.sql.autoBroadcastJoinThreshold to 16m
   • DIncrease spark.executor.memory to 16g
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A data engineer runs a DataFrame aggregation that involves a wide shuffle. The job is slow, and the Spark UI shows that the shuffle stage has 200 tasks, but the cluster only has 16 cores. Many tasks are very small and finish quickly, causing high scheduling overhead. Which change is most appropriate to reduce the overhead?

   • AIncrease spark.executor.memory
   • BIncrease spark.sql.shuffle.partitions to 400
   • CDecrease spark.sql.shuffle.partitions to 16
   • DDecrease spark.executor.memory
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A data engineer tunes a Spark job by increasing spark.executor.memory from 4 GB to 8 GB and decreasing spark.sql.shuffle.partitions from 200 to 50. After re-running the job, the engineer wants to confirm that the changes improved performance. Which metric from the Spark UI is most direct evidence of improvement?

   • AShuffle spill (disk) size decreased
   • BCluster CPU utilization decreased
   • CNumber of tasks increased
   • DDriver memory usage increased
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A data engineer is running a Spark job that performs a large groupBy and then collects the aggregated results to the driver for further processing in a pandas DataFrame. The job fails with an OutOfMemoryError on the driver node. The driver has 8 GB of RAM, and the collected result is approximately 6 GB. Which configuration change is most appropriate?

   • AIncrease spark.executor.memory to 10g
   • BIncrease spark.driver.memory to 10g
   • CDecrease spark.sql.shuffle.partitions to 4
   • DIncrease spark.sql.autoBroadcastJoinThreshold to 10g
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   Which Spark configuration parameter controls the amount of memory allocated to the JVM process that runs the Spark application's main() function and handles task scheduling?

   • Aspark.executor.memory
   • Bspark.default.parallelism
   • Cspark.sql.shuffle.partitions
   • Dspark.driver.memory
   Reveal Solution Discussion Select an answer first