Describe Adaptive Query Execution (AQE) and Its Benefits. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A Spark application joins a large orders table with a customers table. The join key is customer_id, and a small number of customers (e.g., a few high-volume customers) account for a disproportionately large number of orders. This causes a few reduce tasks to process significantly more data than others, leading to stragglers and poor cluster utilization. Which AQE feature directly addresses this problem?

   • ADynamic partition pruning
   • BDynamic skew join optimization
   • CDynamic join strategy selection
   • DCoalescing shuffle partitions
   Reveal Solution Discussion Select an answer first
2. 2foundation · easy

   At what point does Adaptive Query Execution modify the execution plan?

   • AOnly when the query fails and is retried.
   • BAfter the query finishes, to log performance metrics.
   • CBefore the query is submitted to the cluster.
   • DDuring query execution, after shuffle stages complete.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A data engineering team is considering enabling AQE on their production cluster. They are concerned about the impact on their existing queries, which were manually tuned with specific shuffle partition counts and join hints. What is the primary benefit of AQE that would address their concern about manual tuning?

   • AIt increases the number of shuffle partitions to improve parallelism
   • BIt automatically rewrites SQL queries to be more efficient
   • CIt caches intermediate results to avoid recomputation
   • DIt automatically re-optimizes the query plan at runtime based on actual data statistics
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A data engineer is monitoring a Spark job that performs a large sort-merge join. They notice in the Spark UI that a few reduce tasks are processing significantly more data than others, causing the job to take much longer than expected. The data is skewed on the join key. Which AQE feature should they verify is enabled to automatically handle this imbalance?

   • Aspark.sql.autoBroadcastJoinThreshold=10485760
   • Bspark.sql.adaptive.skewJoin.enabled=true
   • Cspark.sql.adaptive.coalescePartitions.enabled=true
   • Dspark.sql.adaptive.partitionPruning.enabled=true
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   Which of the following is a key benefit of Adaptive Query Execution?

   • AIt eliminates the need for any manual query tuning.
   • BIt automatically adjusts query plans to improve performance.
   • CIt replaces the need for data partitioning in the storage layer.
   • DIt guarantees a 100% reduction in query execution time.
   Reveal Solution Discussion Select an answer first