Describe Adaptive Query Execution (AQE) and Its Benefits. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A team is running a join between a large fact table and a small lookup table. The lookup table is usually small enough to be broadcast, but its size varies. The team has set spark.sql.autoBroadcastJoinThreshold to 10MB. On a particular day, the lookup table is 12MB, and the join is running slowly. The team wants Spark to automatically use a broadcast join if possible, without manually changing the threshold. What should they do?

   • AAdd a broadcast hint to the query
   • BEnable AQE by setting spark.sql.adaptive.enabled=true
   • CSet spark.sql.adaptive.coalescePartitions.enabled=true
   • DIncrease spark.sql.autoBroadcastJoinThreshold to 20MB
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   Where can you observe the effects of Adaptive Query Execution in the Spark UI?

   • AIn the Storage tab, by checking cached RDDs
   • BIn the Environment tab, by reviewing configuration settings
   • CIn the Executors tab, by viewing memory usage
   • DIn the SQL tab, by examining the physical query plan
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   Why might AQE switch from a sort-merge join to a broadcast join at runtime?

   • ABecause the query contains a filter condition
   • BBecause the cluster has spare memory
   • CBecause the actual size of one side of the join is small enough to broadcast
   • DBecause the join key is a string type
   Reveal Solution Discussion Select an answer first
5. 14application · medium · select all that apply

   A data engineering team is evaluating the benefits of enabling AQE on their Spark cluster. They want to understand which specific optimizations AQE can perform automatically. Select all the optimizations that are core features of AQE.

   • ASplitting and repartitioning skewed partitions to balance workload
   • BAutomatically choosing between broadcast and sort-merge join strategies
   • CDynamically pruning partitions based on runtime statistics
   • DDynamically adjusting the number of shuffle partitions to reduce overhead
   • EAutomatically rewriting SQL queries to use more efficient syntax
   Check answers Discussion Select an answer first
6. 15expert · hard

   A data engineer is running a large sort-merge join. The join key is heavily skewed, with a few keys having millions of records. They have enabled AQE and the skew join optimization. However, they notice in the Spark UI that the skew is still causing stragglers. What is the most likely reason the skew join optimization is not working?

   • AThe skew join optimization is disabled by default
   • BThe skew ratio is below the threshold set by spark.sql.adaptive.skewJoin.skewedPartitionFactor
   • CThe join is not a sort-merge join; it is a broadcast join
   • DThe skew join optimization only works for broadcast joins, not sort-merge joins
   Reveal Solution Discussion Select an answer first