Configure Spark Partitioning in Distributed Data Processing, Including Shuffles and Partitions ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 5)

Questions 21–25

1. 21foundation · easy

   How can partitioning improve data locality?

   • ABy compressing data to fit on fewer nodes
   • BBy ensuring that data with the same key is stored on the same node, reducing shuffle for subsequent operations
   • CBy sorting data within each partition
   • DBy increasing the number of partitions to spread data across more nodes
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   What is the primary purpose of hash partitioning in Spark?

   • ATo minimize the number of partitions
   • BTo sort data within each partition
   • CTo distribute data evenly across partitions based on a hash of the partition key
   • DTo keep related records on the same node to improve locality
   Reveal Solution Discussion Select an answer first
4. 23application · medium

   A data engineer is joining a large fact table (10 TB) with a small dimension table (50 MB) on a non-key column. The current join causes a full shuffle of the fact table, which is expensive. The engineer wants to minimize shuffle overhead. Which approach should the engineer use?

   • AUse a range partitioner on the fact table only.
   • BUse a sort-merge join by setting spark.sql.join.prefersortmergejoin to true.
   • CUse a broadcast join by hinting the dimension table with broadcast.
   • DRepartition both tables by the join key using repartition(200).
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A Spark job joins a large fact table with a small dimension table. The dimension table is 200 MB, which exceeds the default broadcast threshold. The engineer wants to avoid shuffling the fact table. Which approach should the engineer take?

   • AUse repartition on both tables by the join key.
   • BUse coalesce on the fact table to reduce partitions.
   • CUse a sort-merge join with a higher number of shuffle partitions.
   • DIncrease spark.sql.autoBroadcastJoinThreshold to 300 MB and use a broadcast hint.
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A Spark job reads a large DataFrame and then performs a filter that removes 90% of the rows. The engineer wants to reduce the number of partitions before writing the result to avoid many small files. Which operation should the engineer use?

   • Acoalesce(500)
   • Bcoalesce(10)
   • Crepartition(10)
   • Drepartition(500)
   Reveal Solution Discussion Select an answer first