Describe the Execution Patterns of the Apache Spark™ Engine, Including Actions, Transformations, and Lazy Evaluation. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 3)

Questions 11–15

1. 11application · easy

   A data engineer writes the following code in a notebook: ```python df = spark.read.parquet("data") df_filtered = df.filter(df["age"] > 21) df_mapped = df_filtered.withColumn("age_group", df_filtered["age"] / 10) ``` The engineer then runs `df_mapped.show(5)`. How many Spark jobs will be triggered by this code?

   • AZero, because no action has been called yet.
   • BTwo, because `show()` triggers one job for the transformations and another for the display.
   • COne, because `show()` is the only action and it triggers the entire transformation pipeline.
   • DThree, because each transformation (read, filter, withColumn) triggers its own job.
   Reveal Solution Discussion Select an answer first
2. 12expert · medium

   A Spark application reads a large fact table and a small dimension table, performs a join, then applies a filter, and finally aggregates. The engineer notices that the filter is applied after the join, causing the join to process more data than necessary. They want to optimize the query. Which of the following is the most effective way to leverage Spark's optimization capabilities?

   • AEnable AQE, which will automatically reorder the transformations to apply the filter before the join.
   • BUse `cache()` on the fact table after the filter to avoid re-reading it, but this does not change the join order.
   • CUse `broadcast()` on the dimension table to force a broadcast join, which reduces shuffle but does not address the filter order.
   • DManually reorder the transformations in the code to apply the filter before the join.
   Reveal Solution Discussion Select an answer first
4. 13application · easy

   A data engineer writes the following code: ```python df = spark.read.csv("data.csv") df_filtered = df.filter(df["age"] > 18) df_mapped = df_filtered.withColumn("adult", df_filtered["age"] >= 18) ``` The engineer then calls `df_mapped.count()`. Which of the following operations are actions?

   • A`filter`, `withColumn`, and `count`
   • B`read` and `count`
   • C`read`, `filter`, `withColumn`, and `count`
   • D`count` only
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   In Spark, how does the Directed Acyclic Graph (DAG) relate to the execution of transformations?

   • AThe DAG represents the dependencies between transformations, and Spark uses it to schedule and execute tasks.
   • BThe DAG is built only after an action is called and represents the physical execution plan.
   • CThe DAG is a linear sequence of transformations that must be executed in order.
   • DThe DAG is stored on disk and used only for debugging purposes.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   What is the primary benefit of lazy evaluation in Apache Spark?

   • AIt allows Spark to store all intermediate results in memory for faster reuse.
   • BIt enables Spark to optimize the entire transformation pipeline before executing any computation.
   • CIt ensures that every transformation is executed immediately to avoid data loss.
   • DIt allows Spark to run transformations in parallel on the driver node.
   Reveal Solution Discussion Select an answer first