Describe the Execution Patterns of the Apache Spark™ Engine, Including Actions, Transformations, and Lazy Evaluation. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–19

1. 16application · easy

   A data engineer is working with a large DataFrame and wants to check if the data contains any null values in a specific column. They write the following code: ```python null_count = df.filter(df["col"].isNull()).count() ``` Which of the following statements accurately describes what happens when this code is executed?

   • AThe `filter` transformation is executed immediately, and then the `count` action is executed on the filtered result.
   • BThe `count` action triggers the execution of the `filter` transformation, and the entire pipeline is executed as a single job.
   • CThe `count` action is executed first, and then the `filter` transformation is applied to the result.
   • DThe `filter` transformation is executed first, and then the `count` action is executed on the result, resulting in two separate jobs.
   Reveal Solution Discussion Select an answer first
2. 17expert · medium

   A Spark job processes a large dataset with the following steps: read, filter, join with a small lookup table, groupBy, and agg. The job is slow, and the engineer suspects that the join is causing excessive shuffling. They enable Adaptive Query Execution (AQE). Which of the following optimizations will AQE most likely apply to improve the job's performance?

   • APush the filter down to the data source to reduce the amount of data read.
   • BReorder the transformations to perform the groupBy before the join, reducing the data size.
   • CConvert the join to a broadcast join if the lookup table is small enough, reducing shuffle.
   • DAutomatically cache the lookup table to avoid re-reading it.
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   What happens when an action is called on a Spark DataFrame?

   • ASpark returns the DataFrame without any computation.
   • BOnly the action itself is executed; transformations are ignored.
   • CSpark saves the DataFrame to disk and then executes the action.
   • DSpark executes all the transformations in the lineage up to that point.
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   Which statement correctly describes the difference between a transformation and an action in Apache Spark?

   • ATransformations are executed immediately, while actions are deferred until a transformation is called.
   • BTransformations return a value to the driver program, while actions return a new DataFrame or RDD.
   • CTransformations and actions both return distributed datasets, but actions are executed on the driver only.
   • DTransformations return a new distributed dataset (e.g., DataFrame, RDD) and are lazily evaluated, while actions return a non-distributed value or write data to storage.
   Reveal Solution Discussion Select an answer first