Describe the Execution Patterns of the Apache Spark™ Engine, Including Actions, Transformations, and Lazy Evaluation. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1expert · medium

   A Spark application reads a large dataset, applies a series of transformations, and then calls `count()`. The job fails during execution. The engineer wants to understand the exact sequence of transformations that were applied to the data before the failure. Which of the following is the most reliable way to obtain this information?

   • AUse `df.printSchema()` to view the schema after each transformation.
   • BUse `df.show()` to view the data and infer the transformations.
   • CUse `df.explain(true)` to view the logical and physical plans.
   • DUse `df.rdd.toDebugString()` to view the RDD lineage.
   Reveal Solution Discussion Select an answer first
2. 2application · easy

   A data engineer writes the following code: ```python df = spark.range(1000) df2 = df.filter(df["id"] % 2 == 0) df3 = df2.select("id") ``` They then call `df3.count()`. How many Spark jobs are triggered?

   • AOne, because `count()` triggers the execution of the entire transformation pipeline.
   • BTwo, because `filter` and `select` each trigger their own job.
   • CThree, because each transformation and the action trigger a job.
   • DZero, because no action has been called.
   Reveal Solution Discussion Select an answer first
4. 3application · easy

   A Spark application reads a JSON file, applies a `select` to choose specific columns, then a `filter`, and finally a `groupBy`. The application fails during the `groupBy` stage. The engineer wants to see the logical plan of the DataFrame to understand the sequence of transformations. Which method should they call?

   • A`df.explain(true)`
   • B`df.show()`
   • C`df.printSchema()`
   • D`df.collect()`
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   What does the lineage of an RDD or DataFrame represent in Apache Spark?

   • AThe sequence of transformations that were applied to produce the dataset.
   • BThe schema and data types of the dataset.
   • CThe physical location of each partition on the cluster nodes.
   • DThe list of actions that have been executed on the dataset.
   Reveal Solution Discussion Select an answer first
6. 5expert · medium

   A Spark job reads a large table, applies a filter that removes 90% of the rows, then performs a join with another table, and finally aggregates. The engineer notices that the join is processing a large amount of data and wants to optimize it. Which of the following optimizations would be most effective in reducing the data processed by the join?

   • AEnable AQE, which will automatically push the filter down to the data source.
   • BUse `broadcast()` on the second table to force a broadcast join, which reduces shuffle but not the data processed.
   • CUse `cache()` on the filtered DataFrame to avoid re-reading it, but this does not reduce the join's input size.
   • DManually apply the filter before the join in the code.
   Reveal Solution Discussion Select an answer first