Describe the Architecture of Apache Spark™, Including DataFrame and Dataset Concepts, SparkSession Lifecycle, Caching, Storage Levels, and Garbage Collection. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   How does garbage collection (GC) affect Spark performance?

   • AFrequent GC pauses can slow down task execution and reduce throughput.
   • BGC increases the amount of memory available to executors.
   • CGC has no impact on Spark performance.
   • DGC automatically optimizes the storage level of cached data.
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A Scala developer needs to process a dataset of sensor readings. They want compile-time type safety so that accessing a non-existent column fails at compile time rather than at runtime. They also need to perform relational operations like filter and groupBy. Which API should they use?

   • ADataset[T] with a case class representing the sensor reading schema
   • BDataFrame with a predefined schema
   • CRDD[Row] with manual schema validation
   • DDataset[Row] without a case class
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A Spark application is running on a standalone cluster. The driver node fails mid-job. The cluster manager restarts the driver, but the job does not resume from where it left off. Which component is responsible for scheduling tasks on executors, and why does the job not resume automatically?

   • AThe executors; they store the intermediate results and can rebuild the DAG
   • BThe cluster manager; it should resubmit the job from the last checkpoint
   • CThe driver; it holds the DAG and task scheduling state, so a restart loses the job progress
   • DThe SparkSession; it persists the job state to disk for recovery
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   What is the role of the SparkSession in a Spark application?

   • AIt is responsible for storing data on disk for fault tolerance.
   • BIt is the entry point for programming Spark with the DataFrame and Dataset APIs.
   • CIt manages the cluster resources and schedules tasks.
   • DIt is a low-level API for manipulating RDDs directly.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A developer is migrating a legacy Spark 1.x application to Spark 3.x. The old code creates a SparkContext and then a SQLContext separately to run SQL queries. In the new version, they want to simplify the setup and ensure the application uses a single unified entry point. What should they do?

   • ACreate a SparkSession using SparkSession.builder().appName(...).getOrCreate() and use it for both SQL and DataFrame operations
   • BKeep using SparkContext and SQLContext as before, since they are still the recommended API
   • CUse SparkContext to create DataFrames and SQLContext only for SQL queries
   • DCreate a new SparkSession for every SQL query to avoid state leakage
   Reveal Solution Discussion Select an answer first