Describe the Architecture of Apache Spark™, Including DataFrame and Dataset Concepts, SparkSession Lifecycle, Caching, Storage Levels, and Garbage Collection. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A team caches a large DataFrame with persist(StorageLevel.MEMORY_AND_DISK_SER) to save memory. Later, they notice that queries on the cached data are slower than expected. They suspect serialization overhead. They have enough memory to store the data deserialized. What should they do to improve query performance?

   • AUse cache() instead, which defaults to MEMORY_ONLY
   • BChange the storage level to MEMORY_AND_DISK and re-persist the DataFrame
   • CSwitch to MEMORY_ONLY_SER to keep more data in memory
   • DKeep MEMORY_AND_DISK_SER but increase the executor memory
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A Spark application is running on a cluster with limited resources. The driver is running out of memory because it is collecting large results to the driver using collect(). The team wants to reduce the driver's memory usage without changing the application logic. Which approach is most effective?

   • AUse persist(StorageLevel.DISK_ONLY) to store the data on disk
   • BCache the DataFrame before calling collect()
   • CIncrease the driver memory with spark.driver.memory
   • DUse take(n) instead of collect() to limit the number of rows returned
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   Which storage level stores RDD partitions in memory, and if they do not fit, recomputes them instead of spilling to disk?

   • AMEMORY_AND_DISK
   • BMEMORY_ONLY_SER
   • CMEMORY_ONLY
   • DDISK_ONLY
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A Scala developer is working with a Dataset[Person] where Person is a case class. They need to perform a filter operation on a field that exists in the case class. What is the primary advantage of using a Dataset over a DataFrame in this scenario?

   • ADatasets automatically handle schema evolution
   • BBetter performance because Datasets always use Tungsten
   • CCompile-time type safety, so referencing a non-existent field fails at compile time
   • DDatasets are easier to use in Python
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A developer is debugging a Spark application and wants to understand the relationship between SparkSession, SparkContext, and SQLContext. They notice that after creating a SparkSession, they can access the SparkContext via spark.sparkContext. What does this tell them about the relationship?

   • ASparkContext is a newer API that replaces SparkSession
   • BSQLContext is the parent of SparkSession
   • CSparkSession encapsulates SparkContext and SQLContext, providing a unified entry point
   • DSparkSession is a separate component that has no relation to SparkContext
   Reveal Solution Discussion Select an answer first