Describe Apache Spark and Its Architecture DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 1)

Questions 1–5

1. 1expert · medium

   A Spark job is running slowly. The DAG shows a wide dependency with a shuffle. The team suspects that the shuffle is causing excessive disk I/O. Which configuration change is most likely to reduce shuffle disk I/O?

   • AIncrease spark.executor.memory
   • BIncrease spark.sql.shuffle.partitions
   • CEnable shuffle compression
   • DUse a broadcast join
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A Spark application processes a large dataset. The DAG shows that the job is split into two stages. The first stage completes quickly, but the second stage has many tasks that run slowly. Which of the following is the most likely cause?

   • AThe second stage requires a shuffle, causing data to be transferred across the network
   • BThe first stage used too many executors, leaving few for the second stage
   • CThe driver is performing a collect() operation in the second stage
   • DThe second stage is running on a single core per executor
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   What is the processing model used by Spark Streaming (DStreams)?

   • ABatch processing only, with no streaming capability
   • BSynchronous processing, where the stream is blocked until all data is processed
   • CTrue event-at-a-time processing, where each record is processed individually
   • DMicro-batch processing, where data is collected into small batches and processed as RDDs
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A team is using Spark Streaming to process sensor data. They need to ensure that each event is processed exactly once, even if a node fails. Which feature of Spark Streaming supports this requirement?

   • AIn-memory processing only
   • BMicro-batch processing
   • CDynamic resource allocation
   • DWrite-ahead logs and fault-tolerant receivers
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   In a Spark application, the driver program is responsible for creating RDDs and scheduling tasks. If the driver fails, what happens to the application?

   • AThe executors continue processing the remaining tasks independently
   • BThe application fails and must be restarted
   • CThe cluster manager automatically restarts the driver and resumes the application
   • DThe executors take over the driver's role and continue the application
   Reveal Solution Discussion Select an answer first