Explain the Structured Streaming Engine in Spark, Including Its Functions, Programming Model, Micro-Batch Processing, Exactly-Once Semantics, and Fault Tolerance Mechanisms. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A company is evaluating Structured Streaming for a new project. They need a stream processing engine that can handle both high throughput and moderate latency, and they want to leverage their existing Spark SQL skills. Which statement best describes why Structured Streaming fits this need?

   • AIt provides sub-millisecond latency, making it suitable for real-time applications.
   • BIt requires a separate cluster from batch processing, ensuring isolation.
   • CIt only supports append output mode, simplifying the programming model.
   • DIt is built on the Spark SQL engine, so it uses the same DataFrame API and can achieve high throughput with micro-batch processing.
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A streaming query uses a windowed aggregation with event-time. The team wants to output the final result for each window only once, after the window is closed. Which output mode and watermark configuration should they use?

   • AComplete mode with no watermark.
   • BAppend mode with no watermark.
   • CAppend mode with a watermark that is greater than the maximum expected lateness.
   • DUpdate mode with a watermark that is less than the maximum expected lateness.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A team is designing a streaming pipeline that must handle a burst of 1 million events per second for a few minutes, then drop to near zero. They need low latency during the burst. Which approach best balances cost and performance?

   • AUse a custom batch job that runs every minute to process the burst.
   • BUse a fixed-size cluster sized for the peak burst to ensure low latency.
   • CUse Structured Streaming with a fixed trigger interval and autoscaling enabled on the cluster.
   • DUse a small cluster and accept higher latency during the burst.
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   What is the role of a replayable source in Structured Streaming's fault tolerance?

   • AIt ensures that the source never fails
   • BIt allows the source to be re-read from a specific offset after a failure
   • CIt stores all data in memory for fast access
   • DIt automatically deduplicates data in the source
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A streaming job is experiencing high latency because each micro-batch is taking longer than the trigger interval. The team wants to improve throughput without changing the trigger interval. Which action is most effective?

   • AIncrease the number of partitions in the source and the parallelism of the query.
   • BDisable checkpointing to reduce overhead.
   • CSwitch to a single partition to reduce coordination overhead.
   • DDecrease the trigger interval to process smaller batches.
   Reveal Solution Discussion Select an answer first