Describe Apache Spark - Streaming Processing and Architecture DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 5)

Questions 21–25

1. 21expert · hard

   A financial trading firm uses Spark Streaming to process stock tick data. They require low latency but also need to recover quickly from failures without losing data. They are considering using a 1-second batch interval with checkpointing. Which trade-off should they be most concerned about?

   • AFault tolerance is automatically achieved without checkpointing.
   • BA 1-second batch interval eliminates the need for receivers.
   • CCheckpointing guarantees exactly-once processing without any performance impact.
   • DSmall batch intervals increase scheduling overhead and can reduce throughput.
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A data engineer wants to use Spark Streaming to process real-time clickstream data and then use Spark SQL to query the results. Which approach is most straightforward?

   • AUse DStreams and convert each RDD to a DataFrame for SQL queries.
   • BUse Spark SQL directly on the raw stream without any streaming API.
   • CUse DStreams and register the DStream as a temporary view.
   • DUse Structured Streaming to create a streaming DataFrame and register it as a temporary view.
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   Which Spark library can be used with Spark Streaming to perform real-time machine learning?

   • AMLlib
   • BSpark SQL
   • CGraphX
   • DSpark Core
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A media company uses Spark Streaming to process clickstream data. They set the batch interval to 2 seconds. The cluster has 4 worker nodes, each with 8 cores. The input rate is 10,000 events per second. The team observes that processing time per batch is consistently around 3 seconds, causing a backlog. What is the most appropriate action to reduce processing time without changing the batch interval?

   • AIncrease the number of partitions in the DStream by using more receivers or repartitioning.
   • BDecrease the number of worker nodes to reduce network overhead.
   • CReduce the batch interval to 1 second to process smaller batches faster.
   • DDisable checkpointing to avoid write overhead during each batch.
   Reveal Solution Discussion Select an answer first
6. 25expert · hard

   A healthcare analytics platform uses Spark Streaming to process patient vitals from IoT devices. They require that each reading is processed exactly once, even if a worker node fails mid-batch. They currently use a receiver-based approach with checkpointing enabled. After a recent failure, they observed duplicate processing of some records. Which change would most reliably achieve exactly-once processing?

   • AEnable speculative execution of tasks to handle slow nodes.
   • BIncrease the checkpoint interval to reduce the frequency of writes.
   • CSwitch to the Kafka direct stream API and ensure the sink is idempotent.
   • DUse a larger batch interval to allow more time for processing.
   Reveal Solution Discussion Select an answer first