Perform Streaming Deduplication in Structured Streaming, Both with and Without Watermark Usage. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   How does using a watermark with dropDuplicates affect deduplication in Structured Streaming?

   • AIt makes deduplication happen only on processing time
   • BIt bounds the state to only the watermark interval
   • CIt eliminates the need to specify a key column
   • DIt prevents all duplicate records from ever being emitted
   Reveal Solution Discussion Select an answer first
2. 7application · easy

   A team is deduplicating clickstream data. Each click has a `clickId` and a `clickTime` event-time column. The team wants to use a watermark to handle late data. Which statement about event time and processing time is correct in this context?

   • AThe watermark should be based on processing time to accurately reflect when the click was generated.
   • BProcessing time is when the click occurred, and event time is when it is read; the watermark should be based on processing time.
   • CEvent time and processing time are the same for streaming data, so either can be used.
   • DEvent time is when the click occurred, and processing time is when the click is read by the stream; the watermark should be based on event time.
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   What is the primary benefit of state cleanup in deduplication with a watermark?

   • ASpeeds up the processing of each micro-batch
   • BPrevents unbounded state growth
   • CAllows deduplication without specifying a key
   • DIncreases the accuracy of deduplication
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   What happens to state for keys older than the watermark when using dropDuplicates with a watermark?

   • AIt is discarded
   • BIt is kept indefinitely
   • CIt is moved to a separate storage
   • DIt is used to reprocess old data
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A streaming job deduplicates orders by `orderId` with a 1-hour watermark on `orderTime`. An order with `orderTime` of 10:00 is processed at 10:00. A duplicate of the same order arrives at 11:30. What will happen to the duplicate?

   • AIt will be dropped because the watermark is based on processing time.
   • BIt will be dropped because the orderId is still in state.
   • CIt will cause the stream to fail because the duplicate is too late.
   • DIt will be treated as a new order because the watermark has passed 10:00 and the state was evicted.
   Reveal Solution Discussion Select an answer first