Perform Streaming Deduplication in Structured Streaming, Both with and Without Watermark Usage. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A team is building a deduplication pipeline for financial trades. Each trade has a `tradeId` and an `executionTime` (event time). The team wants to deduplicate trades within a 5-minute watermark. They also need to ensure that the watermark is correctly computed. Which statement about the watermark behavior is correct?

   • AThe watermark is fixed at the start of the stream and does not advance, so all trades are deduplicated against the initial state.
   • BThe watermark is computed per partition, so different partitions may have different watermarks, leading to inconsistent deduplication.
   • CThe watermark advances to the maximum observed event time minus 5 minutes, and any trade with event time older than the watermark is considered late and its state is evicted.
   • DThe watermark is set to the current processing time minus 5 minutes, so trades are deduplicated based on when they were processed.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A company ingests user actions from mobile devices. Each action has a unique `actionId` and a `deviceTime` timestamp. The devices can be offline and send data hours later, so duplicates can arrive very late. The team wants to deduplicate actions but is concerned about memory usage. What is the most important consideration?

   • AThe watermark should be set on deviceTime to guarantee no duplicate is ever missed.
   • BWith a watermark, state is always small and memory usage is never a concern.
   • CWithout a watermark, state grows with every unique actionId, which may exhaust memory over time.
   • DThe deduplication key should include deviceTime to reduce state size.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A streaming pipeline deduplicates payment events by `paymentId` with a 30-minute watermark on `eventTime`. The team finds that a duplicate payment event arriving 45 minutes after the original is processed as a new payment, causing a double charge. They need to decide how to fix this. What is the most appropriate action?

   • AIncrease the watermark to 1 hour to retain state longer and catch the late duplicate.
   • BRemove the watermark entirely to guarantee no duplicate is ever missed.
   • CUse a processing-time watermark instead of event-time to catch the duplicate.
   • DKeep the watermark at 30 minutes and accept that some late duplicates will be missed.
   Reveal Solution Discussion Select an answer first
5. 4expert · hard

   A streaming pipeline deduplicates ad impressions by `impressionId` with a 2-hour watermark on `eventTime`. The team later discovers that some impressions are re-sent by the ad server up to 3 hours after the original event. They need to decide whether to keep the current watermark or adjust it. What is the most important trade-off to consider?

   • AIncreasing the watermark to 3 hours would catch more late duplicates but increase state size and memory usage.
   • BDecreasing the watermark to 1 hour would reduce state but also drop more late duplicates, which is always undesirable.
   • CThe watermark should be set to the maximum processing-time delay observed in the cluster.
   • DThe watermark should be removed entirely to catch all late duplicates without any state growth.
   Reveal Solution Discussion Select an answer first
6. 5expert · hard

   A team is running a deduplication stream without a watermark. They notice the state store is consuming a lot of memory and want to reduce it. They are considering adding a watermark. What is the most important trade-off they must accept?

   • AThe watermark will reduce the number of unique keys that can be deduplicated.
   • BSome late duplicates beyond the watermark will no longer be dropped, reducing deduplication completeness.
   • CThe watermark will increase processing latency because it must wait for more data.
   • DThe watermark will cause the stream to fail if any event is late.
   Reveal Solution Discussion Select an answer first