Perform Streaming Deduplication in Structured Streaming, Both with and Without Watermark Usage. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   What happens to a record that arrives after the watermark in a deduplication query?

   • AIt causes the watermark to be moved forward
   • BIt is stored in a separate state store
   • CIt is used to update the deduplication state
   • DIt is ignored for deduplication purposes
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A company ingests IoT sensor readings from devices that occasionally retransmit the same reading due to network glitches. Each reading has a unique `readingId` and a `timestamp` field set by the device. The team needs to remove duplicate readings from the stream but cannot tolerate dropping any legitimate reading that arrives late, even hours after the original. They plan to run the stream indefinitely. Which approach should they use?

   • AApply withWatermark("timestamp", "2 hours") followed by dropDuplicates("readingId") to bound state while still catching all late duplicates.
   • BApply dropDuplicates("readingId") without a watermark, accepting that state will grow with every unique reading seen.
   • CUse a mapGroupsWithState operation keyed by readingId to manually track all seen IDs in a bounded in-memory cache.
   • DApply dropDuplicates("readingId") and set the Spark session's spark.sql.shuffle.partitions to 1 to reduce memory overhead.
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   How does ignoring late data beyond the watermark affect the result completeness of a deduplication query?

   • AIt guarantees that no duplicates are ever emitted
   • BIt may cause some duplicates to be emitted
   • CIt has no effect on the output
   • DIt causes all late records to be dropped from the output
   Reveal Solution Discussion Select an answer first
5. 14expert · hard

   A streaming job deduplicates transaction events by `transactionId` with a 1-hour watermark on `eventTime`. The team observes that a duplicate transaction that arrives 90 minutes after the original is NOT dropped and is processed as a new record. They also notice the state store size remains stable. Which statement best explains this behavior?

   • AThe state store is full, so Spark stops tracking new keys after a certain limit.
   • BThe duplicate arrived beyond the watermark interval, so its state had already been evicted and the record is treated as new.
   • CThe watermark is applied on processing time, so event-time lateness is not tracked correctly.
   • DdropDuplicates only works on batch DataFrames, not streaming DataFrames.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Which method is used in Structured Streaming to deduplicate records without a watermark?

   • Afilter
   • Bdistinct
   • Cdrop
   • DdropDuplicates
   Reveal Solution Discussion Select an answer first