Perform Streaming Deduplication in Structured Streaming, Both with and Without Watermark Usage. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 5)

Questions 21–24

1. 21expert · hard

   A team is running a deduplication stream with a 1-hour watermark. They monitor the state store and notice it is still growing, even though they expected it to be bounded. Which of the following is the most likely reason?

   • AThe watermark is applied after dropDuplicates, so it has no effect on state cleanup.
   • BThe watermark is not advancing because the event-time column contains null or non-monotonic values.
   • CThe state store grows because dropDuplicates always retains all keys, regardless of watermark.
   • DThe state store grows because the watermark is set to 1 hour but the stream has been running for less than 1 hour.
   Reveal Solution Discussion Select an answer first
2. 22application · easy

   A team is designing a deduplication pipeline for user activity logs. Each log has a `userId`, a `logId`, and a `createdAt` timestamp set by the client application. The team wants to use a watermark to bound state. Which column should they use for the watermark?

   • AcreatedAt, because it represents when the activity actually occurred (event time).
   • BThe time the record is read by the stream, because it is always available and monotonically increasing.
   • CuserId, because it groups events by user and helps with state partitioning.
   • DlogId, because it is unique and can be used to order events.
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   Why is event-time used for watermark-based deduplication instead of processing-time?

   • AEvent-time reflects when the event actually occurred, allowing correct handling of late data
   • BProcessing-time is too variable to be used for any purpose
   • CProcessing-time is not available in Structured Streaming
   • DEvent-time is always consistent across all partitions
   Reveal Solution Discussion Select an answer first
5. 24foundation · easy

   What is a key characteristic of deduplication without a watermark in Structured Streaming?

   • AState is automatically cleaned up after a fixed time
   • BOnly the last 10 minutes of data are considered
   • CDeduplication is performed only on processing time
   • DState grows unboundedly
   Reveal Solution Discussion Select an answer first