Perform Streaming Deduplication in Structured Streaming, Both with and Without Watermark Usage. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   How does a watermark help in state cleanup for deduplication in Structured Streaming?

   • AIt compresses the state to reduce memory usage
   • BIt moves state to disk automatically
   • CIt prevents any state from being created
   • DIt triggers removal of state for keys older than the watermark
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   What does the withWatermark method require as parameters?

   • AA list of columns to deduplicate on
   • BProcessing-time column name and a delay threshold
   • CA threshold and a list of columns to aggregate
   • DEvent-time column name and a delay threshold
   Reveal Solution Discussion Select an answer first
4. 18application · easy

   A team is deduplicating log entries by `logId`. Each log entry has a `generatedAt` timestamp set by the source system. The team wants to use a watermark to limit state, but they are unsure whether to use `generatedAt` or the time the record is read by the stream. What is the correct choice and why?

   • AUse either, since the watermark works the same regardless of the timestamp source.
   • BUse generatedAt because it is event time and reflects when the log was actually created.
   • CUse the stream read time because it is always available and monotonically increasing.
   • DUse generatedAt only if the stream is read in order; otherwise, use processing time.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A team is building a Structured Streaming job that deduplicates user actions using `dropDuplicates("actionId")`. They notice the state store is growing very large and want to enable periodic cleanup. The `actionTime` column contains the time the action occurred. What must they add to the pipeline?

   • ASet spark.sql.streaming.stateStore.providerClass to a memory-optimized provider.
   • BAdd a processing-time column using current_timestamp() and watermark on that column.
   • CCall withWatermark("actionTime", "30 minutes") before dropDuplicates("actionId").
   • DCall dropDuplicates("actionId", "actionTime") to include the timestamp in the deduplication key.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   Which method is used to configure a watermark on an event-time column in Structured Streaming?

   • Awatermark
   • BwithColumn
   • CsetWatermark
   • DwithWatermark
   Reveal Solution Discussion Select an answer first