Create an Append-Only Data Pipeline Capable of Handling Both Batch and Streaming Data Using Delta. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   In a Structured Streaming query writing to Delta, what does the 'append' output mode guarantee?

   • AAll rows are rewritten on every trigger
   • BOnly new rows are written to the sink
   • COnly changed rows are updated in the sink
   • DRows are deleted if they no longer match the query
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   What is the purpose of using watermarks in a streaming pipeline?

   • ATo compress the streaming data
   • BTo increase the throughput of the stream
   • CTo encrypt the streaming data
   • DTo define how long to wait for late-arriving data
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A streaming job writes to a Delta table and uses a checkpoint directory for fault tolerance. Due to a network issue, the job restarts and reprocesses some micro-batches. The team notices duplicate records in the table. What is the most likely cause and solution?

   • AThe table does not have a primary key; add one to deduplicate.
   • BThe watermark is too short; increase it to allow more late data.
   • CThe output mode is set to update; change it to append to avoid duplicates.
   • DThe checkpoint directory is not being used correctly; ensure the same checkpoint location is used on restart.
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   What is the primary purpose of using unique keys or watermarking in an append-only pipeline?

   • ATo encrypt the data at rest
   • BTo compress the data before storage
   • CTo speed up the write process
   • DTo ensure repeated writes do not create duplicate data
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A team manages a Delta table that must be strictly append-only. They have set 'delta.appendOnly' to true. However, they need to perform a one-time correction of erroneous data. What is the best way to handle this?

   • ATemporarily set 'delta.appendOnly' to false, perform the correction, then set it back to true.
   • BCreate a new table with the corrected data and switch the pipeline to use it.
   • CUse the 'MERGE' command to update the erroneous records.
   • DDelete the erroneous records using 'DELETE' and re-append the correct ones.
   Reveal Solution Discussion Select an answer first