Create an Append-Only Data Pipeline Capable of Handling Both Batch and Streaming Data Using Delta. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   Which parameter must be specified when starting a Structured Streaming query to enable checkpointing?

   • AcheckpointLocation
   • Bwatermark
   • CoutputMode
   • Dtrigger
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A Delta table is used to store clickstream data. The schema currently has columns: event_id, user_id, timestamp. A new data source includes an additional column 'session_id'. The pipeline must continue to append data without breaking existing queries, but the team wants to preserve the new column for future use. Which approach should be taken?

   • AManually alter the table to add the column before the write.
   • BEnable schema evolution on the write operation to automatically add the new column.
   • CUse a temporary view to add the column only for the current query.
   • DSet the table property 'delta.schema.autoMerge' to false to reject the new column.
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   What is the default behavior of Delta Lake when a write contains columns that are not present in the target table's schema?

   • AThe table schema is replaced with the new schema
   • BThe write fails with a schema mismatch error
   • CThe extra columns are silently dropped
   • DThe new columns are automatically added to the table
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   A team wants to ingest data from both a batch file source and a streaming Kafka source into the same Delta table. What is the recommended approach?

   • ACreate two separate Delta tables and union them at query time
   • BOnly use batch ingestion and ignore streaming data
   • CUse the same Delta table for both batch writes and Structured Streaming writes
   • DConvert all batch data to streaming before writing
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   A streaming job reads from a file source and writes to a Delta table. The job is restarted after a failure, and some files are reprocessed, causing duplicate records. The team wants to ensure idempotent writes without changing the source system. What is the best approach?

   • AUse a watermark on the file modification time to skip old files.
   • BSet the output mode to complete to overwrite the table each time.
   • CUse the 'dropDuplicates' transformation on the streaming DataFrame.
   • DUse a unique key column and perform a merge with 'notMatched' action to insert only new records.
   Reveal Solution Discussion Select an answer first