Create an Append-Only Data Pipeline Capable of Handling Both Batch and Streaming Data Using Delta. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   In Structured Streaming, which column type is required to use event-time processing with watermarks?

   • AA string column
   • BA boolean column
   • CAn integer column
   • DA timestamp column
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A data platform team needs to ingest data from both a nightly batch export (CSV files) and a real-time event stream into a single Delta table. The pipeline must treat both sources uniformly, appending new records without altering existing ones. Which approach best meets this requirement?

   • AUse a single Structured Streaming query that reads both the CSV files and the event stream as a unioned source.
   • BUse a single batch job that reads the event stream at the end of the day and combines it with the CSV files.
   • CUse a Delta table with a view that unions the batch and streaming data on the fly.
   • DUse two separate pipelines: one batch job for CSV files and one Structured Streaming job for the event stream, both writing to the same Delta table in append mode.
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   What is the primary purpose of checkpointing in a Structured Streaming query?

   • ATo encrypt the streaming data
   • BTo compress the output data
   • CTo enable fault tolerance and recovery
   • DTo speed up the query execution
   Reveal Solution Discussion Select an answer first
5. 9expert · hard

   A streaming pipeline writes to a Delta table and uses a checkpoint directory on DBFS. The team wants to monitor the progress of the stream and ensure it can recover from failures. They also need to ensure that no data is lost or duplicated during recovery. What is the best practice?

   • AUse a temporary checkpoint directory to avoid clutter and rely on the Delta table for recovery.
   • BDisable checkpointing to improve performance and use a unique key to deduplicate on write.
   • CUse a reliable checkpoint location (e.g., cloud storage) and monitor the stream using StreamingQueryListener.
   • DUse the DeltaTable API to monitor the stream and set the checkpoint to a local directory.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   Which option enables automatic schema evolution when writing to a Delta table?

   • ASetting 'spark.sql.sources.partitionOverwriteMode' to 'dynamic'
   • BSetting 'mergeSchema' to 'true' in the write options
   • CSetting 'delta.enableChangeDataFeed' to 'true'
   • DSetting 'delta.autoOptimize' to 'true'
   Reveal Solution Discussion Select an answer first