Design and Implement Data Pipelines DP-750 Practice Questions (Page 8)

Questions 36–39

1. 36expert · hard

   A data pipeline reads from a source, performs a series of transformations, and writes to a Delta table. The team wants to ensure that the pipeline is idempotent, meaning that rerunning it does not produce duplicate data. They also want to handle late-arriving data without causing failures. What should they implement?

   • AUse a delete operation on the Delta table before writing new data.
   • BUse an append operation on the Delta table and rely on the job scheduler to retry on failure.
   • CUse a merge operation on the Delta table and handle late-arriving data with a watermark.
   • DUse a notebook with a try-except block to handle late-arriving data.
   Reveal Solution Discussion Select an answer first
2. 37expert · hard

   A team has a Lakeflow job with a notebook task that processes data from an external API. The API occasionally returns a 429 (rate limit) error. The team has configured the notebook to retry the API call up to 5 times with exponential backoff. However, the job still fails because the notebook task itself fails after the retries are exhausted. The team wants to avoid the job failing and instead send an alert and continue with the next task. What should they do?

   • AModify the notebook to catch the final exception, log an error, and return a success status so the job continues, while sending an alert from within the notebook.
   • BUse a separate alerting task that depends on the notebook task, and configure the notebook task to fail, so the alerting task runs on failure.
   • CSet the notebook task to 'continue on error' in the job configuration, so the job does not fail even if the task fails.
   • DConfigure the Lakeflow job to have a retry policy on the notebook task, so the entire task is retried before failing.
   Reveal Solution Discussion Select an answer first
4. 38application · medium

   A data team needs to build a pipeline that performs a series of complex, interdependent transformations that require custom Python libraries and interactive debugging. The team also needs to implement custom error handling that sends alerts to a specific Slack channel when certain conditions are met. They are considering whether to use notebooks or Lakeflow Spark Declarative Pipelines. Which approach is more appropriate?

   • AUse a notebook because it allows interactive development, custom Python libraries, and you can embed custom error-handling logic that sends Slack alerts.
   • BUse a Lakeflow Spark Declarative Pipeline and write a UDF for the custom logic, and use a separate alerting job to monitor the pipeline.
   • CUse a Lakeflow Spark Declarative Pipeline because it supports custom Python libraries and provides built-in error handling for alerts.
   • DUse a notebook for development, but then convert it to a Lakeflow Spark Declarative Pipeline for production to get the alerting features.
   Reveal Solution Discussion Select an answer first
5. 39application · medium

   A team is designing a Lakeflow Spark Declarative Pipeline that processes streaming data. The pipeline must first parse the raw events, then enrich them with reference data, and finally write the enriched events to a Delta table. The team wants to ensure that the enrichment step only runs after the parsing step has produced output. How should they define the pipeline?

   • ADefine the enrichment step to read from the same source as the parsing step, and rely on the engine to infer the order.
   • BUse two separate Delta Live Tables pipelines, one for parsing and one for enrichment, and schedule them independently.
   • CDefine the parsing and enrichment steps as separate nodes in the pipeline graph, and connect them with an edge so that enrichment depends on parsing.
   • DWrite the parsing and enrichment logic in a single notebook cell, and use a variable to pass the parsed data to the enrichment code.
   Reveal Solution Discussion Select an answer first