Design and Implement Data Pipelines DP-750 Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A team is building a notebook-based pipeline in a Lakeflow job. The pipeline must first ingest data from an external API, then run a data quality check on the ingested data. If the quality check fails, the pipeline should stop and not proceed to the transformation step. The team wants to implement this using notebook tasks and precedence constraints. What is the correct way to design the task logic?

   • ACreate three notebook tasks with no dependencies, and have each task check a global flag to decide whether to run.
   • BCreate two notebook tasks: one for ingestion and one for quality check. Set the quality check task to depend on the ingestion task, and configure the quality check task to fail the job if the check fails. Then add a transformation task that depends on the quality check task.
   • CCreate a single notebook that performs ingestion, quality check, and transformation in one cell, and use a conditional break to stop execution if quality fails.
   • DCreate an ingestion task and a transformation task, and embed the quality check logic inside the transformation task, so the transformation task fails if quality is bad.
   Reveal Solution Discussion Select an answer first
2. 22expert · hard

   A team has a Lakeflow job with multiple tasks. The job processes data from a source that sometimes has schema changes. The team wants to implement error handling that detects schema changes and sends an alert, but does not fail the job. Instead, the job should continue with the previous schema and log the issue. How should they implement this?

   • AUse a Lakeflow Spark Declarative Pipeline with a 'schema evolution' option that automatically adapts to changes.
   • BIn the notebook task, catch the schema change exception, log it, send an alert, and use a fallback schema that was previously saved.
   • CUse a separate monitoring task that runs before the main task and checks the schema, and if a change is detected, it sends an alert and skips the main task.
   • DConfigure the job to fail on schema changes and set up an alert on job failure.
   Reveal Solution Discussion Select an answer first
4. 23application · medium

   A data pipeline processes daily files that occasionally contain malformed records. The team wants to ensure that a malformed record does not cause the entire job to fail, and they want to track the number of malformed records for monitoring. What should they implement?

   • AUse a try-except block in the notebook to catch exceptions and log the malformed records.
   • BAdd a validation step that filters out malformed records before processing.
   • CUse the 'badRecordsPath' option when reading the data to route malformed records to a separate location.
   • DSet the Spark configuration to ignore all errors during the job.
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A pipeline reads raw JSON files, validates records, enriches them with a lookup table, and writes the output to a Delta table. The team wants to ensure that invalid records are quarantined without stopping the entire pipeline. What is the most appropriate order of operations?

   • ARead → Enrich → Quarantine invalid records → Validate → Write to Delta table.
   • BRead → Validate → Enrich → Write to Delta table → Quarantine invalid records.
   • CRead → Enrich → Validate → Write to Delta table → Quarantine invalid records.
   • DRead → Validate → Quarantine invalid records → Enrich valid records → Write to Delta table.
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   In a Lakeflow Job, what is the purpose of configuring a task to retry on failure?

   • ATo send an email notification when the task fails
   • BTo automatically rerun the task if it fails due to transient issues like cluster startup or network glitches
   • CTo ensure the task always succeeds regardless of the error
   • DTo skip the task if it fails and continue with the next task
   Reveal Solution Discussion Select an answer first