Develop a Quarantining Process for Bad Data with Lakeflow Spark Declarative Pipelines, or Autoloader in Classic Jobs. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   In a quarantine workflow, what is the role of a try-catch block in error handling?

   • ATo capture exceptions and route failed records to a quarantine destination
   • BTo validate the schema of the input data
   • CTo optimize the performance of the data transformation
   • DTo monitor the health of the cluster
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A financial services firm is building a Lakeflow Spark Declarative Pipeline to process customer transactions. They need to quarantine records where the transaction amount is negative or the customer ID is missing. Which approach best defines the bad data criteria?

   • AApply a filter that only selects records with positive amounts and non-null customer IDs, discarding the rest.
   • BSet the pipeline to fail if any record violates the schema constraints.
   • CDefine a validation rule in the pipeline that checks for negative amounts and null customer IDs, routing failures to a quarantine sink.
   • DUse a schema with all fields as nullable so no records are considered invalid.
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   A data team is designing a Lakeflow Spark Declarative Pipeline to process customer records. They need to quarantine records that fail either a schema constraint (e.g., non-nullable field) or a business rule (e.g., age between 0 and 120). The team wants to capture the specific reason for quarantine. What is the best way to implement this?

   • AUse a filter that selects only records passing both schema and business rules, discarding the rest.
   • BUse a validation step that checks both schema and business rules, writing failures to a quarantine table with a reason column.
   • CUse a schema with all fields nullable and rely only on business rules for validation.
   • DConfigure the pipeline to fail on the first record that violates any rule.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Which mechanism can be used to capture and store failed records in a quarantine destination during a pipeline run?

   • AA quality monitoring process that writes invalid records to a separate table
   • BA Delta table with a merge operation that updates existing records
   • CA cluster autoscaling policy that adjusts resources based on load
   • DA job notification that sends an email when the pipeline fails
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A financial company has a Lakeflow Spark Declarative Pipeline that processes transactions. The pipeline currently uses a single sink and fails the entire run when any record fails validation. The team wants to quarantine bad records but also needs to ensure that the pipeline continues processing even if the quarantine sink is temporarily unavailable. What is the best way to achieve this?

   • AConfigure the pipeline to fail if the quarantine sink is unavailable, and retry the entire run.
   • BUse a validation step that routes bad records to a quarantine sink and configure the pipeline to continue on sink errors.
   • CUse a try-catch block around the pipeline to catch sink errors and log them.
   • DWrite all records to the main table and use a separate job to move bad records to quarantine.
   Reveal Solution Discussion Select an answer first