Develop a Quarantining Process for Bad Data with Lakeflow Spark Declarative Pipelines, or Autoloader in Classic Jobs. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 4)

Questions 16–19

1. 16application · medium

   A data team has a quarantine table in a Lakeflow Spark Declarative Pipeline that stores records failing validation. They want to automatically delete quarantined records that are older than 30 days to manage storage costs. What is the best way to implement this?

   • ASet the pipeline to fail if the quarantine table exceeds a certain size.
   • BCreate a scheduled job that deletes records from the quarantine table where the quarantine timestamp is older than 30 days.
   • CUse a view that only shows records from the last 30 days, leaving the underlying table unchanged.
   • DModify the pipeline to overwrite the quarantine table with only the latest records each run.
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A healthcare company uses Autoloader in a classic job to ingest patient records. Occasionally, the source files contain records with invalid date formats. The team wants to ensure that these records are captured and stored in a quarantine table without interrupting the processing of valid records. What is the best way to implement this?

   • ASet the 'failOnDataLoss' option to false to prevent job failure on bad records.
   • BUse a foreachBatch sink to apply a validation function that writes invalid records to a quarantine table and valid records to the main table.
   • CConfigure the schema to be permissive so that invalid dates are stored as null.
   • DUse a try-catch block around the Autoloader stream to catch exceptions and write them to a log file.
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   What is a common process for managing quarantined data after it has been stored?

   • ADeleting the quarantine table to save storage
   • BAutomatically merging quarantined records back into the main table
   • CReviewing the quarantined records and deciding whether to reprocess or delete them
   • DIgnoring the quarantined records until the pipeline is rerun
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   Which component in a Lakeflow Spark Declarative Pipeline is typically used to define the quarantine location for bad records?

   • AA cluster configuration that enables error logging
   • BA view that filters out bad records
   • CA separate table defined with a quarantine constraint
   • DA checkpoint that stores pipeline state
   Reveal Solution Discussion Select an answer first