Develop a Quarantining Process for Bad Data with Lakeflow Spark Declarative Pipelines, or Autoloader in Classic Jobs. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A data engineering team has a quarantine table in a Lakeflow Spark Declarative Pipeline that stores records failing validation. They need to periodically review and reprocess these records after fixing the underlying data issues. What is the most effective way to manage the quarantined data?

   • ADelete the quarantine table automatically after each pipeline run to save storage.
   • BModify the pipeline to skip validation checks for previously quarantined records.
   • CStore all quarantined records in a single file and manually review them in a notebook.
   • DCreate a separate job that reads the quarantine table, applies corrections, and writes valid records back to the main table.
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   What is the purpose of defining schema constraints as part of the bad data criteria?

   • ATo automatically reject records that do not conform to the expected schema
   • BTo allow flexible data types for different sources
   • CTo improve the performance of the pipeline by reducing data size
   • DTo store metadata about the data lineage
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   In a Lakeflow Spark Declarative Pipeline, what is the primary purpose of defining a quarantine table or path?

   • ATo provide a backup of the entire input dataset before processing
   • BTo store records that fail validation rules so they can be isolated from the main data flow
   • CTo temporarily cache intermediate results for performance optimization
   • DTo store metadata about the pipeline run, such as timestamps and version numbers
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A telecom company is building a Lakeflow Spark Declarative Pipeline to process call detail records. They need to quarantine records where the call duration is negative or the phone number format is invalid. What is the most appropriate way to define these bad data criteria?

   • AUse a filter that selects only records with positive duration and valid phone numbers, discarding the rest.
   • BUse a validation function that checks for negative duration and regex pattern for phone numbers, routing failures to a quarantine sink.
   • CConfigure the pipeline to fail on the first record that violates the criteria.
   • DSet the schema to accept any string for phone numbers and any integer for duration.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   When using Autoloader in a classic job, what is the typical approach to handle records that fail schema validation?

   • AWrite them to a quarantine table or path for later analysis
   • BStop the entire pipeline and alert the administrator
   • CAutomatically correct the records and continue processing
   • DSkip the records and log a warning in the driver logs
   Reveal Solution Discussion Select an answer first