Develop a Quarantining Process for Bad Data with Lakeflow Spark Declarative Pipelines, or Autoloader in Classic Jobs. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   In a classic Databricks job using Autoloader, how can you route malformed records to a quarantine location?

   • ABy enabling the 'rescue' option in the schema inference
   • BBy using a Delta table with a CHECK constraint to reject bad records
   • CBy setting the 'ignoreCorruptFiles' option to true in the Autoloader configuration
   • DBy configuring a separate Autoloader stream that reads from the same source and writes to a quarantine path
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A logistics company uses Autoloader in a classic Databricks job to ingest JSON files from cloud storage. Some files contain records with missing required fields. The team wants to keep the valid records in the main table and move the malformed records to a quarantine path for later analysis. What is the most efficient way to achieve this?

   • ASet the 'ignoreCorruptFiles' option to true so that malformed files are skipped automatically.
   • BUse Autoloader's 'rescuedDataColumn' option to capture malformed records and then filter them into a separate quarantine path.
   • CConfigure the job to fail on the first malformed record and stop processing.
   • DUse a separate Autoloader stream for each file and manually inspect the output.
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   Why is it important to monitor the volume of quarantined data over time?

   • ATo detect trends that may indicate a systemic data quality issue
   • BTo automatically fix the bad data in the quarantine table
   • CTo reduce the cost of storage by deleting old data
   • DTo ensure the quarantine table does not exceed a certain size
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A manufacturing company uses Autoloader in a classic job to ingest sensor data. Some records have sensor readings that are out of acceptable range. The team wants to quarantine these records and also track how many records are quarantined per run for monitoring. What is the best approach?

   • ASet the 'ignoreCorruptFiles' option to true to skip files with out-of-range readings.
   • BWrite all records to the main table and use a separate job to identify and move out-of-range records to quarantine.
   • CConfigure the job to fail when out-of-range readings are detected.
   • DUse a foreachBatch sink to filter out-of-range records into a quarantine table and update a metrics table with the count of quarantined records.
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A retail company uses a Lakeflow Spark Declarative Pipeline to ingest daily sales transactions. The pipeline currently fails the entire batch when a single record has a negative quantity. The data team wants to isolate invalid records into a separate table so the valid transactions still load. Which pipeline configuration should they implement?

   • AAdd a validation step that filters out records with negative quantity and writes them to a quarantine table using a separate sink.
   • BModify the source schema to allow negative quantities so the records are no longer considered invalid.
   • CUse a try-catch block in the pipeline code to catch the negative quantity exception and log it to a file.
   • DConfigure the pipeline to stop on validation errors and send an alert to the data team for manual correction.
   Reveal Solution Discussion Select an answer first