Use Auto Loader with Schema Enforcement and Schema Evolution in Batch Modes (for Example, Directory Listing or File Notification) to Land Data into UnityCatalog–governed Tables. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 4)

Questions 16–20

1. 16expert · hard

   A data engineering team is migrating from a scheduled batch job that reads all files from an S3 bucket and overwrites a Delta table. They need to ingest new files incrementally and ensure exactly-once processing. They also need to write to a Unity Catalog table. The team wants to minimize the amount of code changes. Which approach should they take?

   • AKeep the batch read but add a `maxFilesPerTrigger` option.
   • BUse a streaming read with `spark.readStream.format('json')` and write to the Unity Catalog table.
   • CReplace the batch read with Auto Loader in streaming mode, writing to the Unity Catalog table with a checkpoint.
   • DUse Auto Loader in batch mode with `trigger(once=True)` and a checkpoint.
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   What is a limitation of directory listing mode in Auto Loader?

   • AIt only works with JSON files
   • BIt can miss files if they are added and then deleted before the next scan
   • CIt cannot be used with Unity Catalog tables
   • DIt requires setting up a cloud notification service like SQS or Event Grid
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A data engineer is using Auto Loader to ingest data from an S3 bucket into a Unity Catalog table. The target table is `sales.raw.orders`. The engineer wants to ensure the data is written to the correct catalog and schema. Which option should be used in the write operation?

   • AUse `spark.catalog.setCurrentDatabase('sales.raw')` before writing.
   • BWrite to a temporary view and then insert into the table.
   • CSpecify the full table name `sales.raw.orders` in the `table` option.
   • DSpecify the catalog and schema in the `catalog` and `schema` options.
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A data engineer is using Auto Loader to ingest CSV files into a Unity Catalog table. The files have a header row, but some files have an extra column that is not in the target table. The engineer wants to enforce the schema and fail the ingestion if any file has a column that does not match the target schema. Which configuration should be used?

   • ASet `cloudFiles.schemaEvolutionMode = 'addNewColumns'` and provide a schema.
   • BSet `cloudFiles.schemaEvolutionMode = 'rescue'` and provide a schema.
   • CSet `cloudFiles.schemaLocation` and let Auto Loader infer the schema.
   • DSet `cloudFiles.schemaEvolutionMode = 'failOnNewColumns'` and provide a schema.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   How does Auto Loader operate in batch mode compared to streaming mode?

   • ABatch mode only supports CSV files, while streaming mode supports all formats
   • BBatch mode processes data in discrete micro-batches, while streaming mode processes data as a continuous stream
   • CBatch mode requires a streaming cluster, while streaming mode works on any cluster
   • DBatch mode processes all available new files in a single batch, while streaming mode processes data continuously as it arrives
   Reveal Solution Discussion Select an answer first