Use Auto Loader with Schema Enforcement and Schema Evolution in Batch Modes (for Example, Directory Listing or File Notification) to Land Data into UnityCatalog–governed Tables. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A data engineer needs to ingest a large backfill of historical CSV files from an Azure Data Lake Storage Gen2 container into a Unity Catalog table. The files are already present and will not be modified. The engineer wants to process them in a single batch and then stop the ingestion. Which Auto Loader configuration should the engineer use?

   • AUse Auto Loader in streaming mode with `cloudFiles.format = 'csv'` and a `processingTime` of 1 hour.
   • BRun Auto Loader in batch mode with `cloudFiles.format = 'csv'` and a `trigger(once=True)`.
   • CUse `spark.read.format('csv').load(path)` and write to the target table with `mode('overwrite')`.
   • DRun a streaming query with Auto Loader and a `maxFilesPerTrigger` of 1000, then stop the stream after the initial batch.
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   What does schema enforcement in Auto Loader do?

   • AIt creates a backup of the schema before ingestion
   • BIt ensures that incoming data conforms to a specified schema, rejecting or handling mismatched records
   • CIt automatically changes the schema of the target table to match the incoming data
   • DIt encrypts the data to protect the schema
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   How does Auto Loader infer the schema of incoming data files?

   • ABy using a machine learning model to predict the schema
   • BBy sampling a subset of the data files to determine the schema
   • CBy requiring the user to manually specify the schema before ingestion
   • DBy reading the schema from a metadata file in the storage location
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A data engineer is setting up Auto Loader to ingest files from an Azure Data Lake Storage Gen2 account. The files are written by an upstream process that does not emit events. The engineer wants to use file notification mode. What must be configured for this to work?

   • AEnable Azure Queue Storage and configure Auto Loader with `cloudFiles.useNotifications = 'true'`.
   • BUse directory listing mode instead, since file notification mode is not supported for ADLS Gen2.
   • CEnable Azure Event Hubs on the storage account and configure Auto Loader with `cloudFiles.useNotifications = 'true'`.
   • DEnable Azure Event Grid on the storage account and configure Auto Loader with `cloudFiles.useNotifications = 'true'`.
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   What is the role of checkpoints in Auto Loader?

   • ATo store the raw data files for backup purposes
   • BTo provide a location for temporary data during processing
   • CTo track which files have been processed and ensure exactly-once processing
   • DTo manage the schema of the target table
   Reveal Solution Discussion Select an answer first