Building and Testing an ETL Pipeline with Lakeflow Spark Declarative Pipelines, SQL, and Apache Spark on the Databricks Platform DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   What is the primary purpose of Autoloader in a Lakeflow Spark Declarative Pipeline?

   • ATo load data from a JDBC source into a Delta table
   • BTo schedule and orchestrate the execution of the pipeline
   • CTo perform complex data transformations on existing data
   • DTo incrementally ingest new files from cloud storage as they arrive
   Reveal Solution Discussion Select an answer first
2. 12expert · hard

   A company has an existing Spark Structured Streaming job that reads from a Kafka topic, performs a windowed aggregation, and writes to a Delta table. They want to migrate this to a Lakeflow Spark Declarative Pipeline. The main challenge is that the streaming job has complex stateful operations and requires exactly-once semantics. What should the data engineer consider?

   • ALakeflow Pipelines support exactly-once semantics but not windowed aggregations.
   • BLakeflow Pipelines can handle this because they are built on Structured Streaming and support stateful operations and exactly-once semantics.
   • CLakeflow Pipelines cannot handle stateful operations, so the team must keep the existing Structured Streaming job.
   • DLakeflow Pipelines only support batch processing, so the team must keep the existing job.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A company is building a Lakeflow Spark Declarative Pipeline. They have a bronze table that ingests raw events via Autoloader. They need a silver table that contains the total number of events per user per day, which should be recomputed from the bronze table each time the pipeline runs. Which table type should they use for the silver table?

   • AA materialized view
   • BA temporary view
   • CA streaming table
   • DA standard Delta table
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   What is the purpose of the 'for/each' control flow operator in a Lakeflow pipeline?

   • ATo configure the compute resources for the pipeline
   • BTo execute a block of code repeatedly over a collection of items
   • CTo define a new table in the pipeline
   • DTo conditionally execute code based on a boolean condition
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A data engineer is configuring a Lakeflow Spark Declarative Pipeline that runs a critical ETL job. The job is idempotent and can be safely retried. However, the engineer is concerned about the cost of retries on a large cluster. They want to balance reliability and cost. What should they do?

   • AEnable retries with a limited number of attempts and a backoff policy.
   • BDisable retries to save cost, even if the job fails.
   • CEnable retries but use a smaller cluster for retries.
   • DEnable retries with an unlimited number of attempts.
   Reveal Solution Discussion Select an answer first