Building and Testing an ETL Pipeline with Lakeflow Spark Declarative Pipelines, SQL, and Apache Spark on the Databricks Platform DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 5)

Questions 21–25

1. 21foundation · easy

   What is the purpose of the 'auto-optimize' configuration in a Lakeflow pipeline?

   • ATo automatically generate documentation for the pipeline
   • BTo automatically fix data quality issues in the pipeline
   • CTo automatically tune the pipeline's compute resources
   • DTo automatically optimize the layout of Delta table files for better performance
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A data engineer needs to automate the deployment of a Lakeflow Spark Declarative Pipeline. The pipeline definition is stored in a Git repository. The engineer wants to update the pipeline in the Databricks workspace whenever a new commit is pushed to the main branch. Which approach should they use?

   • AUse a Databricks Job to run a notebook that reads the pipeline definition from Git and updates it.
   • BUse the Databricks REST API to create a new pipeline each time, ignoring the existing one.
   • CUse the Databricks UI to manually import the pipeline definition after each commit.
   • DUse the Databricks CLI to call the pipelines API to update the pipeline as part of a CI/CD process.
   Reveal Solution Discussion Select an answer first
4. 23application · medium

   A team manages a Lakeflow Spark Declarative Pipeline that runs on a schedule. They need to trigger a one-time run of the pipeline outside the normal schedule after a data backfill. They also want to monitor the run's status and logs. Which approach should they use?

   • AUse the Databricks UI to manually start the pipeline and then check the cluster's Spark UI for logs.
   • BUse the Databricks Jobs API to create a new job that runs the pipeline, then delete the job after it completes.
   • CUse the Databricks CLI to trigger a pipeline update and then query the pipeline events API for status.
   • DUse the Databricks REST API to update the pipeline's schedule to run immediately, then revert it.
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A data engineer needs to orchestrate an ETL workload that consists of a notebook that ingests data, a SQL query that transforms it, and a final notebook that loads it into a reporting table. The workload must run daily at 2 AM, and if the ingestion task fails, the subsequent tasks should not run. Which approach should be used?

   • ACreate a Databricks Job with three tasks (notebook, SQL, notebook) and set the schedule to 2 AM with default retry policy.
   • BCreate a Lakeflow Spark Declarative Pipeline with three separate tables and schedule it with a Databricks Job.
   • CCreate a single notebook that runs all three steps sequentially and schedule it with a Databricks Job.
   • DCreate three separate Databricks Jobs, each with its own schedule, and rely on the order of execution.
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   When would you choose to use raw Structured Streaming instead of a Lakeflow Spark Declarative Pipeline?

   • AWhen you need fine-grained control over the streaming query logic
   • BWhen you need a fully managed, declarative pipeline with built-in monitoring
   • CWhen you need to process data in batch mode only
   • DWhen you want to avoid writing any code
   Reveal Solution Discussion Select an answer first