Monitoring DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A data engineer needs to create a report that shows the daily cost of each job over the past 30 days. They also need to include the cluster type used by each job. They have access to system tables. Which query strategy should they use?

   • AUse the Databricks REST API to get job runs and cluster details, then calculate cost manually.
   • BQuery system.compute.cluster_details to get cluster type and system.jobs.runs to get job runs, but not cost.
   • CJoin system.billing.usage with system.jobs.runs on cluster_id and job_id, then aggregate by date and job_id.
   • DQuery system.billing.usage alone and group by cluster_id to get cost per cluster, then manually map to jobs.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A company wants to build an automated monitoring dashboard that sends alerts when a scheduled job fails or takes longer than a threshold. They plan to use a script that runs every minute. Which approach should they use to retrieve job run status and duration programmatically?

   • AUse the Databricks REST API endpoint 'GET /api/2.1/jobs/runs/list' to fetch recent job runs, then filter by job ID and check the 'state' and 'run_duration' fields.
   • BUse the Databricks CLI command 'databricks jobs list' to list all jobs and their last run status.
   • CUse the Spark UI to view the job run's progress and duration, and parse the HTML to extract the status.
   • DQuery the system.jobs.runs table to retrieve the job run status and duration for the last minute.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A data engineering team notices that their monthly cloud bill has increased significantly. They suspect a specific job is running longer than expected, but they are unsure which cluster it runs on. They want to identify the job's cost and resource usage over the past month using Databricks system tables. Which approach should they take?

   • AQuery the system.access.audit table to find the job's start and end times, then manually calculate the cost based on the cluster's price per hour.
   • BQuery the system.billing.usage table, filtering by the job's cluster ID and time range, then join with system.compute.cluster_details to get cluster metadata.
   • CUse the Databricks CLI command 'databricks clusters list' to view the cluster's current status and then estimate the cost from the displayed runtime.
   • DReview the Spark UI for the job's last run to see the total task time and multiply by the cluster's hourly rate.
   Reveal Solution Discussion Select an answer first
5. 4expert · hard

   A team runs a Lakeflow Spark Declarative Pipeline that processes streaming data and writes to a Delta table. Recently, the pipeline has been failing intermittently due to 'data quality' errors. The team wants to set up automated monitoring that not only alerts on failures but also provides the specific data quality rule that failed. They plan to use a script that polls the pipeline events. Which approach should they use?

   • AQuery the system.pipeline.events table to retrieve all events and filter for 'data_quality' type.
   • BUse the Spark UI to view the pipeline's cluster and look for the failed stage in the job details.
   • CUse the Databricks CLI command 'databricks pipelines get-events' to retrieve the events and parse the output.
   • DPoll the REST API endpoint 'GET /api/2.0/pipelines/{pipeline_id}/events' and parse the event details to extract the data quality rule that failed.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   In the Spark UI, which tab would you use to view the number of tasks that failed for a specific stage?

   • AEnvironment tab
   • BStages tab
   • CExecutors tab
   • DJobs tab
   Reveal Solution Discussion Select an answer first