5.2 Designing Automation and Repeatability PROFESSIONAL-DATA-ENGINEER Practice Questions (Page 3)

Questions 11–14

1. 11application · medium

   A data pipeline in Cloud Composer occasionally fails because a downstream API returns a transient 503 error. The engineer wants to make the pipeline more reliable without manual intervention. What is the most appropriate configuration?

   • ASet the task's trigger_rule to 'all_done'.
   • BIncrease the DAG's schedule_interval to run less frequently.
   • CSet the DAG's catchup=False so past runs are not executed.
   • DSet retries=3 and retry_delay=timedelta(minutes=2) on the task that calls the API.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A data engineer is building a Cloud Composer DAG that must load raw CSV files from Cloud Storage into BigQuery, then run a transformation query, and finally export the results to a second Cloud Storage bucket. The engineer wants to ensure the transformation query only runs after the load job has fully completed. Which DAG structure should be used?

   • ACreate three tasks (load, transform, export) and set dependencies using the >> operator: load >> transform >> export.
   • BDefine three separate Python functions and call them sequentially in a single task using the PythonOperator.
   • CCreate three tasks and set the schedule_interval of each task to trigger 10 minutes after the previous one.
   • DCreate three tasks and use the bitwise OR operator (|) to connect them: load | transform | export.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A data engineer is creating a DAG with two independent tasks: one that cleans data and one that validates data. Both tasks can run in parallel, and a final task must run after both complete. How should the dependencies be defined?

   • Aclean >> final and validate >> final
   • Bclean >> validate >> final
   • Cfinal >> clean and final >> validate
   • Dclean >> validate; final >> validate
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   In Cloud Composer, what is the primary purpose of representing a data pipeline as a directed acyclic graph (DAG)?

   • ATo store the output data of each task in a distributed file system
   • BTo define the order of task execution and the dependencies between tasks, ensuring no circular dependencies exist
   • CTo ensure that every task can run in parallel without any dependencies
   • DTo provide a visual representation of the data schema used by the pipeline
   Reveal Solution Discussion Select an answer first