2.2 Building the Pipelines PROFESSIONAL-DATA-ENGINEER Practice Questions (Page 7)

Questions 31–35

1. 31expert · hard

   A healthcare company processes patient records in a Dataflow pipeline. The data contains missing values for some fields, and the team must decide how to handle them. They have a requirement to preserve as much data as possible while ensuring data quality. They also need to handle late-arriving data in the streaming pipeline. What is the best approach?

   • AImpute missing values, but do not handle late data because it is rare.
   • BFilter out records with missing values and use a fixed watermark to discard late data.
   • CUse Beam transforms to impute missing values with defaults or derived values, and use allowed lateness with a watermark to handle late data.
   • DUse a side input to fill missing values from a reference table, and set the watermark to zero.
   Reveal Solution Discussion Select an answer first
2. 32foundation · easy

   A data engineer needs to run an existing Apache Spark job on Google Cloud. Which service provides a managed environment for this?

   • ADataflow
   • BBigQuery
   • CDataproc
   • DCloud Composer
   Reveal Solution Discussion Select an answer first
4. 33expert · hard

   A company runs a Spark pipeline on Dataproc that processes large-scale log data. The pipeline uses a custom UDF that is computationally expensive. They need to improve performance without increasing the cluster size significantly. They also need to handle a growing volume of data. What should they do?

   • AIncrease the number of worker nodes to handle the load.
   • BUse a larger machine type for the master node.
   • CSwitch to Dataflow to run the same logic.
   • DOptimize the UDF by using built-in Spark functions where possible, and use partitioning and bucketing to reduce shuffle.
   Reveal Solution Discussion Select an answer first
5. 34expert · hard

   A company has a large dataset in BigQuery that needs to be transformed and loaded into a new table with a different schema. The transformation involves joining multiple tables and aggregating data. They need to run this transformation daily, and the cost is a concern. They also need to ensure that the transformation does not affect the source tables. What should they do?

   • AUse a Dataflow pipeline to read from BigQuery, transform, and write to a new table.
   • BUse Cloud Data Fusion to create a pipeline that reads from BigQuery and writes to a new table.
   • CUse a BigQuery scheduled query that creates a new table from the transformation, using partitioning and clustering to optimize cost.
   • DUse a Dataproc cluster with Spark to read from BigQuery and write to a new table.
   Reveal Solution Discussion Select an answer first
6. 35application · medium

   A financial services company receives daily transaction files in Cloud Storage. They need to perform complex transformations (e.g., joining with customer data, aggregating by region) and load the results into a reporting table. The team is proficient in SQL and wants to minimize infrastructure management. What should they use?

   • AUse Cloud Data Fusion to design a pipeline that transforms the data and loads it to BigQuery.
   • BCreate a Dataproc cluster and run Spark SQL jobs to transform the data.
   • CUse Cloud Functions to process each file and write to BigQuery.
   • DLoad the files into BigQuery and use SQL queries with scheduled jobs to perform the transformations.
   Reveal Solution Discussion Select an answer first