2.1 Planning the Data Pipelines PROFESSIONAL-DATA-ENGINEER Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A media streaming company collects user clickstream events from mobile and web clients. They need to (1) power a real-time dashboard showing current viewer counts, (2) feed a recommendation engine that requires enriched user profiles, and (3) retain raw events for regulatory audits. The data engineering team is designing the ingestion and processing pipeline. Which combination of sinks and transformations best satisfies all three requirements?

   • AIngest events directly into BigQuery via the Storage Write API, then use scheduled queries to compute aggregates and overwrite the raw table with enriched data.
   • BStream events to Pub/Sub, use Dataflow to write raw events to Cloud Storage and aggregated counts to BigQuery, and enrich profiles in a separate Dataflow pipeline reading from the same Pub/Sub topic.
   • CUse Cloud Functions to process each event, write the enriched record to Cloud Spanner for the dashboard, and store a copy in Cloud Storage for audits.
   • DStream events to Cloud Storage as JSON files, then run a nightly Dataproc job to aggregate and enrich the data before loading into BigQuery.
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   A pipeline must ingest data from a third-party service that exposes a RESTful endpoint returning JSON. Which type of data source is this?

   • AA database source
   • BA file source
   • CAn API source
   • DA streaming source
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   A multinational corporation needs to ingest data from a legacy mainframe system located in a data center in Europe. The mainframe can only output files via FTP. The data is subject to GDPR, and the company has a policy that all data processing must occur within the EU. They are planning to use a Compute Engine VM in the europe-west1 region to run an FTP server that receives the files, then transfer them to Cloud Storage in the same region. Which additional consideration is most critical for compliance?

   • AEnsure the VM has a static public IP address so the mainframe can reliably connect via FTP.
   • BSet up a Cloud VPN between the mainframe's data center and the VPC to avoid FTP over the public internet.
   • CConfigure the Cloud Storage bucket with a lifecycle policy to delete files after 30 days.
   • DVerify that the FTP connection uses TLS/SSL and that the Cloud Storage bucket uses CMEK with keys stored in the EU.
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A financial services company runs a daily batch pipeline that (1) extracts data from a transactional database, (2) performs complex transformations in Dataproc, (3) loads results into BigQuery, and (4) sends a summary report via email. The pipeline must not start the transformation step until the extraction completes successfully, and the email step must only run after the BigQuery load succeeds. The data engineer wants to use Cloud Composer to orchestrate this workflow. Which DAG design correctly models the dependencies?

   • ACreate four tasks and set dependencies: extract >> transform >> load >> email, with retries configured on the extract and load tasks.
   • BCreate four tasks, set extract and transform to run in parallel, then load and email to run after both complete.
   • CDefine four independent tasks with no dependencies and use a schedule interval to run them sequentially.
   • DUse a single PythonOperator that calls the extraction, transformation, load, and email functions in sequence within one task.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   A pipeline processes daily sales transactions and must compute total revenue per product category. Which transformation is required?

   • AAggregation
   • BFiltering
   • CEnrichment
   • DNormalization
   Reveal Solution Discussion Select an answer first