Describe Data Pipeline Best Practices DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A pipeline ingests data from multiple sources. The team has noticed that some sources occasionally send duplicate records, which inflate the final counts. They want to prevent duplicates from affecting downstream analytics. What should they do?

   • AIgnore the duplicates, as they are rare and do not significantly affect the results.
   • BAdd a deduplication step that removes duplicates based on a unique key, and log the number of duplicates removed.
   • CUse a random sample of the data to reduce the impact of duplicates.
   • DAsk the source systems to stop sending duplicates, without any pipeline changes.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A pipeline ingests user-generated data from a web form. The team has noticed that some records contain invalid email addresses, which later cause failures in a downstream reporting job. They want to catch these issues early. Where should they implement validation checks?

   • AAt every stage of the pipeline, including after ingestion, after transformation, and before load.
   • BOnly at the final load step, right before inserting into the warehouse.
   • COnly during the transformation stage, since that is where the data is modified.
   • DOnly in the downstream reporting job, since that is where the failure occurs.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A pipeline processes customer data that includes personally identifiable information (PII). The team must ensure that only authorized personnel can access the raw data, and that access is logged. Which security practice is most appropriate?

   • AStore the data in a public cloud bucket for easy access.
   • BEncrypt the data at rest and in transit, and implement role-based access control with audit logging.
   • CRemove all PII from the pipeline to avoid security concerns.
   • DRely on the database's default security settings without additional configuration.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   How does structuring pipeline components as modular functions or classes improve testability?

   • AIt reduces the number of tests required
   • BIt allows each component to be tested in isolation
   • CIt eliminates the need for integration testing
   • DIt automatically generates test cases
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Why is data validation important at various stages of a pipeline?

   • ATo reduce the need for logging
   • BTo increase the speed of data processing
   • CTo eliminate the need for error handling
   • DTo prevent downstream issues caused by bad data
   Reveal Solution Discussion Select an answer first