Describe Data Pipeline Best Practices DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A team is developing a pipeline and wants to ensure that any changes to the code can be reviewed and rolled back if necessary. Which practice should they adopt?

   • AKeep the code in a shared folder and edit it directly.
   • BMake backups of the code manually before each change.
   • CUse a deployment tool that overwrites the previous version automatically.
   • DUse a version control system like Git and require pull requests for all changes.
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A data team runs a critical pipeline that loads sales data every hour. They need to be notified immediately if the pipeline fails or if the data loaded is unexpectedly low. Which monitoring and alerting setup best meets this requirement?

   • ACheck the pipeline logs manually every morning.
   • BSet up an alert for any deviation in runtime, but not for row counts.
   • CSet up an alert only for pipeline failures, since that is the most critical issue.
   • DSet up alerts for pipeline failures and for row-count anomalies compared to historical averages.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A nightly Python pipeline extracts data from an external API and loads it into a database. Occasionally, the API returns a 503 error, and the pipeline fails without any record of the incident. The team wants to improve reliability and make failures easier to diagnose. Which combination of practices should they implement?

   • AIncrease the timeout for the API call and log only the final success or failure.
   • BAdd retry logic with exponential backoff and log each attempt with the response status.
   • CSchedule the pipeline to run twice and hope the second run succeeds.
   • DWrap the API call in a try/except block and send an email alert on failure, but do not retry.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A pipeline processes large files that are stored in CSV format. The team has noticed that reading and writing these files is slow, and they want to improve I/O performance. Which change is most effective?

   • AUse a larger file size to reduce the number of files.
   • BSwitch to a columnar storage format like Parquet.
   • CKeep the CSV format but compress the files.
   • DRead the entire file into memory at once.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   Which technique is commonly used to optimize data pipeline performance?

   • AUsing only one processing thread
   • BWriting all data to a single file
   • CDisabling logging
   • DProcessing data in batches
   Reveal Solution Discussion Select an answer first