Describe Python, Reasons to Use, and Its Libraries DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 4)

Questions 16–19

1. 16foundation · easy

   A data engineering team is evaluating programming languages for building data pipelines. Which statement best describes Python's role in data engineering?

   • APython is a domain-specific language designed exclusively for statistical analysis and cannot be used for general programming tasks.
   • BPython is a general-purpose language used for scripting, automation, and data processing, making it a common choice for building data pipelines.
   • CPython is a compiled language that requires explicit memory management, making it unsuitable for rapid data pipeline development.
   • DPython is primarily a query language for databases and is rarely used outside of SQL-like operations.
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A data engineering team is building a pipeline that needs to run a complex, CPU-intensive transformation on a large dataset. The team is considering using Python with pandas. They are evaluating whether to use a single large VM or a cluster of smaller VMs with PySpark. The team's primary goal is to minimize the total cost of the compute resources while meeting a strict processing time deadline. What is the most important factor to consider in this decision?

   • AThe fact that pandas is always faster than PySpark for any dataset size
   • BThe fact that a single large VM is always more cost-effective than a cluster
   • CThe cost of the single large VM versus the combined cost of the cluster, considering the processing time and resource utilization
   • DThe fact that PySpark is always cheaper than a single VM for any workload
   Reveal Solution Discussion Select an answer first
4. 18application · easy

   A data engineering team is building a pipeline to ingest data from a REST API, transform it into a tabular format, and then load it into a database. The team wants to minimize the amount of custom code for HTTP requests and data structuring. Which Python library is specifically designed to simplify making HTTP requests?

   • APySpark
   • Bpandas
   • CNumPy
   • DRequests
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A data engineering team is choosing between Python and Scala for a new, high-throughput streaming pipeline. The team's primary expertise is in Python. The pipeline will process millions of events per second and requires low-latency, stateful processing. The team is considering using PySpark for its familiarity. What is the most significant technical trade-off they should consider?

   • AScala is a dynamically typed language, making it more error-prone for complex streaming logic
   • BScala has a much larger community and more libraries for data engineering than Python
   • CPySpark may introduce higher per-record overhead compared to Scala, potentially impacting latency at very high throughput
   • DPySpark cannot perform stateful streaming operations, while Scala can
   Reveal Solution Discussion Select an answer first