Using Python and Tools for Development DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   Which file format is a built distribution of a Python package that can be installed in Databricks?

   • A.tar.gz
   • B.zip
   • C.jar
   • D.whl
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   What is the primary benefit of using a Pandas UDF (also known as a vectorized UDF) in PySpark?

   • AIt eliminates the need for Spark SQL
   • BIt allows using pandas APIs on distributed data with better performance than row-wise UDFs
   • CIt processes data row-by-row, which is simpler to write
   • DIt automatically caches the entire DataFrame in memory
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A team has developed a custom Python package for internal data transformations. They have built a wheel file (`my_transform-1.0.0-py3-none-any.whl`) and want to install it on all clusters used by their data engineering jobs. The wheel has a dependency on `numpy`, which is already available in the Databricks Runtime. What is the recommended way to install this wheel?

   • AUpload the wheel to a cloud storage location (e.g., S3 or Azure Blob) and install it as a library on each cluster using the library UI.
   • BUse `%pip install /path/to/my_transform-1.0.0-py3-none-any.whl` in a notebook cell and run it once.
   • CAdd the wheel as a library to the cluster using the UI or API, specifying the path to the wheel file in cloud storage.
   • DPlace the wheel in a DBFS location and add it as a library to the cluster's init script so it installs on every cluster start.
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   What is a key performance consideration when using Python UDFs in PySpark?

   • AThey automatically use GPU acceleration
   • BThey incur serialization overhead because data is transferred between JVM and Python
   • CThey require the entire DataFrame to be collected to the driver
   • DThey are always faster than built-in Spark functions
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A data engineer is setting up a cluster for a team that needs to use a specific version of a PyPI package, `scikit-learn==1.2.2`. The cluster uses Databricks Runtime 13.3 LTS, which includes a different version of `scikit-learn`. The engineer wants to ensure the team uses the required version without affecting other teams sharing the same cluster. What is the best approach?

   • AModify the cluster's init script to install the specific version at startup.
   • BCreate a separate cluster for this team and install `scikit-learn==1.2.2` there.
   • CUse `%pip install scikit-learn==1.2.2` in a notebook and instruct the team to run it every time.
   • DInstall `scikit-learn==1.2.2` as a cluster library, which will override the pre-installed version for all users.
   Reveal Solution Discussion Select an answer first