Create and Invoke Pandas UDF. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6expert · hard

   A team is optimizing a Spark job that processes a large DataFrame. They have a Python function that performs a complex calculation on each row. They are considering three options: a regular Python UDF, a SCALAR Pandas UDF, and a GROUPED_MAP Pandas UDF. The calculation is purely row-wise and does not depend on other rows. The DataFrame is not grouped. What is the most appropriate choice?

   • AAny of the above would work equally well
   • BRegular Python UDF
   • CSCALAR Pandas UDF
   • DGROUPED_MAP Pandas UDF
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A data engineer has defined a GROUPED_MAP Pandas UDF named 'calculate_stats' that takes a group of data and returns a DataFrame with 'group_id', 'mean', and 'std' columns. They want to apply it to a DataFrame 'df' grouped by the 'category' column. Which code snippet correctly invokes the UDF?

   • Adf.groupBy('category').agg(calculate_stats)
   • Bdf.apply(calculate_stats, axis=1)
   • Cdf.groupBy('category').apply(calculate_stats)
   • Ddf.select(calculate_stats('category'))
   Reveal Solution Discussion Select an answer first
4. 8foundation · easy

   Which Spark DataFrame transformation can be used to invoke a Pandas UDF and produce a new column?

   • AorderBy
   • Bfilter
   • CwithColumn
   • DgroupBy
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A data scientist is defining a GROUPED_MAP Pandas UDF. They need to ensure that the output DataFrame has a specific schema. How should they specify the schema?

   • AUse a separate schema parameter in the apply() method
   • BThe schema is inferred automatically from the returned DataFrame
   • CPass the schema to the @pandas_udf decorator using the returnType parameter
   • DDefine the schema inside the function and return it with the DataFrame
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A team is deciding between using a regular Python UDF and a Pandas UDF for a complex string-parsing operation on a large DataFrame. They are concerned about performance. What is the primary performance advantage of using a Pandas UDF over a regular UDF?

   • APandas UDFs automatically cache the input data in memory to avoid recomputation
   • BPandas UDFs are written in Scala, which is faster than Python
   • CPandas UDFs process data in batches using vectorized operations, reducing per-row Python overhead
   • DPandas UDFs run on the driver node, which has more resources than worker nodes
   Reveal Solution Discussion Select an answer first