User-Defined Functions and Stateful Operations ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A team is optimizing a Spark job that uses a Python UDF to transform a large DataFrame. The UDF is called on a column with high cardinality. Which change is most likely to improve performance?

   • AIncrease the number of partitions in the DataFrame.
   • BUse a UDF with a return type of StringType instead of a custom case class.
   • CRewrite the UDF using built-in Spark functions where possible.
   • DCache the DataFrame before applying the UDF.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A streaming application uses flatMapGroupsWithState with event-time timeouts. The team wants to ensure that timeouts fire even if the input stream has gaps. What is necessary?

   • AIncrease the timeout duration.
   • BUse a processing-time timeout instead.
   • CDisable watermarking.
   • DSet a watermark that advances based on the event-time of the input.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A streaming application using mapGroupsWithState needs to recover state after a driver failure. What is the role of the StateStore in this scenario?

   • AIt automatically recomputes state from the original input data.
   • BIt persists state to a fault-tolerant store (e.g., HDFS) so that it can be reloaded on restart.
   • CIt stores state in the driver's memory for fast access.
   • DIt keeps state only in memory and loses it on failure.
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   When writing a UDF in Spark, how should null values be handled?

   • AThe UDF should explicitly check for null inputs and decide on an appropriate output, because Spark does not automatically handle nulls inside UDFs.
   • BNull handling is only needed for Python UDFs, not Scala UDFs.
   • CNull values are converted to zero by default in UDFs.
   • DSpark automatically skips null values before invoking the UDF, so no null handling is needed.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   How do you configure a timeout for state in mapGroupsWithState?

   • AUse the withTimeout() method on the DataFrame.
   • BSet the timeout in the streaming query's trigger interval.
   • CSet the timeout duration using GroupState.setTimeoutDuration() and specify the timeout handling in the function.
   • DConfigure a global timeout in spark.conf.
   Reveal Solution Discussion Select an answer first