User-Defined Functions and Stateful Operations ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   Why do UDFs typically have worse performance than built-in Spark functions?

   • AUDFs are executed on the driver, causing a bottleneck.
   • BUDFs are written in a language that Spark cannot compile to JVM bytecode.
   • CUDFs always cause a full shuffle of all data across the cluster.
   • DUDFs are executed as black boxes, preventing Catalyst from applying optimizations like predicate pushdown.
   Reveal Solution Discussion Select an answer first
2. 7application · hard

   A streaming application uses flatMapGroupsWithState with event-time timeouts. The team notices that timeouts are not firing as expected. What is a likely cause?

   • AThe watermark is not advancing because the input data is out of order.
   • BThe state is not being updated in the function.
   • CThe state store is not configured to persist timeouts.
   • DThe timeout duration is set in processing time instead of event time.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A streaming application uses mapGroupsWithState to track user session durations. The state is keyed by user ID. The team notices that the state grows unboundedly because sessions never end. What should they configure to limit state growth?

   • ASet a timeout duration in the GroupState and handle the timeout in the update function.
   • BUse flatMapGroupsWithState instead of mapGroupsWithState.
   • CCall spark.conf.set('spark.sql.streaming.stateStore.minDeltasForSnapshot', 10).
   • DIncrease the number of state store partitions.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A streaming application uses flatMapGroupsWithState to emit alerts when a threshold is crossed. The team wants to ensure that alerts are not emitted repeatedly for the same condition. What should they do?

   • AStore a flag in the state indicating whether the alert has already been emitted.
   • BUse a processing-time timeout to reset the state.
   • CEmit the alert only when the state is updated.
   • DUse mapGroupsWithState instead.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A Spark job uses a UDF to parse a JSON string column. The UDF is slow. The team suspects the UDF is the bottleneck. Which action is most likely to improve performance?

   • ACache the DataFrame before applying the UDF.
   • BIncrease the spark.sql.shuffle.partitions setting.
   • CUse the built-in from_json function with a schema.
   • DUse a UDF with a return type of StringType.
   Reveal Solution Discussion Select an answer first