User-Defined Functions and Stateful Operations ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   What is the key difference between mapGroupsWithState and flatMapGroupsWithState?

   • AflatMapGroupsWithState is stateless, while mapGroupsWithState is stateful.
   • BmapGroupsWithState is used for batch queries, while flatMapGroupsWithState is used for streaming queries.
   • CmapGroupsWithState can output zero or more rows per group, while flatMapGroupsWithState outputs exactly one row per group.
   • DflatMapGroupsWithState can output zero or more rows per group, while mapGroupsWithState outputs exactly one row per group.
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A data engineer needs to apply a custom function to a DataFrame column that returns a struct with two fields. They plan to use a UDF. What is the correct way to define the return type?

   • AUse a StringType and parse the string later.
   • BSpecify a StructType with two fields as the return type in the UDF definition.
   • CLet Spark infer the return type from the function's output.
   • DUse an ArrayType of strings.
   Reveal Solution Discussion Select an answer first
4. 18application · hard

   A streaming job uses flatMapGroupsWithState to aggregate events per key. The requirement is to emit an output row for a key only when no new events arrive for that key for 10 minutes. What is the correct approach?

   • ASet a processing-time timeout of 10 minutes and emit the final aggregation in the timeout handler.
   • BUse a windowed aggregation with a 10-minute slide.
   • CSet an event-time timeout of 10 minutes and rely on watermarking.
   • DUse mapGroupsWithState with a timeout of 10 minutes.
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   What is the role of the StateStore in Spark Structured Streaming?

   • AIt persists and recovers state for stateful operations, providing fault tolerance.
   • BIt caches intermediate results of transformations to speed up queries.
   • CIt stores the streaming query's output data for later retrieval.
   • DIt manages the checkpoint location for the streaming query.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   What is the correct way to invoke a UDF on a DataFrame column in Spark?

   • Adf.transform(myUDF, "existingCol")
   • Bdf.withColumn("newCol", myUDF(col("existingCol")))
   • Cdf.select(myUDF("existingCol"))
   • Ddf.map(row => myUDF(row.getAs[String]("existingCol")))
   Reveal Solution Discussion Select an answer first