Perform Logging and Monitoring of Spark Applications - Publish, Customize, and Analyze Driver Logs and Executor Logs to Diagnose Out-Of-Memory Errors, Cluster Underutilization, Etc. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 7)

Questions 31–35

1. 31expert · hard

   A Spark job is failing with 'java.lang.OutOfMemoryError: Java heap space' on the executors. The team has already increased the executor memory (spark.executor.memory) and the number of executors, but the problem persists. The executor logs show that the OOM occurs during the shuffle phase (e.g., during a groupByKey operation). Which of the following is the most likely cause and the most effective next step?

   • AThe shuffle data is too large for the memory allocated to the shuffle service; increase spark.shuffle.memoryFraction.
   • BThe driver is sending too many tasks to the executor; reduce spark.executor.cores to limit concurrency.
   • CThe data is skewed, causing a single executor to receive too much data during the shuffle; use a salting technique or repartition to balance the data.
   • DThe executor's off-heap memory is too small; increase spark.memory.offHeap.size.
   Reveal Solution Discussion Select an answer first
2. 32foundation · easy

   Which log message is a direct sign of an out-of-memory error in a Spark executor?

   • Aorg.apache.spark.SparkException: Task failed while writing rows
   • Bjava.lang.OutOfMemoryError: Java heap space
   • Cjava.io.FileNotFoundException
   • Djava.net.SocketTimeoutException
   Reveal Solution Discussion Select an answer first
4. 33foundation · easy

   Which log message indicates that a stage has completed successfully?

   • AINFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 123 ms
   • BWARN BlockManager: Removing block
   • CINFO SparkContext: Starting job
   • DERROR TaskSetManager: Task 0.0 in stage 0.0 failed
   Reveal Solution Discussion Select an answer first
5. 34application · medium

   A team is monitoring a long-running streaming Spark application. They want to be alerted if the application stops making progress, for example, if a stage is stuck or if there are repeated task failures. They are already using a log aggregator. What is the most effective way to monitor application progress using logs?

   • AUse the Spark UI's 'Streaming' tab to monitor progress, and manually check it periodically.
   • BParse the driver logs for 'Finished stage' messages and set up an alert if no such message is seen for a defined period.
   • CConfigure the Spark driver to write a heartbeat message to a separate log file every minute, and alert if the heartbeat stops.
   • DParse the executor logs for 'Task completed' messages and alert if the count drops below a threshold.
   Reveal Solution Discussion Select an answer first
6. 35application · medium

   A team is debugging a Spark job that processes data from multiple source systems. They need to trace which specific input record caused a processing failure. The default log format does not include the record identifier. They want to add this context to the log output without changing the application code that generates the logs. What is the most effective way to achieve this?

   • AUse the --conf spark.executor.extraJavaOptions to add -Dlog4j.debug=true, which prints the record ID in the logs.
   • BChange the Spark log level to TRACE, which automatically includes all variable values in the log output.
   • CSet the Spark property spark.log.lineage to true, which adds the input record ID to every log line.
   • DModify the log4j properties to add a custom pattern layout that includes the thread name and a mapped diagnostic context (MDC) key.
   Reveal Solution Discussion Select an answer first