Perform Logging and Monitoring of Spark Applications - Publish, Customize, and Analyze Driver Logs and Executor Logs to Diagnose Out-Of-Memory Errors, Cluster Underutilization, Etc. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 1)

Questions 1–5

1. 1foundation · easy

   Which log pattern indicates cluster underutilization due to low task parallelism?

   • AOnly a few tasks running at a time while many executors are idle
   • BAll executors reporting high GC activity
   • CMany tasks running concurrently across all executors
   • DFrequent task retries due to fetch failures
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A Spark job is failing. The executor logs show a 'org.apache.spark.shuffle.FetchFailedException' for a specific task. The Spark UI shows that the stage has a few failed tasks, but the job eventually succeeds after retries. What is the most likely cause of this error?

   • AThe executor's disk is full, preventing the shuffle data from being written.
   • BThe task is processing a null value, causing a NullPointerException during the shuffle.
   • CThe executor that was serving the shuffle data crashed or was lost, causing the fetch to fail.
   • DThe data being shuffled is too large for the network, causing the fetch to time out.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A team is running a Spark job that processes a large dataset with a key that has a highly skewed distribution. The Spark UI shows that one task in a stage is taking 30 minutes, while all other tasks finish in under a minute. The driver logs show no errors. The team wants to improve the job's performance. Which of the following is the most effective approach?

   • AIncrease the number of executors to provide more parallelism for the skewed task.
   • BReduce the number of partitions to increase the amount of data per partition, which may help balance the load.
   • CIncrease the executor memory to allow the skewed task to process more data without spilling to disk.
   • DUse a salting technique to add a random prefix to the skewed key, then perform the aggregation in two steps.
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   Which type of issue is most likely indicated by repeated `java.lang.OutOfMemoryError: Java heap space` exceptions in the driver log?

   • ADisk space exhaustion on worker nodes
   • BExecutor-side shuffle failure
   • CNetwork connectivity issues between nodes
   • DDriver-side memory exhaustion
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   To include the Spark application ID in every log line, what is the most effective approach?

   • AAdd the application ID to the log4j pattern layout using a conversion character
   • BSet spark.app.name to include the application ID
   • CEnable spark.eventLog.logBlockUpdates
   • DUse a custom log4j appender that prepends the application ID to each message
   Reveal Solution Discussion Select an answer first