Perform Logging and Monitoring of Spark Applications - Publish, Customize, and Analyze Driver Logs and Executor Logs to Diagnose Out-Of-Memory Errors, Cluster Underutilization, Etc. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   Which configuration file is commonly used to set Spark's log4j properties?

   • Aspark-defaults.conf
   • Blog4j.properties
   • Cspark-env.sh
   • Dcore-site.xml
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A company runs Spark jobs on a Databricks workspace and needs to retain driver and executor logs for 90 days for compliance auditing. The logs are currently only available in the cluster's local storage, which is ephemeral and lost when the cluster terminates. They want a centralized, durable location that is easy to query with standard tools. Which solution should they implement?

   • AEnable the Spark History Server and point it to a local directory on the driver node.
   • BUse the Spark UI's 'Logs' tab to manually download logs after each job and store them in a local file server.
   • CConfigure cluster log delivery to write logs to a cloud storage location (e.g., DBFS or an S3 bucket) via the cluster's advanced options.
   • DAdd a log4j appender in the driver code that writes logs to a network share mounted on the cluster.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A team is investigating a memory issue in their Spark job. They want to see the garbage collection (GC) logs for the executors to understand if the JVM is spending too much time on GC. They are using a Databricks cluster. How can they enable and view GC logs?

   • ASet the Spark log level to TRACE for the executor, which automatically includes GC details in the standard logs.
   • BAdd the Spark configuration spark.executor.extraJavaOptions with -verbose:gc -XX:+PrintGCDetails, and then view the logs in the Spark UI's executor logs.
   • CEnable the Spark History Server, which records GC logs for each executor.
   • DUse the Spark UI's 'Executors' tab and click on 'GC Time' to see the detailed GC log output.
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   What does a log showing one executor processing a much larger amount of data than others indicate?

   • AExecutor memory leak
   • BDriver-side bottleneck
   • CNetwork congestion
   • DData skew
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A Spark job is failing. The executor logs show a series of 'java.lang.OutOfMemoryError: Java heap space' exceptions for a specific task, but the task is retried and eventually succeeds. The Spark UI shows that the stage has a high number of task failures. What is the most likely root cause?

   • AThe executor heap size is too small for the data being processed by that task, and the retry succeeds because the data is cached in memory after the first attempt.
   • BThe task is processing a data skew, where one partition is significantly larger than the others, causing the executor to run out of memory.
   • CThe executor logs are being written to a full disk, causing the JVM to crash with an OOM error.
   • DThe driver is sending too many tasks to the executor at once, overwhelming its memory.
   Reveal Solution Discussion Select an answer first