Perform Logging and Monitoring of Spark Applications - Publish, Customize, and Analyze Driver Logs and Executor Logs to Diagnose Out-Of-Memory Errors, Cluster Underutilization, Etc. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   What is a common method to collect Spark executor logs from worker nodes into a central location?

   • AEnabling spark.dynamicAllocation.enabled
   • BUsing a log aggregation service such as YARN's log aggregation
   • CUsing the Spark UI to download logs
   • DSetting spark.executor.instances to 1
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A Spark application is failing. The driver log shows an exception that mentions a 'org.apache.spark.SparkException: Job aborted due to stage failure'. To find the root cause, you need to identify which stage failed and then look at the corresponding executor logs. Which approach is most efficient for correlating the driver log entry with the executor logs?

   • ACheck the driver log for the 'TaskSetManager' messages, which contain the executor ID and the exception, and then look at that executor's log.
   • BSearch the driver log for the exception stack trace and then manually search each executor log for the same exception message.
   • CUse the Spark UI to find the failed stage, then use the 'Executors' tab to view the stdout/stderr logs for the executors that ran tasks in that stage.
   • DLook at the Spark History Server's event log to find the stage ID, then use the YARN ResourceManager to find the executor logs.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A team notices that their Spark job is running much slower than expected. They check the Spark UI and see that the cluster has 10 executors, but only 2 are actively processing tasks at any given time. The other 8 executors are idle. The driver logs show no errors. What is the most likely cause of this underutilization?

   • AThe driver is not parallelizing the work because the application code uses a single-threaded loop to process data.
   • BThe executors are configured with too much memory, causing the JVM to spend too much time on garbage collection.
   • CThe data is not evenly distributed across partitions, causing a few tasks to take much longer than others, leaving other executors idle.
   • DThe network shuffle is bottlenecked, causing tasks to wait for data from other executors.
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   What information in the driver log can help diagnose job scheduling issues?

   • AShuffle block fetch failures
   • BGC pause times on executors
   • CLog entries showing stage submission and task assignment
   • DMessages about task serialization failures
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A company runs Spark jobs on a Databricks workspace. They need to send driver and executor logs to a third-party SIEM (Security Information and Event Management) system for real-time security monitoring. The SIEM system provides an HTTP endpoint that accepts log data. Which approach should they use to publish the logs?

   • AUse the Spark UI's 'Logs' tab to export logs and manually upload them to the SIEM.
   • BConfigure the cluster to deliver logs to cloud storage, and then set up a separate process to periodically read and forward them to the SIEM.
   • CUse a log4j appender in the Spark driver and executor configuration that sends log events directly to the SIEM's HTTP endpoint.
   • DEnable the Spark History Server and configure it to write to the SIEM's database.
   Reveal Solution Discussion Select an answer first