Perform Logging and Monitoring of Spark Applications - Publish, Customize, and Analyze Driver Logs and Executor Logs to Diagnose Out-Of-Memory Errors, Cluster Underutilization, Etc. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16application · medium

   A Spark job is failing with an 'OutOfMemoryError' on the driver. The driver logs show that the error occurs when the code calls a collect() action on a large DataFrame. The team has already increased the driver memory, but the problem persists. What is the most effective next step?

   • ASet spark.driver.maxResultSize to a larger value to allow more data to be sent to the driver.
   • BIncrease the number of partitions of the DataFrame before calling collect() to reduce the amount of data sent to the driver.
   • CIncrease the executor memory to allow the executors to process the data more efficiently before sending it to the driver.
   • DUse a different action, such as take() or write, to avoid collecting all the data to the driver.
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   In a Spark application, where are driver logs typically written when running on a cluster?

   • ATo the executor's stdout/stderr on the worker node
   • BTo the Spark event log directory only
   • CTo the driver node's local file system or to the cluster's log aggregation system
   • DTo the same location as executor logs on each worker node
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   Which log4j component controls the format of log messages (e.g., timestamp, level, logger name)?

   • ALogger
   • BLayout
   • CAppender
   • DFilter
   Reveal Solution Discussion Select an answer first
5. 19application · medium

   A data engineering team is running a nightly Spark batch job that processes sensitive financial data. The job occasionally fails with an obscure error that only appears in the executor logs, but the default log level produces too much noise to find the root cause quickly. The team wants to increase the verbosity of logging specifically for the executors while keeping the driver logs at their current level to avoid overwhelming the driver log aggregator. Which approach should they take?

   • AUse the Spark UI's 'Executors' tab to dynamically change the log level of individual executors to DEBUG during the job run.
   • BSet the Spark property spark.executor.extraJavaOptions to include -Dlog4j.configuration=file:/path/to/executor-log4j.properties, where the custom file sets the root logger to DEBUG.
   • CModify the log4j.properties file in the Spark conf directory to set log4j.rootCategory=DEBUG, and restart the cluster.
   • DAdd a line to the driver code that calls spark.sparkContext.setLogLevel("DEBUG") before the job starts.
   Reveal Solution Discussion Select an answer first
6. 20foundation · easy

   What does a high number of `java.lang.OutOfMemoryError` exceptions in executor logs typically indicate?

   • ANetwork bandwidth saturation
   • BDisk I/O bottlenecks
   • CExecutor-side memory pressure
   • DDriver-side resource contention
   Reveal Solution Discussion Select an answer first