Describe Apache Spark and Its Architecture DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A Spark SQL query joins a large fact table with a small dimension table. The query is slow because the join causes a shuffle. Which Catalyst optimization can help avoid the shuffle?

   • ABroadcast join
   • BColumn pruning
   • CPredicate pushdown
   • DDynamic partition pruning
   Reveal Solution Discussion Select an answer first
2. 12expert · medium

   A company runs Spark on a Mesos cluster. They have a mix of long-running services and short-lived Spark jobs. They want to maximize resource utilization by allowing Spark to release resources when idle. Which Mesos feature should they use?

   • AFine-grained mode
   • BStatic allocation
   • CDynamic resource allocation with Mesos coarse-grained mode
   • DYARN capacity scheduler
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A data engineering team is evaluating Spark for a new data processing pipeline. They need a framework that can handle both batch and streaming workloads, provide in-memory processing for speed, and support multiple languages. Which feature of Spark makes it suitable for this?

   • AIt is a database management system with built-in SQL support
   • BIt is a storage system optimized for columnar data
   • CIt is a message queue for ingesting streaming data
   • DUnified engine for batch and streaming with in-memory computation
   Reveal Solution Discussion Select an answer first
5. 14application · easy

   A Spark job is running with 100 tasks in a stage. The cluster has 10 executors, each with 4 cores. How many tasks can run concurrently?

   • A4
   • B10
   • C40
   • D100
   Reveal Solution Discussion Select an answer first
6. 15expert · medium

   A Spark application is experiencing frequent task failures due to executor loss. The team wants to minimize the impact of executor failures on the overall job. Which configuration is most effective?

   • ADecrease spark.task.maxFailures
   • BIncrease spark.executor.heartbeatInterval
   • CUse a single executor with many cores
   • DEnable speculative execution
   Reveal Solution Discussion Select an answer first