Describe Apache Spark and Its Architecture DATA-ENGINEERING-OPTIMIZE Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A company needs to process real-time clickstream data from a website and update dashboards with near-real-time metrics. They choose Spark Streaming. Which characteristic of Spark Streaming makes it suitable for this use case?

   • AIt processes each event individually as it arrives, ensuring minimal latency
   • BIt provides exactly-once processing without any configuration
   • CIt processes data in micro-batches, providing near-real-time processing with fault tolerance
   • DIt requires a separate cluster from the batch processing engine
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A developer is working with a dataset that has a known schema. They want to perform type-safe operations and benefit from compile-time error checking. Which Spark API should they use?

   • ASpark SQL
   • BDataFrame
   • CDataset
   • DRDD
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A data engineering team runs a Spark application on a YARN cluster. The application fails with an 'OutOfMemory' error in the driver. The driver logs show that the error occurs while the driver is collecting results from executors using collect(). Which configuration change is most likely to resolve the issue?

   • AIncrease spark.executor.memory
   • BIncrease spark.executor.cores
   • CIncrease spark.sql.shuffle.partitions
   • DIncrease spark.driver.memory
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A small team wants to set up a Spark cluster quickly for development purposes without installing Hadoop or any external resource manager. They have a few machines available. Which cluster manager should they use?

   • AYARN
   • BStandalone
   • CKubernetes
   • DMesos
   Reveal Solution Discussion Select an answer first
6. 10expert · medium

   A data engineer is optimizing a Spark job that processes a large dataset. The job uses RDDs and performs multiple transformations. The engineer wants to improve performance by leveraging Catalyst optimizer. Which action should they take?

   • ACache the RDD in memory
   • BIncrease the number of partitions
   • CUse mapPartitions instead of map
   • DConvert the RDD to a DataFrame
   Reveal Solution Discussion Select an answer first