Identify the Features of the Apache Spark Modules, Including Core, Spark SQL, DataFrames, Pandas API on Spark, Structured Streaming, and MLib. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   Which component of Apache Spark is responsible for scheduling tasks, managing the execution of jobs, and providing the foundation upon which all other Spark modules are built?

   • AStructured Streaming
   • BMLlib
   • CSpark Core
   • DSpark SQL
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A team is troubleshooting a performance issue in a Spark job. The job reads a large dataset, performs a series of transformations, and writes the output. They suspect that the Catalyst optimizer is not producing an optimal plan. They want to inspect the physical execution plan to identify potential bottlenecks. Which approach should they take?

   • AUse MLlib's tuning utilities to automatically optimize the execution plan.
   • BUse the Pandas API on Spark to run the job and see the plan.
   • CUse the RDD API to bypass the optimizer and manually control the execution.
   • DUse the explain() method on the DataFrame to view the optimized physical plan.
   Reveal Solution Discussion Select an answer first
4. 18application · medium

   A team is migrating an existing data processing pipeline from a single-node pandas script to Spark. The current script relies heavily on column names and data types to perform type-specific operations. They want to maintain this level of structure while scaling out. Which Spark abstraction should they choose to best match their current workflow?

   • ARDDs, because they are the fundamental data structure and can handle any data type.
   • BMLlib, because it provides a suite of data processing utilities.
   • CStructured Streaming, because it can handle both batch and streaming data.
   • DDataFrames, because they are distributed collections of data organized into named columns with a schema.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A team has a Spark job that reads data from a JDBC source, performs a complex join, and writes the results to a data lake. The job is running slowly, and they suspect that the Catalyst optimizer is not pushing down the filter predicates to the JDBC source. They want to verify this. What should they do?

   • AUse the RDD API to manually implement the filter and join, bypassing the optimizer.
   • BUse MLlib's feature importance to determine if the filter is being applied.
   • CUse the Pandas API on Spark to run the query and see the plan.
   • DUse the explain() method to check if the filter is applied in the physical plan before the JDBC scan.
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A developer is writing a custom transformation that requires fine-grained control over the partitioning of data across the cluster. They need to ensure that records with the same key are processed by the same executor. Which Spark API should they use to have this level of control?

   • ASpark SQL, by writing a query with a specific DISTRIBUTE BY clause.
   • BDataFrames, using the repartition and partitionBy functions.
   • CRDDs, using the partitionBy function with a custom Partitioner.
   • DMLlib, using the VectorAssembler to control data distribution.
   Reveal Solution Discussion Select an answer first