Identify the Features of the Apache Spark Modules, Including Core, Spark SQL, DataFrames, Pandas API on Spark, Structured Streaming, and MLib. ASSOCIATE-DEVELOPER-APACHE-SPARK Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A team is building a real-time dashboard that displays the number of orders per second. They are using Structured Streaming to process the order events. They need to ensure that the output is updated continuously and that the system can recover from failures without losing data. Which feature of Structured Streaming directly supports this requirement?

   • AIts built-in fault-tolerance and exactly-once processing guarantees.
   • BIts use of RDDs for low-level data manipulation.
   • CIts integration with MLlib for predictive analytics.
   • DIts ability to use the DataFrame API for batch processing.
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A data engineering team has a large dataset stored in Parquet files on a cloud object store. They need to run ad-hoc analytical queries that involve joining this data with a small lookup table, filtering rows, and aggregating results. The team is comfortable writing SQL but has limited experience with Scala or Python. Which approach should they use to minimize development effort while leveraging Spark's distributed processing?

   • AUse the Pandas API on Spark to load the data into a pandas DataFrame and perform the operations using pandas functions.
   • BLoad the data as RDDs and implement the join, filter, and aggregation using map and reduceByKey transformations.
   • CUse the Spark SQL API to register the Parquet data as a temporary view and write the join, filter, and aggregation as a SQL query.
   • DUse MLlib's feature transformers to prepare the data and then apply a SQL join within a machine learning pipeline.
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   Which Spark module is designed for scalable and fault-tolerant stream processing and is built on the Spark SQL engine?

   • APandas API on Spark
   • BStructured Streaming
   • CMLlib
   • DSpark Core
   Reveal Solution Discussion Select an answer first
5. 24application · easy

   A data analyst has a large, well-tested Python script that uses pandas for data cleaning and transformation. The script runs too slowly on their single machine, and they want to use Spark to scale it out without rewriting the logic. Which Spark feature is specifically designed for this scenario?

   • AThe Pandas API on Spark, which provides a pandas-compatible API that runs on the Spark engine.
   • BSpark Core, which allows them to run their script as a set of RDD transformations.
   • CSpark SQL, which allows them to convert their pandas code to SQL queries.
   • DMLlib, which provides scalable implementations of common data science algorithms.
   Reveal Solution Discussion Select an answer first
6. 25application · easy

   A data science team wants to train a logistic regression model on a very large dataset that does not fit in memory on a single machine. They need a solution that can scale the training across a cluster. Which Spark module should they use?

   • AStructured Streaming, to train the model on a stream of data.
   • BMLlib, which provides scalable machine learning algorithms like classification and regression.
   • CSpark SQL, to perform the model training using SQL queries.
   • DSpark Core, to implement the gradient descent algorithm from scratch using RDDs.
   Reveal Solution Discussion Select an answer first