Spark DATA-SCIENCE-OPTIMIZE Practice Questions (Page 4)

Questions 16–20

1. 16expert · hard

   A Spark job is processing a large dataset and uses a groupByKey transformation followed by a mapValues to aggregate. The job is running slowly, and the team suspects that the groupByKey is causing excessive shuffling. They want to improve performance without changing the output. Which change should they make?

   • AIncrease the number of executors
   • BUse a map transformation instead of groupByKey
   • CUse a broadcast variable to distribute the aggregation logic
   • DReplace groupByKey with reduceByKey
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A data science team wants to build a recommendation system for an e-commerce platform. They have a large dataset of user-item interactions and need to train a collaborative filtering model at scale. Which Spark MLlib algorithm should they use?

   • AAlternating Least Squares (ALS)
   • BDecision tree classifier
   • CLogistic regression
   • DK-means clustering
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   In Spark SQL, what is the relationship between a DataFrame and a SQL table?

   • AA DataFrame can be registered as a temporary view and then queried with SQL.
   • BA SQL table is converted to an RDD before querying.
   • CA DataFrame is a SQL table that is stored permanently in the metastore.
   • DA DataFrame can only be queried using the DataFrame API, not SQL.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A team is using Spark SQL to query a large dataset stored in Parquet. They notice that queries are slow, and they suspect that the Catalyst optimizer is not able to push down filters effectively. They want to improve query performance. Which action would be most effective?

   • APartition the Parquet table by a frequently filtered column
   • BUse a broadcast join hint in all queries
   • CIncrease the number of executors
   • DConvert the Parquet files to CSV for faster reads
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A data engineering team is running a Spark job that reads a large CSV file, applies a filter to remove invalid records, and then writes the cleaned data to Parquet. The job is taking longer than expected, and the team suspects that too much data is being shuffled across the network. Which action would most directly reduce network shuffle in this scenario?

   • AUse a DataFrame instead of an RDD
   • BIncrease the Spark driver memory
   • CApply the filter before any wide transformation
   • DIncrease the number of executors
   Reveal Solution Discussion Select an answer first