Write Efficient Spark SQL and PySpark Code to Apply Advanced Data Transformations, Including Window Functions, Joins, and Aggregations, to Manipulate and Analyze Large Datasets. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 3)

Questions 11–15

1. 11foundation · easy

   Which Spark SQL clause is used to filter groups after an aggregation?

   • AWHERE
   • BLIMIT
   • CORDER BY
   • DHAVING
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A nightly job aggregates a large `events` table (10 TB) by `event_date` and `event_type`. The table is partitioned by `event_date`. The job currently reads the entire table every night, even though only the last 7 days of data change. Which change reduces the amount of data read and the shuffle size?

   • AIncrease `spark.sql.shuffle.partitions` to 2000.
   • BAdd a filter `WHERE event_date >= date_sub(current_date(), 7)` before the aggregation.
   • CCache the entire `events` table in memory.
   • DRepartition the DataFrame by `event_type` before aggregating.
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   In Spark SQL, which clause is required to define a window over which a window function operates?

   • APARTITION BY ... (without OVER)
   • BORDER BY ... LIMIT ...
   • CGROUP BY ... HAVING ...
   • DOVER (PARTITION BY ... ORDER BY ...)
   Reveal Solution Discussion Select an answer first
5. 14foundation · easy

   Which PySpark DataFrame transformation is used to add a new column or replace an existing one?

   • Adf.withColumn()
   • Bdf.groupBy()
   • Cdf.filter()
   • Ddf.select()
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   What is the primary benefit of bucketing a table on a join key?

   • AIt can avoid shuffles during joins on that key.
   • BIt sorts the data within each partition.
   • CIt reduces the number of files written.
   • DIt automatically compresses the data.
   Reveal Solution Discussion Select an answer first