Write Efficient Spark SQL and PySpark Code to Apply Advanced Data Transformations, Including Window Functions, Joins, and Aggregations, to Manipulate and Analyze Large Datasets. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   Which PySpark DataFrame method performs multiple aggregations in a single pass over the data?

   • Adf.withColumn()
   • Bdf.agg()
   • Cdf.filter()
   • Ddf.select()
   Reveal Solution Discussion Select an answer first
2. 17foundation · easy

   Which Spark SQL clause computes subtotals for every combination of grouping columns, including the grand total?

   • AGROUP BY ROLLUP
   • BGROUP BY CUBE
   • CGROUP BY GROUPING SETS
   • DGROUP BY HAVING
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   Which PySpark DataFrame method is used to perform a join between two DataFrames?

   • Adf.join()
   • Bdf.union()
   • Cdf.concat()
   • Ddf.merge()
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   In Spark SQL, which join type produces the Cartesian product of two tables?

   • AFULL OUTER JOIN
   • BLEFT SEMI JOIN
   • CCROSS JOIN
   • DLEFT JOIN
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A data engineer has a `customers` DataFrame and an `orders` DataFrame. The requirement is to return only customers who have placed at least one order, without including any order columns in the result. Which PySpark operation achieves this with minimal data movement?

   • A`customers.join(orders, 'customer_id', 'left_outer').filter(col('order_id').isNotNull())`
   • B`customers.join(orders, 'customer_id', 'left_anti')`
   • C`customers.join(orders, 'customer_id', 'inner').select(customers.columns)`
   • D`customers.join(orders.select('customer_id').distinct(), 'customer_id', 'left_semi')`
   Reveal Solution Discussion Select an answer first