Compute Summary Statistics on a Spark DataFrame Using .summary() or Dbutils Data Summaries MACHINE-LEARNING-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A data engineer is working with a Spark DataFrame `sales` that has columns `product_id`, `price`, `quantity`, and `region`. They run `sales.summary().show()` and see that the `price` column has a `mean` of 50, `stddev` of 20, `min` of 5, `25%` of 35, `50%` of 48, `75%` of 62, and `max` of 120. They need to identify if there are any potential outliers in the `price` column. Based on the summary statistics, which approach is most appropriate?

   • AFlag any value above the mean plus two standard deviations (50 + 2*20 = 90) as an outlier.
   • BUse the IQR method: calculate IQR = 75% - 25% = 27, then flag values below 35 - 1.5*27 = -5.5 or above 62 + 1.5*27 = 102.5 as outliers.
   • CFlag any value below the 25th percentile or above the 75th percentile as an outlier.
   • DDo not flag any outliers because the min and max are within a reasonable range.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A data engineer runs `df.summary()` on a Spark DataFrame with a column `age`. The output shows `mean` = 35.2, `stddev` = 12.4, `min` = 18, `25%` = 25, `50%` = 33, `75%` = 44, and `max` = 89. Which interpretation of this data is most accurate?

   • AThe distribution is left-skewed because the mean (35.2) is greater than the median (33).
   • BThe distribution has no outliers because the max is 89 and the min is 18.
   • CThe distribution is right-skewed because the mean (35.2) is greater than the median (33).
   • DThe distribution is symmetric because the mean and median are close.
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   A data analyst runs `df.summary()` on a Spark DataFrame with a column `score`. The output shows `count` = 500, `mean` = 70, `stddev` = 15, `min` = 20, `25%` = 60, `50%` = 72, `75%` = 82, `max` = 95. Which statement about the `score` column is most accurate?

   • AThe distribution has no outliers because the min is 20 and the max is 95.
   • BThe distribution is slightly left-skewed because the mean (70) is less than the median (72).
   • CThe distribution is symmetric because the mean and median are close.
   • DThe distribution is right-skewed because the mean (70) is less than the median (72).
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   In the output of df.summary(), which statistic indicates the central tendency of a numeric column?

   • Astddev
   • Bcount
   • Cmin
   • Dmean
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A data analyst has a Spark DataFrame `transactions` with columns `transaction_id`, `user_id`, `amount`, and `timestamp`. They need to check if the `amount` column has any extreme outliers before building a fraud detection model. They want to see the min, max, mean, and standard deviation, as well as the 25th and 75th percentiles. Which single command will provide all of this information?

   • A`transactions.describe('amount')`
   • B`transactions.select('amount').summary()`
   • C`transactions.groupBy('amount').count()`
   • D`transactions.select('amount').agg(min('amount'), max('amount'))`
   Reveal Solution Discussion Select an answer first