Implement Data Cleaning by Reading Bronze Tables with PySpark/SQL, Cleaning Nulls, Standardizing Data Types, and Writing to New Silver Tables. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A bronze table contains transaction data with a `status` column that has values 'completed', 'pending', 'failed', and null. The silver table requires `status` to be non-null, with nulls replaced by 'unknown'. Additionally, the `amount` column has some nulls that should be filled with the median amount. Which approach correctly handles both requirements?

   • ACompute the median of `amount`, then use `df.fillna({'status': 'unknown', 'amount': median_value})`.
   • BUse `df.fillna('unknown')` to fill all nulls with 'unknown'.
   • CUse `df.na.fill('unknown', subset=['status']).na.fill(0, subset=['amount'])`.
   • DUse `df.fillna({'status': 'unknown'})` and then `df.dropna(subset=['amount'])`.
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A data engineer is building a silver table from a bronze table that is updated daily with new records. The silver table must contain only the latest version of each record, identified by a unique `record_id`. The engineer wants to avoid reprocessing the entire bronze table each day. Which strategy should be used?

   • AUse `MERGE` with a delta table to upsert new records by `record_id`.
   • BUse `dropDuplicates(['record_id'])` on the entire bronze table before writing.
   • COverwrite the silver table with the entire cleaned bronze table each day.
   • DAppend all new records to the silver table without deduplication.
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   A DataFrame contains a column 'age' with some null values. The data engineer wants to remove all rows that have a null in the 'age' column. Which PySpark function should be used?

   • Adf.na.drop(how="any")
   • Bdf.filter(df["age"].isNotNull())
   • Cdf.dropna(subset=["age"])
   • Ddf.fillna(value=0, subset=["age"])
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   A data engineer needs to load the bronze table named 'events' into a DataFrame for cleaning. The table is stored in the Databricks Unity Catalog under the 'sales' schema. Which PySpark command correctly reads this table into a DataFrame?

   • Adf = spark.sql("SELECT * FROM events")
   • Bdf = spark.read.table("sales.events")
   • Cdf = spark.read.format("delta").load("/sales/events")
   • Ddf = spark.table("sales.events")
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A bronze table has a `phone` column where missing values are represented as empty strings `''` rather than nulls. The silver table requires `phone` to be either a valid string or null. Which cleaning step should be applied?

   • AUse `df.filter(df['phone'] != '')` to remove rows with empty strings.
   • BUse `df.dropna(subset=['phone'])` to remove rows with null phones.
   • CUse `df.fillna({'phone': 'unknown'})` to fill nulls with a placeholder.
   • DUse `df.withColumn('phone', when(df['phone'] == '', lit(None)).otherwise(df['phone']))` to convert empty strings to null.
   Reveal Solution Discussion Select an answer first