Implement Data Cleaning by Reading Bronze Tables with PySpark/SQL, Cleaning Nulls, Standardizing Data Types, and Writing to New Silver Tables. DATA-ENGINEER-ASSOCIATE Practice Questions (Page 2)

Questions 6–10

1. 6application · medium

   A data engineer needs to read a bronze table using PySpark and then cast the `user_id` column from `string` to `long`. Which code snippet correctly performs both steps?

   • A`df = spark.table('silver.users').withColumn('user_id', col('user_id').cast('long'))`
   • B`df = spark.read.table('bronze.users').withColumn('user_id', col('user_id').cast('int'))`
   • C`df = spark.table('bronze.users').withColumn('user_id', col('user_id').cast('long'))`
   • D`df = spark.sql('SELECT * FROM bronze.users').withColumn('user_id', col('user_id').cast('string'))`
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A data engineer reads a bronze table containing customer records. The `email` column has missing values for 5% of rows, and the `signup_date` column contains strings like '2024-03-15' and '2024-3-5'. The downstream silver table requires `email` to be non-null and `signup_date` to be a proper `date` type. Which approach should the engineer use?

   • AUse `df.fillna({'email': 'unknown@example.com'})` to fill missing emails, then cast `signup_date` to `string` to preserve the original format, and write to the silver table.
   • BUse `df.dropna()` to remove all rows with any null value, then cast `signup_date` to `date` using `df.select(df['signup_date'].cast('date'))` and write the result.
   • CUse `df.filter(df['email'].isNotNull())` to keep only rows with emails, then use `df.withColumn('signup_date', df['signup_date'].cast('timestamp'))` and write to the silver table.
   • DUse `df.dropna(subset=['email'])` to remove rows with missing emails, then cast `signup_date` to `date` using `df.withColumn('signup_date', df['signup_date'].cast('date'))` and write to the silver table.
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A bronze table contains user profiles. The `age` column has some null values, and the `name` column has leading/trailing spaces. The silver table requires `age` to be filled with a default of 0 and `name` to be trimmed. Which PySpark code correctly performs both cleaning steps?

   • A`df.fillna({'age': 0}).withColumn('name', trim(df['name']))`
   • B`df.fillna(0).withColumn('name', df['name'].cast('string'))`
   • C`df.withColumn('age', coalesce(df['age'], lit(0))).withColumn('name', trim(df['name']))`
   • D`df.dropna(subset=['age']).withColumn('name', trim(df['name']))`
   Reveal Solution Discussion Select an answer first
5. 9foundation · easy

   A DataFrame has a column 'price' with null values. The data engineer wants to replace all nulls in 'price' with 0.0. Which PySpark function should be used?

   • Adf.replace(0.0, subset=["price"])
   • Bdf.fillna(0.0, subset=["price"])
   • Cdf.dropna(subset=["price"])
   • Ddf.na.fill(0.0, ["price"])
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A data engineer cleans a bronze table and wants to write the result to a silver table. The silver table should be created if it does not exist, and if it does exist, the new data should be appended. Which write mode should be used?

   • A`df.write.mode('append').saveAsTable('silver.cleaned')`
   • B`df.write.mode('ignore').saveAsTable('silver.cleaned')`
   • C`df.write.mode('overwrite').saveAsTable('silver.cleaned')`
   • D`df.write.mode('error').saveAsTable('silver.cleaned')`
   Reveal Solution Discussion Select an answer first