Design and Implement Data Ingestion Pipelines to Efficiently Ingest a Variety of Data Formats Including Delta Lake, Parquet, ORC, AVRO, JSON, CSV, XML, Text and Binary from Diverse Sources Such as Message Buses and Cloud Storage. DATA-ENGINEER-PROFESSIONAL Practice Questions (Page 7)

Questions 31–34

1. 31foundation · easy

   Which Spark format is used to read binary files into a DataFrame with columns for the file path, modification time, length, and content?

   • Abytes
   • BbinaryFile
   • Cbinary
   • Dfile
   Reveal Solution Discussion Select an answer first
2. 32foundation · easy

   Which option in Auto Loader or DataFrameReader captures columns that are present in the data but not in the specified schema, storing them in a separate column?

   • Amode
   • BschemaEvolution
   • CrescuedDataColumn
   • DcolumnNameOfCorruptRecord
   Reveal Solution Discussion Select an answer first
4. 33foundation · easy

   Which Spark option allows you to read text files with a custom line delimiter instead of the default newline character?

   • AlineSep
   • Bnewline
   • Cseparator
   • Ddelimiter
   Reveal Solution Discussion Select an answer first
5. 34expert · hard

   A financial services company ingests real-time stock trade data from Kafka into a Delta table using Structured Streaming. The data volume is high, and the team is experiencing backpressure issues, causing the streaming job to fall behind. They need to increase throughput while maintaining exactly-once processing semantics. Which approach should they use?

   • AUse a batch read from Kafka instead of Structured Streaming.
   • BIncrease the number of Kafka partitions and the number of Spark executors.
   • CDisable checkpointing to reduce overhead.
   • DDecrease the 'maxOffsetsPerTrigger' option to reduce the amount of data processed per micro-batch.
   Reveal Solution Discussion Select an answer first