Text Preprocessing DATA-SCIENCE-OPTIMIZE Practice Questions (Page 6)

Questions 26–30

1. 26foundation · easy

   Why might you filter out rare words during text preprocessing?

   • ATo convert all words to lowercase
   • BTo reduce vocabulary size and improve model efficiency
   • CTo increase the number of tokens in the text
   • DTo expand contractions
   Reveal Solution Discussion Select an answer first
2. 27application · medium

   A multilingual dataset contains text with inconsistent Unicode encodings, including full-width characters (e.g., 'Ａ' vs 'A') and varying whitespace. The team needs consistent tokens for a cross-lingual model. What should they do?

   • AConvert all text to ASCII using transliteration.
   • BKeep all Unicode characters as-is and rely on the tokenizer to handle them.
   • CNormalize Unicode to a standard form (NFKC), convert full-width to half-width, and collapse multiple spaces.
   • DRemove all non-ASCII characters from the text.
   Reveal Solution Discussion Select an answer first
4. 28application · medium

   A sentiment model for product reviews has a vocabulary of 50,000 tokens, but many appear only once in the training data. The team wants to reduce overfitting and improve generalization. What should they do?

   • AFilter out tokens that appear fewer than 5 times across the corpus.
   • BRemove all stop words to free up vocabulary slots for rare words.
   • CKeep all tokens but assign them random embeddings.
   • DReplace rare tokens with their stems to increase their frequency.
   Reveal Solution Discussion Select an answer first
5. 29expert · hard

   A sentiment model for customer support chats must handle informal text like 'cant', 'wont', and 'im'. The team is debating whether to expand contractions before or after tokenization. The tokenizer splits on whitespace and punctuation. What is the correct approach?

   • AExpand contractions before tokenization using a lookup table.
   • BRemove all contractions and abbreviations from the text.
   • CTokenize first, then expand contractions on the resulting tokens.
   • DLowercase the text and keep contractions as-is.
   Reveal Solution Discussion Select an answer first
6. 30application · medium

   A chatbot training dataset contains informal messages like 'idk', 'gonna', and 'don't'. The team wants to standardize these before training a language model. Which preprocessing step should be applied before tokenization?

   • AExpand contractions and replace informal abbreviations with their full forms.
   • BRemove all contractions and abbreviations from the text.
   • CTokenize first, then expand contractions on the resulting tokens.
   • DLowercase the text and keep contractions as-is.
   Reveal Solution Discussion Select an answer first