Identify Hallucinations, Reasoning Errors and Biases in LLM Output CT-GENAI Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A QA team uses an LLM to generate test cases for a multilingual customer support chatbot. The team notices that the LLM generates more test cases in English than in other languages, even when the prompt asks for equal coverage. The team suspects the bias comes from the training data. They want to mitigate this bias without sacrificing the quality of the English test cases. Which strategy is most effective?

   • AGenerate test cases in English first, then translate them to other languages
   • BIncrease the number of test cases generated to ensure enough non-English cases
   • CUse a multilingual LLM that is specifically trained on diverse language data
   • DManually write non-English test cases to ensure quality
   Reveal Solution Discussion Select an answer first
2. 12expert · hard

   An LLM is used to generate test oracles for a financial trading system. The LLM outputs a test case: 'If the stock price increases by 10%, the system should trigger a buy order. If the stock price decreases by 10%, the system should trigger a sell order.' The team realizes this logic is flawed because it does not account for market conditions. The team needs to classify the error and decide on a detection method. Which classification and detection method is most appropriate?

   • AFactual hallucination; verify the price thresholds against market data
   • BContextual hallucination; compare the output with similar test cases
   • CReasoning error; use step-by-step validation against the trading rules
   • DBias; run fairness metrics on the test cases
   Reveal Solution Discussion Select an answer first
4. 13foundation · easy

   What is a primary source of bias in LLM outputs?

   • ABiased training data
   • BThe model's random number generator
   • CThe user's internet connection speed
   • DThe model's output length
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A team uses an LLM to generate test cases for a discount calculation feature. The LLM outputs: 'If the order total is $100 and the discount is 10%, the final price is $90. Therefore, if the order total is $200 and the discount is 20%, the final price is $160.' The team notices the second calculation is incorrect. What type of error is this?

   • AContextual hallucination
   • BLogical fallacy
   • CFactual hallucination
   • DIncorrect inference
   Reveal Solution Discussion Select an answer first
6. 15expert · hard

   A test team uses an LLM to generate test data for a recruitment platform. The team notices that the LLM generates more test cases with male candidates for technical roles and female candidates for administrative roles. The team wants to address this bias, but they also need to ensure the test data reflects real-world distributions for realistic testing. Which approach best balances bias mitigation with realism?

   • AManually adjust the test data to match the expected real-world distribution
   • BUse a different LLM that is known to be unbiased
   • CKeep the LLM's output as-is to reflect real-world distributions
   • DUse fairness metrics to ensure the test data is balanced across genders, even if it deviates from real-world distributions
   Reveal Solution Discussion Select an answer first