Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 2)

Questions 6–10

1. 6foundation · easy

   What is the main challenge non-deterministic LLM outputs pose for test repeatability?

   • AThe test input must be changed for every execution
   • BThe same test input may produce different outputs on different runs
   • CThe test data must be regenerated before each run
   • DThe test environment must be recreated for each run
   Reveal Solution Discussion Select an answer first
2. 7foundation · easy

   What role does the temperature parameter play in LLM text generation?

   • AIt adjusts the probability distribution over the next token, affecting randomness
   • BIt determines the number of layers in the neural network
   • CIt controls the length of the generated response
   • DIt sets the maximum number of tokens to generate
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A QA team is deploying an LLM-based test oracle in production. They want to detect when the LLM's outputs become unexpectedly variable, which could indicate a problem. What should they implement?

   • AUse a fixed random seed for all production calls.
   • BOnly log successful test cases.
   • CDisable logging to reduce overhead.
   • DLog all LLM inputs and outputs, and monitor the variance of outputs for similar inputs.
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   An LLM is used to extract test case names from natural language descriptions. The output sometimes includes extra whitespace, different capitalization, or trailing punctuation. The test framework compares the extracted names to a list of expected names. What is the best way to handle this variability?

   • ARun the test multiple times and accept the output if it matches any run.
   • BDisable the LLM and manually enter all test case names.
   • CApply a normalization step that trims whitespace, lowercases, and removes punctuation before comparison.
   • DUse the LLM output exactly as-is and update the expected list to match every possible variation.
   Reveal Solution Discussion Select an answer first
6. 10foundation · easy

   Why is version pinning important when using LLMs in software testing?

   • ATo reduce the cost of API usage
   • BTo increase the speed of model inference
   • CTo make the model more accurate
   • DTo ensure the model's behavior remains consistent over time
   Reveal Solution Discussion Select an answer first