Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 4)

Questions 16–20

1. 16foundation · easy

   What is the primary purpose of logging LLM inputs and outputs during testing?

   • ATo diagnose non-deterministic behavior by analyzing patterns in outputs
   • BTo reduce the cost of API calls
   • CTo increase the speed of test execution
   • DTo automatically fix defects in the LLM
   Reveal Solution Discussion Select an answer first
2. 17application · medium

   A test automation engineer is writing a script that uses an LLM to generate test data. The script runs nightly, and the team noticed that the generated data varies slightly between runs, causing some tests to fail intermittently. The engineer wants to reduce this variability without changing the LLM's underlying model. Which action is most effective?

   • AUse a different prompt template that includes more examples.
   • BSet the temperature parameter to 0 and use greedy decoding.
   • CIncrease the max tokens parameter to allow longer responses.
   • DRun the LLM on a different hardware instance each time.
   Reveal Solution Discussion Select an answer first
4. 18foundation · easy

   What should be documented when assessing residual risk of LLM non-determinism?

   • AThe personal opinions of the testers about the LLM
   • BThe total cost of all API calls made during testing
   • CThe exact random seed used for every test run
   • DThe mitigation techniques applied and the remaining risk level
   Reveal Solution Discussion Select an answer first
5. 19foundation · easy

   What is the purpose of using a consensus strategy with multiple LLM samples?

   • ATo increase the speed of generating responses
   • BTo reduce the cost of API calls
   • CTo reduce the impact of occasional non-deterministic errors by selecting the most common output
   • DTo make the model more creative
   Reveal Solution Discussion Select an answer first
6. 20expert · hard

   A test framework uses an LLM to generate expected error messages for negative test cases. The LLM sometimes produces messages that are semantically equivalent but worded differently. The team wants to assert that the error message is correct without being too strict. Which approach is most appropriate?

   • AUse a regex pattern that matches the exact error code and ignore the message text.
   • BUse a semantic similarity score with a threshold to determine if the message is acceptable.
   • CUse a simple 'contains' check for a keyword from the expected message.
   • DUse a hash of the expected message and compare it to the LLM output.
   Reveal Solution Discussion Select an answer first