Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A company's test automation relies on an LLM API. The vendor recently updated the model, and the test outputs changed significantly, causing many tests to fail. The team wants to prevent this from happening again. What is the best action?

   • APin the model version in the API request and test before upgrading.
   • BIncrease the temperature to make the model more flexible.
   • CAutomatically use the latest model version every time.
   • DStop using the LLM API and switch to a rule-based system.
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A QA team is integrating an LLM into their regression test suite. The LLM generates expected outcomes for test cases. The team notices that the same test sometimes passes and sometimes fails because the LLM's output wording changes slightly. What is the most appropriate way to handle this in the test assertions?

   • AReplace the LLM with a rule-based system for all test cases.
   • BUse semantic similarity or regex-based assertions that tolerate wording variations.
   • CRun the test only once and ignore failures.
   • DUse exact string matching on the LLM output.
   Reveal Solution Discussion Select an answer first
4. 3foundation · easy

   What is the purpose of evaluating residual risk after applying mitigation techniques for LLM non-determinism?

   • ATo determine if the remaining variability is acceptable for the testing context
   • BTo increase the temperature to get more diverse outputs
   • CTo eliminate all non-determinism from the LLM
   • DTo switch to a different LLM provider
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   What is the purpose of using deterministic modes or API parameters provided by LLM vendors?

   • ATo ensure consistent outputs for the same input across calls
   • BTo reduce the cost of API usage
   • CTo improve the model's accuracy on complex tasks
   • DTo increase the speed of model inference
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A test automation script uses an LLM to generate a summary of test results. The summary is checked to ensure it contains the key metrics (pass count, fail count, and duration). The LLM sometimes phrases the summary differently, but the metrics are always present. What is the most robust way to assert the correctness of the summary?

   • AUse a simple 'contains' check for the word 'Pass'.
   • BCheck that the summary contains the exact strings 'Pass: 10', 'Fail: 2', and 'Duration: 30s'.
   • CCompare the entire summary to a previously stored summary using a hash.
   • DParse the summary to extract numbers and verify that the values match the expected metrics.
   Reveal Solution Discussion Select an answer first