Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A company is evaluating two LLM vendors for a test automation system. Vendor A offers a 'deterministic mode' that guarantees identical outputs for identical inputs, but it is more expensive. Vendor B does not offer this mode but allows setting temperature and seed. The team needs to decide which vendor to use. What is the most important consideration?

   • AThe speed of the API responses.
   • BThe ability to achieve consistent outputs with the required level of effort.
   • CThe cost difference between the vendors.
   • DThe vendor's customer support quality.
   Reveal Solution Discussion Select an answer first
2. 12foundation · easy

   How does non-deterministic LLM output primarily affect the reliability of automated software tests?

   • AIt reduces the accuracy of the LLM's training data
   • BIt increases the speed of test execution
   • CIt makes test results inconsistent across runs, leading to flaky tests
   • DIt causes the test environment to become unstable
   Reveal Solution Discussion Select an answer first
4. 13expert · hard

   An engineer is troubleshooting why an LLM-based test data generator produces different outputs even when the temperature is set to 0. The API documentation mentions that the model uses a stochastic sampling process that can be influenced by other parameters. Which factor could still cause non-determinism?

   • AThe prompt includes a timestamp that changes each run.
   • BThe random seed is not fixed, and the sampling process uses a random number generator.
   • CThe model version is pinned, but the underlying hardware changes.
   • DThe API automatically retries failed requests, causing duplicate calls.
   Reveal Solution Discussion Select an answer first
5. 14expert · hard

   A team has implemented several mitigations for LLM non-determinism: temperature 0, version pinning, and consensus voting. However, they still observe occasional failures in test assertions due to rare output variations. The team must decide whether to accept this residual risk. What is the most appropriate approach?

   • ARemove all LLM-based tests from the suite.
   • BIgnore the failures and hope they do not recur.
   • CIncrease the temperature to make the model more creative.
   • DDocument the residual risk and define acceptance criteria based on the failure rate.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   Which of the following is an example of post-processing normalization?

   • ADisabling the LLM's API access
   • BIncreasing the temperature to get more varied responses
   • CParsing the LLM output into a structured data format like JSON
   • DUsing a different prompt for each test run
   Reveal Solution Discussion Select an answer first