Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 7)

Questions 31–35

1. 31expert · hard

   A team is using an LLM to generate test oracles for a financial application. The LLM sometimes produces outputs that are semantically correct but phrased differently, causing the test assertions to fail. The team has tried temperature 0 and version pinning, but the issue persists. What is the most effective additional mitigation?

   • AIncrease the temperature to get more diverse outputs.
   • BUse exact string matching on the LLM output.
   • CSwitch to a different LLM that is known to be deterministic.
   • DImplement a consensus mechanism that requires multiple LLM calls to agree on the output.
   Reveal Solution Discussion Select an answer first
2. 32expert · hard

   A company's test automation uses an LLM API. The vendor announces a new model version with improved accuracy, but the team is concerned about changes in output format. They want to adopt the new version while minimizing risk. What is the best approach?

   • AKeep the old version and never upgrade.
   • BImmediately switch to the new version in production to get the accuracy benefits.
   • CTest the new model version in a staging environment and compare outputs with the current version.
   • DUse the new version only for non-critical tests.
   Reveal Solution Discussion Select an answer first
4. 33expert · hard

   An LLM is used to generate test step descriptions from user stories. The descriptions are then compared to a set of expected steps. The team has implemented normalization (lowercasing, trimming, removing punctuation) but still sees failures because the LLM sometimes uses synonyms (e.g., 'click' vs 'press'). What is the best next step?

   • AIncrease the temperature to get more consistent outputs.
   • BRemove the LLM and use a manual process.
   • CUse exact string matching after normalization.
   • DUse a synonym dictionary or semantic similarity to map variations to a canonical form.
   Reveal Solution Discussion Select an answer first
5. 34expert · hard

   A team is monitoring an LLM-based test oracle in production. They have set up logging for inputs and outputs, but they are unsure how to detect non-deterministic behavior. What metric would be most useful?

   • AThe average response time of the API.
   • BThe error rate of the API.
   • CThe number of API calls made per day.
   • DThe standard deviation of output similarity scores for identical inputs.
   Reveal Solution Discussion Select an answer first
6. 35foundation · easy

   What is the goal of post-processing normalization of LLM outputs?

   • ATo increase the length of the generated response
   • BTo improve the model's training data quality
   • CTo convert the output into a standardized format that reduces variability
   • DTo make the output more creative and diverse
   Reveal Solution Discussion Select an answer first