Mitigation of Non-Deterministic Behavior of LLMs CT-GENAI Practice Questions (Page 6)

Questions 26–30

1. 26expert · hard

   A test automation team is using an LLM API that supports a 'seed' parameter. They have set temperature to 0 and fixed the seed, but they still see occasional differences in output. The API documentation states that the seed only affects the sampling process, not the model's internal operations. What could be the cause?

   • AThe API uses a different random number generator on each call.
   • BThe model version is not pinned, and the vendor has updated the model in the background.
   • CThe seed parameter is ignored when temperature is 0.
   • DThe prompt includes a random element that changes each time.
   Reveal Solution Discussion Select an answer first
2. 27application · easy

   A company uses an LLM vendor's API for generating test summaries. The vendor recently introduced a 'deterministic mode' that guarantees identical outputs for identical inputs. The QA team wants to adopt this mode to improve test repeatability. What should they do first?

   • AUse a random seed in the API call to ensure variety.
   • BIncrease the timeout for API calls to allow more processing time.
   • CSwitch to a different LLM vendor that offers a free tier.
   • DUpdate the API call to include the deterministic mode parameter.
   Reveal Solution Discussion Select an answer first
4. 28foundation · easy

   How does a retry mechanism help handle non-deterministic LLM failures?

   • AIt re-runs the request and checks if the new output meets the expected criteria
   • BIt fixes the random seed to ensure the same output
   • CIt increases the temperature to get more varied outputs
   • DIt switches to a different model version
   Reveal Solution Discussion Select an answer first
5. 29application · medium

   A team uses an LLM to classify test failure logs into categories (e.g., 'network', 'assertion', 'timeout'). The classification is sometimes inconsistent, leading to flaky test reports. The team wants to improve the reliability of the classification without changing the model. Which approach is most effective?

   • AUse the first classification result and ignore the rest.
   • BIncrease the temperature to get more diverse classifications.
   • CRemove the LLM and use a random classifier.
   • DRun the classification three times and use the majority result.
   Reveal Solution Discussion Select an answer first
6. 30foundation · easy

   Which technique is commonly used to reduce output variability in LLMs?

   • AIncreasing the temperature to 1.5
   • BRandomly shuffling the prompt tokens
   • CUsing greedy decoding
   • DIncreasing the max token limit
   Reveal Solution Discussion Select an answer first