LLMOps When Deploying and Managing LLMs for Software Testing CT-GENAI Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   An LLM-based test oracle is used to classify UI error messages as 'critical' or 'non-critical'. The team notices that the distribution of classifications has shifted over the past month, but the model's accuracy on a fixed validation set remains unchanged. What should they do first?

   • AIncrease the model's temperature to add randomness and compensate for the shift.
   • BRoll back to the previous model version.
   • CInvestigate the input data distribution and the model's predictions to determine if the shift is due to data drift or model drift.
   • DRetrain the model on the most recent data.
   Reveal Solution Discussion Select an answer first
2. 22foundation · easy

   What is a cost consideration when choosing between a serverless LLM API and a dedicated on-premise deployment for testing?

   • AOn-premise deployments are always cheaper than serverless APIs
   • BServerless APIs charge per request, while on-premise has fixed infrastructure costs
   • CServerless APIs have no cost when idle
   • DOn-premise deployments have no ongoing costs
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   What is the primary purpose of logging LLM inference requests in a test environment?

   • ATo increase the speed of test execution
   • BTo reduce the cost of LLM usage
   • CTo automatically generate new test cases
   • DTo provide an audit trail for debugging and compliance
   Reveal Solution Discussion Select an answer first
5. 24foundation · easy

   What is a common scaling strategy for LLM inference in test environments with variable demand?

   • AManual scaling based on peak load estimates
   • BUsing only CPU-based inference
   • CAutoscaling based on request volume or latency
   • DDisabling scaling to avoid complexity
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A team runs LLM-based test generation as part of their CI/CD pipeline. The workload is bursty: there are long periods of no activity, but when a build is triggered, hundreds of test cases need to be generated quickly. Which cost-optimization strategy is most appropriate?

   • AUse a fixed-size VM that can handle the peak load.
   • BUse a serverless inference service that scales up on demand and scales down when idle.
   • CRun the model on a low-cost CPU-only instance and accept slower generation.
   • DProvision a dedicated GPU cluster that is always on.
   Reveal Solution Discussion Select an answer first