Challenges Testing Complex AI-Based Systems CT-AI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A team is testing a recommendation system that updates its model daily based on new user interactions. A tester reports a failure where the system recommended inappropriate content to a user. When the tester tries to reproduce the issue the next day, the system behaves correctly. What is the most likely reason the failure cannot be reproduced?

   • AThe system is non-deterministic, so outputs vary even with the same input.
   • BThe failure was caused by a temporary network issue.
   • CThe tester did not use the exact same input data.
   • DThe system's continuous learning changed the model, so the original conditions no longer exist.
   Reveal Solution Discussion Select an answer first
2. 2foundation · easy

   Which of the following is a challenge in testing AI-based systems that is directly related to the lack of formal specifications?

   • AThe need to test the system's graphical user interface.
   • BThe requirement to use a specific programming language for test scripts.
   • CThe test oracle problem, where it is unclear what the correct output should be.
   • DThe system's inability to handle large volumes of data.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A team is testing an AI system that generates personalized learning paths for students. The system's output is a sequence of learning activities, and there is no single 'correct' path. The team wants to validate that the paths are effective in improving student outcomes. Which approach is most appropriate for defining a test oracle?

   • AUse a panel of educational experts to rate the quality of the learning paths based on established pedagogical criteria.
   • BUse a simple rule-based system to check that the paths follow a fixed sequence of topics.
   • CMeasure student engagement metrics and assume higher engagement means better learning outcomes.
   • DCompare the generated paths to a set of pre-defined optimal paths created by a single expert.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A team is testing an AI system that recommends job candidates to recruiters. The system was trained on historical hiring data that contains biases against certain groups. The team is now evaluating the system's recommendations for fairness. What is the most important consideration when interpreting the test results?

   • AThe test results are valid because the system was trained on historical data.
   • BThe test results should be ignored because fairness is subjective.
   • CThe test results may reflect the biases in the training data, so the system may not be fair even if it performs well on overall accuracy.
   • DThe test results are invalid because the system is non-deterministic.
   Reveal Solution Discussion Select an answer first
6. 5application · medium

   A team is testing a credit scoring model. The training data contains a significant amount of missing income information for certain demographic groups. The team is now evaluating the model's fairness across different groups. What is the most likely impact of the missing data on the test results?

   • AThe test results will be more accurate because the model has less noise to deal with.
   • BThe test results will be invalid because the data is not normally distributed.
   • CThe model may perform differently for groups with missing data, leading to biased test outcomes.
   • DThe test results will be unaffected because the model can handle missing values.
   Reveal Solution Discussion Select an answer first