Testing Autonomous AI-Based Systems CT-AI Practice Questions (Page 1)

Questions 1–5

1. 1application · medium

   A team is developing an autonomous underwater vehicle (AUV) for ocean exploration. The AUV's navigation system uses a reinforcement learning model trained in simulation. The team wants to validate the model's performance in a variety of ocean conditions before a real deployment. The AUV is expensive and cannot be easily recovered if it malfunctions. What is the most effective way to test the AUV's navigation in diverse ocean conditions?

   • ADeploy the AUV in the ocean and monitor its performance in real-time, aborting the mission if it deviates from the expected path.
   • BUse a high-fidelity simulation that models ocean currents, visibility, and sensor noise to test the navigation model across thousands of virtual scenarios.
   • CPerform formal verification of the reinforcement learning model to prove it will navigate correctly in all ocean conditions.
   • DTest the AUV in a controlled indoor pool to validate the navigation model in a safe environment.
   Reveal Solution Discussion Select an answer first
2. 2foundation · easy

   How do regulatory requirements impact the testing of autonomous AI systems?

   • AThey may mandate specific testing procedures and documentation
   • BThey only apply to systems that are fully autonomous
   • CThey have no impact on testing because AI systems are not regulated
   • DThey require that all testing be done in real-world conditions
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   An autonomous shuttle is being developed for a university campus. The shuttle's perception system uses a deep learning model to detect pedestrians. The team has two testing objectives: (1) ensure the shuttle never collides with a pedestrian, and (2) ensure the shuttle provides a comfortable ride for passengers. The deep learning model is non-deterministic. Which combination of verification and validation approaches is most appropriate?

   • AUse formal verification to prove passenger comfort, and use simulation to demonstrate pedestrian avoidance.
   • BUse extensive simulation to demonstrate that the shuttle avoids pedestrians in a wide range of scenarios, and use real-world testing with a safety driver to evaluate passenger comfort.
   • CUse formal verification to prove the shuttle will never collide with a pedestrian, and use simulation to evaluate passenger comfort.
   • DUse only real-world testing for both objectives, since simulation cannot accurately model pedestrian behavior or passenger comfort.
   Reveal Solution Discussion Select an answer first
5. 4application · medium

   A company is developing an autonomous warehouse robot that navigates between shelves. The robot operates at SAE Level 3 autonomy: it can handle standard routes independently but requires human intervention when it encounters unexpected obstacles. The team wants to test the robot's response to a rare edge case: a worker suddenly stepping into its path. The test must be repeatable and must not endanger warehouse staff. Which testing approach best satisfies these requirements?

   • AUse operational monitoring data from the robot's daily runs and retrospectively analyze whether any worker-stepping events occurred naturally.
   • BPerform a formal verification of the robot's obstacle-avoidance algorithm to mathematically prove it will stop for any worker stepping into its path.
   • CRun a field trial in the active warehouse during off-hours, repeating the worker-stepping scenario 50 times with a safety operator present.
   • DCreate a high-fidelity simulation of the warehouse and inject the worker-stepping scenario as a scripted event, running it 50 times with identical initial conditions.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   What is a key challenge when testing autonomous AI-based systems due to non-determinism?

   • AThe system may produce different outputs for the same input, making it difficult to reproduce and debug failures
   • BThe system's performance cannot be measured
   • CThe system cannot be tested in a simulated environment
   • DThe system requires a large amount of labeled data for testing
   Reveal Solution Discussion Select an answer first