Scoping an AI Red Team Exercise and Threat Modeling COASP Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A financial firm is red teaming an AI-based fraud detection system. The system uses a trained model served via a REST API, with features computed from transaction data stored in a data lake. The team has limited time and must prioritize. Which attack surface should be considered the highest priority for a realistic external attacker?

   • AThe REST API endpoint, because it is the only externally reachable component and can be probed for adversarial inputs and API-level vulnerabilities.
   • BThe model training pipeline, because poisoning the training data would have the broadest impact on the model's behavior.
   • CThe data lake, because it stores all the training and inference data and is a high-value target for data exfiltration.
   • DThe feature computation logic, because it is custom code and likely contains bugs that could be exploited.
   Reveal Solution Discussion Select an answer first
2. 22expert · hard

   A red team is scoping an exercise for an AI-based credit scoring system. The system is used by a bank and is subject to fair lending regulations. The team has identified two high-risk threats: (1) an attacker could manipulate input data to get a loan approved fraudulently, and (2) the model could exhibit bias against a protected class, leading to regulatory penalties. The team has limited time and must choose one threat to test. Which threat should be prioritized?

   • ANeither threat, because the red team should focus on technical vulnerabilities, not business risks.
   • BBoth threats equally, because they are both high risk.
   • CThe fraud threat, because it directly causes financial loss to the bank.
   • DThe bias threat, because regulatory penalties and reputational damage could be severe, and the bank has a compliance obligation to address it.
   Reveal Solution Discussion Select an answer first
4. 23expert · hard

   A red team is prioritizing threats for an AI-powered medical diagnosis system. The team has identified the following threats: (1) an attacker crafts adversarial images to cause misdiagnosis, (2) an insider steals patient data from the training set, (3) the model is unavailable due to a DDoS attack on the API. The hospital's top priority is patient safety. Which threat should be ranked highest?

   • AAll threats should be ranked equally, as they all have significant impact.
   • BDDoS attack on the API, because it makes the system unavailable.
   • CAdversarial images causing misdiagnosis, because it directly affects patient safety.
   • DInsider theft of patient data, because it violates privacy regulations.
   Reveal Solution Discussion Select an answer first
5. 24application · medium

   A retail company is red teaming an AI chatbot that handles customer orders. The company's business goal is to reduce customer service costs, but the security team is concerned about prompt injection leading to unauthorized actions. The red team lead must align the exercise with business goals. Which approach best achieves this?

   • AFocus the exercise on prompt injection attacks that could cause the chatbot to perform unauthorized actions, and measure the potential financial impact of such attacks.
   • BTest the chatbot's response accuracy on a wide range of customer queries, since that is what affects customer satisfaction.
   • CRun a generic penetration test on the chatbot's infrastructure without considering AI-specific threats.
   • DExclude prompt injection from the scope because it is a known issue and the company wants to focus on cost savings.
   Reveal Solution Discussion Select an answer first
6. 25application · medium

   A red team is assessing an AI-powered autonomous vehicle perception system. They have identified the following threats: (A) adversarial stickers on stop signs causing misclassification, (B) GPS spoofing to alter the vehicle's location, (C) a software bug in the UI that shows incorrect speed, (D) physical tampering with the camera lens. The team must prioritize based on risk. Which threat should be ranked highest?

   • APhysical tampering with the camera lens, because it is simple and requires no technical skill.
   • BGPS spoofing, because it is easy to execute and affects navigation.
   • CAdversarial stickers on stop signs, because they directly target the AI perception system and could cause a critical safety failure.
   • DThe UI bug, because it is easy to fix and could confuse the driver.
   Reveal Solution Discussion Select an answer first