Scoping an AI Red Team Exercise and Threat Modeling COASP Practice Questions (Page 4)

Questions 16–20

1. 16expert · hard

   A security architect is threat modeling an AI system that uses a large language model (LLM) to generate code snippets for developers. The system is integrated with a corporate code repository and can push code changes. The architect must decide which threat modeling framework to use. The team is familiar with STRIDE but wants to ensure AI-specific threats are covered. What is the best approach?

   • AUse MITRE ATLAS as the primary framework, and supplement with STRIDE for traditional infrastructure threats.
   • BUse MITRE ATLAS only, because it is the only framework that can identify threats to AI systems.
   • CUse a custom framework based on the team's experience, since existing frameworks are too generic.
   • DUse STRIDE only, because it is a well-established framework and covers all types of threats.
   Reveal Solution Discussion Select an answer first
2. 17expert · hard

   A red team is scoping an exercise for an AI-powered autonomous drone navigation system. The system uses a reinforcement learning model trained in simulation. The team must decide whether to include the simulation environment in the scope. The simulation is used for both training and testing. The team has limited time. What is the most appropriate scope decision?

   • AInclude the simulation environment in scope, as it is a critical component for both training and testing the model.
   • BInclude only the deployed drone's onboard model, and exclude all development and testing environments.
   • CInclude the simulation, but only for testing the model's performance, not for security testing.
   • DExclude the simulation environment, as it is not part of the deployed system.
   Reveal Solution Discussion Select an answer first
4. 18expert · hard

   A company is red teaming an AI system that recommends content to users. The system uses a collaborative filtering model and a content-based model. The red team has a limited budget and must decide whether to include both models or focus on one. The content-based model is newer and less tested, while the collaborative filtering model is critical to revenue. The company's main concern is user trust. Which scope decision best balances risk and budget?

   • ATest only the collaborative filtering model, because it is critical to revenue and any vulnerability would have the highest impact.
   • BSkip both models and test the recommendation API infrastructure, since that is common to both.
   • CInclude both models but allocate more testing effort to the content-based model, since it is less tested and could have unknown vulnerabilities.
   • DTest only the content-based model, because it is newer and more likely to have vulnerabilities.
   Reveal Solution Discussion Select an answer first
5. 19expert · hard

   A healthcare organization is red teaming an AI system that predicts patient readmission risk. The system uses patient data and is subject to HIPAA. The red team wants to test the system's resilience to adversarial attacks, but the organization's legal team is concerned about privacy. The red team lead must align the exercise with both security and compliance goals. What is the best approach?

   • AConduct the exercise on the production system but only during off-peak hours to minimize impact.
   • BUse a synthetic dataset that mirrors the statistical properties of the real data, and conduct the exercise in a sandboxed environment that is not connected to the production network.
   • CUse a small sample of real patient data, but only for the red team, and ensure it is deleted after the exercise.
   • DSkip the adversarial testing and focus only on compliance audits, since privacy is more important.
   Reveal Solution Discussion Select an answer first
6. 20application · medium

   A healthcare startup is deploying an AI-based diagnostic assistant that processes patient images and provides preliminary findings to clinicians. The compliance team requires that the red team exercise not access live patient data, and the business wants to test the system's resilience against adversarial image perturbations. What should be the primary scope boundary for this exercise?

   • ALimit the exercise to the model's API endpoint only, without testing the image preprocessing steps, to avoid touching data pipelines.
   • BTest only the model's output layer by sending random noise to the API, since the goal is to see if the model misclassifies.
   • CInclude the full production environment with live patient data to ensure realistic testing, since the model is already deployed.
   • DUse only synthetic or de-identified images that mimic the distribution of real patient data, and focus on the model's image preprocessing and inference pipeline.
   Reveal Solution Discussion Select an answer first