AI System Hacking Methodologies, Frameworks, and Risk Implications COASP Practice Questions (Page 3)

Questions 11–15

1. 11expert · hard

   A security team is conducting an offensive assessment of an AI system that detects fraudulent transactions. The team has limited time and budget. They need to identify the most critical attack surface and prioritize their testing. They have access to the system's API documentation and a few legitimate transaction samples. Which approach best balances thoroughness with resource constraints?

   • AFocus solely on traditional web application vulnerabilities in the API, such as SQL injection and XSS, ignoring AI-specific attacks.
   • BPerform black-box testing by sending crafted transactions to the API and observing responses, then use the results to infer model behavior and craft targeted adversarial examples.
   • CAttempt to extract the model's training data through model inversion attacks, then use the data to retrain a surrogate model.
   • DRequest full access to the model's training data and internal infrastructure to perform white-box testing.
   Reveal Solution Discussion Select an answer first
2. 12application · medium

   A red team is assessing an AI-powered email filtering system. During the reconnaissance phase, they discover that the model is trained on a public dataset of emails and that the system has an API that allows users to submit feedback. The red team wants to identify potential attack vectors. Which finding is MOST relevant to the weaponization phase?

   • AThe model is hosted on a cloud provider that has had past security breaches.
   • BThe public dataset contains emails from various sources, including some that are not relevant to the organization.
   • CThe API rate limit is set to 100 requests per minute.
   • DThe feedback API can be used to submit crafted emails that may influence the model's future updates.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A penetration tester is conducting an offensive assessment of an AI-powered spam filter. The tester wants to craft an email that bypasses the filter while remaining readable to humans. According to the AI hacking methodology, which phase does this activity fall under?

   • ACommand and control
   • BWeaponization
   • CExploitation
   • DReconnaissance
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   A security team is using MITRE ATLAS to evaluate the risk of an attack on their AI-based hiring tool. They identify a technique that involves an attacker submitting a specially crafted resume that causes the model to rank an unqualified candidate as the top choice. Which MITRE ATLAS tactic does this technique fall under?

   • AExecution, because the attacker is causing the model to execute a malicious action.
   • BInitial Access, because the attacker is submitting a resume to the system.
   • CML Attack Staging, because the attacker is preparing the malicious resume.
   • DImpact, because the attacker is manipulating the model's output to achieve a negative outcome.
   Reveal Solution Discussion Select an answer first
6. 15foundation · easy

   In the systematic methodology for attacking AI systems, which activity is typically performed during the reconnaissance phase?

   • ADeploying a backdoor trigger that causes misclassification during inference
   • BCrafting adversarial examples that will be delivered to the target model
   • CExecuting a data poisoning attack by injecting malicious samples into the training pipeline
   • DCollecting information about the target model's architecture, training data, and deployed environment
   Reveal Solution Discussion Select an answer first