AI System Hacking Methodologies, Frameworks, and Risk Implications COASP Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A security consultant is assessing an AI-based spam filter. The consultant has completed the reconnaissance phase and is now in the weaponization phase. The consultant has identified that the model is a natural language processing (NLP) model and that the system accepts email text as input. The consultant wants to craft an attack that will cause the model to classify a phishing email as legitimate. Which approach is MOST appropriate for the weaponization phase?

   • APoison the training data by injecting phishing emails labeled as legitimate.
   • BPerform model inversion to reconstruct the training data and find a phishing email that is misclassified.
   • CDirectly send the phishing email to the spam filter to see if it is classified as legitimate.
   • DCreate a phishing email with carefully crafted language that is designed to bypass the model's classification.
   Reveal Solution Discussion Select an answer first
2. 2expert · hard

   A company uses an AI model for credit scoring. They have discovered that an attacker has been querying the model with carefully crafted inputs to infer the decision boundaries and steal the model. The company must decide how to respond while minimizing business disruption. Which response best balances security and business continuity?

   • AReplace the model with a completely new model trained on different data.
   • BImplement rate limiting and anomaly detection on API queries, then retrain the model with additional defensive techniques such as differential privacy.
   • CImmediately shut down the API and take the model offline for a full forensic investigation.
   • DIgnore the attack because model theft does not cause direct financial loss.
   Reveal Solution Discussion Select an answer first
4. 3expert · hard

   A security analyst is investigating a suspected attack on an AI-based medical diagnosis system. The system has been misdiagnosing certain conditions. The analyst must determine whether the cause is data poisoning, model drift, or an adversarial attack. Which approach is most effective in distinguishing between these causes?

   • AAnalyze the model's performance on a validation dataset over time and compare it with the training data's integrity.
   • BCheck the system's logs for unusual access patterns to determine if an attacker is actively querying the system.
   • CConsult with the medical staff to see if the misdiagnoses are due to changes in patient demographics.
   • DImmediately retrain the model on new data and assume it was data poisoning.
   Reveal Solution Discussion Select an answer first
5. 4expert · hard

   A company's AI-powered customer service chatbot is found to be leaking sensitive customer information through prompt injection attacks. The company must decide on a mitigation strategy that balances security with user experience. Which approach is most effective?

   • AImplement input filtering and output sanitization to block malicious prompts and prevent sensitive data leakage.
   • BOnly allow the chatbot to respond to predefined questions and block all other inputs.
   • CAdd a disclaimer to the chatbot's responses stating that the information may be inaccurate.
   • DDisable the chatbot entirely and replace it with human agents.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   Which of the following is a potential business risk resulting from a successful data poisoning attack on an AI system?

   • AThe AI model may make incorrect decisions, leading to financial losses or reputational damage
   • BThe AI model's inference speed increases, causing performance degradation
   • CThe AI model's source code becomes publicly available, violating intellectual property rights
   • DThe AI model's training data is encrypted by ransomware, halting operations
   Reveal Solution Discussion Select an answer first