AI Security Testing, Evaluation, and Hardening COASP Practice Questions (Page 2)

Questions 6–10

1. 6expert · hard

   A team is evaluating a model for detecting phishing URLs. The model has high accuracy on known phishing sites but fails on newly crafted ones. The team has two options: (1) increase the model's sensitivity to catch more phishing sites, or (2) add a post-processing filter that checks URLs against a blocklist. The team wants to minimize false positives while improving detection of new threats. Which approach is more effective?

   • ARetrain the model with more phishing examples, which is the only way to improve detection
   • BIncrease the model's sensitivity, which will catch more phishing but also increase false positives
   • CUse both approaches simultaneously, accepting the higher false-positive rate
   • DAdd a blocklist filter, which can be updated with new threats without affecting the model's false-positive rate
   Reveal Solution Discussion Select an answer first
2. 7application · medium

   A team is comparing two versions of a malware-detection model. Version 1 has higher accuracy on clean files but is vulnerable to evasion. Version 2 has slightly lower clean accuracy but is more robust to evasion. The deployment environment is a high-risk network where attackers frequently use obfuscation. Which metric should the team prioritize in their evaluation?

   • ARobustness accuracy, which measures performance on adversarial samples
   • BTraining time, since faster training is always better
   • CModel size, since smaller models are easier to deploy
   • DClean accuracy, since it reflects the model's performance on normal files
   Reveal Solution Discussion Select an answer first
4. 8application · medium

   A company is required by industry standards to regularly test its AI models for adversarial robustness. The team has completed the first round of testing and found several vulnerabilities. What is the NEXT step to align with compliance best practices?

   • ADocument the findings and create a remediation plan with timelines
   • BImmediately deploy the model to production since testing is complete
   • CDelete the test results to avoid regulatory scrutiny
   • DWait for the next audit before taking any action
   Reveal Solution Discussion Select an answer first
5. 9application · medium

   A security team is evaluating two different AI models for a spam-detection system. Model A has a higher clean accuracy but is vulnerable to adversarial perturbations, while Model B has slightly lower clean accuracy but is more robust. The team needs to choose a model for deployment, prioritizing security. Which evaluation approach should the team use to make the decision?

   • ACompare the models' attack success rates (ASR) under the same adversarial attack, and choose the model with the lower ASR.
   • BSelect the model with the lower false positive rate on a standard spam dataset, as this is the most important metric for spam detection.
   • CChoose Model B because it is more robust, regardless of its clean accuracy, since security is the top priority.
   • DChoose Model A because its higher clean accuracy indicates it will perform better in real-world scenarios.
   Reveal Solution Discussion Select an answer first
6. 10application · medium

   A retail company is deploying an AI model to recommend products to customers. The security team wants to conduct a security assessment of the model before launch, but they have limited time and need to focus on the most critical vulnerabilities. The model is trained on customer purchase history and is accessible via a public API. Which testing activity should the team prioritize?

   • ATest the model's recommendation accuracy on a standard benchmark dataset to ensure it provides relevant recommendations.
   • BTest the model's integration with the company's customer database to ensure it can retrieve and process data correctly.
   • CTest the model for model extraction attacks, where an attacker could query the API to reconstruct the model's training data or decision boundaries.
   • DTest the model's response time under high load to ensure it can handle a large number of API requests.
   Reveal Solution Discussion Select an answer first