AI Security Testing, Evaluation, and Hardening COASP Practice Questions (Page 5)

Questions 21–25

1. 21application · medium

   A financial services company has deployed a natural language processing (NLP) model to screen customer support tickets for potential fraud. The model was recently found to be vulnerable to adversarial text perturbations that cause it to misclassify fraudulent requests as legitimate. The team needs to harden the model without significantly degrading its performance on clean inputs. Which combination of defensive measures would be most appropriate to implement first?

   • AApply adversarial training by augmenting the training set with adversarial examples generated from the current model, then retrain the model.
   • BImplement a rule-based filter that blocks any ticket containing words that were used in known adversarial examples.
   • CSwitch to a larger, more complex model architecture to increase the model's capacity and make it inherently more robust to attacks.
   • DDeploy an input sanitization layer that removes all punctuation and converts text to lowercase before feeding it to the model.
   Reveal Solution Discussion Select an answer first
2. 22application · medium

   A security engineer is testing a production image-classification model that powers a retail product-search feature. The team wants to identify whether the model is vulnerable to adversarial examples that could cause misclassification of product photos. The engineer has access to the model's training data and can query the model API, but does not have access to the model's internal weights. Which approach would most effectively assess the model's vulnerability to evasion attacks under these constraints?

   • AUse a black-box attack library like Foolbox to generate transferable adversarial examples using a substitute model trained on the same data, then test them against the live API.
   • BPerform a data-poisoning attack by injecting malicious samples into the training set to corrupt the model's learned features before deployment.
   • CRun a white-box gradient-based attack such as Projected Gradient Descent (PGD) directly against the production model's loss function to craft optimal perturbations.
   • DEvaluate the model's robustness by measuring its clean accuracy on a held-out validation set and comparing it to the training accuracy to detect overfitting.
   Reveal Solution Discussion Select an answer first
4. 23foundation · easy

   What is the purpose of input sanitization in AI system hardening?

   • ATo reduce the model's inference latency
   • BTo remove or neutralize malicious or malformed inputs before they reach the model
   • CTo increase the model's accuracy on clean data
   • DTo encrypt the model's parameters
   Reveal Solution Discussion Select an answer first
5. 24expert · hard

   A hospital's diagnostic AI model suddenly begins misclassifying chest X-rays. Logs show a spike in unusual queries from a single IP address just before the degradation. The model is served via an API, and the training pipeline is separate. The incident response team must contain the threat while keeping the service available for urgent clinical use. Which action should be taken FIRST?

   • AShut down the API and retrain the model on a sanitized dataset
   • BBlock the suspicious IP address at the API gateway and roll back to the last known-good model version
   • CDisconnect the training pipeline and quarantine all recent training data
   • DIncrease logging verbosity and continue monitoring to gather more evidence
   Reveal Solution Discussion Select an answer first
6. 25foundation · easy

   What is the primary goal of AI security testing?

   • ATo verify that the model complies with all relevant data protection regulations
   • BTo ensure the AI model achieves high accuracy on the training dataset
   • CTo benchmark the model's performance against other models on standard datasets
   • DTo identify and mitigate vulnerabilities that could be exploited by adversaries to compromise the model's integrity, availability, or confidentiality
   Reveal Solution Discussion Select an answer first