AI Security Testing, Evaluation, and Hardening COASP Practice Questions (Page 8)

Questions 36–40

1. 36application · medium

   A startup is developing an AI-powered chatbot for customer service. The team wants to conduct a security assessment of the chatbot before launch, but they have limited resources and need to focus on the most critical vulnerabilities. The chatbot processes user inputs and generates responses, and it is connected to a backend database. Which testing activity should the team prioritize?

   • ATest the chatbot's ability to handle a high volume of concurrent users to ensure it can scale during peak demand.
   • BTest the chatbot for prompt injection attacks that could manipulate the model into revealing sensitive information or performing unauthorized actions.
   • CTest the chatbot's integration with the backend database to ensure it can retrieve and store information correctly.
   • DTest the chatbot's response accuracy on a standard customer service benchmark to ensure it provides correct answers.
   Reveal Solution Discussion Select an answer first
2. 37application · medium

   A cybersecurity consulting firm has been hired to perform an AI security assessment for a client's autonomous vehicle perception system. The client wants to understand the system's vulnerabilities to adversarial attacks before deployment. The consulting team has access to the model's architecture and training data, but the client has restricted the team from running attacks that could damage the production system. Which testing approach should the team use to balance thoroughness with safety?

   • APerform a static code review of the model's implementation to identify potential vulnerabilities without executing any attacks.
   • BUse a third-party AI security testing service that guarantees no harm to the production system, without specifying the testing methodology.
   • CRun live adversarial attacks against the production system to observe real-world behavior, but only during low-traffic periods.
   • DConduct offline adversarial testing using a replica of the model in a sandboxed environment, simulating attacks such as evasion and poisoning.
   Reveal Solution Discussion Select an answer first
4. 38application · medium

   A financial institution has an AI model that approves or denies loan applications. The model has been found to be biased against a specific demographic group, leading to regulatory scrutiny. The institution needs to respond to this incident and ensure compliance with fair lending laws. Which action should the institution take first?

   • ADisable the AI model and process all loan applications manually to avoid further regulatory issues.
   • BImmediately retrain the model on a new dataset that is balanced across all demographic groups.
   • CConduct a thorough investigation to determine the root cause of the bias, including reviewing the training data and model evaluation metrics.
   • DNotify the regulatory authority about the bias incident immediately, without conducting an internal investigation first.
   Reveal Solution Discussion Select an answer first
5. 39application · medium

   A research team is developing a new AI model for detecting malware in network traffic. They want to integrate AI security testing into their development pipeline to catch vulnerabilities early. The team is looking for a tool that can automate the generation of adversarial examples and evaluate the model's robustness. Which tool or framework would be most suitable for this purpose?

   • AUse a cloud-based AI service that provides automated security testing, but without specifying the underlying methodology.
   • BUse a standard unit testing framework like pytest to write test cases for the model's expected outputs.
   • CUse the Adversarial Robustness Toolbox (ART) to generate adversarial examples and evaluate the model's robustness against various attack types.
   • DUse a general-purpose fuzzing tool like AFL to generate random inputs and test the model's behavior on unexpected data.
   Reveal Solution Discussion Select an answer first
6. 40application · medium

   A team is planning an AI security test for a recommendation system that uses collaborative filtering. They want to identify whether an attacker could influence the recommendations by injecting fake user profiles. Which testing methodology should they use?

   • AModel inversion attack to infer user preferences from outputs
   • BEvasion attack simulation that modifies input queries to the model
   • CExtraction attack simulation to reconstruct the training data
   • DPoisoning attack simulation that injects malicious data into the training set
   Reveal Solution Discussion Select an answer first