AI and Machine Learning Fundamentals from an Offensive Security Perspective COASP Practice Questions (Page 6)

Questions 26–30

1. 26application · medium

   A security engineer at a financial firm is testing the robustness of a fraud-detection model that classifies transactions as legitimate or fraudulent. The engineer wants to evaluate how the model responds to inputs that are intentionally perturbed to mimic a fraudster's attempt to avoid detection. The engineer has access to the model's training data and can query the model as a black box. Which approach best simulates a realistic evasion attempt against this model?

   • AExtract the model's decision boundary by querying it with many random inputs and then craft a single universal perturbation that flips all predictions.
   • BGenerate adversarial examples using the Fast Gradient Sign Method (FGSM) on the model's loss function, then test the model's predictions on those examples.
   • CReplace the model's softmax layer with a linear layer and retrain the final layer on the original training data.
   • DInject a small number of mislabeled transactions into the training set and retrain the model to see if accuracy drops.
   Reveal Solution Discussion Select an answer first
2. 27foundation · easy

   What is the role of a 'model' in a machine learning system?

   • AIt is the raw dataset used to train the algorithm.
   • BIt is the evaluation metric used to measure accuracy.
   • CIt is the hardware that runs the training process.
   • DIt is the algorithm that processes input data to produce predictions or decisions.
   Reveal Solution Discussion Select an answer first
4. 28foundation · easy

   In mapping the attack surface of an AI system, which layer is primarily responsible for serving predictions to end users and is often exposed via an API?

   • ATraining data storage
   • BFeature extraction module
   • CInference endpoint
   • DModel training cluster
   Reveal Solution Discussion Select an answer first
5. 29expert · hard

   A red team is assessing a company's AI-powered recommendation system. The system uses a deep learning model trained on user behavior data, and the model is served via a public API. The red team has already identified that the model is a black box, but they have access to the API and can send unlimited queries. The red team's objective is to create a surrogate model that mimics the original model's behavior, which they plan to use to craft adversarial examples. The team is also concerned about being detected by the company's monitoring system, which tracks API usage patterns. Which approach best balances the need to create a surrogate model while minimizing the risk of detection?

   • AUse a pre-trained public model as a surrogate without querying the target API, assuming it is similar enough.
   • BDistribute queries over a long period with random intervals, mimicking normal user traffic, and use the responses to train the surrogate model.
   • CExploit the API's error messages to infer the model's architecture, then build a surrogate from that information.
   • DSend a large number of queries in a short burst to quickly gather data, then train the surrogate model offline.
   Reveal Solution Discussion Select an answer first
6. 30application · medium

   A red team is assessing an organization's AI-based fraud detection system. The team has identified that the model's training data is stored in an unencrypted cloud bucket and that the model is served via an API with no rate limiting. Which attack surface component is the red team most directly exploiting by accessing the training data?

   • AInfrastructure component
   • BInference component
   • CData component
   • DModel component
   Reveal Solution Discussion Select an answer first