AI and Machine Learning Fundamentals from an Offensive Security Perspective COASP Practice Questions (Page 1)

Questions 1–5

1. 1expert · hard

   A security architect is designing a defense for an AI system that uses a model trained on sensitive customer data. The architect must choose between two controls: (1) encrypting the training data at rest and in transit, and (2) implementing differential privacy during training. The primary threat is an attacker who gains access to the model's predictions and wants to infer information about individual customers. Which control is more effective against this threat?

   • AEncryption, as it protects the data from unauthorized access
   • BDifferential privacy, as it limits the information the model can reveal about training data
   • CNeither is effective against inference attacks
   • DBoth are equally effective
   Reveal Solution Discussion Select an answer first
2. 2application · medium

   A data scientist is building a malware detection model and wants to ensure that an attacker cannot corrupt the model's training data to make it classify malware as benign. Which defensive technique directly addresses this threat?

   • AInput perturbation at inference time
   • BRate limiting on the prediction API
   • CModel ensembling to improve accuracy
   • DData sanitization and outlier detection during training
   Reveal Solution Discussion Select an answer first
4. 3application · medium

   During a penetration test, an analyst has successfully gained access to a machine learning model's prediction API. The analyst now wants to systematically map the model's behavior to identify potential evasion opportunities. Which step of the standard hacking methodology does this activity align with?

   • AVulnerability analysis
   • BPost-exploitation
   • CReconnaissance and information gathering
   • DExploitation
   Reveal Solution Discussion Select an answer first
5. 4foundation · easy

   From an offensive security perspective, why are AI/ML systems considered a unique attack target compared to traditional software?

   • AThey do not require authentication, so anyone can access them.
   • BTheir behavior is determined by training data and learned parameters, which can be manipulated in ways that are not possible with hard-coded logic.
   • CThey are always deployed on cloud infrastructure, which is inherently more vulnerable.
   • DThey are written in Python, which is inherently insecure.
   Reveal Solution Discussion Select an answer first
6. 5foundation · easy

   In the context of offensive security, how does the standard penetration testing methodology (e.g., reconnaissance, scanning, exploitation) apply to AI/ML systems?

   • AIt applies only to the infrastructure layer, ignoring the model and data components.
   • BIt does not apply because AI/ML systems are not vulnerable to traditional attacks.
   • CIt applies with additional steps for AI-specific attacks, such as probing the model's behavior and testing for adversarial vulnerabilities.
   • DIt requires replacing all standard phases with entirely new AI-specific phases.
   Reveal Solution Discussion Select an answer first