Using Automation and AI for Detection Engineering and Incident Response GASAE Practice Questions (Page 3)

Questions 11–15

1. 11application · medium

   A SOC is evaluating an AI model that will be used to prioritize alerts. The model was trained on a dataset where 95% of the alerts were from a single, noisy detection rule. The team is concerned that the model will be biased toward this rule. What is the most effective way to address this bias during the model's evaluation?

   • AEvaluate the model's performance separately on alerts from the noisy rule and on alerts from all other rules.
   • BAsk the analysts to manually review a sample of the model's predictions to see if they agree with the priorities.
   • CRemove all alerts from the noisy rule from the training dataset and retrain the model.
   • DEnsure the model is evaluated on a random sample of all alerts, which will naturally include a proportional number from the noisy rule.
   Reveal Solution Discussion Select an answer first
2. 12expert · hard

   A security team is automating its detection engineering process. They have an AI model that generates candidate rules, a CI/CD pipeline that tests them, and a manual approval step before deployment. The team is seeing a bottleneck: the AI model generates hundreds of candidate rules per week, but the analysts only have time to review and approve a small fraction of them. What is the most effective way to address this bottleneck without sacrificing the quality of the rules that are deployed?

   • AReduce the number of rules the AI model generates by making it more conservative.
   • BUse the CI/CD pipeline to automatically filter out rules that fail basic syntax checks or have a very high false positive rate in the test environment.
   • CHave the AI model rank the rules by its own confidence score, and have analysts only review the top 10%.
   • DRemove the manual approval step and let the CI/CD pipeline deploy all rules that pass automated testing.
   Reveal Solution Discussion Select an answer first
4. 13application · medium

   A detection engineer wants to use a machine learning model to generate new detection rules from historical attack telemetry. The team is concerned about the model producing rules that are too broad and would generate excessive false positives. What is the most effective way to constrain the model's output while still leveraging AI?

   • ALet the model generate as many rules as possible and then have analysts manually test each one.
   • BUse a simple rule template that only matches exact IP addresses from the attack telemetry.
   • CDeploy all generated rules to a test SIEM and monitor alert volume for a week before deciding which to keep.
   • DTrain the model on labeled benign and malicious events and require that generated rules achieve a minimum precision threshold on a validation set before review.
   Reveal Solution Discussion Select an answer first
5. 14application · medium

   An organization has an automated playbook that responds to a specific type of phishing alert. The playbook's first step is to automatically delete the reported email from all user mailboxes. The security team is concerned that this action is too aggressive and could delete legitimate emails if the alert is a false positive. Which modification to the playbook best addresses this concern?

   • AChange the first step to move the email to a restricted 'quarantine' folder instead of permanently deleting it, allowing for recovery if needed.
   • BKeep the deletion step but add a step that immediately notifies the user whose email was deleted.
   • CRemove the automated response and require an analyst to manually delete the email after review.
   • DAdd a step to the playbook that checks if the email contains any attachments before deleting it.
   Reveal Solution Discussion Select an answer first
6. 15application · medium

   A company's incident response (IR) team uses a legacy ticketing system and manual processes. They are introducing an AI-powered tool that can analyze an alert and suggest a response playbook. The team wants to integrate this tool into their existing workflow with minimal disruption. What is the most effective first step for this integration?

   • AConfigure the AI tool to automatically create a ticket in the legacy system with its suggested playbook, but leave the final execution of the playbook to the human analyst.
   • BReplace the legacy ticketing system with a new, modern platform that has native AI integration.
   • CProvide the AI tool's suggestions to analysts through a separate, standalone dashboard that is not connected to the ticketing system.
   • DHave the AI tool directly execute its suggested playbook in the production environment and then create a ticket for documentation.
   Reveal Solution Discussion Select an answer first