A test automation engineer is setting up a process to collect data from multiple sources after each test run. Which data collection approach is most likely to provide a comprehensive view of test results and system behavior?

Collect only the application's error logs

Collect only the pass/fail status from the automation tool's summary report

Collect test execution logs from the automation tool and resource utilization metrics from the SUT

Collect only the test environment configuration files

A team notices that their automated regression suite has a high pass rate, but defects are still escaping to production. They suspect the suite is not effective at catching real issues. Which metric would best help them evaluate the suite's ability to detect defects?

The average time it takes to execute the entire suite

The number of test cases in the suite that are marked as 'automated'

The percentage of automated tests that are flaky (intermittently failing)

The number of defects found by the automated suite compared to total defects found by all testing activities

A test automation team is defining the reporting objectives for a new automation initiative. The stakeholders want to know if the automation is reducing the time to release. Which reporting objective is most aligned with this need?

Track the number of test cases automated each sprint

Track the number of defects found by the automation team

Track the total execution time of the automated regression suite and the time saved compared to manual execution

Track the number of test environments available

A test automation team has been tracking the pass rate of their suite for six months. They notice that the pass rate has been steadily declining from 95% to 80%. They also see that the number of tests marked as 'flaky' has increased. What is the most appropriate FIRST action to take?

Report the declining pass rate to management and recommend stopping all automated testing

Immediately increase the number of test environments to reduce contention

Analyze the flaky tests to identify common root causes, such as timing issues or environment instability

Remove all flaky tests from the suite to improve the pass rate

Which metric would best indicate the efficiency of a test automation solution?

Number of test cases that failed

Average execution time per test case

Number of test cases that passed

Total number of test cases in the suite

Certified Tester Advanced Level Test Automation Engineering

Domain 6Objective 2

Objective: Analyze Data from the Test Automation Solution and the System Under Test CTAL-TAE Practice Questions (Page 6)

Part of the Domain 6: Test Automation Reporting and Metrics domain, which makes up ~12% of our current practice bank. ISTQB does not publish an official question count, but from its 90-minute exam (~35–60 total, ~4–7 in this domain), expect 1–2 from this objective — we provide 34 practice questions to prepare you well beyond it. (estimate)

34questions here

7free pages

7concepts

Questions 26–30

26application · medium
A test automation engineer is setting up a process to collect data from multiple sources after each test run. Which data collection approach is most likely to provide a comprehensive view of test results and system behavior?
Select an answer first
27application · medium
A team notices that their automated regression suite has a high pass rate, but defects are still escaping to production. They suspect the suite is not effective at catching real issues. Which metric would best help them evaluate the suite's ability to detect defects?
Select an answer first
28application · medium
A test automation team is defining the reporting objectives for a new automation initiative. The stakeholders want to know if the automation is reducing the time to release. Which reporting objective is most aligned with this need?
Select an answer first
29application · medium
A test automation team has been tracking the pass rate of their suite for six months. They notice that the pass rate has been steadily declining from 95% to 80%. They also see that the number of tests marked as 'flaky' has increased. What is the most appropriate FIRST action to take?
Select an answer first
30foundation · easy
Which metric would best indicate the efficiency of a test automation solution?
Select an answer first

Finished these 5 questions?

Review the revealed explanations, or continue through the curriculum.

Next 5 questions →Back to CTAL-TAE

Free Basic Practice is a study aid with revealable answers — not a scored exam. Examers.io is independent and not affiliated with or endorsed by ISTQB. “CTAL-TAE” is a trademark of its owner, used for identification only.