Test reports for the Rational Quality Manager integration

Test reports describe the test coverage, approval status, and results for linked test plans and test cases in IBM® Rational® Quality Manager.

IBM Rational DOORS® includes two types of reports for integrated test plans and test cases in Rational Quality Manager. The Test Coverage report helps ensure traceability for validation of requirements in test plans and test cases. This report also shows the approval status of linked test cases, which teams can use to determine the impact of additional changes to requirements. The Test Results report displays the test verdict; pass, failure, and deferred status; error records; and other results. The Test Coverage and Test Results reports can be generated for each module view that is associated with a test plan.

Test Coverage report

The Test Coverage report, which is also known as the Traceability report, provides this information: The Test Coverage report is implemented by the module traceability view. The traceability view is an unfiltered, default view of the module with an additional column that shows test plans and test cases that are linked to each requirement.

Test Results report

The Test Results report, which is also known as the Statement of Quality report, provides this information:

The Test Results report is based on a module view. The following image shows a portion of a report.

The image is a screen capture of a Test Results report. The headings and result values are described below.

The report includes these columns and test result values.

Table 1. Test Results report column headings and values
Column Heading Description
Test Cases The ID and name of each test case in the test plan and its linked requirement. This column includes the verdict for each test case. The format is (Test Case Id) Test Case Name: Verdict.
Test Status The test status column provides an indication of coverage and the status of linked test cases. This column displays one of these values:
  • Blank. No test cases in the test plan that are linked to the requirement.
  • Approved. Test cases are in the test plan, linked to the requirement, and approved.
  • Not Approved. Test cases are in the test plan and linked to the requirement, but at least one test case is not approved. The test case's status might be draft, under review, in progress, or a similar status.
Verdict The verdict for tests based on the last run of each execution record for test cases that are linked to the requirement. Verdict is a Rational Quality Manager term.
Passed The number of passed execution records, based on the last run, that are linked to each test case. The format is Test Case Id: Passed Execution Records of Total Execution Records.
Deferred The number of deferred execution records, based on the last run, that are linked to each test case. The format is Test Case Id: Deferred Execution Records of Total Execution Records.
Error The number of error execution records, based on the last run, that are linked to each test case. The format is Test Case Id: Error Execution Records of Total Execution Records.
Failed The number of failed execution records, based on the last run, that are linked to each test case. The format is Test Case Id: Failed Execution Records of Total Execution Records.
Perm Failed The number of perm failed execution records, based on the last run, that are linked to each test case. The format is Test Case Id: Perm Failed Execution Records of Total Execution Records.
Verdict Changed Whether the verdict changed after the report was last run. This column shows one of two possible values:
  • Blank. The verdict did not changed after the report was last run.
  • Yes. The verdict changed after the report was run. You can view the history to analyze the change.
Only the Test Status, Verdict, and Verdict Changed columns are summarized at the parent headings in the Rational DOORS view. The rules for determining the verdict for the following statistics columns when the headings have no object text are as follows:
  • Passed: The number of passed test cases that are linked to requirements under the heading.
  • Deferred: The number of deferred test cases that are linked to requirements under the heading.
  • Error: The number of error test cases that are linked to requirements under the heading.
  • Failed: The number of failed test cases that are linked to requirements under the heading.
  • Perm Failed: The number of perm failed test cases that are linked to requirements under the heading.

For a requirement with object text, the statistics columns contain the number of execution records from the last run for each test case that is linked to the requirement. For headings with no object text, the statistics columns contain the number of test cases that are linked to the requirement with the verdict.

The values for each column are updated when the report is run. You cannot select which columns to update, but you can show and hide columns in the view.

Verdicts are based on the weighted values of the state of the test results for each test case, as shown in the following table. The lowest weight is 1 and the highest weight is 5.
Table 2. States for test results the weight of each state
Weight State Description
1 Passed A test has actually been run and the result is known to be passed.
2 Deferred A test failed, but the result of the test is deferred to a future date.
3 Error A test ran, but something happened beyond the control of the test, such as a power outage, and the results are inconclusive.
4 Failed A test ran and failed. A known defect exists in the test, requirement, or in system under test.
5 Perm Failed A test failed and nothing can cause this test to pass.
In a collection of execution record test results that is based on the last run of the test case, the combined value for the test case is set to the value that holds the highest weighting. The highest weighted value overrides all of the other results and determines the test verdict. For example, in a list of results, if a single result is Perm Failed, the overall result is Perm Failed.

Feedback