Rational TestMate Release Note
The following areas are covered in these Release Notes:
General Information About This Release
This document uses the following terms to designate the various platforms on which Rational TestMate (hereafter TestMate) runs.
This release requires Apex release 3.0.0 or 3.0.0B. Refer to the Rational Apex/Summit Installation Guide for further information about hardware and software configurations.
Rational Software Corporation provides the following documentation with this release.
Printed documentation (some also provided online):
These documents can be accessed through the Help:Manuals command from any directory viewer or the List of Documents entry in the button bar at the top of each HTML document. A subset of these documents is provided in hard-copy.
Postscript versions of these manuals are also provided and can be printed using the Help:Print Manuals command.
In some cases, you may note discrepancies between the printed documentation and the online documentation. In these cases, the online documentation is the most up-to-date and should be followed.
For a list of available manual pages, enter the following command in an xterm window:
man testmate
An xterm window can be created from the Tools:Shell command.
In some cases, you may note discrepancies between the printed documentation and the online documentation. In these cases, the online documentation is the most up-to-date and should be followed.
[Next] [Prev] [TOC] [Index]
Rational TestMate Release Note
Upgrading from Previous Releases
After the 3.0 installation is complete, you will need to upgrade existing subsystems containing TestMate artifacts, such as .tc files. This section describes how to upgrade subsystems created by earlier releases of TestMate.
There is a new Ada 83 version of the testmate.ss API view
$APEX_BASE/ada/testmate.ss/tm_<arch>.ada83.3.0.0 .rel
Any user view in which Ada test cases are used must import this new TestMate API view; the import change must be performed where necessary using the TestMate Setup dialog or the appropriate Apex import command.
There is a new Ada 95 version of the testmate.ss API view
$APEX_BASE/ada/testmate.ss/tm_<arch>.ada95.3.0.0 .rel
Any user view in which Ada test cases are used must import this new TestMate API view; the import change must be performed where necessary using the TestMate Setup dialog or the appropriate Apex import command.
There is a new C/C++ version of the testmate.ss API view
$APEX_BASE/c++/testmate.ss/tm_<arch>.3.0.0.rel
Any user view in which C or C++ test cases are used must import this new TestMate API view; the import change must be performed where necessary using the TestMate Setup dialog or the appropriate Apex import command.
There is a new version of the TestMate customization keys for Ada 83, Ada95, C/C++, and Ada Cross. Typically all views are setup to reference the new keys indirectly via the TestMate.sw file switch TM_Customization_Key. In the event that a site specific key is used or this switch is hardcoded to a value other than the default value, the appropriate modifications must be made to the TestMate.sw file to either use the new keys directory provided in this release, or to update the site specific keys referenced by the TestMate.sw file to reflect any modifications made to the new keys.
The new keys directories are in:
$APEX_BASE/{c++,ada}/keys/tm_standard.3.0.0
For TestMate Ada Cross via Apex non-RCI cross compilers, the keys directory is in
$APEX_BASE/{c++,ada}/keys/tm_cross.3.0.0
For TestMate Ada Cross via Apex RCI, the keys directory is in
$APEX_BASE/{c++,ada}/keys/tm_rci.3.0.0
Existing TestMate-created testing views will not automatically be updated to import the new testmate.ss API view. Users should either delete the testing views or change the imports manually.
TestMate 3.0.0 is upwardly compatible with TestMate 2.4.6, i.e. TestMate artifacts created using TestMate 2.4.6 can be directly used by TestMate 3.0.0 without being upgraded. However, test cases and test lists which have been created/modified using TestMate 3.0.0 may not be used again by TestMate 2.4.6. There is no requirement to upgrade any of the TestMate artifacts for 3.0.0.
In TestMate 3.0.0, a summary field has been added to test cases and test lists. The tmupgrade command (described below) can be used to initialize this field to an initial value based upon the first line of the description field. In addition, the switches FT_HIDDEN, FTGEN, STGEN, LIVE_UNITS, and TEMPLATE_PLACEHOLDER_SUB are no longer used by TestMate and will be deleted from any TestMate.sw files by the tmupgrade command. It is recommended that tmupgrade only be used if it is desirable to initialize the summary field or these switches have been defined in the TestMate switch file (warnings will be issued by TestMate commands if these switches are encountered; leaving them defined will have no other affect besides issuing these warnings).
The tmupgrade command can be run either directly from the command line in an Apex xterm, or indirectly via a dialog interface accessible from the TestMate:Misc:Upgrade to new release... menu.
The objects operated on can be views, directories, or any TestMate specific object. In the case of a directory or view, all objects contained within it will be processed, recursively processing any subdirectories.
Objects which are under Apex CMVC control will only be modified if the proper TestMate CMVC switches are set in the view in which the objects reside, or if the global session switch is set, or if they are currently checked out. See the TestMate switch information for AUTO_CHECKIN and AUTO_CHECKOUT.
For TestMate 3.0.0, only test cases, test lists, and TestMate.sw files will be examined, and, if necessary, modified.
Rational TestMate Release Note
Prior releases used the term "Unit test cases" to describe Ada test cases that have individual columns for entry of input data of each subprogram under test input parameter, and to enter expected values of output parameters and function results. These test cases are now called "Data-driven test cases".
tm_build has been renamed to tmbuild .
Under certain circumstances it may be necessary to perform manual inspection of uncovered code and, through analysis, mark certain elements as being covered.
It is now possible to mark a coverage element in a coverage result object as being "manually covered". This can be done either via the GUI or from the command line. For detailed information on the GUI and command line interface see Modifying Coverage Results in the TestMate User's Guide.
In past releases, the only mechanism provided to modify a test result for a test case in a test result object was from the command line. Test Result values can now be modified directly in the test results editor. For more information see the TestMate User's Guide.
All online documentation is now provided in HTML. Netscape 4.0.14 is provided with Apex.
A Netscape window is raised when you first invoke the online help system. If you already have Netscape running, the current session is used.
Tutorial information has been expanded, particularly for Ada test cases and automated unit testing features of TestMate.
Earlier versions of TestMate supported a feature called replication, which allowed you to quickly create closely related test cases. This feature has been merged into the more general-purpose pattern expansion feature within the integrated test editor.
The Test List Editor and Unit Test List Editor have been integrated. Test cases of all types can now be edited directly in the Table View of the new integrated Test Editor. Column sets coupled with new View:Hide/Show menu items make it quick and easy to show the columns you're interest in and hide the ones you're not. Pattern expansion (via Edit:FIll Pattern and Edit:Fill Permuted) is available in the Table View to assist in populating test cases with parameter values, input and output files, etc.
Column resizing is even easier than before. Simply press the middle mouse button anywhere in the column annotation area and drag the column to the desired size.
Column rearranging has been removed. Rows can still be rearranged, and the new order will be made permanent if the test list is saved.
You can now add/delete/modify unit parameters, stubs and stub parameters to support iterative development. The Edit:Columns:Insert Unit Parameter and Edit:Columns:Insert Stub menu items bring up dialogs to guide you through the process. For stubs, you simply place the function or procedure definition in the input box and click OK. The test cases in the list will be updated with the new stub definition or unit parameters. The Edit:Columns:Column Attributes menu item brings up a dialog with the selected column's current profile. Edit the fields and click OK - the new attributes will appear in the column and the test cases in the list will be updated with the new profile.
When adding stubs, a column for each stub parameter (if any) will be added. A column for the stub's name will be added at the start of the parameter list, immediately following the Unit Under Test column. This column serves two purposes. For stubs with a parameterless profile, this column indicates that a stub is present for this test list. In addition, if you wish to remove the stub, you can simply remove this column. There is no need to select and remove the stub's parameter columns.
The new integrated Test Editor also replaces the Test Case Editor (TCE). The Form View of the Test Editor gives you access to the all the same fields as the TCE, but now you don't have to leave the Test Editor to modify them. The Form View gives a structured view of the test case body. Additional panels provide access to the I/O files, execution context, termination controls (for program test cases), description, keyword, requirements and summary fields. A shuttle interface at the top of the Form View allows you to navigate through the list of test cases quickly, with buttons for beginning of list, end of list, next test case and previous test case.
The TestMate:Setup dialog has been modified to used tabs to help reduce the amount of real estate it requires and to provide a more logical and appealing grouping of related switches.
In the Form View, in the Descriptions panel, is a task field. Now you can associate a Summit/TM task with a test case.
Likewise, in the List Attributes panel, you can find a task field for associating a Summit/TM task with the test list or test list group.
You can now control whether TestMate generates exception handling code in Ada test scripts. Some compilers or runtimes do not support conventional Ada exceptions (for example, some safety-critical runtime variants). Generation of exception handling code is controlled by the switch NO_TARGET_EXCEPTION_SUPPORT.See the TesTMate Customization and Configuration Guide.
Structural test generation has been removed from TestMate.
The Ada toolkit API has been replaced with TestMate shell.
TestMate 3.0.0 introduces a scripting language, TestMate shell, which consists of extensions to Apex shell that allow you to easily access the contents of test cases, test lists, test results, and coverage files. For more information see the Command Reference TestMate Shell chapter.
The following table summarizes the changes to the TestMate switches:
The following switches are no longer supported in the 3.0.0 release of TestMate.
TestMate is shipped with model views for the standard Ada83 and Ada95 runtimes. To build the appropriate views for use with the threaded versions of these runtimes shipped with Solaris, see the tmbuild command in the TestMate Customization and Configuration Guide. This command can be used to construct new testmate.ss and model.ss views.
The trap based coverage analyzer has been configured to work with the Solaris Threaded runtimes (ada83 or ada95). Currently this is the only platform for which the trap based coverage analyzer will work with threaded runtimes.
Rational TestMate Release Note
Defects Resolved in Release 3.0.0
The defects described in this section have been resolved in TestMate 3.0.0.T
Rational TestMate Release Note
TestMate Ada Cross and TestMate Ada MCDC
Please be sure to read the Coverage Analyzer section later in these release notes for information regarding TestMate Ada Cross and TestMate Ada MCDC interaction with Apex for Tornado.
The following TestMate Ada features are not supported with TestMate Ada Cross:
For data driven tests (those which use a TestMate generated testing view) or execution using the instrumented coverage analyzer, setting the memory pool for TMS must be done by manually setting the TARGET_TMS_POOL switch in the source view in which the link/execution is initiated.
Program test cases are used to drive executables. In the case of RCI, these executables may be text or binary files that need to be downloaded to a target processor.
TestMate may not locate these executables properly in some RCI environments. In order to make program test cases execute properly for RCI targets, make sure you do the following:
For each test driver in the RCI directory, create a symbolic link from the target test driver to one of the locations on your test driver path (as specified in your TEST_DRIVER_LOC switch). The name of the symbolic link _must_ match the name of the file containing the Ada main procedure, minus the ".2.ada" suffix.
For example, if you have an Apex RCI to VADScross environment, and have a test driver named foo.vox, whose main subprogram is located in <view>/Tests/drivers/foo.2.ada , you must create a symbolic link from the file foo.vox to <view>/Tests/drivers/foo :
% ln -s <target_rci_directory>/foo.vox <view>/Tests/drivers/foo
(when executing the "ln" commands, you should substitute the correct values for the placeholders shown above).
When locating the Ada source for a test driver, TestMate first adds ".2.ada" to the end of the test driver and starts traversing the Ada library from that point (in other words, it assumes that the test driver executable file is located in the same location as the corresponding .2.ada file, and that they have the same simple name).
By creating a symbolic link from the executable to the place that TestMate expects to find the file (and with the correct name), you essentially work around a limitation that TestMate would otherwise have with running program test cases for RCI targets.
The test result group display does not summarize the coverage totals across all test results in the display. To obtain a summary, select all the test results in the image and use TestMate:Test Evaluation:Coverage Summary to get a coverage summary.
If you are using TestMate with Apex 3.0.0, the TestMate Options button on the Link Dialog is broken. Please use the command line interface to link objects that require TestMate options.
Alternatively, you can edit the Link dialog to fix the problem. The fix requires modification to two commands in the dialog.
This problem has been fixed in Apex 3.0.0B.
It is possible to get unprintable characters embedded in entered text and this is not detected by the editors. These unprintable items manifest themselves as compilation errors detected when a generated script is compiled and/or linked. To correct this condition, visit the unit test list and/or test case and change the offending field.
In previous TestMate releases, this was often caused by the insertion of the carriage return character. TestMate now interprets the carriage return character as a command to complete editing of the current cell and move one cell below. To insert a carriage return in a cell, you need to press ALT-RETURN.
The Controls operators only work for textual comparisons. Numeric comparisons such as filters for all "coverage < 80%" will not work.
The test case signature and pattern information for a test list is kept in a special test case associated with every test list. This special test case has the string "_skeleton.tc" as part of its file name. You should never delete this test case nor attempt to insert this test case into a test list.
If elements (test cases or test lists) referenced by the test list (test list group) have been removed, relocated, or had their permissions changed, you may not be able to visit the test list. Depending on which element is missing, one of the following situations may arise:
Rational TestMate Release Note
Prior to TestMate 3.0 the test case driver, evaluation action, epilog, precondition and prolog fields had three fields each. These were the Result field, Procedure field, and Parameters field. These have been collapsed into a single field which is parsed internally to derive the result, procedure and parameters fields. This section describes the language constraints associated with these cells.
There are three varieties of syntax which can appear in one of these test case fields. The syntax is described below using quasi-regular expressions. In the expression to follow a set of bracket enclosing an element means the element is optional. A `*' symbol following an element means the element can appear 0 or more times.
The format for program test cases is:
[<result> =] [<procedure> [<arg> [<arg>]*]]
<result> is optional, args can only appear if a valid <procedure> value is present, and the null string is accepted.
The format for an ada test case is:
[<result> :=] [<procedure> [ ( <arg> [,<arg>]* ) ]]
<result> is optional, args can appear only if a procedure appears and must be enclosed in parens, and the null string is accepted. Further the values for <result> and <procedure> simple names.The values for <result> and <procedure> may be simple identifiers, or identifiers with selectors, for example, r = p.f(x).
The format for a C or C++ test case is:
[<result> =] [<procedure> ( [ <arg>, [<arg>]* ] ) ]]
Paramters can only appear if a procedure appears and must be enclosed in parenthesis. The parenthesis must appear even in the absense of arguments. This differs from the ada syntax above. The values for <result> and <procedure> may be simple identifiers, or identifiers with selectors, for example. r := p.f(x). The scope resolution operator is also available for <procedure> values, for example, r := c::p.f(x).
TestMate does not currently support data-driven test lists for C or C++ units, nor does it support test data generation. Unit testing may still be performed using C and C++ test cases, however test drivers must be written by hand, rather than automatically generated by TestMate.
This section describes limitations of the data-driven testing features of TestMate. If these limitations apply to an Ada unit that you wish to test, consider using a non-data-driven Ada test case instead, and write an Ada driver manually. For more information see the chapter on "Ada Test Cases" in the TestMate User's Guide.
TestMate will either fail to generate a compilable script, or it will fail to generate the script for subprograms which have task type. (including function return types).
This restriction also applies to stubs with task type parameters or function return types.
Unconstrained output parameters are not supported due to a current limitation in creating an appropriate constrained output target object. An example would be an out parameter of string type.
Global variables of an anonymous type are not yet supported (for example Xarr : array(1..10) of integer; ). The user is encouraged to create a declared type and use that instead of the anonymous type:
type Xarr_Type is array(1..10) of integer;
This release does not support data-driven tests of tasks. All other test cases support testing of tasks and task types (for example, an Ada test case may call a custom driver that calls task entries).
This restriction also applies to stubs -- automated stubs cannot be tasks.
Automatic pattern generation for hidden global variables and testing of hidden subprograms is not supported for generic instantiations or units under test that have subunits.
Stub values for generic instantiation subprograms are not yet supported. The user can workaround this limitation by declaring the generic instantiation unit and generic unit as live.
A private type in a generic specification cannot be de-encapsulated currently. Note that generic formals used in instantiations can be private types with proper de-encapsulation.
If a composite type (record or array) includes a component which would require a surrogate routine (e.g. for de-encapsulation), then the composite type requires a de-encapsulation. This de-encapsulation is not yet supported. Note that de-encapsulation is also required if the user tests a floating point component using a "range" test.
A private type with non-default initialized discriminants cannot yet be handled with a de-encapsulation. A warning will be issued and the user should use external values of this type as parameters (rather than having TestMate create a surrogate based on the user-entered literal value).
Test data generation supports subprograms with parameters of most Ada types, or types derived from them. However, the following types are not yet supported:
TestMate cannot currently generate test data for the following program units:
Rational TestMate Release Note
The initial releases of "Apex for VxWorks 3.0.0" does not contain the necessary support for the trap based coverage analyzer to work, and will result in undefined symbols when attempts are made to link in the TestMate trap based coverage analyzer. This will require a subsequent patch release of "Apex for VxWorks 3.0.0".
Data Driven Tests (those which generate scripts in a testing view) or execution using the instrumented coverage analyzer requires the manual setting of the TARGET_COV_POOL switch in the TestMate.sw file located in the source view. Setting this switch in the link or run dialogs will have no affect.
A window exists in which decision coverage can be missed due to two tasks executing the same object code decision at the "same time". If each task takes a different decision target, it is possible that one of the targets will not be recorded, resulting in no coverage for that decision when in fact there was coverage.
There is a problem in C/C++ trap based coverage analysis of a unit which contains multiple includes of the same file via different names, where the file exists in the same view. For example, foo.c contains:
#include "myheader.h"
and the myheader.h file contains:
#include "utilities/macros.h"
This will result in an unhandled exception when TestMate processes the foo.c file. The workaround is to make all #includes use the same simple naming notation "macros.h"
This is currently not supported.
This is currently not supported. Statement and decision coverage provide more information than line information, and should be used instead.
Trap baed coverage analysis is currently not supported for inlined subprograms for both Ada and C/C++.
Coverage for templates is currently not supported.
In order to collect trap based coverage on a compilation unit which contains, or is itself, a replicated generic, one must specify the compilation unit and the set of compilation units in the closure of the main program which instantiates the desired replicated generic.
Under some conditions of optimized code, source level decision coverage using the trap based coverage analyzer may indicate that a decision has been optimized away if the explicit target of the decision has been optimized away.
Elaboration segments/decisions for generics are incorrectly annotated as having no object code. The source span for these elaboration segments covers the entire spec or body. If one displays the Ada source coverage based upon the summary totals for a unit which has 0% coverage for the elaboration code, the entire unit will be highlighted as uncovered. The coverage totals for each unique subprogram are correct, and as a workaround, one can display the Ada source coverage on a subprogram by subprogram basis (just not the entire unit at once).
Under some conditions, a return statement which starts a segment will be highlighted as optimized away, when in fact it has not. Under this condition, there is no way of knowing whether or not it was executed. There are two simple conditions in which this is known to occur - returning the value passed and returning a literal. For example:
function Foo (A_Value : Boolean) return Boolean is
begin
return A_Value;
end Foo;
function Foo return Boolean is
A work around is to preceed the return statement with another statement which causes code to be generated.
Coverage analysis is not supported outside the normal Ada tasking implementation; i.e. if a user program (Ada, C, or C++) performs a UNIX fork thereby creating another UNIX process, coverage analysis on the new process is not supported and may result in erroneous coverage results for the originating process.
Coverage for C/C++ programs which use threading is not currently supported.
CM information is not collected for C/C++ units specified as units under test.
In the C/C++ test case - Includes section, use of the notation " #include <myfile.h> " will result in a warning message during script generation:
!!! Unknown Placeholder <myfile.h> ignored
Use of the "Test Case Generation" option with the MCDC Analyzer primitive may cause premature termination of this operation.
When generating a coverage file report that includes a CASE statement with a Boolean expression the display for this decision points will include an error message. Subsequent metrics information will be off by the condition totals for these expressions.
The Explain messages associated with MCDC Boolean Expressions in the Ada Object Editor display are incorrectly formatted. The "Condition" and "Set" specification is being displayed after the independence set they are associated with.
The maximum number of conditions supported for any single boolean expression is 24 for this release. Expressions with more then 24 subconditions are rejected with an appropriate error message.
The default algorithm utilized to generate boolean pair tables does not recognize coupled subconditions. Coupled subconditions are subconditions that are directly dependent upon each other (such as two subconditions that are the same boolean variable). The result of this is that when coupled subconditions are present, impossible to achieve pairs will be displayed and complete coverage will not be attainable without manual evaluation.
Overloading package standard may hinder the instrumentation of source level constructs that require type conversion. For instance, instrumentation of an expression of a type derived from boolean will attempt to convert the expression to type "standard.boolean" which may fail at compilation time if the overloaded standard package does not properly remap type boolean to the expected predefined type.
Visible objects in nested generic instantiations (more then one level deep) whose type is based on a generic formal whose actual type is based on standard.boolean fail to instrument properly. The type used in type conversions is the formal type name instead of the actual (based on boolean) type name.
Floating point attributes used as CASE statement choices are not currently statically evaluated by MCDC processing. For instance:
case My_Obj is
when My_Float'Mantissa | My_Float'emax =>
null;
when others =>
null;
end case;
In the above example the `Mantissa and `Emax attributes will be rejected with an appropriate error message and processing will continue.
The use of Boolean'First or Boolean'Last in expressions will result in the (sub)expression being treated as a variable with coverage side-effects.
Drivers which terminate with an Unhandled exception ( C++ ) are not correctly processed by the Test Execution Server. A workaround is to change the default value of the FAILED_WITH_EXCEPTION status mapping. This should be set to the value 32774. See "User-defined Results" in the TestMate Customization and Configuration Guide for more information.
Rational TestMate Release Note