Test

Test
Prev	Appendix B. + Porting and Maintenance + +	Next

+The libstdc++ testsuite includes testing for standard conformance, +regressions, ABI, and performance. +

Organization

Directory Layout

+ The directory libsrcdir/testsuite contains the + individual test cases organized in sub-directories corresponding to + chapters of the C++ standard (detailed below), the dejagnu test + harness support files, and sources to various testsuite utilities + that are packaged in a separate testing library. +

+ All test cases for functionality required by the runtime components + of the C++ standard (ISO 14882) are files within the following + directories. +

+17_intro
+18_support
+19_diagnostics
+20_util
+21_strings
+22_locale
+23_containers
+25_algorithms
+26_numerics
+27_io
+28_regex
+29_atomics
+30_threads
+

+ In addition, the following directories include test files: +

+tr1		  Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
+backward	  Tests for backwards compatibility and deprecated features.
+demangle	  Tests for __cxa_demangle, the IA 64 C++ ABI demangler
+ext		  Tests for extensions.
+performance	  Tests for performance analysis, and performance regressions.
+

+ Some directories don't have test files, but instead contain + auxiliary information: +

+config		  Files for the dejagnu test harness.
+lib		  Files for the dejagnu test harness.
+libstdc++*     	  Files for the dejagnu test harness.
+data		  Sample text files for testing input and output.
+util		  Files for libtestc++, utilities and testing routines.
+

+ Within a directory that includes test files, there may be + additional subdirectories, or files. Originally, test cases + were appended to one file that represented a particular section + of the chapter under test, and was named accordingly. For + instance, to test items related to 21.3.6.1 - + basic_string::find [lib.string::find] in the standard, + the following was used: +

+21_strings/find.cc
+

+ However, that practice soon became a liability as the test cases + became huge and unwieldy, and testing new or extended + functionality (like wide characters or named locales) became + frustrating, leading to aggressive pruning of test cases on some + platforms that covered up implementation errors. Now, the test + suite has a policy of one file, one test case, which solves the + above issues and gives finer grained results and more manageable + error debugging. As an example, the test case quoted above + becomes: +

+21_strings/basic_string/find/char/1.cc
+21_strings/basic_string/find/char/2.cc
+21_strings/basic_string/find/char/3.cc
+21_strings/basic_string/find/wchar_t/1.cc
+21_strings/basic_string/find/wchar_t/2.cc
+21_strings/basic_string/find/wchar_t/3.cc
+

+ All new tests should be written with the policy of one test + case, one file in mind. +

Naming Conventions

+ In addition, there are some special names and suffixes that are + used within the testsuite to designate particular kinds of + tests. +

+ _xin.cc +
+ This test case expects some kind of interactive input in order + to finish or pass. At the moment, the interactive tests are not + run by default. Instead, they are run by hand, like: +
```
+g++ 27_io/objects/char/3_xin.cc
+cat 27_io/objects/char/3_xin.in | a.out
+     
```
+ .in +
+ This file contains the expected input for the corresponding + _xin.cc test case. +
+ _neg.cc +
+ This test case is expected to fail: it's a negative test. At the + moment, these are almost always compile time errors. +
+ char +
+ This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the char instantiation of a + template. +
+ wchar_t +
+ This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing the wchar_t instantiation of + a template. Some hosts do not support wchar_t + functionality, so for these targets, all of these tests will not + be run. +
+ thread +
+ This can either be a directory name or part of a longer file + name, and indicates that this file, or the files within this + directory are testing situations where multiple threads are + being used. +
+ performance +
+ This can either be an enclosing directory name or part of a + specific file name. This indicates a test that is used to + analyze runtime performance, for performance regression testing, + or for other optimization related analysis. At the moment, these + test cases are not run by default. +

Running the Testsuite

Basic

+ You can check the status of the build without installing it + using the dejagnu harness, much like the rest of the gcc + tools.

 make check

in the libbuilddir directory.

 make check-target-libstdc++-v3

in the gccbuilddir directory. +

+ These commands are functionally equivalent and will create a + 'testsuite' directory underneath + libbuilddir containing the results of the + tests. Two results files will be generated: + libstdc++.sum, which is a PASS/FAIL summary for each + test, and libstdc++.log which is a log of + the exact command line passed to the compiler, the compiler + output, and the executable output (if any). +

+ Archives of test results for various versions and platforms are + available on the GCC website in the build + status section of each individual release, and are also + archived on a daily basis on the gcc-testresults + mailing list. Please check either of these places for a similar + combination of source version, operating system, and host CPU. +

Variations

+ There are several options for running tests, including testing + the regression tests, testing a subset of the regression tests, + testing the performance tests, testing just compilation, testing + installed tools, etc. In addition, there is a special rule for + checking the exported symbols of the shared library. +

+ To debug the dejagnu test harness during runs, try invoking with a + specific argument to the variable RUNTESTFLAGS, as below. +

+make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
+

+ or +

+make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
+

+ To run a subset of the library tests, you will need to generate + the testsuite_files file by running + make testsuite_files in the + libbuilddir/testsuite directory, described + below. Edit the file to remove the tests you don't want and + then run the testsuite as normal. +

+ There are two ways to run on a simulator: set up DEJAGNU to point to a + specially crafted site.exp, or pass down --target_board flags. +

+ Example flags to pass down for various embedded builds are as follows: +

+      --target=powerpc-eabism (libgloss/sim)
+make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
+
+--target=calmrisc32 (libgloss/sid)
+make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
+
+--target=xscale-elf (newlib/sim)
+make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
+

+ Also, here is an example of how to run the libstdc++ testsuite + for a multilibed build directory with different ABI settings: +

+make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
+

+ You can run the tests with a compiler and library that have + already been installed. Make sure that the compiler (e.g., + g++) is in your PATH. If you are + using shared libraries, then you must also ensure that the + directory containing the shared version of libstdc++ is in your + LD_LIBRARY_PATH, or equivalent. If your GCC source + tree is at /path/to/gcc, then you can run the tests + as follows: +

+runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
+

+ The testsuite will create a number of files in the directory in + which you run this command,. Some of those files might use the + same name as files created by other testsuites (like the ones + for GCC and G++), so you should not try to run all the + testsuites in parallel from the same directory. +

+ In addition, there are some testing options that are mostly of + interest to library maintainers and system integrators. As such, + these tests may not work on all cpu and host combinations, and + may need to be executed in the + libbuilddir/testsuite directory. These + options include, but are not necessarily limited to, the + following: +

+   make testsuite_files
+

+ Five files are generated that determine what test files + are run. These files are: +

+ testsuite_files +
+ This is a list of all the test cases that will be run. Each + test case is on a separate line, given with an absolute path + from the libsrcdir/testsuite directory. +
+ testsuite_files_interactive +
+ This is a list of all the interactive test cases, using the + same format as the file list above. These tests are not run + by default. +
+ testsuite_files_performance +
+ This is a list of all the performance test cases, using the + same format as the file list above. These tests are not run + by default. +
+ testsuite_thread +
+ This file indicates that the host system can run tests which + involved multiple threads. +
+ testsuite_wchar_t +
+ This file indicates that the host system can run the wchar_t + tests, and corresponds to the macro definition + _GLIBCXX_USE_WCHAR_T in the file c++config.h. +

+   make check-abi
+

+ The library ABI can be tested. This involves testing the shared + library against an ABI-defining previous version of symbol + exports. +

+   make check-compile
+

+ This rule compiles, but does not link or execute, the + testsuite_files test cases and displays the + output on stdout. +

+   make check-performance
+

+ This rule runs through the + testsuite_files_performance test cases and + collects information for performance analysis and can be used to + spot performance regressions. Various timing information is + collected, as well as number of hard page faults, and memory + used. This is not run by default, and the implementation is in + flux. +

+ We are interested in any strange failures of the testsuite; + please email the main libstdc++ mailing list if you see + something odd or have questions. +

+ To run the libstdc++ test suite under the debug mode, edit + libstdc++-v3/scripts/testsuite_flags to add the + compile-time flag -D_GLIBCXX_DEBUG to the + result printed by the --build-cxx + option. Additionally, add the + -D_GLIBCXX_DEBUG_PEDANTIC flag to turn on + pedantic checking. The libstdc++ test suite should produce + precisely the same results under debug mode that it does under + release mode: any deviation indicates an error in either the + library or the test suite. +

+ The parallel + mode can be tested in much the same manner, substituting + -D_GLIBCXX_PARALLEL for + -D_GLIBCXX_DEBUG in the previous paragraph. +

+ Or, just run the testsuites with CXXFLAGS + set to -D_GLIBCXX_DEBUG or + -D_GLIBCXX_PARALLEL. +

Writing a new test case

+ The first step in making a new test case is to choose the correct + directory and file name, given the organization as previously + described. +

+ All files are copyright the FSF, and GPL'd: this is very + important. The first copyright year should correspond to the date + the file was checked in to SVN. +

+ As per the dejagnu instructions, always return 0 from main to + indicate success. +

+ A bunch of utility functions and classes have already been + abstracted out into the testsuite utility library, + libtestc++. To use this functionality, just include the + appropriate header file: the library or specific object files will + automatically be linked in as part of the testsuite run. +

+ For a test that needs to take advantage of the dejagnu test + harness, what follows below is a list of special keyword that + harness uses. Basically, a test case contains dg-keywords (see + dg.exp) indicating what to do and what kinds of behavior are to be + expected. New test cases should be written with the new style + DejaGnu framework in mind. +

+ To ease transition, here is the list of dg-keyword documentation + lifted from dg.exp. +

+# The currently supported options are:
+#
+# dg-prms-id N
+#	set prms_id to N
+#
+# dg-options "options ..." [{ target selector }]
+#	specify special options to pass to the tool (eg: compiler)
+#
+# dg-do do-what-keyword [{ target/xfail selector }]
+#	`do-what-keyword' is tool specific and is passed unchanged to
+#	${tool}-dg-test.  An example is gcc where `keyword' can be any of:
+#	preprocess|compile|assemble|link|run
+#	and will do one of: produce a .i, produce a .s, produce a .o,
+#	produce an a.out, or produce an a.out and run it (the default is
+#	compile).
+#
+# dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
+#	indicate an error message <regexp> is expected on this line
+#	(the test fails if it doesn't occur)
+#	Linenum=0 for general tool messages (eg: -V arg missing).
+#	"." means the current line.
+#
+# dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
+#	indicate a warning message <regexp> is expected on this line
+#	(the test fails if it doesn't occur)
+#
+# dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
+#	indicate a bogus error message <regexp> use to occur here
+#	(the test fails if it does occur)
+#
+# dg-build regexp comment [{ target/xfail selector }]
+#	indicate the build use to fail for some reason
+#	(errors covered here include bad assembler generated, tool crashes,
+#	and link failures)
+#	(the test fails if it does occur)
+#
+# dg-excess-errors comment [{ target/xfail selector }]
+#	indicate excess errors are expected (any line)
+#	(this should only be used sparingly and temporarily)
+#
+# dg-output regexp [{ target selector }]
+#	indicate the expected output of the program is <regexp>
+#	(there may be multiple occurrences of this, they are concatenated)
+#
+# dg-final { tcl code }
+#	add some tcl code to be run at the end
+#	(there may be multiple occurrences of this, they are concatenated)
+#	(unbalanced braces must be \-escaped)
+#
+# "{ target selector }" is a list of expressions that determine whether the
+# test succeeds or fails for a particular target, or in some cases whether the
+# option applies for a particular target.  If the case of `dg-do' it specifies
+# whether the test case is even attempted on the specified target.
+#
+# The target selector is always optional.  The format is one of:
+#
+# { xfail *-*-* ... } - the test is expected to fail for the given targets
+# { target *-*-* ... } - the option only applies to the given targets
+#
+# At least one target must be specified, use *-*-* for "all targets".
+# At present it is not possible to specify both `xfail' and `target'.
+# "native" may be used in place of "*-*-*".
+
+Example 1: Testing compilation only
+// { dg-do compile }
+
+Example 2: Testing for expected warnings on line 36, which all targets fail
+// { dg-warning "string literals" "" { xfail *-*-* } 36
+
+Example 3: Testing for expected warnings on line 36
+// { dg-warning "string literals" "" { target *-*-* } 36
+
+Example 4: Testing for compilation errors on line 41
+// { dg-do compile }
+// { dg-error "no match for" "" { target *-*-* } 41 }
+
+Example 5: Testing with special command line settings, or without the
+use of pre-compiled headers, in particular the stdc++.h.gch file. Any
+options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
+up in the normal.exp file.
+// { dg-options "-O0" { target *-*-* } }
+

+ More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. +

Test Harness and Utilities

Dejagnu Harness Details

+ Underlying details of testing for conformance and regressions are + abstracted via the GNU Dejagnu package. This is similar to the + rest of GCC. +

This is information for those looking at making changes to the testsuite +structure, and/or needing to trace dejagnu's actions with --verbose. This +will not be useful to people who are "merely" adding new tests to the existing +structure. +

The first key point when working with dejagnu is the idea of a "tool". +Files, directories, and functions are all implicitly used when they are +named after the tool in use. Here, the tool will always be "libstdc++". +

The lib subdir contains support routines. The +lib/libstdc++.exp file ("support library") is loaded +automagically, and must explicitly load the others. For example, files can +be copied from the core compiler's support directory into lib. +

Some routines in lib/libstdc++.exp are callbacks, some are +our own. Callbacks must be prefixed with the name of the tool. To easily +distinguish the others, by convention our own routines are named "v3-*". +

The next key point when working with dejagnu is "test files". Any +directory whose name starts with the tool name will be searched for test files. +(We have only one.) In those directories, any .exp file is +considered a test file, and will be run in turn. Our main test file is called +normal.exp; it runs all the tests in testsuite_files using the +callbacks loaded from the support library. +

The config directory is searched for any particular "target +board" information unique to this library. This is currently unused and sets +only default variables. +

Utilities

+ The testsuite directory also contains some files that implement + functionality that is intended to make writing test cases easier, + or to avoid duplication, or to provide error checking in a way that + is consistent across platforms and test harnesses. A stand-alone + executable, called abi_check, and a static + library called libtestc++ are + constructed. Both of these items are not installed, and only used + during testing. +

+ These files include the following functionality: +

+ testsuite_abi.h, + testsuite_abi.cc, + testsuite_abi_check.cc +
+ Creates the executable abi_check. + Used to check correctness of symbol versioning, visibility of + exported symbols, and compatibility on symbols in the shared + library, for hosts that support this feature. More information + can be found in the ABI documentation here +
+ testsuite_allocator.h, + testsuite_allocator.cc +
+ Contains specialized allocators that keep track of construction + and destruction. Also, support for overriding global new and + delete operators, including verification that new and delete + are called during execution, and that allocation over max_size + fails. +
+ testsuite_character.h +
+ Contains std::char_traits and + std::codecvt specializations for a user-defined + POD. +
+ testsuite_hooks.h, + testsuite_hooks.cc +
+ A large number of utilities, including: +
- VERIFY
- set_memory_limits
- verify_demangle
- run_tests_wrapped_locale
- run_tests_wrapped_env
- try_named_locale
- try_mkfifo
- func_callback
- counter
- copy_tracker
- copy_constructor
- assignment_operator
- destructor
- pod_char, pod_int and associated char_traits specializations
+ testsuite_io.h +
+ Error, exception, and constraint checking for + std::streambuf, std::basic_stringbuf, std::basic_filebuf. +
+ testsuite_iterators.h +
+ Wrappers for various iterators. +
+ testsuite_performance.h +
+ A number of class abstractions for performance counters, and + reporting functions including: +
- time_counter
- resource_counter
- report_performance