(The `site.exp' file contains a handful of useful variables like host
and target triplets, and pathnames.)
+Parallel testing
+****************
+
+If not testing with a remote host (in DejaGnu's sense), you can run
+the GDB test suite in a fully parallel mode. In this mode, each .exp
+file runs separately and maybe simultaneously. The test suite ensures
+that all the temporary files created by the test suite do not clash,
+by putting them into separate directories. This mode is primarily
+intended for use by the Makefile.
+
+For GNU make, the Makefile tries to run the tests in parallel mode if
+any -j option is given. For a non-GNU make, tests are not
+parallelized.
+
+If RUNTESTFLAGS is not empty, then by default the tests are
+serialized. This can be overridden by either using the
+`check-parallel' target in the Makefile, or by setting FORCE_PARALLEL
+to any non-empty value:
+
+ make check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
+ make check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL=1
+
+If you want to use runtest directly instead of using the Makefile, see
+the description of GDB_PARALLEL below.
+
+Racy testcases
+**************
+
+Sometimes, new testcases are added to the testsuite that are not
+entirely deterministic, and can randomly pass or fail. We call them
+"racy testcases", and they can be bothersome when one is comparing
+different testsuite runs. In order to help identifying them, it is
+possible to run the tests several times in a row and ask the testsuite
+machinery to analyze the results. To do that, you need to specify the
+RACY_ITER environment variable to make:
+
+ make check RACY_ITER=5 -j4
+
+The value assigned to RACY_ITER represents the number of times you
+wish to run the tests in sequence (in the example above, the entire
+testsuite will be executed 5 times in a row, in parallel). It is also
+possible to check just a specific test:
+
+ make check TESTS='gdb.base/default.exp' RACY_ITER=3
+
+One can also decide to call the Makefile rules by hand inside the
+gdb/testsuite directory, e.g.:
+
+ make check-paralell-racy -j4
+
+In which case the value of the DEFAULT_RACY_ITER variable (inside
+gdb/testsuite/Makefile.in) will be used to determine how many
+iterations will be run.
+
+After running the tests, you shall see a file name 'racy.sum' in the
+gdb/testsuite directory. You can also inspect the generated *.log and
+*.sum files by looking into the gdb/testsuite/racy_ouputs directory.
+
+If you already have *.sum files generated from previous testsuite runs
+and you would like to analyze them without having to run the testsuite
+again, you can also use the 'analyze-racy-logs.py' script directly.
+It is located in the gdb/testsuite/ directory, and it expects a list
+of two or more *.sum files to be provided as its argument. For
+example:
+
+ ./gdb/testsuite/analyze-racy-logs.py testsuite-01/gdb.sum \
+ testsuite-02/gdb.sum testsuite-03/gdb.sum
+
+The script will output its analysis report to the standard output.
+
Running the Performance Tests
*****************************
GDB_PARALLEL
-When testing natively (that is, not with a remote host), you can run
-the GDB test suite in a fully parallel mode. In this mode, each .exp
-file runs separately and maybe simultaneously. The test suite will
-ensure that all the temporary files created by the test suite do not
-clash, by putting them into separate directories. This mode is
-primarily intended for use by the Makefile.
-
-To use this mode, set the GDB_PARALLEL on the runtest command line.
-Before starting the tests, you must ensure that the directories cache,
-outputs, and temp in the test suite build directory are either empty
-or have been deleted. cache in particular is used to share data
-across invocations of runtest, and files there may affect the test
-results. Note that the Makefile automatically does these deletions.
+To use parallel testing mode without using the the Makefile, set
+GDB_PARALLEL on the runtest command line to "yes". Before starting
+the tests, you must ensure that the directories cache, outputs, and
+temp in the test suite build directory are either empty or have been
+deleted. cache in particular is used to share data across invocations
+of runtest, and files there may affect the test results. The Makefile
+automatically does these deletions.
+
+FORCE_PARALLEL
+
+Setting FORCE_PARALLEL to any non-empty value forces parallel testing
+mode even if RUNTESTFLAGS is not empty.
+
+FORCE_SEPARATE_MI_TTY
+
+Setting FORCE_MI_SEPARATE_UI to 1 forces all MI testing to start GDB
+in console mode, with MI running on a separate TTY, on a secondary UI
+started with "new-ui".
GDB_INOTIFY
If using GNU make then the contents are wildcard-expanded using
GNU make's $(wildcard) function. Test paths must be fully specified,
relative to the "testsuite" subdirectory. This allows one to run all
-tests in a subdirectory by passing "gdb.subdir/*.exp".
+tests in a subdirectory by passing "gdb.subdir/*.exp", or more simply
+by using the check-gdb.subdir target in the Makefile.
+
If for some strange reason one wanted to run all tests that begin with
the letter "d" that is also possible: TESTS="*/d*.exp".
If not using GNU make then the value is passed directly to runtest.
If not specified, all tests are run.
+READ1
+
+This make (not runtest) variable is used to specify whether the
+testsuite preloads the read1.so library into expect. Any non-empty
+value means true. See "Race detection" below.
+
+GDB_TEST_SOCKETHOST
+
+This variable can provide the hostname/address that should be used
+when performing GDBserver-related tests. This is useful in some
+situations, e.g., when you want to test the IPv6 connectivity of GDB
+and GDBserver, or when using a different hostname/address is needed.
+For example, to make GDB and GDBserver use IPv6-only connections, you
+can do:
+
+ make check TESTS="gdb.server/*.exp" RUNTESTFLAGS='GDB_TEST_SOCKETHOST=tcp6:[::1]'
+
+Note that only a hostname/address can be provided, without a port
+number.
+
+TS
+
+This variable turns on the timestamp printing for each line of "make
+check". Note that the timestamp will be printed on stdout output
+only. In other words, there will be no timestamp output on either
+gdb.sum and gdb.log files. If you would like to enable timestamp
+printing, you can do:
+
+ make check TS=1
+
+TS_FORMAT
+
+You can provide a custom format for timestamp printing with this
+variable. The format must be a string compatible with "strftime".
+This variable is only useful when the TS variable is also provided.
+If you would like to change the output format of the timestamp, you
+can do:
+
+ make check TS=1 TS_FORMAT='[%b %H:%S]'
+
+Race detection
+**************
+
+The testsuite includes a mechanism that helps detect test races.
+
+For example, say the program running under expect outputs "abcd", and
+a test does something like this:
+
+ expect {
+ "a.*c" {
+ }
+ "b" {
+ }
+ "a" {
+ }
+ }
+
+Which case happens to match depends on what expect manages to read
+into its internal buffer in one go. If it manages to read three bytes
+or more, then the first case matches. If it manages to read two
+bytes, then the second case matches. If it manages to read only one
+byte, then the third case matches.
+
+To help detect these cases, the race detection mechanism preloads a
+library into expect that forces the `read' system call to always
+return at most 1 byte.
+
+To enable this, either pass a non-empty value in the READ1 make
+variable, or use the check-read1 make target instead of check.
+
+Examples:
+
+ make -j10 check-read1 TESTS="*/paginate-*.exp"
+ make -j10 check READ1="1"
+
Testsuite Configuration
***********************
The predefined trace state variables the board has.
+gdb,no_thread_names
+
+ The target doesn't support thread names.
+
+gdb,nopie_flag
+
+ The flag required to force the compiler to produce non-position-independent
+ executables.
Testsuite Organization
**********************
gdb.<lang>
Language-specific tests for any language besides C. Examples are
-gdb.cp for C++ and gdb.java for Java.
+gdb.cp for C++ and gdb.rust for Rust.
gdb.<platform>
Non-portable tests. The tests are specific to a specific
-configuration (host or target), such as HP-UX or eCos. Example is
-gdb.hp, for HP-UX.
+configuration (host or target), such as eCos.
gdb.arch
running the test at all. This is the better option if the limitation
is intrinsic to the environment, rather than a bug expected to be
fixed in the near future.
+
+Local vs Remote vs Native
+*************************
+
+It's unfortunately easy to get confused in the testsuite about what's
+native and what's not, what's remote and what's not. The confusion is
+caused by the overlap in vocabulary between DejaGnu and GDB.
+
+From a DejaGnu point of view:
+
+ - native: the host or target board is considered native if the its
+ triplet is the same as the build system's triplet,
+
+ - remote: the host or target board is considered remote if it's
+ running on a different machine, and thus require ssh, for example,
+ to run commands, versus simply running commands directly.
+
+Note that they are not mutually exclusive, as you can have a remote
+machine that has the same triplet as the build machine.
+
+From a GDB point of view:
+
+ - native: when GDB uses system calls such as ptrace to interact
+ directly with processes on the same system its running on,
+
+ - remote: when GDB speaks the RSP (Remote Serial Protocol) with
+ another program doing the ptrace stuff.
+
+Note that they are mutually exclusive. An inferior can only be either
+debugged with the native target, or with the remote target a specific
+time.
+
+That means that there are cases where the target is not remote for
+DejaGnu, but is remote for GDB (e.g. running GDBserver on the same
+machine).
+
+You can also have a remote target for DejaGnu, but native for GDB
+(e.g. building on x86 a GDB that runs on ARM and running the
+testsuite with a remote host).
+
+Therefore, care must be taken to check for the right kind of remote.
+Use [is_remote target] to check whether the DejaGnu target board is
+remote. When what you really want to know is whether GDB is using the
+remote protocol, because feature X is only available when GDB debugs
+natively, check gdb_protocol instead.
projects / deliverable / binutils-gdb.git / blobdiff