Implement timestamp'ed output on "make check"

[deliverable/binutils-gdb.git] / gdb / testsuite / README

This is a collection of tests for GDB.

The file gdb/README contains basic instructions on how to run the
testsuite, while this file documents additional options and controls
that are available.  The GDB wiki may also have some pages with ideas
and suggestions.


Running the Testsuite
*********************

There are two ways to run the testsuite and pass additional parameters
to DejaGnu.  The first is to do `make check' in the main build
directory and specifying the makefile variable `RUNTESTFLAGS':

         make check RUNTESTFLAGS='TRANSCRIPT=y gdb.base/a2-run.exp'

The second is to cd to the testsuite directory and invoke the DejaGnu
`runtest' command directly.

        cd testsuite
        make site.exp
        runtest TRANSCRIPT=y

(The `site.exp' file contains a handful of useful variables like host
and target triplets, and pathnames.)

Parallel testing
****************

If not testing with a remote host (in DejaGnu's sense), you can run
the GDB test suite in a fully parallel mode.  In this mode, each .exp
file runs separately and maybe simultaneously.  The test suite ensures
that all the temporary files created by the test suite do not clash,
by putting them into separate directories.  This mode is primarily
intended for use by the Makefile.

For GNU make, the Makefile tries to run the tests in parallel mode if
any -j option is given.  For a non-GNU make, tests are not
parallelized.

If RUNTESTFLAGS is not empty, then by default the tests are
serialized.  This can be overridden by either using the
`check-parallel' target in the Makefile, or by setting FORCE_PARALLEL
to any non-empty value:

        make check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
        make check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL=1

If you want to use runtest directly instead of using the Makefile, see
the description of GDB_PARALLEL below.

Racy testcases
**************

Sometimes, new testcases are added to the testsuite that are not
entirely deterministic, and can randomly pass or fail.  We call them
"racy testcases", and they can be bothersome when one is comparing
different testsuite runs.  In order to help identifying them, it is
possible to run the tests several times in a row and ask the testsuite
machinery to analyze the results.  To do that, you need to specify the
RACY_ITER environment variable to make:

        make check RACY_ITER=5 -j4

The value assigned to RACY_ITER represents the number of times you
wish to run the tests in sequence (in the example above, the entire
testsuite will be executed 5 times in a row, in parallel).  It is also
possible to check just a specific test:

        make check TESTS='gdb.base/default.exp' RACY_ITER=3

One can also decide to call the Makefile rules by hand inside the
gdb/testsuite directory, e.g.:

        make check-paralell-racy -j4

In which case the value of the DEFAULT_RACY_ITER variable (inside
gdb/testsuite/Makefile.in) will be used to determine how many
iterations will be run.

After running the tests, you shall see a file name 'racy.sum' in the
gdb/testsuite directory.  You can also inspect the generated *.log and
*.sum files by looking into the gdb/testsuite/racy_ouputs directory.

If you already have *.sum files generated from previous testsuite runs
and you would like to analyze them without having to run the testsuite
again, you can also use the 'analyze-racy-logs.py' script directly.
It is located in the gdb/testsuite/ directory, and it expects a list
of two or more *.sum files to be provided as its argument.  For
example:

        ./gdb/testsuite/analyze-racy-logs.py testsuite-01/gdb.sum \
          testsuite-02/gdb.sum testsuite-03/gdb.sum

The script will output its analysis report to the standard output.

Running the Performance Tests
*****************************

GDB Testsuite includes performance test cases, which are not run together
with other test cases, because performance test cases are slow and need
a quiet system.  There are two ways to run the performance test cases.
The first is to do `make check-perf' in the main build directory:

        make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8"

The second is to cd to the testsuite directory and invoke the DejaGnu
`runtest' command directly.

        cd testsuite
        make site.exp
        runtest GDB_PERFTEST_MODE=both GDB_PERFTEST_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8

Only "compile", "run" and "both" are valid to GDB_PERFTEST_MODE.  They
stand for "compile tests only", "run tests only", and "compile and run
tests" respectively.  "both" is the default.  GDB_PERFTEST_TIMEOUT
specify the timeout, which is 3000 in default.  The result of
performance test is appended in `testsuite/perftest.log'.

Testsuite Parameters
********************

The following parameters are DejaGNU variables that you can set to
affect the testsuite run globally.

TRANSCRIPT

You may find it useful to have a transcript of the commands that the
testsuite sends to GDB, for instance if GDB crashes during the run,
and you want to reconstruct the sequence of commands.

If the DejaGNU variable TRANSCRIPT is set (to any value), each
invocation of GDB during the test run will get a transcript file
written into the DejaGNU output directory.  The file will have the
name transcript.<n>, where <n> is an integer.  The first line of the
file shows the invocation command with all the options passed to it,
while subsequent lines are the GDB commands.  A `make check' might
look like this:

      make check RUNTESTFLAGS=TRANSCRIPT=y

The transcript may not be complete, as for instance tests of command
completion may show only partial command lines.

GDB

By default, the testsuite exercises the GDB in the build directory,
but you can set GDB to be a pathname to a different version.  For
instance,

    make check RUNTESTFLAGS=GDB=/usr/bin/gdb

runs the testsuite on the GDB in /usr/bin.

GDBSERVER

You can set GDBSERVER to be a particular GDBserver of interest, so for
instance

    make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"

checks both the installed GDB and GDBserver.

INTERNAL_GDBFLAGS

Command line options passed to all GDB invocations.

The default is "-nw -nx".

`-nw' disables any of the windowed interfaces.
`-nx' disables ~/.gdbinit, so that it doesn't interfere with
the tests.

This is actually considered an internal variable, and you
won't normally want to change it.  However, in some situations,
this may be tweaked as a last resort if the testsuite doesn't
have direct support for the specifics of your environment.
The testsuite does not override a value provided by the user.

As an example, when testing an installed GDB that has been
configured with `--with-system-gdbinit', like by default,
you do not want ~/.gdbinit to interfere with tests, but, you
may want the system .gdbinit file loaded.  As there's no way to
ask the testsuite, or GDB, to load the system gdbinit but
not ~/.gdbinit, a workaround is then to remove `-nx' from
INTERNAL_GDBFLAGS, and point $HOME at a directory without
a .gdbinit.  For example:

        cd testsuite
        HOME=`pwd` runtest \
          GDB=/usr/bin/gdb \
          GDBSERVER=/usr/bin/gdbserver \
          INTERNAL_GDBFLAGS=-nw

GDB_PARALLEL

To use parallel testing mode without using the the Makefile, set
GDB_PARALLEL on the runtest command line to "yes".  Before starting
the tests, you must ensure that the directories cache, outputs, and
temp in the test suite build directory are either empty or have been
deleted.  cache in particular is used to share data across invocations
of runtest, and files there may affect the test results.  The Makefile
automatically does these deletions.

FORCE_PARALLEL

Setting FORCE_PARALLEL to any non-empty value forces parallel testing
mode even if RUNTESTFLAGS is not empty.

FORCE_SEPARATE_MI_TTY

Setting FORCE_MI_SEPARATE_UI to 1 forces all MI testing to start GDB
in console mode, with MI running on a separate TTY, on a secondary UI
started with "new-ui".

GDB_INOTIFY

For debugging parallel mode, it is handy to be able to see when a test
case writes to a file outside of its designated output directory.

If you have the inotify-tools package installed, you can set the
GDB_INOTIFY variable on the runtest command line.  This will cause the
test suite to watch for parallel-unsafe file creations and report
them, both to stdout and in the test suite log file.

This setting is only meaningful in conjunction with GDB_PARALLEL.

TESTS

This variable is used to specify which set of tests to run.
It is passed to make (not runtest) and its contents are a space separated
list of tests to run.

If using GNU make then the contents are wildcard-expanded using
GNU make's $(wildcard) function.  Test paths must be fully specified,
relative to the "testsuite" subdirectory.  This allows one to run all
tests in a subdirectory by passing "gdb.subdir/*.exp", or more simply
by using the check-gdb.subdir target in the Makefile.

If for some strange reason one wanted to run all tests that begin with
the letter "d" that is also possible: TESTS="*/d*.exp".

Do not write */*.exp to specify all tests (assuming all tests are only
nested one level deep, which is not necessarily true).  This will pick up
.exp files in ancillary directories like "lib" and "config".
Instead write gdb.*/*.exp.

Example:

        make -j10 check TESTS="gdb.server/[s-w]*.exp */x*.exp"

If not using GNU make then the value is passed directly to runtest.
If not specified, all tests are run.

READ1

This make (not runtest) variable is used to specify whether the
testsuite preloads the read1.so library into expect.  Any non-empty
value means true.  See "Race detection" below.

GDB_TEST_SOCKETHOST

This variable can provide the hostname/address that should be used
when performing GDBserver-related tests.  This is useful in some
situations, e.g., when you want to test the IPv6 connectivity of GDB
and GDBserver, or when using a different hostname/address is needed.
For example, to make GDB and GDBserver use IPv6-only connections, you
can do:

        make check TESTS="gdb.server/*.exp" RUNTESTFLAGS='GDB_TEST_SOCKETHOST=tcp6:[::1]'

Note that only a hostname/address can be provided, without a port
number.

TS

This variable turns on the timestamp printing for each line of "make
check".  Note that the timestamp will be printed on stdout output
only.  In other words, there will be no timestamp output on either
gdb.sum and gdb.log files.  If you would like to enable timestamp
printing, you can do:

        make check TS=1

TS_FORMAT

You can provide a custom format for timestamp printing with this
variable.  The format must be a string compatible with "strftime".
This variable is only useful when the TS variable is also provided.
If you would like to change the output format of the timestamp, you
can do:

        make check TS=1 TS_FORMAT='[%b %H:%S]'

Race detection
**************

The testsuite includes a mechanism that helps detect test races.

For example, say the program running under expect outputs "abcd", and
a test does something like this:

  expect {
    "a.*c" {
    }
    "b" {
    }
    "a" {
    }
  }

Which case happens to match depends on what expect manages to read
into its internal buffer in one go.  If it manages to read three bytes
or more, then the first case matches.  If it manages to read two
bytes, then the second case matches.  If it manages to read only one
byte, then the third case matches.

To help detect these cases, the race detection mechanism preloads a
library into expect that forces the `read' system call to always
return at most 1 byte.

To enable this, either pass a non-empty value in the READ1 make
variable, or use the check-read1 make target instead of check.

Examples:

        make -j10 check-read1 TESTS="*/paginate-*.exp"
        make -j10 check READ1="1"

Testsuite Configuration
***********************

It is possible to adjust the behavior of the testsuite by defining
the global variables listed below, either in a `site.exp' file,
or in a board file.

gdb_test_timeout

Defining this variable changes the default timeout duration used
during communication with GDB.  More specifically, the global variable
used during testing is `timeout', but this variable gets reset to
`gdb_test_timeout' at the beginning of each testcase, which ensures
that any local change to `timeout' in a testcase does not affect
subsequent testcases.

This global variable comes in handy when the debugger is slower than
normal due to the testing environment, triggering unexpected `TIMEOUT'
test failures.  Examples include when testing on a remote machine, or
against a system where communications are slow.

If not specifically defined, this variable gets automatically defined
to the same value as `timeout' during the testsuite initialization.
The default value of the timeout is defined in the file
`testsuite/config/unix.exp' (at least for Unix hosts; board files may
have their own values).

gdb_reverse_timeout

Defining this variable changes the default timeout duration when tests
under gdb.reverse directory are running.  Process record and reverse
debugging is so slow that its tests have unexpected `TIMEOUT' test
failures.  This global variable is useful to bump up the value of
`timeout' for gdb.reverse tests and doesn't cause any delay where
actual failures happen in the rest of the testsuite.


Board Settings
**************

DejaGNU includes the concept of a "board file", which specifies
testing details for a particular target (which are often bare circuit
boards, thus the name).

In the GDB testsuite specifically, the board file may include a
number of "board settings" that test cases may check before deciding
whether to exercise a particular feature.  For instance, a board
lacking any I/O devices, or perhaps simply having its I/O devices
not wired up, should set `noinferiorio'.

Here are the supported board settings:

gdb,cannot_call_functions

  The board does not support inferior call, that is, invoking inferior
  functions in GDB.

gdb,can_reverse

  The board supports reverse execution.

gdb,no_hardware_watchpoints

  The board does not support hardware watchpoints.

gdb,nofileio

  GDB is unable to intercept target file operations in remote and
  perform them on the host.

gdb,noinferiorio

  The board is unable to provide I/O capability to the inferior.

gdb,noresults

  A program will not return an exit code or result code (or the value
  of the result is undefined, and should not be looked at).

gdb,nosignals

  The board does not support signals.

gdb,skip_huge_test

  Skip time-consuming tests on the board with slow connection.

gdb,skip_float_tests

  Skip tests related to floating point.

gdb,use_precord

  The board supports process record.

gdb_init_command
gdb_init_commands

  Commands to send to GDB every time a program is about to be run.  The
  first of these settings defines a single command as a string.  The
  second defines a TCL list of commands being a string each.  The commands
  are sent one by one in a sequence, first from `gdb_init_command', if any,
  followed by individual commands from `gdb_init_command', if any, in this
  list's order.

gdb_server_prog

  The location of GDBserver.  If GDBserver somewhere other than its
  default location is used in test, specify the location of GDBserver in
  this variable.  The location is a file name for GDBserver, and may be
  either absolute or relative to the testsuite subdirectory of the build
  directory.

in_proc_agent

  The location of the in-process agent (used for fast tracepoints and
  other special tests).  If the in-process agent of interest is anywhere
  other than its default location, set this variable.  The location is a
  filename, and may be either absolute or relative to the testsuite
  subdirectory of the build directory.

noargs

  GDB does not support argument passing for inferior.

no_long_long

  The board does not support type long long.

use_cygmon

  The board is running the monitor Cygmon.

use_gdb_stub

  The tests are running with a GDB stub.

exit_is_reliable

  Set to true if GDB can assume that letting the program run to end
  reliably results in program exits being reported as such, as opposed
  to, e.g., the program ending in an infinite loop or the board
  crashing/resetting.  If not set, this defaults to $use_gdb_stub.  In
  other words, native targets are assumed reliable by default, and
  remote stubs assumed unreliable.

gdb,predefined_tsv

  The predefined trace state variables the board has.

gdb,no_thread_names

  The target doesn't support thread names.

Testsuite Organization
**********************

The testsuite is entirely contained in `gdb/testsuite'.  The main
directory of the testsuite includes some makefiles and configury, but
these are minimal, and used for little besides cleaning up, since the
tests themselves handle the compilation of the programs that GDB will
run.

The file `testsuite/lib/gdb.exp' contains common utility procs useful
for all GDB tests, while the directory testsuite/config contains
configuration-specific files, typically used for special-purpose
definitions of procs like `gdb_load' and `gdb_start'.

The tests themselves are to be found in directories named
'testsuite/gdb.* and subdirectories of those.  The names of the test
files must always end with ".exp".  DejaGNU collects the test files by
wildcarding in the test directories, so both subdirectories and
individual files typically get chosen and run in alphabetical order.

The following lists some notable types of subdirectories and what they
are for.  Since DejaGNU finds test files no matter where they are
located, and since each test file sets up its own compilation and
execution environment, this organization is simply for convenience and
intelligibility.

gdb.base

This is the base testsuite.  The tests in it should apply to all
configurations of GDB (but generic native-only tests may live here).
The test programs should be in the subset of C that is both valid
ANSI/ISO C, and C++.

gdb.<lang>

Language-specific tests for any language besides C.  Examples are
gdb.cp for C++ and gdb.rust for Rust.

gdb.<platform>

Non-portable tests.  The tests are specific to a specific
configuration (host or target), such as eCos.

gdb.arch

Architecture-specific tests that are (usually) cross-platform.

gdb.<subsystem>

Tests that exercise a specific GDB subsystem in more depth.  For
instance, gdb.disasm exercises various disassemblers, while
gdb.stabs tests pathways through the stabs symbol reader.

gdb.perf

GDB performance tests.

Writing Tests
*************

In many areas, the GDB tests are already quite comprehensive; you
should be able to copy existing tests to handle new cases.  Be aware
that older tests may use obsolete practices but have not yet been
updated.

You should try to use `gdb_test' whenever possible, since it includes
cases to handle all the unexpected errors that might happen.  However,
it doesn't cost anything to add new test procedures; for instance,
gdb.base/exprs.exp defines a `test_expr' that calls `gdb_test'
multiple times.

Only use `send_gdb' and `gdb_expect' when absolutely necessary.  Even
if GDB has several valid responses to a command, you can use
`gdb_test_multiple'.  Like `gdb_test', `gdb_test_multiple' recognizes
internal errors and unexpected prompts.

Do not write tests which expect a literal tab character from GDB.  On
some operating systems (e.g. OpenBSD) the TTY layer expands tabs to
spaces, so by the time GDB's output reaches `expect' the tab is gone.

The source language programs do *not* need to be in a consistent
style.  Since GDB is used to debug programs written in many different
styles, it's worth having a mix of styles in the testsuite; for
instance, some GDB bugs involving the display of source lines might
never manifest themselves if the test programs used GNU coding style
uniformly.

Some testcase results need more detailed explanation:

KFAIL

Use KFAIL for known problem of GDB itself.  You must specify the GDB
bug report number, as in these sample tests:

        kfail "gdb/13392" "continue to marker 2"

or

        setup_kfail gdb/13392 "*-*-*"
        kfail "continue to marker 2"


XFAIL

Short for "expected failure", this indicates a known problem with the
environment.  This could include limitations of the operating system,
compiler version, and other components.

This example from gdb.base/attach-pie-misread.exp is a sanity check
for the target environment:

        # On x86_64 it is commonly about 4MB.
        if {$stub_size > 25000000} {
            xfail "stub size $stub_size is too large"
            return
        }

You should provide bug report number for the failing component of the
environment, if such bug report is available, as with this example
referring to a GCC problem:

          if {[test_compiler_info {gcc-[0-3]-*}]
              || [test_compiler_info {gcc-4-[0-5]-*}]} {
              setup_xfail "gcc/46955" *-*-*
          }
          gdb_test "python print ttype.template_argument(2)" "&C::c"

Note that it is also acceptable, and often preferable, to avoid
running the test at all.  This is the better option if the limitation
is intrinsic to the environment, rather than a bug expected to be
fixed in the near future.

Local vs Remote vs Native
*************************

It's unfortunately easy to get confused in the testsuite about what's
native and what's not, what's remote and what's not.  The confusion is
caused by the overlap in vocabulary between DejaGnu and GDB.

From a DejaGnu point of view:

 - native: the host or target board is considered native if the its
   triplet is the same as the build system's triplet,

 - remote: the host or target board is considered remote if it's
   running on a different machine, and thus require ssh, for example,
   to run commands, versus simply running commands directly.

Note that they are not mutually exclusive, as you can have a remote
machine that has the same triplet as the build machine.

From a GDB point of view:

 - native: when GDB uses system calls such as ptrace to interact
   directly with processes on the same system its running on,

 - remote: when GDB speaks the RSP (Remote Serial Protocol) with
   another program doing the ptrace stuff.

Note that they are mutually exclusive.  An inferior can only be either
debugged with the native target, or with the remote target a specific
time.

That means that there are cases where the target is not remote for
DejaGnu, but is remote for GDB (e.g. running GDBserver on the same
machine).

You can also have a remote target for DejaGnu, but native for GDB
(e.g.  building on x86 a GDB that runs on ARM and running the
testsuite with a remote host).

Therefore, care must be taken to check for the right kind of remote.
Use [is_remote target] to check whether the DejaGnu target board is
remote.  When what you really want to know is whether GDB is using the
remote protocol, because feature X is only available when GDB debugs
natively, check gdb_protocol instead.