gdb/testsuite/README

   1 This is a collection of tests for GDB.
   2
   3 The file gdb/README contains basic instructions on how to run the
   4 testsuite, while this file documents additional options and controls
   5 that are available.  The GDB wiki may also have some pages with ideas
   6 and suggestions.
   7
   8
   9 Running the Testsuite
  10 *********************
  11
  12 There are two ways to run the testsuite and pass additional parameters
  13 to DejaGnu.  The first is to do `make check' in the main build
  14 directory and specifying the makefile variable `RUNTESTFLAGS':
  15
  16          make check RUNTESTFLAGS='TRANSCRIPT=y gdb.base/a2-run.exp'
  17
  18 The second is to cd to the testsuite directory and invoke the DejaGnu
  19 `runtest' command directly.
  20
  21         cd testsuite
  22         make site.exp
  23         runtest TRANSCRIPT=y
  24
  25 (The `site.exp' file contains a handful of useful variables like host
  26 and target triplets, and pathnames.)
  27
  28 Testsuite Parameters
  29 ********************
  30
  31 The following parameters are DejaGNU variables that you can set to
  32 affect the testsuite run globally.
  33
  34 TRANSCRIPT
  35
  36 You may find it useful to have a transcript of the commands that the
  37 testsuite sends to GDB, for instance if GDB crashes during the run,
  38 and you want to reconstruct the sequence of commands.
  39
  40 If the DejaGNU variable TRANSCRIPT is set (to any value), each
  41 invocation of GDB during the test run will get a transcript file
  42 written into the DejaGNU output directory.  The file will have the
  43 name transcript.<n>, where <n> is an integer.  The first line of the
  44 file shows the invocation command with all the options passed to it,
  45 while subsequent lines are the GDB commands.  A `make check' might
  46 look like this:
  47
  48       make check RUNTESTFLAGS=TRANSCRIPT=y
  49
  50 The transcript may not be complete, as for instance tests of command
  51 completion may show only partial command lines.
  52
  53 GDB
  54
  55 By default, the testsuite exercises the GDB in the build directory,
  56 but you can set GDB to be a pathname to a different version.  For
  57 instance,
  58
  59     make check RUNTESTFLAGS=GDB=/usr/bin/gdb
  60
  61 runs the testsuite on the GDB in /usr/bin.
  62
  63 GDBSERVER
  64
  65 You can set GDBSERVER to be a particular GDBserver of interest, so for
  66 instance
  67
  68     make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"
  69
  70 checks both the installed GDB and GDBserver.
  71
  72 INTERNAL_GDBFLAGS
  73
  74 Command line options passed to all GDB invocations.
  75
  76 The default is "-nw -nx".
  77
  78 `-nw' disables any of the windowed interfaces.
  79 `-nx' disables ~/.gdbinit, so that it doesn't interfere with
  80 the tests.
  81
  82 This is actually considered an internal variable, and you
  83 won't normally want to change it.  However, in some situations,
  84 this may be tweaked as a last resort if the testsuite doesn't
  85 have direct support for the specifics of your environment.
  86 The testsuite does not override a value provided by the user.
  87
  88 As an example, when testing an installed GDB that has been
  89 configured with `--with-system-gdbinit', like by default,
  90 you do not want ~/.gdbinit to interfere with tests, but, you
  91 may want the system .gdbinit file loaded.  As there's no way to
  92 ask the testsuite, or GDB, to load the system gdbinit but
  93 not ~/.gdbinit, a workaround is then to remove `-nx' from
  94 INTERNAL_GDBFLAGS, and point $HOME at a directory without
  95 a .gdbinit.  For example:
  96
  97         cd testsuite
  98         HOME=`pwd` runtest \
  99           GDB=/usr/bin/gdb \
 100           GDBSERVER=/usr/bin/gdbserver \
 101           INTERNAL_GDBFLAGS=-nw
 102
 103 GDB_PARALLEL
 104
 105 When testing natively (that is, not with a remote host), you can run
 106 the GDB test suite in a fully parallel mode.  In this mode, each .exp
 107 file runs separately and maybe simultaneously.  The test suite will
 108 ensure that all the temporary files created by the test suite do not
 109 clash, by putting them into separate directories.  This mode is
 110 primarily intended for use by the Makefile.
 111
 112 To use this mode, set the GDB_PARALLEL on the runtest command line.
 113 Before starting the tests, you must ensure that the directories cache,
 114 outputs, and temp in the test suite build directory are either empty
 115 or have been deleted.  cache in particular is used to share data
 116 across invocations of runtest, and files there may affect the test
 117 results.  Note that the Makefile automatically does these deletions.
 118
 119 GDB_INOTIFY
 120
 121 For debugging parallel mode, it is handy to be able to see when a test
 122 case writes to a file outside of its designated output directory.
 123
 124 If you have the inotify-tools package installed, you can set the
 125 GDB_INOTIFY variable on the runtest command line.  This will cause the
 126 test suite to watch for parallel-unsafe file creations and report
 127 them, both to stdout and in the test suite log file.
 128
 129 This setting is only meaningful in conjunction with GDB_PARALLEL.
 130
 131
 132 Testsuite Configuration
 133 ***********************
 134
 135 It is possible to adjust the behavior of the testsuite by defining
 136 the global variables listed below, either in a `site.exp' file,
 137 or in a board file.
 138
 139 gdb_test_timeout
 140
 141 Defining this variable changes the default timeout duration used
 142 during communication with GDB.  More specifically, the global variable
 143 used during testing is `timeout', but this variable gets reset to
 144 `gdb_test_timeout' at the beginning of each testcase, which ensures
 145 that any local change to `timeout' in a testcase does not affect
 146 subsequent testcases.
 147
 148 This global variable comes in handy when the debugger is slower than
 149 normal due to the testing environment, triggering unexpected `TIMEOUT'
 150 test failures.  Examples include when testing on a remote machine, or
 151 against a system where communications are slow.
 152
 153 If not specifically defined, this variable gets automatically defined
 154 to the same value as `timeout' during the testsuite initialization.
 155 The default value of the timeout is defined in the file
 156 `testsuite/config/unix.exp' (at least for Unix hosts; board files may
 157 have their own values).
 158
 159
 160 Board Settings
 161 **************
 162
 163 DejaGNU includes the concept of a "board file", which specifies
 164 testing details for a particular target (which are often bare circuit
 165 boards, thus the name).
 166
 167 In the GDB testsuite specifically, the board file may include a
 168 number of "board settings" that test cases may check before deciding
 169 whether to exercise a particular feature.  For instance, a board
 170 lacking any I/O devices, or perhaps simply having its I/O devices
 171 not wired up, should set `noinferiorio'.
 172
 173 Here are the supported board settings:
 174
 175 gdb,cannot_call_functions
 176
 177   The board does not support inferior call, that is, invoking inferior
 178   functions in GDB.
 179
 180 gdb,can_reverse
 181
 182   The board supports reverse execution.
 183
 184 gdb,no_hardware_watchpoints
 185
 186   The board does not support hardware watchpoints.
 187
 188 gdb,nofileio
 189
 190   GDB is unable to intercept target file operations in remote and
 191   perform them on the host.
 192
 193 gdb,noinferiorio
 194
 195   The board is unable to provide I/O capability to the inferior.
 196
 197 gdb,noresults
 198
 199   A program will not return an exit code or result code (or the value
 200   of the result is undefined, and should not be looked at).
 201
 202 gdb,nosignals
 203
 204   The board does not support signals.
 205
 206 gdb,skip_huge_test
 207
 208   Skip time-consuming tests on the board with slow connection.
 209
 210 gdb,skip_float_tests
 211
 212   Skip tests related to floating point.
 213
 214 gdb,use_precord
 215
 216   The board supports process record.
 217
 218 gdb_server_prog
 219
 220   The location of GDBserver.  If GDBserver somewhere other than its
 221   default location is used in test, specify the location of GDBserver in
 222   this variable.  The location is a file name for GDBserver, and may be
 223   either absolute or relative to the testsuite subdirectory of the build
 224   directory.
 225
 226 in_proc_agent
 227
 228   The location of the in-process agent (used for fast tracepoints and
 229   other special tests).  If the in-process agent of interest is anywhere
 230   other than its default location, set this variable.  The location is a
 231   filename, and may be either absolute or relative to the testsuite
 232   subdirectory of the build directory.
 233
 234 noargs
 235
 236   GDB does not support argument passing for inferior.
 237
 238 no_long_long
 239
 240   The board does not support type long long.
 241
 242 use_cygmon
 243
 244   The board is running the monitor Cygmon.
 245
 246 use_gdb_stub
 247
 248   The tests are running with a GDB stub.
 249
 250 exit_is_reliable
 251
 252   Set to true if GDB can assume that letting the program run to end
 253   reliably results in program exits being reported as such, as opposed
 254   to, e.g., the program ending in an infinite loop or the board
 255   crashing/resetting.  If not set, this defaults to $use_gdb_stub.  In
 256   other words, native targets are assumed reliable by default, and
 257   remote stubs assumed unreliable.
 258
 259 gdb,predefined_tsv
 260
 261   The predefined trace state variables the board has.
 262
 263
 264 Testsuite Organization
 265 **********************
 266
 267 The testsuite is entirely contained in `gdb/testsuite'.  The main
 268 directory of the testsuite includes some makefiles and configury, but
 269 these are minimal, and used for little besides cleaning up, since the
 270 tests themselves handle the compilation of the programs that GDB will
 271 run.
 272
 273 The file `testsuite/lib/gdb.exp' contains common utility procs useful
 274 for all GDB tests, while the directory testsuite/config contains
 275 configuration-specific files, typically used for special-purpose
 276 definitions of procs like `gdb_load' and `gdb_start'.
 277
 278 The tests themselves are to be found in directories named
 279 'testsuite/gdb.* and subdirectories of those.  The names of the test
 280 files must always end with ".exp".  DejaGNU collects the test files by
 281 wildcarding in the test directories, so both subdirectories and
 282 individual files typically get chosen and run in alphabetical order.
 283
 284 The following lists some notable types of subdirectories and what they
 285 are for.  Since DejaGNU finds test files no matter where they are
 286 located, and since each test file sets up its own compilation and
 287 execution environment, this organization is simply for convenience and
 288 intelligibility.
 289
 290 gdb.base
 291
 292 This is the base testsuite.  The tests in it should apply to all
 293 configurations of GDB (but generic native-only tests may live here).
 294 The test programs should be in the subset of C that is both valid
 295 ANSI/ISO C, and C++.
 296
 297 gdb.<lang>
 298
 299 Language-specific tests for any language besides C.  Examples are
 300 gdb.cp for C++ and gdb.java for Java.
 301
 302 gdb.<platform>
 303
 304 Non-portable tests.  The tests are specific to a specific
 305 configuration (host or target), such as HP-UX or eCos.  Example is
 306 gdb.hp, for HP-UX.
 307
 308 gdb.arch
 309
 310 Architecture-specific tests that are (usually) cross-platform.
 311
 312 gdb.<subsystem>
 313
 314 Tests that exercise a specific GDB subsystem in more depth.  For
 315 instance, gdb.disasm exercises various disassemblers, while
 316 gdb.stabs tests pathways through the stabs symbol reader.
 317
 318 Writing Tests
 319 *************
 320
 321 In many areas, the GDB tests are already quite comprehensive; you
 322 should be able to copy existing tests to handle new cases.  Be aware
 323 that older tests may use obsolete practices but have not yet been
 324 updated.
 325
 326 You should try to use `gdb_test' whenever possible, since it includes
 327 cases to handle all the unexpected errors that might happen.  However,
 328 it doesn't cost anything to add new test procedures; for instance,
 329 gdb.base/exprs.exp defines a `test_expr' that calls `gdb_test'
 330 multiple times.
 331
 332 Only use `send_gdb' and `gdb_expect' when absolutely necessary.  Even
 333 if GDB has several valid responses to a command, you can use
 334 `gdb_test_multiple'.  Like `gdb_test', `gdb_test_multiple' recognizes
 335 internal errors and unexpected prompts.
 336
 337 Do not write tests which expect a literal tab character from GDB.  On
 338 some operating systems (e.g. OpenBSD) the TTY layer expands tabs to
 339 spaces, so by the time GDB's output reaches `expect' the tab is gone.
 340
 341 The source language programs do *not* need to be in a consistent
 342 style.  Since GDB is used to debug programs written in many different
 343 styles, it's worth having a mix of styles in the testsuite; for
 344 instance, some GDB bugs involving the display of source lines might
 345 never manifest themselves if the test programs used GNU coding style
 346 uniformly.
 347
 348 Some testcase results need more detailed explanation:
 349
 350 KFAIL
 351
 352 Use KFAIL for known problem of GDB itself.  You must specify the GDB
 353 bug report number, as in these sample tests:
 354
 355         kfail "gdb/13392" "continue to marker 2"
 356
 357 or
 358
 359         setup_kfail gdb/13392 "*-*-*"
 360         kfail "continue to marker 2"
 361
 362
 363 XFAIL
 364
 365 Short for "expected failure", this indicates a known problem with the
 366 environment.  This could include limitations of the operating system,
 367 compiler version, and other components.
 368
 369 This example from gdb.base/attach-pie-misread.exp is a sanity check
 370 for the target environment:
 371
 372         # On x86_64 it is commonly about 4MB.
 373         if {$stub_size > 25000000} {
 374             xfail "stub size $stub_size is too large"
 375             return
 376         }
 377
 378 You should provide bug report number for the failing component of the
 379 environment, if such bug report is available, as with this example
 380 referring to a GCC problem:
 381
 382           if {[test_compiler_info {gcc-[0-3]-*}]
 383               || [test_compiler_info {gcc-4-[0-5]-*}]} {
 384               setup_xfail "gcc/46955" *-*-*
 385           }
 386           gdb_test "python print ttype.template_argument(2)" "&C::c"
 387
 388 Note that it is also acceptable, and often preferable, to avoid
 389 running the test at all.  This is the better option if the limitation
 390 is intrinsic to the environment, rather than a bug expected to be
 391 fixed in the near future.