gdb/testsuite/README

   1 This is a collection of tests for GDB.
   2
   3 The file gdb/README contains basic instructions on how to run the
   4 testsuite, while this file documents additional options and controls
   5 that are available.  The GDB wiki may also have some pages with ideas
   6 and suggestions.
   7
   8
   9 Running the Testsuite
  10 *********************
  11
  12 There are two ways to run the testsuite and pass additional parameters
  13 to DejaGnu.  The first is to do `make check' in the main build
  14 directory and specifying the makefile variable `RUNTESTFLAGS':
  15
  16          make check RUNTESTFLAGS='TRANSCRIPT=y gdb.base/a2-run.exp'
  17
  18 The second is to cd to the testsuite directory and invoke the DejaGnu
  19 `runtest' command directly.
  20
  21         cd testsuite
  22         make site.exp
  23         runtest TRANSCRIPT=y
  24
  25 (The `site.exp' file contains a handful of useful variables like host
  26 and target triplets, and pathnames.)
  27
  28 Testsuite Parameters
  29 ********************
  30
  31 The following parameters are DejaGNU variables that you can set to
  32 affect the testsuite run globally.
  33
  34 TRANSCRIPT
  35
  36 You may find it useful to have a transcript of the commands that the
  37 testsuite sends to GDB, for instance if GDB crashes during the run,
  38 and you want to reconstruct the sequence of commands.
  39
  40 If the DejaGNU variable TRANSCRIPT is set (to any value), each
  41 invocation of GDB during the test run will get a transcript file
  42 written into the DejaGNU output directory.  The file will have the
  43 name transcript.<n>, where <n> is an integer.  The first line of the
  44 file shows the invocation command with all the options passed to it,
  45 while subsequent lines are the GDB commands.  A `make check' might
  46 look like this:
  47
  48       make check RUNTESTFLAGS=TRANSCRIPT=y
  49
  50 The transcript may not be complete, as for instance tests of command
  51 completion may show only partial command lines.
  52
  53 GDB
  54
  55 By default, the testsuite exercises the GDB in the build directory,
  56 but you can set GDB to be a pathname to a different version.  For
  57 instance,
  58
  59     make check RUNTESTFLAGS=GDB=/usr/bin/gdb
  60
  61 runs the testsuite on the GDB in /usr/bin.
  62
  63 GDBSERVER
  64
  65 You can set GDBSERVER to be a particular GDBserver of interest, so for
  66 instance
  67
  68     make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"
  69
  70 checks both the installed GDB and GDBserver.
  71
  72 INTERNAL_GDBFLAGS
  73
  74 Command line options passed to all GDB invocations.
  75
  76 The default is "-nw -nx".
  77
  78 `-nw' disables any of the windowed interfaces.
  79 `-nx' disables ~/.gdbinit, so that it doesn't interfere with
  80 the tests.
  81
  82 This is actually considered an internal variable, and you
  83 won't normally want to change it.  However, in some situations,
  84 this may be tweaked as a last resort if the testsuite doesn't
  85 have direct support for the specifics of your environment.
  86 The testsuite does not override a value provided by the user.
  87
  88 As an example, when testing an installed GDB that has been
  89 configured with `--with-system-gdbinit', like by default,
  90 you do not want ~/.gdbinit to interfere with tests, but, you
  91 may want the system .gdbinit file loaded.  As there's no way to
  92 ask the testsuite, or GDB, to load the system gdbinit but
  93 not ~/.gdbinit, a workaround is then to remove `-nx' from
  94 INTERNAL_GDBFLAGS, and point $HOME at a directory without
  95 a .gdbinit.  For example:
  96
  97         cd testsuite
  98         HOME=`pwd` runtest \
  99           GDB=/usr/bin/gdb \
 100           GDBSERVER=/usr/bin/gdbserver \
 101           INTERNAL_GDBFLAGS=-nw
 102
 103 GDB_PARALLEL
 104
 105 When testing natively (that is, not with a remote host), you can run
 106 the GDB test suite in a fully parallel mode.  In this mode, each .exp
 107 file runs separately and maybe simultaneously.  The test suite will
 108 ensure that all the temporary files created by the test suite do not
 109 clash, by putting them into separate directories.  This mode is
 110 primarily intended for use by the Makefile.
 111
 112 To use this mode, set the GDB_PARALLEL on the runtest command line.
 113 Before starting the tests, you must ensure that the directories cache,
 114 outputs, and temp in the test suite build directory are either empty
 115 or have been deleted.  cache in particular is used to share data
 116 across invocations of runtest, and files there may affect the test
 117 results.  Note that the Makefile automatically does these deletions.
 118
 119 GDB_INOTIFY
 120
 121 For debugging parallel mode, it is handy to be able to see when a test
 122 case writes to a file outside of its designated output directory.
 123
 124 If you have the inotify-tools package installed, you can set the
 125 GDB_INOTIFY variable on the runtest command line.  This will cause the
 126 test suite to watch for parallel-unsafe file creations and report
 127 them, both to stdout and in the test suite log file.
 128
 129 This setting is only meaningful in conjunction with GDB_PARALLEL.
 130
 131
 132 Testsuite Configuration
 133 ***********************
 134
 135 It is possible to adjust the behavior of the testsuite by defining
 136 the global variables listed below, either in a `site.exp' file,
 137 or in a board file.
 138
 139 gdb_test_timeout
 140
 141 Defining this variable changes the default timeout duration used
 142 during communication with GDB.  More specifically, the global variable
 143 used during testing is `timeout', but this variable gets reset to
 144 `gdb_test_timeout' at the beginning of each testcase, which ensures
 145 that any local change to `timeout' in a testcase does not affect
 146 subsequent testcases.
 147
 148 This global variable comes in handy when the debugger is slower than
 149 normal due to the testing environment, triggering unexpected `TIMEOUT'
 150 test failures.  Examples include when testing on a remote machine, or
 151 against a system where communications are slow.
 152
 153 If not specifically defined, this variable gets automatically defined
 154 to the same value as `timeout' during the testsuite initialization.
 155 The default value of the timeout is defined in the file
 156 `testsuite/config/unix.exp' (at least for Unix hosts; board files may
 157 have their own values).
 158
 159
 160 Board Settings
 161 **************
 162
 163 DejaGNU includes the concept of a "board file", which specifies
 164 testing details for a particular target (which are often bare circuit
 165 boards, thus the name).
 166
 167 In the GDB testsuite specifically, the board file may include a
 168 number of "board settings" that test cases may check before deciding
 169 whether to exercise a particular feature.  For instance, a board
 170 lacking any I/O devices, or perhaps simply having its I/O devices
 171 not wired up, should set `noinferiorio'.
 172
 173 Here are the supported board settings:
 174
 175 gdb,cannot_call_functions
 176
 177   The board does not support inferior call, that is, invoking inferior
 178   functions in GDB.
 179
 180 gdb,can_reverse
 181
 182   The board supports reverse execution.
 183
 184 gdb,no_hardware_watchpoints
 185
 186   The board does not support hardware watchpoints.
 187
 188 gdb,nofileio
 189
 190   GDB is unable to intercept target file operations in remote and
 191   perform them on the host.
 192
 193 gdb,noinferiorio
 194
 195   The board is unable to provide I/O capability to the inferior.
 196
 197 gdb,noresults
 198
 199   A program will not return an exit code or result code (or the value
 200   of the result is undefined, and should not be looked at).
 201
 202 gdb,nosignals
 203
 204   The board does not support signals.
 205
 206 gdb,skip_huge_test
 207
 208   Skip time-consuming tests on the board with slow connection.
 209
 210 gdb,skip_float_tests
 211
 212   Skip tests related to floating point.
 213
 214 gdb,use_precord
 215
 216   The board supports process record.
 217
 218 gdb_server_prog
 219
 220   The location of GDBserver.  If GDBserver somewhere other than its
 221   default location is used in test, specify the location of GDBserver in
 222   this variable.  The location is a file name for GDBserver, and may be
 223   either absolute or relative to the testsuite subdirectory of the build
 224   directory.
 225
 226 in_proc_agent
 227
 228   The location of the in-process agent (used for fast tracepoints and
 229   other special tests).  If the in-process agent of interest is anywhere
 230   other than its default location, set this variable.  The location is a
 231   filename, and may be either absolute or relative to the testsuite
 232   subdirectory of the build directory.
 233
 234 noargs
 235
 236   GDB does not support argument passing for inferior.
 237
 238 no_long_long
 239
 240   The board does not support type long long.
 241
 242 use_cygmon
 243
 244   The board is running the monitor Cygmon.
 245
 246 use_gdb_stub
 247
 248   The tests are running with a GDB stub.
 249
 250 gdb,predefined_tsv
 251
 252   The predefined trace state variables the board has.
 253
 254
 255 Testsuite Organization
 256 **********************
 257
 258 The testsuite is entirely contained in `gdb/testsuite'.  The main
 259 directory of the testsuite includes some makefiles and configury, but
 260 these are minimal, and used for little besides cleaning up, since the
 261 tests themselves handle the compilation of the programs that GDB will
 262 run.
 263
 264 The file `testsuite/lib/gdb.exp' contains common utility procs useful
 265 for all GDB tests, while the directory testsuite/config contains
 266 configuration-specific files, typically used for special-purpose
 267 definitions of procs like `gdb_load' and `gdb_start'.
 268
 269 The tests themselves are to be found in directories named
 270 'testsuite/gdb.* and subdirectories of those.  The names of the test
 271 files must always end with ".exp".  DejaGNU collects the test files by
 272 wildcarding in the test directories, so both subdirectories and
 273 individual files typically get chosen and run in alphabetical order.
 274
 275 The following lists some notable types of subdirectories and what they
 276 are for.  Since DejaGNU finds test files no matter where they are
 277 located, and since each test file sets up its own compilation and
 278 execution environment, this organization is simply for convenience and
 279 intelligibility.
 280
 281 gdb.base
 282
 283 This is the base testsuite.  The tests in it should apply to all
 284 configurations of GDB (but generic native-only tests may live here).
 285 The test programs should be in the subset of C that is both valid
 286 ANSI/ISO C, and C++.
 287
 288 gdb.<lang>
 289
 290 Language-specific tests for any language besides C.  Examples are
 291 gdb.cp for C++ and gdb.java for Java.
 292
 293 gdb.<platform>
 294
 295 Non-portable tests.  The tests are specific to a specific
 296 configuration (host or target), such as HP-UX or eCos.  Example is
 297 gdb.hp, for HP-UX.
 298
 299 gdb.arch
 300
 301 Architecture-specific tests that are (usually) cross-platform.
 302
 303 gdb.<subsystem>
 304
 305 Tests that exercise a specific GDB subsystem in more depth.  For
 306 instance, gdb.disasm exercises various disassemblers, while
 307 gdb.stabs tests pathways through the stabs symbol reader.
 308
 309 Writing Tests
 310 *************
 311
 312 In many areas, the GDB tests are already quite comprehensive; you
 313 should be able to copy existing tests to handle new cases.  Be aware
 314 that older tests may use obsolete practices but have not yet been
 315 updated.
 316
 317 You should try to use `gdb_test' whenever possible, since it includes
 318 cases to handle all the unexpected errors that might happen.  However,
 319 it doesn't cost anything to add new test procedures; for instance,
 320 gdb.base/exprs.exp defines a `test_expr' that calls `gdb_test'
 321 multiple times.
 322
 323 Only use `send_gdb' and `gdb_expect' when absolutely necessary.  Even
 324 if GDB has several valid responses to a command, you can use
 325 `gdb_test_multiple'.  Like `gdb_test', `gdb_test_multiple' recognizes
 326 internal errors and unexpected prompts.
 327
 328 Do not write tests which expect a literal tab character from GDB.  On
 329 some operating systems (e.g. OpenBSD) the TTY layer expands tabs to
 330 spaces, so by the time GDB's output reaches `expect' the tab is gone.
 331
 332 The source language programs do *not* need to be in a consistent
 333 style.  Since GDB is used to debug programs written in many different
 334 styles, it's worth having a mix of styles in the testsuite; for
 335 instance, some GDB bugs involving the display of source lines might
 336 never manifest themselves if the test programs used GNU coding style
 337 uniformly.
 338
 339 Some testcase results need more detailed explanation:
 340
 341 KFAIL
 342
 343 Use KFAIL for known problem of GDB itself.  You must specify the GDB
 344 bug report number, as in these sample tests:
 345
 346         kfail "gdb/13392" "continue to marker 2"
 347
 348 or
 349
 350         setup_kfail gdb/13392 "*-*-*"
 351         kfail "continue to marker 2"
 352
 353
 354 XFAIL
 355
 356 Short for "expected failure", this indicates a known problem with the
 357 environment.  This could include limitations of the operating system,
 358 compiler version, and other components.
 359
 360 This example from gdb.base/attach-pie-misread.exp is a sanity check
 361 for the target environment:
 362
 363         # On x86_64 it is commonly about 4MB.
 364         if {$stub_size > 25000000} {
 365             xfail "stub size $stub_size is too large"
 366             return
 367         }
 368
 369 You should provide bug report number for the failing component of the
 370 environment, if such bug report is available, as with this example
 371 referring to a GCC problem:
 372
 373           if {[test_compiler_info {gcc-[0-3]-*}]
 374               || [test_compiler_info {gcc-4-[0-5]-*}]} {
 375               setup_xfail "gcc/46955" *-*-*
 376           }
 377           gdb_test "python print ttype.template_argument(2)" "&C::c"
 378
 379 Note that it is also acceptable, and often preferable, to avoid
 380 running the test at all.  This is the better option if the limitation
 381 is intrinsic to the environment, rather than a bug expected to be
 382 fixed in the near future.