Commit | Line | Data |
---|---|---|
b866c52d SS |
1 | This is a collection of tests for GDB. |
2 | ||
3 | The file gdb/README contains basic instructions on how to run the | |
4 | testsuite, while this file documents additional options and controls | |
5 | that are available. The GDB wiki may also have some pages with ideas | |
6 | and suggestions. | |
7 | ||
8 | ||
9 | Running the Testsuite | |
10 | ********************* | |
11 | ||
12 | There are two ways to run the testsuite and pass additional parameters | |
13 | to DejaGnu. The first is to do `make check' in the main build | |
14 | directory and specifying the makefile variable `RUNTESTFLAGS': | |
15 | ||
16 | make check RUNTESTFLAGS='TRANSCRIPT=y gdb.base/a2-run.exp' | |
17 | ||
18 | The second is to cd to the testsuite directory and invoke the DejaGnu | |
19 | `runtest' command directly. | |
20 | ||
21 | cd testsuite | |
22 | make site.exp | |
23 | runtest TRANSCRIPT=y | |
24 | ||
25 | (The `site.exp' file contains a handful of useful variables like host | |
26 | and target triplets, and pathnames.) | |
27 | ||
71c0c615 YQ |
28 | Running the Performance Tests |
29 | ***************************** | |
30 | ||
31 | GDB Testsuite includes performance test cases, which are not run together | |
32 | with other test cases, because performance test cases are slow and need | |
33 | a quiet system. There are two ways to run the performance test cases. | |
34 | The first is to do `make check-perf' in the main build directory: | |
35 | ||
36 | make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8" | |
37 | ||
38 | The second is to cd to the testsuite directory and invoke the DejaGnu | |
39 | `runtest' command directly. | |
40 | ||
41 | cd testsuite | |
42 | make site.exp | |
43 | runtest GDB_PERFTEST_MODE=both GDB_PERFTEST_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8 | |
44 | ||
45 | Only "compile", "run" and "both" are valid to GDB_PERFTEST_MODE. They | |
46 | stand for "compile tests only", "run tests only", and "compile and run | |
47 | tests" respectively. "both" is the default. GDB_PERFTEST_TIMEOUT | |
48 | specify the timeout, which is 3000 in default. The result of | |
49 | performance test is appended in `testsuite/perftest.log'. | |
50 | ||
b866c52d SS |
51 | Testsuite Parameters |
52 | ******************** | |
53 | ||
54 | The following parameters are DejaGNU variables that you can set to | |
55 | affect the testsuite run globally. | |
56 | ||
57 | TRANSCRIPT | |
58 | ||
59 | You may find it useful to have a transcript of the commands that the | |
60 | testsuite sends to GDB, for instance if GDB crashes during the run, | |
61 | and you want to reconstruct the sequence of commands. | |
62 | ||
63 | If the DejaGNU variable TRANSCRIPT is set (to any value), each | |
64 | invocation of GDB during the test run will get a transcript file | |
65 | written into the DejaGNU output directory. The file will have the | |
66 | name transcript.<n>, where <n> is an integer. The first line of the | |
67 | file shows the invocation command with all the options passed to it, | |
68 | while subsequent lines are the GDB commands. A `make check' might | |
69 | look like this: | |
70 | ||
71 | make check RUNTESTFLAGS=TRANSCRIPT=y | |
72 | ||
73 | The transcript may not be complete, as for instance tests of command | |
74 | completion may show only partial command lines. | |
75 | ||
76 | GDB | |
77 | ||
78 | By default, the testsuite exercises the GDB in the build directory, | |
79 | but you can set GDB to be a pathname to a different version. For | |
80 | instance, | |
81 | ||
82 | make check RUNTESTFLAGS=GDB=/usr/bin/gdb | |
83 | ||
84 | runs the testsuite on the GDB in /usr/bin. | |
85 | ||
86 | GDBSERVER | |
87 | ||
88 | You can set GDBSERVER to be a particular GDBserver of interest, so for | |
89 | instance | |
90 | ||
91 | make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver" | |
92 | ||
93 | checks both the installed GDB and GDBserver. | |
94 | ||
95 | INTERNAL_GDBFLAGS | |
96 | ||
97 | Command line options passed to all GDB invocations. | |
98 | ||
99 | The default is "-nw -nx". | |
100 | ||
101 | `-nw' disables any of the windowed interfaces. | |
102 | `-nx' disables ~/.gdbinit, so that it doesn't interfere with | |
103 | the tests. | |
104 | ||
105 | This is actually considered an internal variable, and you | |
106 | won't normally want to change it. However, in some situations, | |
107 | this may be tweaked as a last resort if the testsuite doesn't | |
108 | have direct support for the specifics of your environment. | |
109 | The testsuite does not override a value provided by the user. | |
110 | ||
111 | As an example, when testing an installed GDB that has been | |
112 | configured with `--with-system-gdbinit', like by default, | |
113 | you do not want ~/.gdbinit to interfere with tests, but, you | |
114 | may want the system .gdbinit file loaded. As there's no way to | |
115 | ask the testsuite, or GDB, to load the system gdbinit but | |
116 | not ~/.gdbinit, a workaround is then to remove `-nx' from | |
117 | INTERNAL_GDBFLAGS, and point $HOME at a directory without | |
118 | a .gdbinit. For example: | |
119 | ||
120 | cd testsuite | |
121 | HOME=`pwd` runtest \ | |
122 | GDB=/usr/bin/gdb \ | |
123 | GDBSERVER=/usr/bin/gdbserver \ | |
124 | INTERNAL_GDBFLAGS=-nw | |
125 | ||
126 | GDB_PARALLEL | |
127 | ||
128 | When testing natively (that is, not with a remote host), you can run | |
129 | the GDB test suite in a fully parallel mode. In this mode, each .exp | |
130 | file runs separately and maybe simultaneously. The test suite will | |
131 | ensure that all the temporary files created by the test suite do not | |
132 | clash, by putting them into separate directories. This mode is | |
133 | primarily intended for use by the Makefile. | |
134 | ||
135 | To use this mode, set the GDB_PARALLEL on the runtest command line. | |
136 | Before starting the tests, you must ensure that the directories cache, | |
137 | outputs, and temp in the test suite build directory are either empty | |
138 | or have been deleted. cache in particular is used to share data | |
139 | across invocations of runtest, and files there may affect the test | |
140 | results. Note that the Makefile automatically does these deletions. | |
141 | ||
142 | GDB_INOTIFY | |
143 | ||
144 | For debugging parallel mode, it is handy to be able to see when a test | |
145 | case writes to a file outside of its designated output directory. | |
146 | ||
147 | If you have the inotify-tools package installed, you can set the | |
148 | GDB_INOTIFY variable on the runtest command line. This will cause the | |
149 | test suite to watch for parallel-unsafe file creations and report | |
150 | them, both to stdout and in the test suite log file. | |
151 | ||
152 | This setting is only meaningful in conjunction with GDB_PARALLEL. | |
153 | ||
c17ef0d5 DE |
154 | TESTS |
155 | ||
156 | This variable is used to specify which set of tests to run. | |
157 | It is passed to make (not runtest) and its contents are a space separated | |
158 | list of tests to run. | |
159 | ||
160 | If using GNU make then the contents are wildcard-expanded using | |
161 | GNU make's $(wildcard) function. Test paths must be fully specified, | |
162 | relative to the "testsuite" subdirectory. This allows one to run all | |
163 | tests in a subdirectory by passing "gdb.subdir/*.exp". | |
164 | If for some strange reason one wanted to run all tests that begin with | |
165 | the letter "d" that is also possible: TESTS="*/d*.exp". | |
166 | ||
167 | Do not write */*.exp to specify all tests (assuming all tests are only | |
168 | nested one level deep, which is not necessarily true). This will pick up | |
169 | .exp files in ancillary directories like "lib" and "config". | |
170 | Instead write gdb.*/*.exp. | |
171 | ||
172 | Example: | |
173 | ||
174 | make -j10 check TESTS="gdb.server/[s-w]*.exp */x*.exp" | |
175 | ||
176 | If not using GNU make then the value is passed directly to runtest. | |
177 | If not specified, all tests are run. | |
b866c52d SS |
178 | |
179 | Testsuite Configuration | |
180 | *********************** | |
181 | ||
182 | It is possible to adjust the behavior of the testsuite by defining | |
183 | the global variables listed below, either in a `site.exp' file, | |
184 | or in a board file. | |
185 | ||
186 | gdb_test_timeout | |
187 | ||
188 | Defining this variable changes the default timeout duration used | |
189 | during communication with GDB. More specifically, the global variable | |
190 | used during testing is `timeout', but this variable gets reset to | |
191 | `gdb_test_timeout' at the beginning of each testcase, which ensures | |
192 | that any local change to `timeout' in a testcase does not affect | |
193 | subsequent testcases. | |
194 | ||
195 | This global variable comes in handy when the debugger is slower than | |
196 | normal due to the testing environment, triggering unexpected `TIMEOUT' | |
197 | test failures. Examples include when testing on a remote machine, or | |
198 | against a system where communications are slow. | |
199 | ||
200 | If not specifically defined, this variable gets automatically defined | |
201 | to the same value as `timeout' during the testsuite initialization. | |
202 | The default value of the timeout is defined in the file | |
203 | `testsuite/config/unix.exp' (at least for Unix hosts; board files may | |
204 | have their own values). | |
205 | ||
206 | ||
207 | Board Settings | |
208 | ************** | |
209 | ||
210 | DejaGNU includes the concept of a "board file", which specifies | |
211 | testing details for a particular target (which are often bare circuit | |
212 | boards, thus the name). | |
213 | ||
214 | In the GDB testsuite specifically, the board file may include a | |
215 | number of "board settings" that test cases may check before deciding | |
216 | whether to exercise a particular feature. For instance, a board | |
217 | lacking any I/O devices, or perhaps simply having its I/O devices | |
218 | not wired up, should set `noinferiorio'. | |
219 | ||
220 | Here are the supported board settings: | |
221 | ||
222 | gdb,cannot_call_functions | |
223 | ||
224 | The board does not support inferior call, that is, invoking inferior | |
225 | functions in GDB. | |
226 | ||
227 | gdb,can_reverse | |
228 | ||
229 | The board supports reverse execution. | |
230 | ||
231 | gdb,no_hardware_watchpoints | |
232 | ||
233 | The board does not support hardware watchpoints. | |
234 | ||
235 | gdb,nofileio | |
236 | ||
237 | GDB is unable to intercept target file operations in remote and | |
238 | perform them on the host. | |
239 | ||
240 | gdb,noinferiorio | |
241 | ||
242 | The board is unable to provide I/O capability to the inferior. | |
243 | ||
244 | gdb,noresults | |
245 | ||
246 | A program will not return an exit code or result code (or the value | |
247 | of the result is undefined, and should not be looked at). | |
248 | ||
249 | gdb,nosignals | |
250 | ||
251 | The board does not support signals. | |
252 | ||
253 | gdb,skip_huge_test | |
254 | ||
255 | Skip time-consuming tests on the board with slow connection. | |
256 | ||
257 | gdb,skip_float_tests | |
258 | ||
259 | Skip tests related to floating point. | |
260 | ||
261 | gdb,use_precord | |
262 | ||
263 | The board supports process record. | |
264 | ||
265 | gdb_server_prog | |
266 | ||
267 | The location of GDBserver. If GDBserver somewhere other than its | |
268 | default location is used in test, specify the location of GDBserver in | |
269 | this variable. The location is a file name for GDBserver, and may be | |
270 | either absolute or relative to the testsuite subdirectory of the build | |
271 | directory. | |
272 | ||
273 | in_proc_agent | |
274 | ||
275 | The location of the in-process agent (used for fast tracepoints and | |
276 | other special tests). If the in-process agent of interest is anywhere | |
277 | other than its default location, set this variable. The location is a | |
278 | filename, and may be either absolute or relative to the testsuite | |
279 | subdirectory of the build directory. | |
280 | ||
281 | noargs | |
282 | ||
283 | GDB does not support argument passing for inferior. | |
284 | ||
285 | no_long_long | |
286 | ||
287 | The board does not support type long long. | |
288 | ||
289 | use_cygmon | |
290 | ||
291 | The board is running the monitor Cygmon. | |
292 | ||
293 | use_gdb_stub | |
294 | ||
295 | The tests are running with a GDB stub. | |
296 | ||
b477a5e6 PA |
297 | exit_is_reliable |
298 | ||
299 | Set to true if GDB can assume that letting the program run to end | |
300 | reliably results in program exits being reported as such, as opposed | |
301 | to, e.g., the program ending in an infinite loop or the board | |
302 | crashing/resetting. If not set, this defaults to $use_gdb_stub. In | |
303 | other words, native targets are assumed reliable by default, and | |
304 | remote stubs assumed unreliable. | |
305 | ||
b866c52d SS |
306 | gdb,predefined_tsv |
307 | ||
308 | The predefined trace state variables the board has. | |
309 | ||
310 | ||
311 | Testsuite Organization | |
312 | ********************** | |
313 | ||
314 | The testsuite is entirely contained in `gdb/testsuite'. The main | |
315 | directory of the testsuite includes some makefiles and configury, but | |
316 | these are minimal, and used for little besides cleaning up, since the | |
317 | tests themselves handle the compilation of the programs that GDB will | |
318 | run. | |
319 | ||
320 | The file `testsuite/lib/gdb.exp' contains common utility procs useful | |
321 | for all GDB tests, while the directory testsuite/config contains | |
322 | configuration-specific files, typically used for special-purpose | |
323 | definitions of procs like `gdb_load' and `gdb_start'. | |
324 | ||
325 | The tests themselves are to be found in directories named | |
326 | 'testsuite/gdb.* and subdirectories of those. The names of the test | |
327 | files must always end with ".exp". DejaGNU collects the test files by | |
328 | wildcarding in the test directories, so both subdirectories and | |
329 | individual files typically get chosen and run in alphabetical order. | |
330 | ||
331 | The following lists some notable types of subdirectories and what they | |
332 | are for. Since DejaGNU finds test files no matter where they are | |
333 | located, and since each test file sets up its own compilation and | |
334 | execution environment, this organization is simply for convenience and | |
335 | intelligibility. | |
336 | ||
337 | gdb.base | |
338 | ||
339 | This is the base testsuite. The tests in it should apply to all | |
340 | configurations of GDB (but generic native-only tests may live here). | |
341 | The test programs should be in the subset of C that is both valid | |
342 | ANSI/ISO C, and C++. | |
343 | ||
344 | gdb.<lang> | |
345 | ||
346 | Language-specific tests for any language besides C. Examples are | |
347 | gdb.cp for C++ and gdb.java for Java. | |
348 | ||
349 | gdb.<platform> | |
350 | ||
351 | Non-portable tests. The tests are specific to a specific | |
352 | configuration (host or target), such as HP-UX or eCos. Example is | |
353 | gdb.hp, for HP-UX. | |
354 | ||
355 | gdb.arch | |
356 | ||
357 | Architecture-specific tests that are (usually) cross-platform. | |
358 | ||
359 | gdb.<subsystem> | |
360 | ||
361 | Tests that exercise a specific GDB subsystem in more depth. For | |
362 | instance, gdb.disasm exercises various disassemblers, while | |
363 | gdb.stabs tests pathways through the stabs symbol reader. | |
364 | ||
71c0c615 YQ |
365 | gdb.perf |
366 | ||
367 | GDB performance tests. | |
368 | ||
b866c52d SS |
369 | Writing Tests |
370 | ************* | |
371 | ||
372 | In many areas, the GDB tests are already quite comprehensive; you | |
373 | should be able to copy existing tests to handle new cases. Be aware | |
374 | that older tests may use obsolete practices but have not yet been | |
375 | updated. | |
376 | ||
377 | You should try to use `gdb_test' whenever possible, since it includes | |
378 | cases to handle all the unexpected errors that might happen. However, | |
379 | it doesn't cost anything to add new test procedures; for instance, | |
380 | gdb.base/exprs.exp defines a `test_expr' that calls `gdb_test' | |
381 | multiple times. | |
382 | ||
383 | Only use `send_gdb' and `gdb_expect' when absolutely necessary. Even | |
384 | if GDB has several valid responses to a command, you can use | |
385 | `gdb_test_multiple'. Like `gdb_test', `gdb_test_multiple' recognizes | |
386 | internal errors and unexpected prompts. | |
387 | ||
388 | Do not write tests which expect a literal tab character from GDB. On | |
389 | some operating systems (e.g. OpenBSD) the TTY layer expands tabs to | |
390 | spaces, so by the time GDB's output reaches `expect' the tab is gone. | |
391 | ||
392 | The source language programs do *not* need to be in a consistent | |
393 | style. Since GDB is used to debug programs written in many different | |
394 | styles, it's worth having a mix of styles in the testsuite; for | |
395 | instance, some GDB bugs involving the display of source lines might | |
396 | never manifest themselves if the test programs used GNU coding style | |
397 | uniformly. | |
398 | ||
399 | Some testcase results need more detailed explanation: | |
400 | ||
401 | KFAIL | |
402 | ||
403 | Use KFAIL for known problem of GDB itself. You must specify the GDB | |
404 | bug report number, as in these sample tests: | |
405 | ||
406 | kfail "gdb/13392" "continue to marker 2" | |
407 | ||
408 | or | |
409 | ||
410 | setup_kfail gdb/13392 "*-*-*" | |
411 | kfail "continue to marker 2" | |
412 | ||
413 | ||
414 | XFAIL | |
415 | ||
416 | Short for "expected failure", this indicates a known problem with the | |
417 | environment. This could include limitations of the operating system, | |
418 | compiler version, and other components. | |
419 | ||
420 | This example from gdb.base/attach-pie-misread.exp is a sanity check | |
421 | for the target environment: | |
422 | ||
423 | # On x86_64 it is commonly about 4MB. | |
424 | if {$stub_size > 25000000} { | |
425 | xfail "stub size $stub_size is too large" | |
426 | return | |
427 | } | |
428 | ||
429 | You should provide bug report number for the failing component of the | |
430 | environment, if such bug report is available, as with this example | |
431 | referring to a GCC problem: | |
432 | ||
433 | if {[test_compiler_info {gcc-[0-3]-*}] | |
434 | || [test_compiler_info {gcc-4-[0-5]-*}]} { | |
435 | setup_xfail "gcc/46955" *-*-* | |
436 | } | |
437 | gdb_test "python print ttype.template_argument(2)" "&C::c" | |
438 | ||
439 | Note that it is also acceptable, and often preferable, to avoid | |
440 | running the test at all. This is the better option if the limitation | |
441 | is intrinsic to the environment, rather than a bug expected to be | |
442 | fixed in the near future. |