Babeltrace Debug Info Analysis ----------------------------- The babeltrace debug info analysis is a set of features which allow mapping events from a trace to their location in source code or within a binary file, based on their `ip` (instruction pointer) field. Prerequisites ------------- In order to install a version of babeltrace with debug info support, the following libraries are required: * libelf * libdw Both of them are provided by the elfutils project (https://fedorahosted.org/elfutils/), and can be installed through the elfutils package on Ubuntu, Debian, RHEL, and others. Compiling for Debug Info Analysis --------------------------------- Traced programs on which debug info analysis is to be performed can be compiled in a few different ways, and still lead to useful results. Ideally, one should compile the program in debug mode, which is achieved on gcc by simply using the `-g` flag. This generates debug information in the operating system's native format, which is then used by babeltrace to map an event's source location to a file and line number, and the name of the surrounding function. Do note that only debug information in DWARF format, version 2 or later, is currently supported by babeltrace. Use the `-gdwarf` or `-gdwarf-(VERSION)` to explicitly generate DWARF debug information. If the executable is not compiled with `-g` or an equivalent option enabled, and thus no DWARF information is available, babeltrace will use ELF symbols from the executable. Instead of providing source file, line number and function name, however, the analysis will provide the name of the nearest function symbol, plus an offset in bytes to the location in the executable from which the event originated. If the executable has neither ELF symbols nor DWARF information, babeltrace will be unable to map an event to its source location and will simply display the instruction pointer (address), as in prior versions of babeltrace. Getting the Right Tracer ------------------------ Debug info analysis is performed automatically by babeltrace, provided the trace contains sufficient information. In order to be able to trace all the necessary information, the following software is required: * lttng-ust version 2.8.0 or later * lttng-tools, corresponding version You can get these from source at: * https://github.com/lttng/lttng-ust * https://github.com/lttng/lttng-tools Ubuntu users also have the option of installing via the LTTng daily PPA: * https://launchpad.net/~lttng/+archive/ubuntu/daily Tracing for Debug Info Analysis ------------------------------- Babeltrace needs some extra information from contexts, namely ip and vpid, to perform its analysis. These can be enabled after the creation of a tracing session as follows: $ lttng add-context --userspace --type ip --type vpid The tracing can then be performed as it normally would. Once the trace is collected, it can the be read by babeltrace for analysis. Analysing the Trace ------------------- To perform the analysis, the trace can simply be read as it normally would: $ babeltrace Debug info analysis is on by default and will automatically print the extra source location information if it can find it. A sample output may look like this: [...] [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 } [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9", src = "libhello.c:13" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 } [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c", src = "libhello.c:20" }, vpid = 28719 }, { some_field = 1729 } [...] The interesting part is the debug_info section of the context: debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" } This is the expected output for events generated by an executable for which DWARF information is available. It shows the name of the binary and offset to the tracepoint, the name of the function containing the tracepoint instance which generated the event ("foo") and the offset within the function, and its source location ("libhello.c", line 7). The second event in the sample output is of the same type ("my_first_tracepoint"), but it was generated by a different tracepoint instance, hence the different source location (line 13) and function ("bar"). The third event, of a different type, also shows debug information. If DWARF info is absent, but ELF symbols are not stripped, the output will instead look like this: [...] [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 } [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 } [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c" }, vpid = 28719 }, { some_field = 1729 } [...] The debug information now provides both binary and function location information, but no source location information, as this requires DWARF. The function names are in fact resolved using ELF symbols, so there may be a discrepancy with those provided by DWARF (e.g. in the case of mangling). Paths to the binary and to the source location (if any) can be expanded by using the command-line option --debug-info-full-path. Otherwise, only the filename is shown. Debug Info and Dynamic Loading ------------------------------ Babeltrace can resolve addresses of events originating from dynamically loaded libraries, provided that some extra information is collected at tracing time. This can be achieved by preloading LTTng UST's libdl helper when launching the program to be traced, like so: $ LD_PRELOAD="liblttng-ust-dl.so" The tracing and analysis can now be performed as described in prior sections, and events from tracepoints in dlopened libraries will be resolved automatically by babeltrace. Separate Debug Info ------------------- It is possible to store DWARF debug information separate from an executable, whether for concerns of file size, or simply to facilitate the sharing of the debug information. This is usually achieved via one of two mechanisms, namely build ID and debug link. Both methods permit separate executables and debug information. Their use and operation is described in GDB's documentation at: https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html Babeltrace will find separate debug files automatically, provided they follow the requirements described in the documentation above. The debug information lookup order is the same as GDB's, that is first debug info is looked for within the executable, then through the build ID method in the standard /usr/lib/debug/.build-id/ location, and finally in the various possible debug link locations. The first debug information file found is used. The --debug-info-dir command-line option can be used to override the default /usr/lib/debug/ directory used in build ID and debug link lookups. Multiple debug info directories are currently not supported. Target Prefix ------------- The debug info analysis uses the paths to the executables as collected during tracing as one mechanism to resolve DWARF or ELF information. If the trace was taken on a separate machine, for instance, it is possible to use --debug-info-target-prefix to specify a prefix directory, representing the root of the target filesystem, which will then be used for lookups. For example, if an executable was located at /usr/bin/foo on the target system, it could be placed at /home/efficios/target/usr/bin/foo on the system on which the analysis is performed. In this case, the prefix is /home/efficios/target/.