| 1 | Babeltrace Debug Info Analysis |
| 2 | ----------------------------- |
| 3 | |
| 4 | The babeltrace debug info analysis is a set of features which allow |
| 5 | mapping events from a trace to their location in source code or within |
| 6 | a binary file, based on their `ip` (instruction pointer) field. |
| 7 | |
| 8 | Prerequisites |
| 9 | ------------- |
| 10 | |
| 11 | In order to install a version of babeltrace with debug info support, |
| 12 | the following libraries are required: |
| 13 | |
| 14 | * libelf |
| 15 | * libdw |
| 16 | |
| 17 | Both of them are provided by the elfutils project |
| 18 | (https://fedorahosted.org/elfutils/), and can be installed through the |
| 19 | elfutils package on Ubuntu, Debian, RHEL, and others. |
| 20 | |
| 21 | Compiling for Debug Info Analysis |
| 22 | --------------------------------- |
| 23 | |
| 24 | Traced programs on which debug info analysis is to be performed can be |
| 25 | compiled in a few different ways, and still lead to useful results. |
| 26 | |
| 27 | Ideally, one should compile the program in debug mode, which is |
| 28 | achieved on gcc by simply using the `-g` flag. This generates debug |
| 29 | information in the operating system's native format, which is then |
| 30 | used by babeltrace to map an event's source location to a file and |
| 31 | line number, and the name of the surrounding function. |
| 32 | |
| 33 | Do note that only debug information in DWARF format, version 2 or |
| 34 | later, is currently supported by babeltrace. Use the `-gdwarf` or |
| 35 | `-gdwarf-(VERSION)` to explicitly generate DWARF debug information. |
| 36 | |
| 37 | If the executable is not compiled with `-g` or an equivalent option |
| 38 | enabled, and thus no DWARF information is available, babeltrace will |
| 39 | use ELF symbols from the executable. Instead of providing source file, |
| 40 | line number and function name, however, the analysis will provide the |
| 41 | name of the nearest function symbol, plus an offset in bytes to the |
| 42 | location in the executable from which the event originated. |
| 43 | |
| 44 | If the executable has neither ELF symbols nor DWARF information, |
| 45 | babeltrace will be unable to map an event to its source location and |
| 46 | will simply display the instruction pointer (address), as in prior |
| 47 | versions of babeltrace. |
| 48 | |
| 49 | Getting the Right Tracer |
| 50 | ------------------------ |
| 51 | |
| 52 | Debug info analysis is performed automatically by babeltrace, provided |
| 53 | the trace contains sufficient information. In order to be able to |
| 54 | trace all the necessary information, the following software is |
| 55 | required: |
| 56 | |
| 57 | * lttng-ust version 2.8.0 or later |
| 58 | * lttng-tools, corresponding version |
| 59 | |
| 60 | You can get these from source at: |
| 61 | |
| 62 | * https://github.com/lttng/lttng-ust |
| 63 | * https://github.com/lttng/lttng-tools |
| 64 | |
| 65 | Ubuntu users also have the option of installing via the LTTng daily |
| 66 | PPA: |
| 67 | |
| 68 | * https://launchpad.net/~lttng/+archive/ubuntu/daily |
| 69 | |
| 70 | Tracing for Debug Info Analysis |
| 71 | ------------------------------- |
| 72 | |
| 73 | Babeltrace needs some extra information from contexts, namely ip and |
| 74 | vpid, to perform its analysis. These can be enabled after the creation |
| 75 | of a tracing session as follows: |
| 76 | |
| 77 | $ lttng add-context --userspace --type ip --type vpid |
| 78 | |
| 79 | The tracing can then be performed as it normally would. Once the trace |
| 80 | is collected, it can the be read by babeltrace for analysis. |
| 81 | |
| 82 | Analysing the Trace |
| 83 | ------------------- |
| 84 | |
| 85 | To perform the analysis, the trace can simply be read as it normally |
| 86 | would: |
| 87 | |
| 88 | $ babeltrace <path/to/trace> |
| 89 | |
| 90 | Debug info analysis is on by default and will automatically print the |
| 91 | extra source location information if it can find it. A sample output |
| 92 | may look like this: |
| 93 | |
| 94 | [...] |
| 95 | [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 } |
| 96 | [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9", src = "libhello.c:13" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 } |
| 97 | [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c", src = "libhello.c:20" }, vpid = 28719 }, { some_field = 1729 } |
| 98 | [...] |
| 99 | |
| 100 | The interesting part is the debug_info section of the context: |
| 101 | |
| 102 | debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" } |
| 103 | |
| 104 | This is the expected output for events generated by an executable for |
| 105 | which DWARF information is available. It shows the name of the binary |
| 106 | and offset to the tracepoint, the name of the function containing the |
| 107 | tracepoint instance which generated the event ("foo") and the offset |
| 108 | within the function, and its source location ("libhello.c", line 7). |
| 109 | |
| 110 | The second event in the sample output is of the same type |
| 111 | ("my_first_tracepoint"), but it was generated by a different |
| 112 | tracepoint instance, hence the different source location (line 13) and |
| 113 | function ("bar"). |
| 114 | |
| 115 | The third event, of a different type, also shows debug information. |
| 116 | |
| 117 | If DWARF info is absent, but ELF symbols are not stripped, the output |
| 118 | will instead look like this: |
| 119 | |
| 120 | [...] |
| 121 | [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 } |
| 122 | [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 } |
| 123 | [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c" }, vpid = 28719 }, { some_field = 1729 } |
| 124 | [...] |
| 125 | |
| 126 | The debug information now provides both binary and function location |
| 127 | information, but no source location information, as this requires |
| 128 | DWARF. The function names are in fact resolved using ELF symbols, so |
| 129 | there may be a discrepancy with those provided by DWARF (e.g. in the |
| 130 | case of mangling). |
| 131 | |
| 132 | Paths to the binary and to the source location (if any) can be |
| 133 | expanded by using the command-line option |
| 134 | --debug-info-full-path. Otherwise, only the filename is shown. |
| 135 | |
| 136 | Debug Info and Dynamic Loading |
| 137 | ------------------------------ |
| 138 | |
| 139 | Babeltrace can resolve addresses of events originating from |
| 140 | dynamically loaded libraries, provided that some extra information is |
| 141 | collected at tracing time. |
| 142 | |
| 143 | This can be achieved by preloading LTTng UST's libdl helper when |
| 144 | launching the program to be traced, like so: |
| 145 | |
| 146 | $ LD_PRELOAD="liblttng-ust-dl.so" <path/to/executable> |
| 147 | |
| 148 | The tracing and analysis can now be performed as described in prior |
| 149 | sections, and events from tracepoints in dlopened libraries will be |
| 150 | resolved automatically by babeltrace. |
| 151 | |
| 152 | Separate Debug Info |
| 153 | ------------------- |
| 154 | |
| 155 | It is possible to store DWARF debug information separate from an |
| 156 | executable, whether for concerns of file size, or simply to facilitate |
| 157 | the sharing of the debug information. |
| 158 | |
| 159 | This is usually achieved via one of two mechanisms, namely build ID |
| 160 | and debug link. Both methods permit separate executables and debug |
| 161 | information. Their use and operation is described in GDB's |
| 162 | documentation at: |
| 163 | |
| 164 | https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html |
| 165 | |
| 166 | Babeltrace will find separate debug files automatically, provided they |
| 167 | follow the requirements described in the documentation above. The |
| 168 | debug information lookup order is the same as GDB's, that is first |
| 169 | debug info is looked for within the executable, then through the build |
| 170 | ID method in the standard /usr/lib/debug/.build-id/ location, and |
| 171 | finally in the various possible debug link locations. The first debug |
| 172 | information file found is used. |
| 173 | |
| 174 | The --debug-info-dir command-line option can be used to override the |
| 175 | default /usr/lib/debug/ directory used in build ID and debug link |
| 176 | lookups. Multiple debug info directories are currently not supported. |
| 177 | |
| 178 | Target Prefix |
| 179 | ------------- |
| 180 | |
| 181 | The debug info analysis uses the paths to the executables as collected |
| 182 | during tracing as one mechanism to resolve DWARF or ELF |
| 183 | information. If the trace was taken on a separate machine, for |
| 184 | instance, it is possible to use --debug-info-target-prefix to specify |
| 185 | a prefix directory, representing the root of the target filesystem, |
| 186 | which will then be used for lookups. For example, if an executable was |
| 187 | located at /usr/bin/foo on the target system, it could be placed at |
| 188 | /home/efficios/target/usr/bin/foo on the system on which the analysis |
| 189 | is performed. In this case, the prefix is /home/efficios/target/. |