1 .TH "LTTNG" "1" "December 3rd, 2012" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path PATH"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
86 \fBadd-context\fP [OPTIONS]
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel. For instance,
91 you could ask the tracer to add the PID information for all events in a
92 channel. You can also add performance monitoring unit counters (perf PMU) using
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
100 # lttng add-context \-k \-t prio \-t perf:branch-misses \\
101 \-t perf:cache-misses
104 Please take a look at the help (\-h/\-\-help) for a detailed list of available
107 If no channel is given (\-c), the context is added to all channels that were
108 already enabled. If the session has no channel, a default channel is created.
109 Otherwise the context will be added only to the given channel (\-c).
111 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
118 Show summary of possible options and commands.
120 .BR "\-s, \-\-session NAME"
121 Apply on session name.
123 .BR "\-c, \-\-channel NAME"
124 Apply on channel name.
126 .BR "\-k, \-\-kernel"
127 Apply for the kernel tracer
129 .BR "\-u, \-\-userspace"
130 Apply for the user-space tracer
132 .BR "\-t, \-\-type TYPE"
133 Context type. You can repeat this option on the command line. Please
134 use "lttng add-context \-h" to list all available types.
139 \fBcalibrate\fP [OPTIONS]
141 Quantify LTTng overhead
143 The LTTng calibrate command can be used to find out the combined average
144 overhead of the LTTng tracer and the instrumentation mechanisms used. This
145 overhead can be calibrated in terms of time or using any of the PMU performance
146 counter available on the system.
148 For now, the only calibration implemented is that of the kernel function
149 instrumentation (kretprobes).
151 * Calibrate kernel function instrumentation
153 Let's use an example to show this calibration. We use an i7 processor with 4
154 general-purpose PMU registers. This information is available by issuing dmesg,
155 looking for "generic registers".
157 This sequence of commands will gather a trace executing a kretprobe hooked on
158 an empty function, gathering PMU counters LLC (Last Level Cache) misses
159 information (see lttng add-context \-\-help to see the list of available PMU
163 # lttng create calibrate-function
164 # lttng enable-event calibrate \-\-kernel \\
165 \-\-function lttng_calibrate_kretprobe
166 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \\
167 \-t perf:LLC-store-misses \\
168 \-t perf:LLC-prefetch-misses
170 # for a in $(seq 1 10); do \\
171 lttng calibrate \-\-kernel \-\-function;
174 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* \\
178 The output from babeltrace can be saved to a text file and opened in a
179 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
180 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
181 counters are per-CPU, so scheduling events would need to be present to account
182 for migration between CPU. Therefore, for calibration purposes, only events
183 staying on the same CPU must be considered.
185 The average result, for the i7, on 10 samples:
189 perf_LLC_load_misses: 5.0 0.577
190 perf_LLC_store_misses: 1.6 0.516
191 perf_LLC_prefetch_misses: 9.0 14.742
194 As we can notice, the load and store misses are relatively stable across runs
195 (their standard deviation is relatively low) compared to the prefetch misses.
196 We can conclude from this information that LLC load and store misses can be
197 accounted for quite precisely, but prefetches within a function seems to behave
198 too erratically (not much causality link between the code executed and the CPU
199 prefetch activity) to be accounted for.
205 Show summary of possible options and commands.
207 .BR "\-k, \-\-kernel"
208 Apply for the kernel tracer
210 .BR "\-u, \-\-userspace"
211 Apply for the user-space tracer
214 Dynamic function entry/return probe (default)
219 .IP \fBcreate\fP [NAME] [OPTIONS]
221 Create tracing session.
223 A tracing session contains channel(s) which contains event(s). It is domain
224 agnostic meaning that you can enable channels and events for either the
225 user-space tracer and/or the kernel tracer. It acts like a container
226 aggregating multiple tracing sources.
228 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
229 containing the current session name. If NAME is omitted, a session name is
230 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
232 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
235 The $HOME environment variable can be overridden by defining the environment
236 variable LTTNG_HOME. This is useful when the user running the commands has
237 a non-writeable home directory.
243 Show summary of possible options and commands.
245 .BR "\-\-list-options"
246 Simple listing of options
248 .BR "\-o, \-\-output PATH"
249 Specify output path for traces
252 Traces will not be outputed
255 Set the session in snapshot mode. Created in no-output mode and uses the
256 URL, if one, as the default snapshot output. Every channel will be set
257 in overwrite mode and with mmap output (splice not supported).
260 .BR "\-U, \-\-set-url=URL"
261 Set URL for the consumer output destination. It is persistent for the
262 session lifetime. Redo the command to change it. This will set both data
263 and control URL for network.
265 .BR "\-C, \-\-ctrl-url=URL"
266 Set control path URL. (Must use -D also)
268 .BR "\-D, \-\-data-url=URL"
269 Set data path URL. (Must use -C also)
271 Using these options, each API call can be controlled individually. For
272 instance, \-C does not enable the consumer automatically. You'll need the \-e
277 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
279 Supported protocols are (proto):
282 Local filesystem full path.
286 This will use the default network transport layer which is TCP for both
287 control (PORT1) and data port (PORT2). The default ports are
288 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
292 Can only be used with -C and -D together
294 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
299 # lttng create -U net://192.168.1.42
301 Uses TCP and default ports for the given destination.
304 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
306 Uses TCP, default ports and IPv6.
309 # lttng create s1 -U net://myhost.com:3229
311 Create session s1 and set its consumer to myhost.com on port 3229 for control.
316 \fBdestroy\fP [NAME] [OPTIONS]
318 Teardown tracing session
320 Free memory on the session daemon and tracer side. It's gone!
322 If NAME is omitted, the session name is taken from the .lttngrc file.
328 Show summary of possible options and commands.
333 .BR "\-\-list-options"
334 Simple listing of options
339 \fBenable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
341 Enable tracing channel
343 To enable an event, you must enable both the event and the channel that
346 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
349 Exactly one of \-k or -u must be specified.
351 It is important to note that if a certain type of buffers is used, the session
352 will be set with that type and all other subsequent channel needs to have the
355 Note that once the session has been started and enabled on the tracer side,
356 it's not possible anymore to enable a new channel for that session.
364 .BR "\-\-list-options"
365 Simple listing of options
367 .BR "\-s, \-\-session NAME"
368 Apply on session name
370 .BR "\-k, \-\-kernel"
371 Apply to the kernel tracer
373 .BR "\-u, \-\-userspace"
374 Apply to the user-space tracer
377 Discard event when subbuffers are full (default)
380 Flight recorder mode : overwrites events when subbuffers are full
382 .BR "\-\-subbuf-size SIZE"
383 Subbuffer size in bytes {+k,+M,+G}.
384 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
385 Rounded up to the next power of 2.
387 The minimum subbuffer size, for each tracer, is the max value between
388 the default above and the system page size. You can issue this command
389 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
391 .BR "\-\-num-subbuf NUM"
392 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4,
393 metadata: 2) Rounded up to the next power of 2.
395 .BR "\-\-switch-timer USEC"
396 Switch subbuffer timer interval in µsec.
397 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
399 .BR "\-\-read-timer USEC"
400 Read timer interval in µsec.
401 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
403 .BR "\-\-output TYPE"
404 Channel output type. Possible values: mmap, splice
405 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
407 .BR "\-\-buffers-uid"
408 Use per UID buffer (\-u only). Buffers are shared between applications
409 that have the same UID.
411 .BR "\-\-buffers-pid"
412 Use per PID buffer (\-u only). Each application has its own buffers.
414 .BR "\-\-buffers-global"
415 Use shared buffer for the whole system (\-k only)
417 .BR "\-C, \-\-tracefile-size SIZE"
418 Maximum size of each tracefile within a stream (in bytes).
419 0 means unlimited. (default: 0)
421 .BR "\-W, \-\-tracefile-count COUNT"
422 Used in conjunction with \-C option, this will limit the number of files
423 created to the specified count. 0 means unlimited. (default: 0)
428 $ lttng enable-channel -k -C 4096 -W 32 chan1
430 For each stream, the maximum size of each trace file will be 4096 bytes, and
431 there will be a maximum of 32 different files. The file count is appended after
432 the stream number as seen in the following example. The last trace file is
433 smaller than 4096 since it was not completely filled.
436 ~/lttng-traces/[...]/chan1_0_0 (4096)
437 ~/lttng-traces/[...]/chan1_0_1 (4096)
438 ~/lttng-traces/[...]/chan1_0_2 (3245)
439 ~/lttng-traces/[...]/chan1_1_0 (4096)
444 $ lttng enable-channel -k -C 4096
446 This will create trace files of 4096 bytes and will create new ones as long as
447 there is data available.
452 \fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]
456 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
457 omitted, a default channel named '\fBchannel0\fP' is created and the event is
458 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
459 using the wildcard "*".
461 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
468 Show summary of possible options and commands.
470 .BR "\-\-list-options"
471 Simple listing of options
473 .BR "\-s, \-\-session NAME"
474 Apply on session name
476 .BR "\-c, \-\-channel NAME"
477 Apply on channel name
480 Enable all tracepoints and syscalls. This actually enable a single
483 .BR "\-k, \-\-kernel"
484 Apply for the kernel tracer
486 .BR "\-u, \-\-userspace"
487 Apply for the user-space tracer
490 Tracepoint event (default). Userspace tracer supports wildcards at end
491 of string. Don't forget to quote to deal with bash expansion.
498 .BR "\-\-loglevel NAME"
499 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
501 .BR "\-\-loglevel-only NAME"
502 Tracepoint loglevel (only this loglevel).
503 The loglevel or loglevel-only options should be combined with a
504 tracepoint name or tracepoint wildcard.
506 .BR "\-\-probe (addr | symbol | symbol+offset)"
507 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
508 or hexadecimal (0xNNN...)
510 .BR "\-\-function (addr | symbol | symbol+offset)"
511 Dynamic function entry/return probe. Addr and offset can be octal
512 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
515 System call event. Enabling syscalls tracing (kernel tracer), you will
516 not be able to disable them with disable-event. This is a known
517 limitation. You can disable the entire channel to do the trick.
519 .BR "\-\-filter 'expression'"
520 Set a filter on a newly enabled event. Filter expression on event
521 fields and context. Event recording depends on evaluation. Only
522 specify on first activation of a given event within a session.
523 Filter only allowed when enabling events within a session before
524 tracing is started. If the filter fails to link with the event
525 within the traced domain, the event will be discarded.
526 Currently, filter is only implemented for the user-space tracer.
531 'intfield > 500 && intfield < 503'
532 '(strfield == "test" || intfield != 10) && intfield > 33'
533 'doublefield > 1.1 && intfield < 5.3'
536 Wildcards are allowed at the end of strings:
538 In string literals, the escape character is a '\\'. Use '\\*' for
539 the '*' character, and '\\\\' for the '\\' character. Wildcard
540 match any sequence of characters, including an empty sub-string
541 (match 0 or more characters).
543 Context information can be used for filtering. The examples below show
544 usage of context filtering on process name (with a wildcard), process ID
545 range, and unique thread ID for filtering. The process and thread ID of
546 running applications can be found under columns "PID" and "LWP" of the
550 '$ctx.procname == "demo*"'
551 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
559 \fBdisable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
561 Disable tracing channel
563 Disabling a channel makes all event(s) in that channel to stop tracing. You can
564 enable it back by calling \fBlttng enable-channel NAME\fP again.
566 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
573 Show summary of possible options and commands.
575 .BR "\-\-list-options"
576 Simple listing of options
578 .BR "\-s, \-\-session NAME"
579 Apply on session name
581 .BR "\-k, \-\-kernel"
582 Apply for the kernel tracer
584 .BR "\-u, \-\-userspace"
585 Apply for the user-space tracer
590 \fBdisable-event\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
592 Disable tracing event
594 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
597 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
604 Show summary of possible options and commands.
606 .BR "\-\-list-options"
607 Simple listing of options
609 .BR "\-s, \-\-session NAME"
610 Apply on session name
612 .BR "\-a, \-\-all-events"
613 Disable all events. This does NOT disable "*" but rather every known
614 events of the session.
616 .BR "\-k, \-\-kernel"
617 Apply for the kernel tracer
619 .BR "\-u, \-\-userspace"
620 Apply for the user-space tracer
625 \fBlist\fP [OPTIONS] [SESSION [SESSION OPTIONS]]
627 List tracing session information.
629 With no arguments, it will list available tracing session(s).
631 With the session name, it will display the details of the session including
632 the trace file path, the associated channels and their state (activated
633 and deactivated), the activated events and more.
635 With \-k alone, it will list all available kernel events (except the system
637 With \-u alone, it will list all available user-space events from registered
638 applications. Here is an example of 'lttng list \-u':
641 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
642 ust_tests_hello:tptest_sighandler (type: tracepoint)
643 ust_tests_hello:tptest (type: tracepoint)
646 You can now enable any event listed by using the name :
647 \fBust_tests_hello:tptest\fP.
653 Show summary of possible options and commands.
655 .BR "\-\-list-options"
656 Simple listing of options
658 .BR "\-k, \-\-kernel"
661 .BR "\-u, \-\-userspace"
662 Select user-space domain.
668 .BR "\-c, \-\-channel NAME"
669 List details of a channel
671 .BR "\-d, \-\-domain"
672 List available domain(s)
677 \fBset-session\fP NAME [OPTIONS]
679 Set current session name
681 Will change the session name in the .lttngrc file.
687 Show summary of possible options and commands.
689 .BR "\-\-list-options"
690 Simple listing of options
695 \fBsnapshot\fP [OPTIONS] ACTION
697 Snapshot command for LTTng session.
703 Show summary of possible options and commands.
705 .BR "\-\-list-options"
706 Simple listing of options
712 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
714 Setup and add an snapshot output for a session. Output are the destination
715 where the snapshot will be sent. Only one output is permitted. To change it,
716 you'll need to delete it and add back the new one.
719 \fBdel-output\fP ID | NAME [-s <NAME>]
721 Delete an output for a session using the ID. You can either specify the
722 output's ID that can be found with list-output or the name.
725 \fBlist-output\fP [-s <NAME>]
727 List the output of a session. Attributes of the output are printed.
730 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
732 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
733 used instead of a previously added output. Specifying only a name or/and a max
734 size will override the current output values. For instance, you can record a
735 snapshot with a custom maximum size or with a different name.
738 $ lttng add-output -n mysnapshot file:///data/snapshot
740 $ lttng snapshot record -n new_name_snapshot
743 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
744 rather then in mysnapshot*/
747 .B DETAILED ACTION OPTIONS
750 .BR "\-s, \-\-session NAME"
751 Apply to session name.
753 .BR "\-n, \-\-name NAME"
754 Name of the snapshot's output.
756 .BR "\-m, \-\-max-size SIZE"
757 Maximum size in bytes of the snapshot. The maxium size does not include
760 .BR "\-C, \-\-ctrl-url URL"
761 Set control path URL. (Must use -D also)
763 .BR "\-D, \-\-data-url URL"
764 Set data path URL. (Must use -C also)
769 \fBstart\fP [NAME] [OPTIONS]
773 It will start tracing for all tracers for a specific tracing session.
774 If NAME is omitted, the session name is taken from the .lttngrc file.
780 Show summary of possible options and commands.
782 .BR "\-\-list-options"
783 Simple listing of options
787 .IP "\fBstop\fP [NAME] [OPTIONS]"
791 It will stop tracing for all tracers for a specific tracing session. Before
792 returning, the command checks for data availability meaning that it will wait
793 until the trace is readable for the session. Use \-\-no-wait to avoid this
796 If NAME is omitted, the session name is taken from the .lttngrc file.
803 Show summary of possible options and commands.
805 Simple listing of options
807 Don't wait for data availability.
814 Show version information
821 Show summary of possible options and commands.
823 Simple listing of options
828 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
830 View traces of a tracing session
832 By default, the babeltrace viewer will be used for text viewing.
834 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
844 Simple listing of options
845 \-t, \-\-trace-path PATH
846 Trace directory path for the viewer
848 Specify viewer and/or options to use
849 This will completely override the default viewers so
850 please make sure to specify the full command. The trace
851 directory path of the session will be appended at the end
856 On success 0 is returned and a positive value on error. Value of 1 means a command
857 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
858 something went wrong during the command.
860 Any other value above 10, please refer to
861 .BR <lttng/lttng-error.h>
862 for a detailed list or use lttng_strerror() to get a human readable string of
866 .SH "ENVIRONMENT VARIABLES"
869 Note that all command line options override environment variables.
873 .IP "LTTNG_SESSIOND_PATH"
874 Allows one to specify the full session daemon binary path to lttng command line
875 tool. You can also use \-\-sessiond-path option having the same effect.
879 .BR lttng-sessiond(8),
881 .BR lttng-health-check(3)
884 If you encounter any issues or usability problem, please report it on our
885 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
886 at https://bugs.lttng.org which is a bugtracker.
890 lttng is distributed under the GNU General Public License version 2. See the file
893 A Web site is available at http://lttng.org for more information on the LTTng
896 You can also find our git tree at http://git.lttng.org.
898 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
900 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
905 Thanks to Yannick Brosseau without whom this project would never have been so
906 lean and mean! Also thanks to the Ericsson teams working on tracing which
907 helped us greatly with detailed bug reports and unusual test cases.
909 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
910 maintainer) and Jon Bernard for our Debian packages.
912 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
913 Montreal for the LTTng journey.
918 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
919 David Goulet. More people have since contributed to it. It is currently
920 maintained by David Goulet <dgoulet@efficios.com>.