1 .TH "LTTNG" "1" "December 3rd, 2012" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path PATH"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel. For instance,
91 you could ask the tracer to add the PID information for all events in a
92 channel. You can also add performance monitoring unit counters (perf PMU) using
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
100 # lttng add-context \-k \-t prio \-t perf:branch-misses \\
101 \-t perf:cache-misses
104 Please take a look at the help (\-h/\-\-help) for a detailed list of available
107 If no channel is given (\-c), the context is added to all channels that were
108 already enabled. If the session has no channel, a default channel is created.
109 Otherwise the context will be added only to the given channel (\-c).
111 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
118 Show summary of possible options and commands.
120 .BR "\-s, \-\-session NAME"
121 Apply on session name.
123 .BR "\-c, \-\-channel NAME"
124 Apply on channel name.
126 .BR "\-k, \-\-kernel"
127 Apply for the kernel tracer
129 .BR "\-u, \-\-userspace"
130 Apply for the user-space tracer
132 .BR "\-t, \-\-type TYPE"
133 Context type. You can repeat this option on the command line. Please
134 use "lttng add-context \-h" to list all available types.
139 \fBcalibrate\fP [OPTIONS]
141 Quantify LTTng overhead
143 The LTTng calibrate command can be used to find out the combined average
144 overhead of the LTTng tracer and the instrumentation mechanisms used. This
145 overhead can be calibrated in terms of time or using any of the PMU performance
146 counter available on the system.
148 For now, the only calibration implemented is that of the kernel function
149 instrumentation (kretprobes).
151 * Calibrate kernel function instrumentation
153 Let's use an example to show this calibration. We use an i7 processor with 4
154 general-purpose PMU registers. This information is available by issuing dmesg,
155 looking for "generic registers".
157 This sequence of commands will gather a trace executing a kretprobe hooked on
158 an empty function, gathering PMU counters LLC (Last Level Cache) misses
159 information (see lttng add-context \-\-help to see the list of available PMU
163 # lttng create calibrate-function
164 # lttng enable-event calibrate \-\-kernel \\
165 \-\-function lttng_calibrate_kretprobe
166 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \\
167 \-t perf:LLC-store-misses \\
168 \-t perf:LLC-prefetch-misses
170 # for a in $(seq 1 10); do \\
171 lttng calibrate \-\-kernel \-\-function;
174 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* \\
178 The output from babeltrace can be saved to a text file and opened in a
179 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
180 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
181 counters are per-CPU, so scheduling events would need to be present to account
182 for migration between CPU. Therefore, for calibration purposes, only events
183 staying on the same CPU must be considered.
185 The average result, for the i7, on 10 samples:
189 perf_LLC_load_misses: 5.0 0.577
190 perf_LLC_store_misses: 1.6 0.516
191 perf_LLC_prefetch_misses: 9.0 14.742
194 As we can notice, the load and store misses are relatively stable across runs
195 (their standard deviation is relatively low) compared to the prefetch misses.
196 We can conclude from this information that LLC load and store misses can be
197 accounted for quite precisely, but prefetches within a function seems to behave
198 too erratically (not much causality link between the code executed and the CPU
199 prefetch activity) to be accounted for.
205 Show summary of possible options and commands.
207 .BR "\-k, \-\-kernel"
208 Apply for the kernel tracer
210 .BR "\-u, \-\-userspace"
211 Apply for the user-space tracer
214 Dynamic function entry/return probe (default)
218 .IP "\fBcreate\fP [NAME] [OPTIONS]
220 Create tracing session.
222 A tracing session contains channel(s) which contains event(s). It is domain
223 agnostic meaning that you can enable channels and events for either the
224 user-space tracer and/or the kernel tracer. It acts like a container
225 aggregating multiple tracing sources.
227 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
228 containing the current session name. If NAME is omitted, a session name is
229 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
231 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
234 The $HOME environment variable can be overridden by defining the environment
235 variable LTTNG_HOME. This is useful when the user running the commands has
236 a non-writeable home directory.
243 Show summary of possible options and commands.
245 Simple listing of options
247 Specify output path for traces
249 Traces will not be outputed
251 Set the session in snapshot mode. Created in no-output mode
252 and uses the URL, if one, as the default snapshot output.
253 Every channel will be set in overwrite mode and with mmap
254 output (splice not supported).
256 Using these options, each API call can be controlled individually. For
257 instance, \-C does not enable the consumer automatically. You'll need the \-e
261 Set URL for the consumer output destination. It is persistent for the
262 session lifetime. Redo the command to change it. This will set both
263 data and control URL for network.
264 \-C, \-\-ctrl-url=URL
265 Set control path URL. (Must use -D also)
266 \-D, \-\-data-url=URL
267 Set data path URL. (Must use -C also)
271 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
273 Supported protocols are (proto):
275 Local filesystem full path.
278 This will use the default network transport layer which is TCP for both
279 control (PORT1) and data port (PORT2). The default ports are
280 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
283 Can only be used with -C and -D together
285 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
289 # lttng create -U net://192.168.1.42
290 Uses TCP and default ports for the given destination.
292 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
293 Uses TCP, default ports and IPv6.
295 # lttng create s1 -U net://myhost.com:3229
296 Create session s1 and set its consumer to myhost.com on port 3229 for control.
301 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
303 Teardown tracing session
305 Free memory on the session daemon and tracer side. It's gone!
307 If NAME is omitted, the session name is taken from the .lttngrc file.
314 Show summary of possible options and commands.
318 Simple listing of options
323 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
325 Enable tracing channel
327 To enable an event, you must enable both the event and the channel that
330 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
333 It is important to note that if a certain type of buffers is used, the session
334 will be set with that type and all other subsequent channel needs to have the
337 Note that once the session has been started and enabled on the tracer side,
338 it's not possible anymore to enable a new channel for that session.
347 Simple listing of options
348 \-s, \-\-session NAME
349 Apply on session name
351 Apply to the kernel tracer
353 Apply to the user-space tracer
356 Discard event when subbuffers are full (default)
358 Flight recorder mode : overwrites events when subbuffers are full
360 Subbuffer size in bytes {+k,+M,+G}
361 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
362 Rounded up to the next power of 2.
364 The minimum subbuffer size, for each tracer, is the max value between
365 the default above and the system page size. You can issue this command
366 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
368 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4, metadata: 2)
369 Rounded up to the next power of 2.
370 \-\-switch-timer USEC
371 Switch subbuffer timer interval in µsec.
372 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
374 Read timer interval in µsec.
375 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
377 Channel output type. Possible values: mmap, splice
378 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
380 Use per UID buffer (\-u only). Buffers are shared between applications
381 that have the same UID.
383 Use per PID buffer (\-u only). Each application has its own buffers.
385 Use shared buffer for the whole system (\-k only)
386 \-C, \-\-tracefile-size SIZE
387 Maximum size of each tracefile within a stream (in bytes).
388 0 means unlimited. (default: 0)
389 \-W, \-\-tracefile-count COUNT
390 Used in conjunction with \-C option, this will limit the number
391 of files created to the specified count. 0 means unlimited. (default: 0)
395 $ lttng enable-channel -C 4096 -W 32 chan1
396 For each stream, the maximum size of each trace file will be 4096 bytes, and
397 there will be a maximum of 32 different files. The file count is appended after
398 the stream number as seen in the following example. The last trace file is
399 smaller than 4096 since it was not completely filled.
401 ~/lttng-traces/[...]/chan1_0_0 (4096)
402 ~/lttng-traces/[...]/chan1_0_1 (4096)
403 ~/lttng-traces/[...]/chan1_0_2 (3245)
404 ~/lttng-traces/[...]/chan1_1_0 (4096)
407 $ lttng enable-channel -C 4096
408 This will create trace files of 4096 bytes and will create new ones as long as
409 there is data available.
414 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
418 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
419 omitted, a default channel named '\fBchannel0\fP' is created and the event is
420 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
421 using the wildcard "*".
423 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
431 Show summary of possible options and commands.
433 Simple listing of options
434 \-s, \-\-session NAME
435 Apply on session name
436 \-c, \-\-channel NAME
437 Apply on channel name
439 Enable all tracepoints and syscalls. This actually enable a single
442 Apply for the kernel tracer
444 Apply for the user-space tracer
447 Tracepoint event (default)
448 - userspace tracer supports wildcards at end of string. Don't forget to
449 quote to deal with bash expansion.
454 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
455 \-\-loglevel-only NAME
456 Tracepoint loglevel (only this loglevel).
458 The loglevel or loglevel-only options should be combined with a
459 tracepoint name or tracepoint wildcard.
460 \-\-probe [addr | symbol | symbol+offset]
461 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
462 or hexadecimal (0xNNN...)
463 \-\-function [addr | symbol | symbol+offset]
464 Dynamic function entry/return probe. Addr and offset can be octal
465 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
467 System call event. Enabling syscalls tracing (kernel tracer), you will
468 not be able to disable them with disable-event. This is a known
469 limitation. You can disable the entire channel to do the trick.
471 \-\-filter 'expression'
472 Set a filter on a newly enabled event. Filter expression on event
473 fields and context. Event recording depends on evaluation. Only
474 specify on first activation of a given event within a session.
475 Filter only allowed when enabling events within a session before
476 tracing is started. If the filter fails to link with the event
477 within the traced domain, the event will be discarded.
478 Currently, filter is only implemented for the user-space tracer.
482 'intfield > 500 && intfield < 503'
483 '(stringfield == "test" || intfield != 10) && intfield > 33'
484 'doublefield > 1.1 && intfield < 5.3'
486 Wildcards are allowed at the end of strings:
488 In string literals, the escape character is a '\\'. Use '\\*' for
489 the '*' character, and '\\\\' for the '\\' character. Wildcard
490 match any sequence of characters, including an empty sub-string
491 (match 0 or more characters).
493 Context information can be used for filtering. The examples
494 below show usage of context filtering on process name (with a
495 wildcard), process ID range, and unique thread ID for filtering.
496 The process and thread ID of running applications can be found
497 under columns "PID" and "LWP" of the "ps -eLf" command.
499 '$ctx.procname == "demo*"'
500 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
504 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
506 Disable tracing channel
508 Disabling a channel makes all event(s) in that channel to stop tracing. You can
509 enable it back by calling \fBlttng enable-channel NAME\fP again.
511 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
519 Show summary of possible options and commands.
521 Simple listing of options
522 \-s, \-\-session NAME
523 Apply on session name
525 Apply for the kernel tracer
527 Apply for the user-space tracer
530 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
532 Disable tracing event
534 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
537 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
545 Show summary of possible options and commands.
547 Simple listing of options
548 \-s, \-\-session NAME
549 Apply on session name
551 Disable all events. This does NOT disable "*" but rather
552 every known events of the session.
554 Apply for the kernel tracer
556 Apply for the user-space tracer
559 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
561 List tracing session information.
563 With no arguments, it will list available tracing session(s).
565 With the session name, it will display the details of the session including
566 the trace file path, the associated channels and their state (activated
567 and deactivated), the activated events and more.
569 With \-k alone, it will list all available kernel events (except the system
571 With \-u alone, it will list all available user-space events from registered
572 applications. Here is an example of 'lttng list \-u':
574 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
575 ust_tests_hello:tptest_sighandler (type: tracepoint)
576 ust_tests_hello:tptest (type: tracepoint)
578 You can now enable any event listed by using the name :
579 \fBust_tests_hello:tptest\fP.
586 Show summary of possible options and commands.
588 Simple listing of options
592 Select user-space domain.
596 \-c, \-\-channel NAME
597 List details of a channel
599 List available domain(s)
602 .IP "\fBset-session\fP NAME"
604 Set current session name
606 Will change the session name in the .lttngrc file.
613 Show summary of possible options and commands.
615 Simple listing of options
620 .IP "\fBsnapshot\fP ACTION"
622 Snapshot command for LTTng session.
629 Show summary of possible options and commands.
631 Simple listing of options
637 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
639 Setup and add an snapshot output for a session. Output are the destination
640 where the snapshot will be sent. Only one output is permitted. To change it,
641 you'll need to delete it and add back the new one.
643 \fBdel-output\fP ID | NAME [-s <NAME>]
645 Delete an output for a session using the ID. You can either specify the
646 output's ID that can be found with list-output or the name.
648 \fBlist-output\fP [-s <NAME>]
650 List the output of a session. Attributes of the output are printed.
652 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
654 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
655 used instead of a previously added output. Specifying only a name or/and a max
656 size will override the current output values. For instance, you can record a
657 snapshot with a custom maximum size or with a different name.
659 $ lttng add-output -n mysnapshot file:///data/snapshot
661 $ lttng snapshot record -n new_name_snapshot
663 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
664 rather then in mysnapshot*/
670 \-s, \-\-session NAME
671 Apply to session name.
673 Name of the snapshot's output.
674 \-m, \-\-max-size SIZE
675 Maximum size in bytes of the snapshot. The maxium size does not
676 include the metadata file.
677 \-C, \-\-ctrl-url URL
678 Set control path URL. (Must use -D also)
679 \-D, \-\-data-url URL
680 Set data path URL. (Must use -C also)
685 .IP "\fBstart\fP [NAME] [OPTIONS]"
689 It will start tracing for all tracers for a specific tracing session.
691 If NAME is omitted, the session name is taken from the .lttngrc file.
698 Show summary of possible options and commands.
700 Simple listing of options
705 .IP "\fBstop\fP [NAME] [OPTIONS]"
709 It will stop tracing for all tracers for a specific tracing session. Before
710 returning, the command checks for data availability meaning that it will wait
711 until the trace is readable for the session. Use \-\-no-wait to avoid this
714 If NAME is omitted, the session name is taken from the .lttngrc file.
721 Show summary of possible options and commands.
723 Simple listing of options
725 Don't wait for data availability.
732 Show version information
739 Show summary of possible options and commands.
741 Simple listing of options
746 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
748 View traces of a tracing session
750 By default, the babeltrace viewer will be used for text viewing.
752 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
762 Simple listing of options
763 \-t, \-\-trace-path PATH
764 Trace directory path for the viewer
766 Specify viewer and/or options to use
767 This will completely override the default viewers so
768 please make sure to specify the full command. The trace
769 directory path of the session will be appended at the end
774 On success 0 is returned and a positive value on error. Value of 1 means a command
775 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
776 something went wrong during the command.
778 Any other value above 10, please refer to
779 .BR <lttng/lttng-error.h>
780 for a detailed list or use lttng_strerror() to get a human readable string of
784 .SH "ENVIRONMENT VARIABLES"
787 Note that all command line options override environment variables.
791 .IP "LTTNG_SESSIOND_PATH"
792 Allows one to specify the full session daemon binary path to lttng command line
793 tool. You can also use \-\-sessiond-path option having the same effect.
797 .BR lttng-sessiond(8),
799 .BR lttng-health-check(3)
802 If you encounter any issues or usability problem, please report it on our
803 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
804 at https://bugs.lttng.org which is a bugtracker.
808 lttng is distributed under the GNU General Public License version 2. See the file
811 A Web site is available at http://lttng.org for more information on the LTTng
814 You can also find our git tree at http://git.lttng.org.
816 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
818 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
823 Thanks to Yannick Brosseau without whom this project would never have been so
824 lean and mean! Also thanks to the Ericsson teams working on tracing which
825 helped us greatly with detailed bug reports and unusual test cases.
827 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
828 maintainer) and Jon Bernard for our Debian packages.
830 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
831 Montreal for the LTTng journey.
836 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
837 David Goulet. More people have since contributed to it. It is currently
838 maintained by David Goulet <dgoulet@efficios.com>.