1 .TH "LTTNG" "1" "July 18th, 2013" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
9 lttng [OPTIONS] <COMMAND>
13 The LTTng project aims at providing highly efficient tracing tools for Linux.
14 It's tracers help tracking down performance issues and debugging problems
15 involving multiple concurrent processes and threads. Tracing across multiple
16 systems is also possible.
18 The \fBlttng\fP command line tool from the lttng-tools package is used to control
19 both kernel and user-space tracing. Every interactions with the tracer should
20 be done by this tool or by the liblttng-ctl provided with the lttng-tools
23 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
24 which allows you to interact with multiple tracers (kernel and user-space)
25 inside the same container, a tracing session. Traces can be gathered from the
26 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
27 those traces is done using the babeltrace(1) text viewer.
29 We introduce the notion of \fBtracing domains\fP which is essentially a type of
30 tracer (kernel or user space for now). In the future, we could see a third
31 tracer being for instance an hypervisor. For some commands, you'll need to
32 specify on which domain the command applies (-u or -k). For instance, enabling
33 a kernel event, you must specify the kernel domain to the command so we know
34 for which tracer this event is for.
36 In order to trace the kernel, the session daemon needs to be running as root.
37 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
38 in that group can interact with the root session daemon and thus trace the
39 kernel. Session daemons can co-exist meaning that you can have a session daemon
40 running as Alice that can be used to trace her applications along side with a
41 root daemon or even a Bob daemon. We highly recommend to start the session
42 daemon at boot time for stable and long term tracing.
44 Every user-space applications instrumented with lttng-ust(3), will
45 automatically register to the session daemon. This feature gives you the
46 ability to list available traceable applications and tracepoints on a per user
47 basis. (See \fBlist\fP command).
51 This program follow the usual GNU command line syntax with long options starting with
52 two dashes. Below is a summary of the available options.
57 Show summary of possible options and commands.
59 .BR "\-v, \-\-verbose"
61 Three levels of verbosity are available which are triggered by putting additional v to
62 the option (\-vv or \-vvv)
65 Suppress all messages (even errors).
67 .BR "\-g, \-\-group NAME"
68 Set unix tracing group name. (default: tracing)
70 .BR "\-n, \-\-no-sessiond"
71 Don't automatically spawn a session daemon.
73 .BR "\-\-sessiond\-path PATH"
74 Set session daemon full binary path.
76 .BR "\-\-list\-options"
77 Simple listing of lttng options.
79 .BR "\-\-list\-commands"
80 Simple listing of lttng commands.
84 \fBadd-context\fP [OPTIONS]
86 Add context to event(s) and/or channel(s).
88 A context is basically extra information appended to a channel. For instance,
89 you could ask the tracer to add the PID information for all events in a
90 channel. You can also add performance monitoring unit counters (perf PMU) using
93 For example, this command will add the context information 'prio' and two perf
94 counters (hardware branch misses and cache misses), to all events in the trace
98 # lttng add-context \-k \-t prio \-t perf:branch-misses \\
102 Please take a look at the help (\-h/\-\-help) for a detailed list of available
105 If no channel is given (\-c), the context is added to all channels that were
106 already enabled. If the session has no channel, a default channel is created.
107 Otherwise the context will be added only to the given channel (\-c).
109 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
116 Show summary of possible options and commands.
118 .BR "\-s, \-\-session NAME"
119 Apply on session name.
121 .BR "\-c, \-\-channel NAME"
122 Apply on channel name.
124 .BR "\-k, \-\-kernel"
125 Apply for the kernel tracer
127 .BR "\-u, \-\-userspace"
128 Apply for the user-space tracer
130 .BR "\-t, \-\-type TYPE"
131 Context type. You can repeat this option on the command line. Please
132 use "lttng add-context \-h" to list all available types.
137 \fBcalibrate\fP [OPTIONS]
139 Quantify LTTng overhead
141 The LTTng calibrate command can be used to find out the combined average
142 overhead of the LTTng tracer and the instrumentation mechanisms used. This
143 overhead can be calibrated in terms of time or using any of the PMU performance
144 counter available on the system.
146 For now, the only calibration implemented is that of the kernel function
147 instrumentation (kretprobes).
149 * Calibrate kernel function instrumentation
151 Let's use an example to show this calibration. We use an i7 processor with 4
152 general-purpose PMU registers. This information is available by issuing dmesg,
153 looking for "generic registers".
155 This sequence of commands will gather a trace executing a kretprobe hooked on
156 an empty function, gathering PMU counters LLC (Last Level Cache) misses
157 information (see lttng add-context \-\-help to see the list of available PMU
161 # lttng create calibrate-function
162 # lttng enable-event calibrate \-\-kernel \\
163 \-\-function lttng_calibrate_kretprobe
164 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \\
165 \-t perf:LLC-store-misses \\
166 \-t perf:LLC-prefetch-misses
168 # for a in $(seq 1 10); do \\
169 lttng calibrate \-\-kernel \-\-function;
172 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* \\
176 The output from babeltrace can be saved to a text file and opened in a
177 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
178 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
179 counters are per-CPU, so scheduling events would need to be present to account
180 for migration between CPU. Therefore, for calibration purposes, only events
181 staying on the same CPU must be considered.
183 The average result, for the i7, on 10 samples:
187 perf_LLC_load_misses: 5.0 0.577
188 perf_LLC_store_misses: 1.6 0.516
189 perf_LLC_prefetch_misses: 9.0 14.742
192 As we can notice, the load and store misses are relatively stable across runs
193 (their standard deviation is relatively low) compared to the prefetch misses.
194 We can conclude from this information that LLC load and store misses can be
195 accounted for quite precisely, but prefetches within a function seems to behave
196 too erratically (not much causality link between the code executed and the CPU
197 prefetch activity) to be accounted for.
203 Show summary of possible options and commands.
205 .BR "\-k, \-\-kernel"
206 Apply for the kernel tracer
208 .BR "\-u, \-\-userspace"
209 Apply for the user-space tracer
212 Dynamic function entry/return probe (default)
217 .IP \fBcreate\fP [NAME] [OPTIONS]
219 Create tracing session.
221 A tracing session contains channel(s) which contains event(s). It is domain
222 agnostic meaning that you can enable channels and events for either the
223 user-space tracer and/or the kernel tracer. It acts like a container
224 aggregating multiple tracing sources.
226 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
227 containing the current session name. If NAME is omitted, a session name is
228 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
230 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
233 The $HOME environment variable can be overridden by defining the environment
234 variable LTTNG_HOME. This is useful when the user running the commands has
235 a non-writeable home directory.
241 Show summary of possible options and commands.
243 .BR "\-\-list-options"
244 Simple listing of options
246 .BR "\-o, \-\-output PATH"
247 Specify output path for traces
250 Traces will not be outputed
253 Set the session in snapshot mode. Created in no-output mode and uses the
254 URL, if one, as the default snapshot output. Every channel will be set
255 in overwrite mode and with mmap output (splice not supported).
258 .BR "\-U, \-\-set-url=URL"
259 Set URL for the consumer output destination. It is persistent for the
260 session lifetime. Redo the command to change it. This will set both data
261 and control URL for network.
263 .BR "\-C, \-\-ctrl-url=URL"
264 Set control path URL. (Must use -D also)
266 .BR "\-D, \-\-data-url=URL"
267 Set data path URL. (Must use -C also)
269 Using these options, each API call can be controlled individually. For
270 instance, \-C does not enable the consumer automatically. You'll need the \-e
275 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
277 Supported protocols are (proto):
280 Local filesystem full path.
284 This will use the default network transport layer which is TCP for both
285 control (PORT1) and data port (PORT2). The default ports are
286 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
290 Can only be used with -C and -D together
292 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
297 # lttng create -U net://192.168.1.42
299 Uses TCP and default ports for the given destination.
302 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
304 Uses TCP, default ports and IPv6.
307 # lttng create s1 -U net://myhost.com:3229
309 Create session s1 and set its consumer to myhost.com on port 3229 for control.
314 \fBdestroy\fP [NAME] [OPTIONS]
316 Teardown tracing session
318 Free memory on the session daemon and tracer side. It's gone!
320 If NAME is omitted, the session name is taken from the .lttngrc file.
326 Show summary of possible options and commands.
331 .BR "\-\-list-options"
332 Simple listing of options
337 \fBenable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
339 Enable tracing channel
341 To enable an event, you must enable both the event and the channel that
344 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
347 Exactly one of \-k or -u must be specified.
349 It is important to note that if a certain type of buffers is used, the session
350 will be set with that type and all other subsequent channel needs to have the
353 Note that once the session has been started and enabled on the tracer side,
354 it's not possible anymore to enable a new channel for that session.
362 .BR "\-\-list-options"
363 Simple listing of options
365 .BR "\-s, \-\-session NAME"
366 Apply on session name
368 .BR "\-k, \-\-kernel"
369 Apply to the kernel tracer
371 .BR "\-u, \-\-userspace"
372 Apply to the user-space tracer
375 Discard event when subbuffers are full (default)
378 Flight recorder mode : overwrites events when subbuffers are full
380 .BR "\-\-subbuf-size SIZE"
381 Subbuffer size in bytes {+k,+M,+G}.
382 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
383 Rounded up to the next power of 2.
385 The minimum subbuffer size, for each tracer, is the max value between
386 the default above and the system page size. You can issue this command
387 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
389 .BR "\-\-num-subbuf NUM"
390 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4,
391 metadata: 2) Rounded up to the next power of 2.
393 .BR "\-\-switch-timer USEC"
394 Switch subbuffer timer interval in µsec.
395 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
397 .BR "\-\-read-timer USEC"
398 Read timer interval in µsec.
399 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
401 .BR "\-\-output TYPE"
402 Channel output type. Possible values: mmap, splice
403 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
405 .BR "\-\-buffers-uid"
406 Use per UID buffer (\-u only). Buffers are shared between applications
407 that have the same UID.
409 .BR "\-\-buffers-pid"
410 Use per PID buffer (\-u only). Each application has its own buffers.
412 .BR "\-\-buffers-global"
413 Use shared buffer for the whole system (\-k only)
415 .BR "\-C, \-\-tracefile-size SIZE"
416 Maximum size of each tracefile within a stream (in bytes).
417 0 means unlimited. (default: 0)
419 .BR "\-W, \-\-tracefile-count COUNT"
420 Used in conjunction with \-C option, this will limit the number of files
421 created to the specified count. 0 means unlimited. (default: 0)
426 $ lttng enable-channel -k -C 4096 -W 32 chan1
428 For each stream, the maximum size of each trace file will be 4096 bytes, and
429 there will be a maximum of 32 different files. The file count is appended after
430 the stream number as seen in the following example. The last trace file is
431 smaller than 4096 since it was not completely filled.
434 ~/lttng-traces/[...]/chan1_0_0 (4096)
435 ~/lttng-traces/[...]/chan1_0_1 (4096)
436 ~/lttng-traces/[...]/chan1_0_2 (3245)
437 ~/lttng-traces/[...]/chan1_1_0 (4096)
442 $ lttng enable-channel -k -C 4096
444 This will create trace files of 4096 bytes and will create new ones as long as
445 there is data available.
450 \fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]
454 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
455 omitted, a default channel named '\fBchannel0\fP' is created and the event is
456 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
457 using the wildcard "*".
459 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
466 Show summary of possible options and commands.
468 .BR "\-\-list-options"
469 Simple listing of options
471 .BR "\-s, \-\-session NAME"
472 Apply on session name
474 .BR "\-c, \-\-channel NAME"
475 Apply on channel name
478 Enable all tracepoints and syscalls. This actually enable a single
481 .BR "\-k, \-\-kernel"
482 Apply for the kernel tracer
484 .BR "\-u, \-\-userspace"
485 Apply for the user-space tracer
488 Tracepoint event (default). Userspace tracer supports wildcards at end
489 of string. Don't forget to quote to deal with bash expansion.
496 .BR "\-\-loglevel NAME"
497 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
499 .BR "\-\-loglevel-only NAME"
500 Tracepoint loglevel (only this loglevel).
501 The loglevel or loglevel-only options should be combined with a
502 tracepoint name or tracepoint wildcard.
504 .BR "\-\-probe (addr | symbol | symbol+offset)"
505 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
506 or hexadecimal (0xNNN...)
508 .BR "\-\-function (addr | symbol | symbol+offset)"
509 Dynamic function entry/return probe. Addr and offset can be octal
510 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
513 System call event. Enabling syscalls tracing (kernel tracer), you will
514 not be able to disable them with disable-event. This is a known
515 limitation. You can disable the entire channel to do the trick.
517 .BR "\-\-filter 'expression'"
518 Set a filter on a newly enabled event. Filter expression on event
519 fields and context. Event recording depends on evaluation. Only
520 specify on first activation of a given event within a session.
521 Filter only allowed when enabling events within a session before
522 tracing is started. If the filter fails to link with the event
523 within the traced domain, the event will be discarded.
524 Currently, filter is only implemented for the user-space tracer.
529 'intfield > 500 && intfield < 503'
530 '(strfield == "test" || intfield != 10) && intfield > 33'
531 'doublefield > 1.1 && intfield < 5.3'
534 Wildcards are allowed at the end of strings:
536 In string literals, the escape character is a '\\'. Use '\\*' for
537 the '*' character, and '\\\\' for the '\\' character. Wildcard
538 match any sequence of characters, including an empty sub-string
539 (match 0 or more characters).
541 Context information can be used for filtering. The examples below show
542 usage of context filtering on process name (with a wildcard), process ID
543 range, and unique thread ID for filtering. The process and thread ID of
544 running applications can be found under columns "PID" and "LWP" of the
548 '$ctx.procname == "demo*"'
549 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
557 \fBdisable-channel\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
559 Disable tracing channel
561 Disabling a channel makes all event(s) in that channel to stop tracing. You can
562 enable it back by calling \fBlttng enable-channel NAME\fP again.
564 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
571 Show summary of possible options and commands.
573 .BR "\-\-list-options"
574 Simple listing of options
576 .BR "\-s, \-\-session NAME"
577 Apply on session name
579 .BR "\-k, \-\-kernel"
580 Apply for the kernel tracer
582 .BR "\-u, \-\-userspace"
583 Apply for the user-space tracer
588 \fBdisable-event\fP NAME[,NAME2,...] (\-k | \-u) [OPTIONS]
590 Disable tracing event
592 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
595 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
602 Show summary of possible options and commands.
604 .BR "\-\-list-options"
605 Simple listing of options
607 .BR "\-s, \-\-session NAME"
608 Apply on session name
610 .BR "\-a, \-\-all-events"
611 Disable all events. This does NOT disable "*" but rather every known
612 events of the session.
614 .BR "\-k, \-\-kernel"
615 Apply for the kernel tracer
617 .BR "\-u, \-\-userspace"
618 Apply for the user-space tracer
623 \fBlist\fP [OPTIONS] [SESSION [SESSION OPTIONS]]
625 List tracing session information.
627 With no arguments, it will list available tracing session(s).
629 With the session name, it will display the details of the session including
630 the trace file path, the associated channels and their state (activated
631 and deactivated), the activated events and more.
633 With \-k alone, it will list all available kernel events (except the system
635 With \-u alone, it will list all available user-space events from registered
636 applications. Here is an example of 'lttng list \-u':
639 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
640 ust_tests_hello:tptest_sighandler (type: tracepoint)
641 ust_tests_hello:tptest (type: tracepoint)
644 You can now enable any event listed by using the name :
645 \fBust_tests_hello:tptest\fP.
651 Show summary of possible options and commands.
653 .BR "\-\-list-options"
654 Simple listing of options
656 .BR "\-k, \-\-kernel"
659 .BR "\-u, \-\-userspace"
660 Select user-space domain.
666 .BR "\-c, \-\-channel NAME"
667 List details of a channel
669 .BR "\-d, \-\-domain"
670 List available domain(s)
675 \fBset-session\fP NAME [OPTIONS]
677 Set current session name
679 Will change the session name in the .lttngrc file.
685 Show summary of possible options and commands.
687 .BR "\-\-list-options"
688 Simple listing of options
693 \fBsnapshot\fP [OPTIONS] ACTION
695 Snapshot command for LTTng session.
701 Show summary of possible options and commands.
703 .BR "\-\-list-options"
704 Simple listing of options
710 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
712 Setup and add an snapshot output for a session. Output are the destination
713 where the snapshot will be sent. Only one output is permitted. To change it,
714 you'll need to delete it and add back the new one.
717 \fBdel-output\fP ID | NAME [-s <NAME>]
719 Delete an output for a session using the ID. You can either specify the
720 output's ID that can be found with list-output or the name.
723 \fBlist-output\fP [-s <NAME>]
725 List the output of a session. Attributes of the output are printed.
728 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
730 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
731 used instead of a previously added output. Specifying only a name or/and a max
732 size will override the current output values. For instance, you can record a
733 snapshot with a custom maximum size or with a different name.
736 $ lttng add-output -n mysnapshot file:///data/snapshot
738 $ lttng snapshot record -n new_name_snapshot
741 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
742 rather then in mysnapshot*/
745 .B DETAILED ACTION OPTIONS
748 .BR "\-s, \-\-session NAME"
749 Apply to session name.
751 .BR "\-n, \-\-name NAME"
752 Name of the snapshot's output.
754 .BR "\-m, \-\-max-size SIZE"
755 Maximum size in bytes of the snapshot. The maxium size does not include
758 .BR "\-C, \-\-ctrl-url URL"
759 Set control path URL. (Must use -D also)
761 .BR "\-D, \-\-data-url URL"
762 Set data path URL. (Must use -C also)
767 \fBstart\fP [NAME] [OPTIONS]
771 It will start tracing for all tracers for a specific tracing session.
772 If NAME is omitted, the session name is taken from the .lttngrc file.
778 Show summary of possible options and commands.
780 .BR "\-\-list-options"
781 Simple listing of options
786 \fBstop\fP [NAME] [OPTIONS]
790 It will stop tracing for all tracers for a specific tracing session. Before
791 returning, the command checks for data availability meaning that it will wait
792 until the trace is readable for the session. Use \-\-no-wait to avoid this
795 If NAME is omitted, the session name is taken from the .lttngrc file.
801 Show summary of possible options and commands.
803 .BR "\-\-list-options"
804 Simple listing of options
806 Don't wait for data availability.
813 Show version information
819 Show summary of possible options and commands.
821 .BR "\-\-list-options"
822 Simple listing of options
827 \fBview\fP [SESSION_NAME] [OPTIONS]
829 View traces of a tracing session. By default, the babeltrace viewer
830 will be used for text viewing. If SESSION_NAME is omitted, the session
831 name is taken from the .lttngrc file.
839 .BR "\-\-list-options"
840 Simple listing of options
842 .BR "\-t, \-\-trace-path PATH"
843 Trace directory path for the viewer
845 .BR "\-e, \-\-viewer CMD"
846 Specify viewer and/or options to use This will completely override the
847 default viewers so please make sure to specify the full command. The
848 trace directory path of the session will be appended at the end to the
855 On success 0 is returned and a positive value on error. Value of 1 means a command
856 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
857 something went wrong during the command.
859 Any other value above 10, please refer to
860 .BR "<lttng/lttng-error.h>"
861 for a detailed list or use lttng_strerror() to get a human readable string of
865 .SH "ENVIRONMENT VARIABLES"
868 Note that all command line options override environment variables.
872 .IP "LTTNG_SESSIOND_PATH"
873 Allows one to specify the full session daemon binary path to lttng command line
874 tool. You can also use \-\-sessiond-path option having the same effect.
880 .BR lttng-sessiond(8),
882 .BR lttng-health-check(3)
887 If you encounter any issues or usability problem, please report it on our
888 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
889 at https://bugs.lttng.org which is a bugtracker.
895 lttng is distributed under the GNU General Public License version 2. See the file
898 A Web site is available at http://lttng.org for more information on the LTTng
901 You can also find our git tree at http://git.lttng.org.
903 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
905 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
910 Thanks to Yannick Brosseau without whom this project would never have been so
911 lean and mean! Also thanks to the Ericsson teams working on tracing which
912 helped us greatly with detailed bug reports and unusual test cases.
914 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
915 maintainer) and Jon Bernard for our Debian packages.
917 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
918 Montreal for the LTTng journey.
923 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
924 David Goulet. More people have since contributed to it. It is currently
925 maintained by David Goulet <dgoulet@efficios.com>.