1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
4 v2.12, 25 February 2021
7 include::../common/copyright.txt[]
10 include::../common/welcome.txt[]
13 include::../common/audience.txt[]
17 === What's in this documentation?
19 The LTTng Documentation is divided into the following sections:
21 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
22 rudiments of software tracing and the rationale behind the
25 Skip this section if you’re familiar with software tracing and with the
28 * **<<installing-lttng,Installation>>** describes the steps to
29 install the LTTng packages on common Linux distributions and from
32 Skip this section if you already properly installed LTTng on your target
35 * **<<getting-started,Quick start>>** is a concise guide to
36 getting started quickly with LTTng kernel and user space tracing.
38 We recommend this section if you're new to LTTng or to software tracing
41 Skip this section if you're not new to LTTng.
43 * **<<core-concepts,Core concepts>>** explains the concepts at
46 It's a good idea to become familiar with the core concepts
47 before attempting to use the toolkit.
49 * **<<plumbing,Components of LTTng>>** describes the various components
50 of the LTTng machinery, like the daemons, the libraries, and the
51 command-line interface.
52 * **<<instrumenting,Instrumentation>>** shows different ways to
53 instrument user applications and the Linux kernel.
55 Instrumenting source code is essential to provide a meaningful
58 Skip this section if you don't have a programming background.
60 * **<<controlling-tracing,Tracing control>>** is divided into topics
61 which demonstrate how to use the vast array of features that
62 LTTng{nbsp}{revision} offers.
63 * **<<reference,Reference>>** contains reference tables.
64 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
65 to LTTng or to the field of software tracing.
68 include::../common/convention.txt[]
71 include::../common/acknowledgements.txt[]
75 == What's new in LTTng{nbsp}{revision}?
77 LTTng{nbsp}{revision} bears the name _Ta Meilleure_, a Northeast IPA
78 beer brewed by https://lagabiere.com/[Lagabière]. Translating to ``Your
79 best one'', this beer gives out strong aromas of passion fruit, lemon,
80 and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a
81 smooth lingering hop bitterness.
83 New features and changes in LTTng{nbsp}{revision}:
87 * Clear the contents of one or more <<tracing-session,tracing sessions>>
88 without having to destroy and reconfigure them
89 with the new man:lttng-clear(1) command.
91 This is especially useful to clear the tracing data of a tracing session
92 between attempts to reproduce a problem.
94 See <<clear,Clear a tracing session>>.
96 * Before LTTng{nbsp}{revision}, the man:lttng-track(1) and
97 man:lttng-untrack(1) commands used to add and remove process IDs
98 (PIDs) to a whitelist so that LTTng would only trace processes with
101 LTTng{nbsp}{revision} adds Unix user IDs (UIDs) and Unix group IDs
102 (GIDs) to the available <<pid-tracking,process attributes to track>>.
103 You can specify numeric user/group IDs and user/group names to track,
108 $ lttng track --userspace --vuid=http,999 --vgid=mysql,9
111 While you can also track UIDs and GIDs with the
112 opt:lttng-enable-event(1):--filter option of the `enable-event` command,
113 this dedicated process attribute tracking approach reduces tracing
114 overhead and prevents the creation of <<def-sub-buffer,sub-buffers>> for
115 the users and groups which LTTng doesn't track.
117 In the command manual pages, the term ``whitelist'' is renamed to
118 ``inclusion set'' to clarify the concept.
120 * The <<lttng-relayd,relay daemon>> can now maintain many files
121 virtually opened without using as many file descriptors (FD). It does
122 so by closing and reopening FDs as needed.
124 This feature is meant as a workaround for users who can't bump the
125 system limit because of permission restrictions.
127 The new opt:lttng-relayd(8):--fd-pool-size relay daemon option
128 sets the maximum number of simultaneously opened file descriptors
129 (using the soft `RLIMIT_NOFILE` resource limit of the process by
130 default; see man:getrlimit(2)).
132 * By default, the relay daemon writes its traces under a predefined
134 +$LTTNG_HOME/lttng-traces/__host__/__session__/__domain__+, with:
141 <<tracing-session,Tracing session>> name.
144 <<domain,Tracing domain>> name (`ust` or `kernel`).
147 Change this hierarchy to group traces by tracing session name rather
149 (+$LTTNG_HOME/lttng-traces/__session__/__host__/__domain__+) with the
150 new opt:lttng-relayd(8):--group-output-by-session option of the
153 This feature is especially useful if you're tracing two or more hosts,
154 having different hostnames, which share the same tracing session name as
155 part of their configuration. In this scenario, you can use a descriptive
156 tracing session name (for example, `connection-hang`) across a fleet of
157 machines streaming to a single relay daemon.
159 * The relay daemon has a new opt:lttng-relayd(8):--working-directory
160 option to override its working directory.
162 Linux kernel tracing::
164 * New instrumentation hooks to trace the entry and exit tracepoints of
165 the network reception code paths of the Linux kernel.
167 Use the resulting event records to identify the bounds of a network
168 reception and link the events that occur in the interim (for example,
169 wake-ups) to a specific network reception instance. You can also
170 analyze the latency of the network stack thanks to those event records.
172 * The `thread` field of the `irqaction` structure, which specifies the
173 process to wake up when a threaded interrupt request (IRQ) occurs, is
174 now part of the `lttng_statedump_interrupt` event record.
176 Use this information to discover which processes handle the various
177 IRQs. You can also associate the events occurring in the context of
178 those processes with their respective IRQ.
180 * New `lttng_statedump_cpu_topology` tracepoint to record the active
183 Use this information to discover which CPUs are SMT siblings or part of
184 the same socket. As of LTTng{nbsp}{revision}, only the x86 architecture
185 is supported since all architectures describe their topologies
188 The `architecture` field of the tracepoint is statically defined and
189 exists for all architecture implementations. Analysis tools can
190 therefore anticipate the layout of the event record.
192 Event record example:
196 lttng_statedump_cpu_topology:
202 model_name: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
208 * New product UUID metadata environment field, `product_uuid`,
209 which LTTng copies from the
210 https://en.wikipedia.org/wiki/Desktop_Management_Interface[Desktop
211 Management Interface] (DMI).
213 Use this environment field to uniquely identify a machine (virtual or
214 physical) in order to correlate traces from multiple virtual machines.
220 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
221 generation_ is a modern toolkit for tracing Linux systems and
222 applications. So your first question might be:
229 As the history of software engineering progressed and led to what
230 we now take for granted--complex, numerous and
231 interdependent software applications running in parallel on
232 sophisticated operating systems like Linux--the authors of such
233 components, software developers, began feeling a natural
234 urge to have tools that would ensure the robustness and good performance
235 of their masterpieces.
237 One major achievement in this field is, inarguably, the
238 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
239 an essential tool for developers to find and fix bugs. But even the best
240 debugger won't help make your software run faster, and nowadays, faster
241 software means either more work done by the same hardware, or cheaper
242 hardware for the same work.
244 A _profiler_ is often the tool of choice to identify performance
245 bottlenecks. Profiling is suitable to identify _where_ performance is
246 lost in a given software. The profiler outputs a profile, a statistical
247 summary of observed events, which you may use to discover which
248 functions took the most time to execute. However, a profiler won't
249 report _why_ some identified functions are the bottleneck. Bottlenecks
250 might only occur when specific conditions are met, conditions that are
251 sometimes impossible to capture by a statistical profiler, or impossible
252 to reproduce with an application altered by the overhead of an
253 event-based profiler. For a thorough investigation of software
254 performance issues, a history of execution is essential, with the
255 recorded values of variables and context fields you choose, and
256 with as little influence as possible on the instrumented software. This
257 is where tracing comes in handy.
259 _Tracing_ is a technique used to understand what goes on in a running
260 software system. The software used for tracing is called a _tracer_,
261 which is conceptually similar to a tape recorder. When recording,
262 specific instrumentation points placed in the software source code
263 generate events that are saved on a giant tape: a _trace_ file. You
264 can trace user applications and the operating system at the same time,
265 opening the possibility of resolving a wide range of problems that would
266 otherwise be extremely challenging.
268 Tracing is often compared to _logging_. However, tracers and loggers are
269 two different tools, serving two different purposes. Tracers are
270 designed to record much lower-level events that occur much more
271 frequently than log messages, often in the range of thousands per
272 second, with very little execution overhead. Logging is more appropriate
273 for a very high-level analysis of less frequent events: user accesses,
274 exceptional conditions (errors and warnings, for example), database
275 transactions, instant messaging communications, and such. Simply put,
276 logging is one of the many use cases that can be satisfied with tracing.
278 The list of recorded events inside a trace file can be read manually
279 like a log file for the maximum level of detail, but it is generally
280 much more interesting to perform application-specific analyses to
281 produce reduced statistics and graphs that are useful to resolve a
282 given problem. Trace viewers and analyzers are specialized tools
285 In the end, this is what LTTng is: a powerful, open source set of
286 tools to trace the Linux kernel and user applications at the same time.
287 LTTng is composed of several components actively maintained and
288 developed by its link:/community/#where[community].
291 [[lttng-alternatives]]
292 === Alternatives to noch:{LTTng}
294 Excluding proprietary solutions, a few competing software tracers
297 https://github.com/dtrace4linux/linux[dtrace4linux]::
298 A port of Sun Microsystems' DTrace to Linux.
300 The cmd:dtrace tool interprets user scripts and is responsible for
301 loading code into the Linux kernel for further execution and collecting
304 https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
305 A subsystem in the Linux kernel in which a virtual machine can
306 execute programs passed from the user space to the kernel.
308 You can attach such programs to tracepoints and kprobes thanks to a
309 system call, and they can output data to the user space when executed
310 thanks to different mechanisms (pipe, VM register values, and eBPF maps,
313 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
314 The de facto function tracer of the Linux kernel.
316 Its user interface is a set of special files in sysfs.
318 https://perf.wiki.kernel.org/[perf]::
319 A performance analysis tool for Linux which supports hardware
320 performance counters, tracepoints, as well as other counters and
323 The controlling utility of perf is the cmd:perf command line/text UI
326 http://linux.die.net/man/1/strace[strace]::
327 A command-line utility which records system calls made by a
328 user process, as well as signal deliveries and changes of process
331 strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
332 fulfill its function.
334 http://www.sysdig.org/[sysdig]::
335 Like SystemTap, uses scripts to analyze Linux kernel events.
337 You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
338 sysdig executes them while it traces the system or afterwards. The
339 interface of sysdig is the cmd:sysdig command-line tool as well as the
340 text UI-based cmd:csysdig tool.
342 https://sourceware.org/systemtap/[SystemTap]::
343 A Linux kernel and user space tracer which uses custom user scripts
344 to produce plain text traces.
346 SystemTap converts the scripts to the C language, and then compiles them
347 as Linux kernel modules which are loaded to produce trace data. The
348 primary user interface of SystemTap is the cmd:stap command-line tool.
350 The main distinctive features of LTTng is that it produces correlated
351 kernel and user space traces, as well as doing so with the lowest
352 overhead amongst other solutions. It produces trace files in the
353 http://diamon.org/ctf[CTF] format, a file format optimized
354 for the production and analyses of multi-gigabyte data.
356 LTTng is the result of more than 10{nbsp}years of active open source
357 development by a community of passionate developers. LTTng is currently
358 available on major desktop and server Linux distributions.
360 The main interface for tracing control is a single command-line tool
361 named cmd:lttng. The latter can create several tracing sessions, enable
362 and disable events on the fly, filter events efficiently with custom
363 user expressions, start and stop tracing, and much more. LTTng can
364 record the traces on the file system or send them over the network, and
365 keep them totally or partially. You can view the traces once tracing
366 becomes inactive or in real-time.
368 <<installing-lttng,Install LTTng now>> and
369 <<getting-started,start tracing>>!
375 **LTTng** is a set of software <<plumbing,components>> which interact to
376 <<instrumenting,instrument>> the Linux kernel and user applications, and
377 to <<controlling-tracing,control tracing>> (start and stop
378 tracing, enable and disable event rules, and the rest). Those
379 components are bundled into the following packages:
382 Libraries and command-line interface to control tracing.
385 Linux kernel modules to instrument and trace the kernel.
388 Libraries and Java/Python packages to instrument and trace user
391 Most distributions mark the LTTng-modules and LTTng-UST packages as
392 optional when installing LTTng-tools (which is always required). In the
393 following sections, we always provide the steps to install all three,
396 * You only need to install LTTng-modules if you intend to trace the
398 * You only need to install LTTng-UST if you intend to trace user
402 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 25{nbsp}February{nbsp}2021.
404 |Distribution |Available in releases
406 |https://www.ubuntu.com/[Ubuntu]
407 |<<ubuntu,Ubuntu 20.10 _Groovy Gorilla_>>.
409 Ubuntu{nbsp}16.04 _Xenial Xerus_, Ubuntu{nbsp}18.04 _Bionic Beaver_,
410 and Ubuntu{nbsp}20.04 _Focal Fossa_:
411 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
413 |https://www.debian.org/[Debian]
414 |<<debian,Debian "bullseye" (testing)>>.
416 |https://getfedora.org/[Fedora]
417 |xref:fedora[Fedora{nbsp}33, Fedora{nbsp}34, and Fedora{nbsp}35].
419 |https://www.archlinux.org/[Arch Linux]
420 |<<arch-linux,_Community_ repository and AUR>>.
422 |https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
423 |See http://packages.efficios.com/[EfficiOS Enterprise Packages].
425 |https://alpinelinux.org/[Alpine Linux]
426 |<<alpine-linux,Alpine Linux{nbsp}3.12 and Alpine Linux{nbsp}3.13>>.
428 |https://buildroot.org/[Buildroot]
429 |xref:buildroot[Buildroot{nbsp}2020.08 and Buildroot{nbsp}2020.11].
431 |https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
432 https://www.yoctoproject.org/[Yocto]
433 |<<oe-yocto,Yocto Project{nbsp}3.2 _Gatesgarth_>>
434 (`openembedded-core` layer).
439 === [[ubuntu-official-repositories]]Ubuntu
441 LTTng{nbsp}{revision} is available on Ubuntu{nbsp}20.10 _Groovy
442 Gorilla_. For previous supported releases of Ubuntu, <<ubuntu-ppa,use
443 the LTTng Stable{nbsp}{revision} PPA>>.
445 To install LTTng{nbsp}{revision} on Ubuntu{nbsp}20.10 _Groovy Gorilla_:
447 . Install the main LTTng{nbsp}{revision} packages:
452 # apt-get install lttng-tools
453 # apt-get install lttng-modules-dkms
454 # apt-get install liblttng-ust-dev
458 . **If you need to instrument and trace
459 <<java-application,Java applications>>**, install the LTTng-UST
465 # apt-get install liblttng-ust-agent-java
469 . **If you need to instrument and trace
470 <<python-application,Python{nbsp}3 applications>>**, install the
471 LTTng-UST Python agent:
476 # apt-get install python3-lttngust
482 === Ubuntu: noch:{LTTng} Stable {revision} PPA
484 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
485 Stable{nbsp}{revision} PPA] offers the latest stable
486 LTTng{nbsp}{revision} packages for Ubuntu{nbsp}16.04 _Xenial Xerus_,
487 Ubuntu{nbsp}18.04 _Bionic Beaver_, and Ubuntu{nbsp}20.04 _Focal Fossa_.
489 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
492 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
498 # apt-add-repository ppa:lttng/stable-2.12
503 . Install the main LTTng{nbsp}{revision} packages:
508 # apt-get install lttng-tools
509 # apt-get install lttng-modules-dkms
510 # apt-get install liblttng-ust-dev
514 . **If you need to instrument and trace
515 <<java-application,Java applications>>**, install the LTTng-UST
521 # apt-get install liblttng-ust-agent-java
525 . **If you need to instrument and trace
526 <<python-application,Python{nbsp}3 applications>>**, install the
527 LTTng-UST Python agent:
532 # apt-get install python3-lttngust
540 To install LTTng{nbsp}{revision} on Debian "bullseye" (testing):
542 . Install the main LTTng{nbsp}{revision} packages:
547 # apt-get install lttng-modules-dkms
548 # apt-get install liblttng-ust-dev
549 # apt-get install lttng-tools
553 . **If you need to instrument and trace <<java-application,Java
554 applications>>**, install the LTTng-UST Java agent:
559 # apt-get install liblttng-ust-agent-java
563 . **If you need to instrument and trace <<python-application,Python
564 applications>>**, install the LTTng-UST Python agent:
569 # apt-get install python3-lttngust
577 To install LTTng{nbsp}{revision} on Fedora{nbsp}33, Fedora{nbsp}34, or
580 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
586 # yum install lttng-tools
587 # yum install lttng-ust
591 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
597 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
598 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
599 cd lttng-modules-2.12.* &&
601 sudo make modules_install &&
607 .Java and Python application instrumentation and tracing
609 If you need to instrument and trace <<java-application,Java
610 applications>> on Fedora, you need to build and install
611 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
612 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
613 `--enable-java-agent-all` options to the `configure` script, depending
614 on which Java logging framework you use.
616 If you need to instrument and trace <<python-application,Python
617 applications>> on Fedora, you need to build and install
618 LTTng-UST{nbsp}{revision} from source and pass the
619 `--enable-python-agent` option to the `configure` script.
626 LTTng-UST{nbsp}{revision} is available in the _community_
627 repository of Arch Linux, while LTTng-tools{nbsp}{revision} and
628 LTTng-modules{nbsp}{revision} are available in the
629 https://aur.archlinux.org/[AUR].
631 To install LTTng{nbsp}{revision} on Arch Linux, using
632 https://github.com/Jguer/yay[yay] for the AUR packages:
634 . Install the main LTTng{nbsp}{revision} packages:
639 # pacman -Sy lttng-ust
640 $ yay -Sy lttng-tools
641 $ yay -Sy lttng-modules
645 . **If you need to instrument and trace <<python-application,Python
646 applications>>**, install the LTTng-UST Python agent:
651 # pacman -Sy python-lttngust
652 # pacman -Sy python2-lttngust
660 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
661 Alpine Linux{nbsp}3.12 or Alpine Linux{nbsp}3.13:
663 . Add the LTTng packages:
668 # apk add lttng-tools
669 # apk add lttng-ust-dev
673 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
679 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
680 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
681 cd lttng-modules-2.12.* &&
683 sudo make modules_install &&
692 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2020.08 or
693 Buildroot{nbsp}2020.11:
695 . Launch the Buildroot configuration tool:
704 . In **Kernel**, check **Linux kernel**.
705 . In **Toolchain**, check **Enable WCHAR support**.
706 . In **Target packages**{nbsp}→ **Debugging, profiling and benchmark**,
707 check **lttng-modules** and **lttng-tools**.
708 . In **Target packages**{nbsp}→ **Libraries**{nbsp}→
709 **Other**, check **lttng-libust**.
713 === OpenEmbedded and Yocto
715 LTTng{nbsp}{revision} recipes are available in the
716 https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
717 layer for Yocto Project{nbsp}3.2 _Gatesgarth_ under the following names:
723 With BitBake, the simplest way to include LTTng recipes in your target
724 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
727 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
732 . Select a machine and an image recipe.
733 . Click **Edit image recipe**.
734 . Under the **All recipes** tab, search for **lttng**.
735 . Check the desired LTTng recipes.
738 [[building-from-source]]
739 === Build from source
741 To build and install LTTng{nbsp}{revision} from source:
743 . Using the package manager of your distribution, or from source,
744 install the following dependencies of LTTng-tools and LTTng-UST:
747 * https://sourceforge.net/projects/libuuid/[libuuid]
748 * http://directory.fsf.org/wiki/Popt[popt]
749 * http://liburcu.org/[Userspace RCU]
750 * http://www.xmlsoft.org/[libxml2]
751 * **Optional**: https://github.com/numactl/numactl[numactl]
754 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
760 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
761 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
762 cd lttng-modules-2.12.* &&
764 sudo make modules_install &&
769 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
775 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
776 tar -xf lttng-ust-latest-2.12.tar.bz2 &&
777 cd lttng-ust-2.12.* &&
785 Add `--disable-numa` to `./configure` if you don't have
786 https://github.com/numactl/numactl[numactl].
790 .Java and Python application tracing
792 If you need to instrument and trace <<java-application,Java
793 applications>>, pass the `--enable-java-agent-jul`,
794 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
795 `configure` script, depending on which Java logging framework you use.
797 If you need to instrument and trace <<python-application,Python
798 applications>>, pass the `--enable-python-agent` option to the
799 `configure` script. You can set the `PYTHON` environment variable to the
800 path to the Python interpreter for which to install the LTTng-UST Python
808 By default, LTTng-UST libraries are installed to
809 dir:{/usr/local/lib}, which is the de facto directory in which to
810 keep self-compiled and third-party libraries.
812 When <<building-tracepoint-providers-and-user-application,linking an
813 instrumented user application with `liblttng-ust`>>:
815 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
817 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
818 man:gcc(1), man:g++(1), or man:clang(1).
822 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
828 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
829 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
830 cd lttng-tools-2.12.* &&
838 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
839 previous steps automatically for a given version of LTTng and confine
840 the installed files in a specific directory. This can be useful to test
841 LTTng without installing it on your system.
847 This is a short guide to get started quickly with LTTng kernel and user
850 Before you follow this guide, make sure to <<installing-lttng,install>>
853 This tutorial walks you through the steps to:
855 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
856 . <<tracing-your-own-user-application,Trace a user application>> written
858 . <<viewing-and-analyzing-your-traces,View and analyze the
862 [[tracing-the-linux-kernel]]
863 === Trace the Linux kernel
865 The following command lines start with the `#` prompt because you need
866 root privileges to trace the Linux kernel. You can also trace the kernel
867 as a regular user if your Unix user is a member of the
868 <<tracing-group,tracing group>>.
870 . Create a <<tracing-session,tracing session>> which writes its traces
871 to dir:{/tmp/my-kernel-trace}:
876 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
880 . List the available kernel tracepoints and system calls:
885 # lttng list --kernel
886 # lttng list --kernel --syscall
890 . Create <<event,event rules>> which match the desired instrumentation
891 point names, for example the `sched_switch` and `sched_process_fork`
892 tracepoints, and the man:open(2) and man:close(2) system calls:
897 # lttng enable-event --kernel sched_switch,sched_process_fork
898 # lttng enable-event --kernel --syscall open,close
902 Create an event rule which matches _all_ the Linux kernel
903 tracepoints with the opt:lttng-enable-event(1):--all option
904 (this will generate a lot of data when tracing):
909 # lttng enable-event --kernel --all
913 . <<basic-tracing-session-control,Start tracing>>:
922 . Do some operation on your system for a few seconds. For example,
923 load a website, or list the files of a directory.
924 . <<creating-destroying-tracing-sessions,Destroy>> the current
934 The man:lttng-destroy(1) command doesn't destroy the trace data; it
935 only destroys the state of the tracing session.
937 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
938 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
939 session>>). You need to stop tracing to make LTTng flush the remaining
940 trace data and make the trace readable.
942 . For the sake of this example, make the recorded trace accessible to
948 # chown -R $(whoami) /tmp/my-kernel-trace
952 See <<viewing-and-analyzing-your-traces,View and analyze the
953 recorded events>> to view the recorded events.
956 [[tracing-your-own-user-application]]
957 === Trace a user application
959 This section steps you through a simple example to trace a
960 _Hello world_ program written in C.
962 To create the traceable user application:
964 . Create the tracepoint provider header file, which defines the
965 tracepoints and the events they can generate:
971 #undef TRACEPOINT_PROVIDER
972 #define TRACEPOINT_PROVIDER hello_world
974 #undef TRACEPOINT_INCLUDE
975 #define TRACEPOINT_INCLUDE "./hello-tp.h"
977 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
980 #include <lttng/tracepoint.h>
990 ctf_string(my_string_field, my_string_arg)
991 ctf_integer(int, my_integer_field, my_integer_arg)
995 #endif /* _HELLO_TP_H */
997 #include <lttng/tracepoint-event.h>
1001 . Create the tracepoint provider package source file:
1007 #define TRACEPOINT_CREATE_PROBES
1008 #define TRACEPOINT_DEFINE
1010 #include "hello-tp.h"
1014 . Build the tracepoint provider package:
1019 $ gcc -c -I. hello-tp.c
1023 . Create the _Hello World_ application source file:
1030 #include "hello-tp.h"
1032 int main(int argc, char *argv[])
1036 puts("Hello, World!\nPress Enter to continue...");
1039 * The following getchar() call is only placed here for the purpose
1040 * of this demonstration, to pause the application in order for
1041 * you to have time to list its tracepoints. It's not needed
1047 * A tracepoint() call.
1049 * Arguments, as defined in hello-tp.h:
1051 * 1. Tracepoint provider name (required)
1052 * 2. Tracepoint name (required)
1053 * 3. my_integer_arg (first user-defined argument)
1054 * 4. my_string_arg (second user-defined argument)
1056 * Notice the tracepoint provider and tracepoint names are
1057 * NOT strings: they are in fact parts of variables that the
1058 * macros in hello-tp.h create.
1060 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
1062 for (x = 0; x < argc; ++x) {
1063 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
1066 puts("Quitting now!");
1067 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
1074 . Build the application:
1083 . Link the application with the tracepoint provider package,
1084 `liblttng-ust`, and `libdl`:
1089 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1093 Here's the whole build process:
1096 .Build steps of the user space tracing tutorial.
1097 image::ust-flow.png[]
1099 To trace the user application:
1101 . Run the application with a few arguments:
1106 $ ./hello world and beyond
1115 Press Enter to continue...
1119 . Start an LTTng <<lttng-sessiond,session daemon>>:
1124 $ lttng-sessiond --daemonize
1128 Note that a session daemon might already be running, for example as a
1129 service that the service manager of the distribution started.
1131 . List the available user space tracepoints:
1136 $ lttng list --userspace
1140 You see the `hello_world:my_first_tracepoint` tracepoint listed
1141 under the `./hello` process.
1143 . Create a <<tracing-session,tracing session>>:
1148 $ lttng create my-user-space-session
1152 . Create an <<event,event rule>> which matches the
1153 `hello_world:my_first_tracepoint` event name:
1158 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1162 . <<basic-tracing-session-control,Start tracing>>:
1171 . Go back to the running `hello` application and press Enter. The
1172 program executes all `tracepoint()` instrumentation points and exits.
1173 . <<creating-destroying-tracing-sessions,Destroy>> the current
1183 The man:lttng-destroy(1) command doesn't destroy the trace data; it
1184 only destroys the state of the tracing session.
1186 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1187 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
1188 session>>). You need to stop tracing to make LTTng flush the remaining
1189 trace data and make the trace readable.
1191 By default, LTTng saves the traces in
1192 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
1193 where +__name__+ is the tracing session name. The
1194 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
1196 See <<viewing-and-analyzing-your-traces,View and analyze the
1197 recorded events>> to view the recorded events.
1200 [[viewing-and-analyzing-your-traces]]
1201 === View and analyze the recorded events
1203 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
1204 kernel>> and <<tracing-your-own-user-application,Trace a user
1205 application>> tutorials, you can inspect the recorded events.
1207 There are many tools you can use to read LTTng traces:
1209 https://babeltrace.org/[Babeltrace{nbsp}2]::
1210 A rich, flexible trace manipulation toolkit which includes
1211 a versatile command-line interface
1212 (https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]),
1213 a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C library],
1214 and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1215 so that you can easily process or convert an LTTng trace with
1218 The Babeltrace{nbsp}2 project ships with a
1219 https://babeltrace.org/docs/v2.0/man7/babeltrace2-plugin-ctf.7/[plugin]
1220 which supports the format of the traces which LTTng produces,
1221 https://diamon.org/ctf/[CTF].
1223 http://tracecompass.org/[Trace Compass]::
1224 A graphical user interface for viewing and analyzing any type of
1225 logs or traces, including those of LTTng.
1227 https://github.com/lttng/lttng-analyses[**LTTng analyses**]::
1228 An experimental project which includes many high-level analyses of
1229 LTTng kernel traces, like scheduling statistics, interrupt
1230 frequency distribution, top CPU usage, and more.
1232 NOTE: This section assumes that LTTng saved the traces it recorded
1233 during the previous tutorials to their default location, in the
1234 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1235 environment variable defaults to `$HOME` if not set.
1238 [[viewing-and-analyzing-your-traces-bt]]
1239 ==== Use the cmd:babeltrace2 command-line tool
1241 The simplest way to list all the recorded events of an LTTng trace is to
1243 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
1248 $ babeltrace2 ~/lttng-traces/my-user-space-session*
1251 cmd:babeltrace2 finds all traces recursively within the given path and
1252 prints all their events, sorting them chronologically.
1254 Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1259 $ babeltrace2 /tmp/my-kernel-trace | grep _switch
1262 Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1263 the recorded events:
1267 $ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1271 [[viewing-and-analyzing-your-traces-bt-python]]
1272 ==== Use the Babeltrace{nbsp}2 Python bindings
1274 The <<viewing-and-analyzing-your-traces-bt,text output of
1275 cmd:babeltrace2>> is useful to isolate events by simple matching using
1276 man:grep(1) and similar utilities. However, more elaborate filters, such
1277 as keeping only event records with a field value falling within a
1278 specific range, are not trivial to write using a shell. Moreover,
1279 reductions and even the most basic computations involving multiple event
1280 records are virtually impossible to implement.
1282 Fortunately, Babeltrace{nbsp}2 ships with
1283 https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1284 which make it easy to read the event records of an LTTng trace
1285 sequentially and compute the desired information.
1287 The following script accepts an LTTng Linux kernel trace path as its
1288 first argument and prints the short names of the top five running
1289 processes on CPU{nbsp}0 during the whole trace:
1299 # Get the trace path from the first command-line argument.
1300 it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1302 # This counter dictionary will hold execution times:
1304 # Task command name -> Total execution time (ns)
1305 exec_times = collections.Counter()
1307 # This holds the last `sched_switch` timestamp.
1311 # We only care about event messages.
1312 if type(msg) is not bt2._EventMessageConst:
1315 # Event of the event message.
1318 # Keep only `sched_switch` events.
1319 if event.cls.name != 'sched_switch':
1322 # Keep only events which occurred on CPU 0.
1323 if event.packet.context_field['cpu_id'] != 0:
1326 # Event timestamp (ns).
1327 cur_ts = msg.default_clock_snapshot.ns_from_origin
1333 # (Short) name of the previous task command.
1334 prev_comm = str(event.payload_field['prev_comm'])
1336 # Initialize an entry in our dictionary if not yet done.
1337 if prev_comm not in exec_times:
1338 exec_times[prev_comm] = 0
1340 # Compute previous command execution time.
1341 diff = cur_ts - last_ts
1343 # Update execution time of this command.
1344 exec_times[prev_comm] += diff
1346 # Update last timestamp.
1350 for name, ns in exec_times.most_common(5):
1351 print('{:20}{} s'.format(name, ns / 1e9))
1354 if __name__ == '__main__':
1362 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1368 swapper/0 48.607245889 s
1369 chromium 7.192738188 s
1370 pavucontrol 0.709894415 s
1371 Compositor 0.660867933 s
1372 Xorg.bin 0.616753786 s
1375 Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1376 since we weren't using the CPU that much when tracing, its first
1377 position in the list makes sense.
1381 == [[understanding-lttng]]Core concepts
1383 From a user's perspective, the LTTng system is built on a few concepts,
1384 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1385 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1386 Understanding how those objects relate to eachother is key in mastering
1389 The core concepts are:
1391 * <<tracing-session,Tracing session>>
1392 * <<domain,Tracing domain>>
1393 * <<channel,Channel and ring buffer>>
1394 * <<"event","Instrumentation point, event rule, event, and event record">>
1400 A _tracing session_ is a stateful dialogue between you and
1401 a <<lttng-sessiond,session daemon>>. You can
1402 <<creating-destroying-tracing-sessions,create a new tracing
1403 session>> with the `lttng create` command.
1405 Most of what you do when you control LTTng tracers happens within a
1406 tracing session. In particular, a tracing session:
1409 * Has its own set of trace files.
1410 * Has its own state of activity (started or stopped).
1411 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1413 * Has its own <<channel,channels>> to which are associated their own
1414 <<event,event rules>>.
1415 * Has its own <<pid-tracking,process attribute tracking>> inclusion
1419 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1420 image::concepts.png[]
1422 Those attributes and objects are completely isolated between different
1425 A tracing session is analogous to a cash machine session:
1426 the operations you do on the banking system through the cash machine do
1427 not alter the data of other users of the same system. In the case of
1428 the cash machine, a session lasts as long as your bank card is inside.
1429 In the case of LTTng, a tracing session lasts from the `lttng create`
1430 command to the `lttng destroy` command.
1433 .Each Unix user has its own set of tracing sessions.
1434 image::many-sessions.png[]
1437 [[tracing-session-mode]]
1438 ==== Tracing session mode
1440 LTTng can send the generated trace data to different locations. The
1441 _tracing session mode_ dictates where to send it. The following modes
1442 are available in LTTng{nbsp}{revision}:
1445 LTTng writes the traces to the file system of the machine it traces
1448 Network streaming mode::
1449 LTTng sends the traces over the network to a
1450 <<lttng-relayd,relay daemon>> running on a remote system.
1453 LTTng doesn't write the traces by default.
1455 Instead, you can request LTTng to <<taking-a-snapshot,take a snapshot>>,
1456 that is, a copy of the current sub-buffers of the tracing session, and
1457 to write it to the file system of the target or to send it over the
1458 network to a <<lttng-relayd,relay daemon>> running on a remote system.
1461 This mode is similar to the network streaming mode, but a live
1462 trace viewer can connect to the distant relay daemon to
1463 <<lttng-live,view event records as LTTng generates them>>.
1469 A _tracing domain_ is a namespace for event sources. A tracing domain
1470 has its own properties and features.
1472 There are currently five available tracing domains:
1476 * `java.util.logging` (JUL)
1480 You must specify a tracing domain when using some commands to avoid
1481 ambiguity. For example, since all the domains support named tracepoints
1482 as event sources (instrumentation points that you manually insert in the
1483 source code), you need to specify a tracing domain when
1484 <<enabling-disabling-events,creating an event rule>> because all the
1485 tracing domains could have tracepoints with the same names.
1487 You can create <<channel,channels>> in the Linux kernel and user space
1488 tracing domains. The other tracing domains have a single default
1493 === Channel and ring buffer
1495 A _channel_ is an object which is responsible for a set of ring buffers.
1496 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1497 tracer emits an event, it can record it to one or more
1498 sub-buffers. The attributes of a channel determine what to do when
1499 there's no space left for a new event record because all sub-buffers
1500 are full, where to send a full sub-buffer, and other behaviours.
1502 A channel is always associated to a <<domain,tracing domain>>. The
1503 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1504 a default channel which you can't configure.
1506 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1507 an event, it records it to the sub-buffers of all
1508 the enabled channels with a satisfied event rule, as long as those
1509 channels are part of active <<tracing-session,tracing sessions>>.
1512 [[channel-buffering-schemes]]
1513 ==== Per-user vs. per-process buffering schemes
1515 A channel has at least one ring buffer _per CPU_. LTTng always
1516 records an event to the ring buffer associated to the CPU on which it
1519 Two _buffering schemes_ are available when you
1520 <<enabling-disabling-channels,create a channel>> in the
1521 user space <<domain,tracing domain>>:
1523 Per-user buffering::
1524 Allocate one set of ring buffers--one per CPU--shared by all the
1525 instrumented processes of each Unix user.
1529 .Per-user buffering scheme.
1530 image::per-user-buffering.png[]
1533 Per-process buffering::
1534 Allocate one set of ring buffers--one per CPU--for each
1535 instrumented process.
1539 .Per-process buffering scheme.
1540 image::per-process-buffering.png[]
1543 The per-process buffering scheme tends to consume more memory than the
1544 per-user option because systems generally have more instrumented
1545 processes than Unix users running instrumented processes. However, the
1546 per-process buffering scheme ensures that one process having a high
1547 event throughput won't fill all the shared sub-buffers of the same
1550 The Linux kernel tracing domain has only one available buffering scheme
1551 which is to allocate a single set of ring buffers for the whole system.
1552 This scheme is similar to the per-user option, but with a single, global
1553 user ``running'' the kernel.
1556 [[channel-overwrite-mode-vs-discard-mode]]
1557 ==== Overwrite vs. discard event record loss modes
1559 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1560 arc in the following animations) of the ring buffer of a specific
1561 channel. When there's no space left in a sub-buffer, the tracer marks it
1562 as consumable (red) and another, empty sub-buffer starts receiving the
1563 following event records. A <<lttng-consumerd,consumer daemon>>
1564 eventually consumes the marked sub-buffer (returns to white).
1567 [role="docsvg-channel-subbuf-anim"]
1572 In an ideal world, sub-buffers are consumed faster than they are filled,
1573 as it is the case in the previous animation. In the real world,
1574 however, all sub-buffers can be full at some point, leaving no space to
1575 record the following events.
1577 By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when
1578 no empty sub-buffer is available, it is acceptable to lose event records
1579 when the alternative would be to cause substantial delays in the
1580 execution of the instrumented application. LTTng privileges performance
1581 over integrity; it aims at perturbing the target system as little as
1582 possible in order to make tracing of subtle race conditions and rare
1583 interrupt cascades possible.
1585 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1586 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1587 example>> to learn how to use the blocking mode.
1589 When it comes to losing event records because no empty sub-buffer is
1590 available, or because the <<opt-blocking-timeout,blocking timeout>> is
1591 reached, the _event record loss mode_ of the channel determines what to
1592 do. The available event record loss modes are:
1595 Drop the newest event records until the tracer releases a sub-buffer.
1597 This is the only available mode when you specify a
1598 <<opt-blocking-timeout,blocking timeout>>.
1601 Clear the sub-buffer containing the oldest event records and start
1602 writing the newest event records there.
1604 This mode is sometimes called _flight recorder mode_ because it's
1606 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1607 always keep a fixed amount of the latest data.
1609 Which mechanism you should choose depends on your context: prioritize
1610 the newest or the oldest event records in the ring buffer?
1612 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1613 as soon as a there's no space left for a new event record, whereas in
1614 discard mode, the tracer only discards the event record that doesn't
1617 In discard mode, LTTng increments a count of lost event records when an
1618 event record is lost and saves this count to the trace. In overwrite
1619 mode, since LTTng{nbsp}2.8, LTTng increments a count of lost sub-buffers
1620 when a sub-buffer is lost and saves this count to the trace. In this
1621 mode, LTTng doesn't write to the trace the exact number of lost event
1622 records in those lost sub-buffers. Trace analyses can use saved
1623 discarded event record and sub-buffer counts of the trace to decide
1624 whether or not to perform the analyses even if trace data is known to be
1627 There are a few ways to decrease your probability of losing event
1629 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1630 how to fine-tune the sub-buffer count and size of a channel to virtually
1631 stop losing event records, though at the cost of greater memory usage.
1634 [[channel-subbuf-size-vs-subbuf-count]]
1635 ==== Sub-buffer count and size
1637 When you <<enabling-disabling-channels,create a channel>>, you can
1638 set its number of sub-buffers and their size.
1640 Note that there is noticeable CPU overhead introduced when
1641 switching sub-buffers (marking a full one as consumable and switching
1642 to an empty one for the following events to be recorded). Knowing this,
1643 the following list presents a few practical situations along with how
1644 to configure the sub-buffer count and size for them:
1646 * **High event throughput**: In general, prefer bigger sub-buffers to
1647 lower the risk of losing event records.
1649 Having bigger sub-buffers also ensures a lower
1650 <<channel-switch-timer,sub-buffer switching frequency>>.
1652 The number of sub-buffers is only meaningful if you create the channel
1653 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1654 other sub-buffers are left unaltered.
1656 * **Low event throughput**: In general, prefer smaller sub-buffers
1657 since the risk of losing event records is low.
1659 Because events occur less frequently, the sub-buffer switching frequency
1660 should remain low and thus the overhead of the tracer shouldn't be a
1663 * **Low memory system**: If your target system has a low memory
1664 limit, prefer fewer first, then smaller sub-buffers.
1666 Even if the system is limited in memory, you want to keep the
1667 sub-buffers as big as possible to avoid a high sub-buffer switching
1670 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1671 which means event data is very compact. For example, the average
1672 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1673 sub-buffer size of 1{nbsp}MiB is considered big.
1675 The previous situations highlight the major trade-off between a few big
1676 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1677 frequency vs. how much data is lost in overwrite mode. Assuming a
1678 constant event throughput and using the overwrite mode, the two
1679 following configurations have the same ring buffer total size:
1682 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1687 * **Two sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1688 switching frequency, but if a sub-buffer overwrite happens, half of
1689 the event records so far (4{nbsp}MiB) are definitely lost.
1690 * **Eight sub-buffers of 1{nbsp}MiB each**: Expect four times the
1691 overhead of the tracer as the previous configuration, but if a
1692 sub-buffer overwrite happens, only the eighth of event records so far
1693 are definitely lost.
1695 In discard mode, the sub-buffers count parameter is pointless: use two
1696 sub-buffers and set their size according to the requirements of your
1700 [[channel-switch-timer]]
1701 ==== Switch timer period
1703 The _switch timer period_ is an important configurable attribute of
1704 a channel to ensure periodic sub-buffer flushing.
1706 When the _switch timer_ expires, a sub-buffer switch happens. Set
1707 the switch timer period attribute when you
1708 <<enabling-disabling-channels,create a channel>> to ensure that LTTng
1709 consumes and commits trace data to trace files or to a distant relay
1710 daemon periodically in case of a low event throughput.
1713 [role="docsvg-channel-switch-timer"]
1718 This attribute is also convenient when you use big sub-buffers to cope
1719 with a sporadic high event throughput, even if the throughput is
1723 [[channel-read-timer]]
1724 ==== Read timer period
1726 By default, the LTTng tracers use a notification mechanism to signal a
1727 full sub-buffer so that a consumer daemon can consume it. When such
1728 notifications must be avoided, for example in real-time applications,
1729 use the _read timer_ of the channel instead. When the read timer fires,
1730 the <<lttng-consumerd,consumer daemon>> checks for full, consumable
1734 [[tracefile-rotation]]
1735 ==== Trace file count and size
1737 By default, trace files can grow as large as needed. Set the maximum
1738 size of each trace file that a channel writes when you
1739 <<enabling-disabling-channels,create a channel>>. When the size of a
1740 trace file reaches the fixed maximum size of the channel, LTTng creates
1741 another file to contain the next event records. LTTng appends a file
1742 count to each trace file name in this case.
1744 If you set the trace file size attribute when you create a channel, the
1745 maximum number of trace files that LTTng creates is _unlimited_ by
1746 default. To limit them, set a maximum number of trace files. When the
1747 number of trace files reaches the fixed maximum count of the channel,
1748 the oldest trace file is overwritten. This mechanism is called _trace
1753 Even if you don't limit the trace file count, you can't assume that
1754 LTTng doesn't manage any trace file.
1756 In other words, there is no safe way to know if LTTng still holds a
1757 given trace file open with the trace file rotation feature.
1759 The only way to obtain an unmanaged, self-contained LTTng trace before
1760 you <<creating-destroying-tracing-sessions,destroy>> the tracing session
1761 is with the <<session-rotation,tracing session rotation>> feature
1762 (available since LTTng{nbsp}2.11).
1767 === Instrumentation point, event rule, event, and event record
1769 An _event rule_ is a set of conditions which must be **all** satisfied
1770 for LTTng to record an occuring event.
1772 You set the conditions when you <<enabling-disabling-events,create
1775 You always attach an event rule to a <<channel,channel>> when you create
1778 When an event passes the conditions of an event rule, LTTng records it
1779 in one of the sub-buffers of the attached channel.
1781 The available conditions, as of LTTng{nbsp}{revision}, are:
1783 * The event rule _is enabled_.
1784 * The type of the instrumentation point _is{nbsp}T_.
1785 * The name of the instrumentation point (sometimes called _event name_)
1786 _matches{nbsp}N_, but _isn't{nbsp}E_.
1787 * The log level of the instrumentation point _is as severe as{nbsp}L_, or
1788 _is exactly{nbsp}L_.
1789 * The fields of the payload of the event _satisfy_ a filter
1790 expression{nbsp}__F__.
1792 As you can see, all the conditions but the dynamic filter are related to
1793 the status of the event rule or to the instrumentation point, not to the
1794 occurring events. This is why, without a filter, checking if an event
1795 passes an event rule isn't a dynamic task: when you create or modify an
1796 event rule, all the tracers of its tracing domain enable or disable the
1797 instrumentation points themselves once. This is possible because the
1798 attributes of an instrumentation point (type, name, and log level) are
1799 defined statically. In other words, without a dynamic filter, the tracer
1800 _doesn't evaluate_ the arguments of an instrumentation point unless it
1801 matches an enabled event rule.
1803 Note that, for LTTng to record an event, the <<channel,channel>> to
1804 which a matching event rule is attached must also be enabled, and the
1805 <<tracing-session,tracing session>> owning this channel must be active
1809 .Logical path from an instrumentation point to an event record.
1810 image::event-rule.png[]
1812 .Event, event record, or event rule?
1814 With so many similar terms, it's easy to get confused.
1816 An **event** is the consequence of the execution of an _instrumentation
1817 point_, like a tracepoint that you manually place in some source code,
1818 or a Linux kernel kprobe. An event is said to _occur_ at a specific
1819 time. Different actions can be taken upon the occurrence of an event,
1820 like record the payload of the event to a buffer.
1822 An **event record** is the representation of an event in a sub-buffer. A
1823 tracer is responsible for capturing the payload of an event, current
1824 context variables, the ID of the event, and its timestamp. LTTng
1825 can append this sub-buffer to a trace file.
1827 An **event rule** is a set of conditions which must _all_ be satisfied
1828 for LTTng to record an occuring event. Events still occur without
1829 satisfying event rules, but LTTng doesn't record them.
1834 == Components of noch:{LTTng}
1836 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1837 to call LTTng a simple _tool_ since it is composed of multiple
1838 interacting components. This section describes those components,
1839 explains their respective roles, and shows how they connect together to
1840 form the LTTng ecosystem.
1842 The following diagram shows how the most important components of LTTng
1843 interact with user applications, the Linux kernel, and you:
1846 .Control and trace data paths between LTTng components.
1847 image::plumbing.png[]
1849 The LTTng project incorporates:
1851 * **LTTng-tools**: Libraries and command-line interface to
1852 control tracing sessions.
1853 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1854 ** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1855 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1856 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1857 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1858 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1860 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1861 headers to instrument and trace any native user application.
1862 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1863 *** `liblttng-ust-libc-wrapper`
1864 *** `liblttng-ust-pthread-wrapper`
1865 *** `liblttng-ust-cyg-profile`
1866 *** `liblttng-ust-cyg-profile-fast`
1867 *** `liblttng-ust-dl`
1868 ** User space tracepoint provider source files generator command-line
1869 tool (man:lttng-gen-tp(1)).
1870 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1871 Java applications using `java.util.logging` or
1872 Apache log4j{nbsp}1.2 logging.
1873 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1874 Python applications using the standard `logging` package.
1875 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1877 ** LTTng kernel tracer module.
1878 ** Tracing ring buffer kernel modules.
1879 ** Probe kernel modules.
1880 ** LTTng logger kernel module.
1884 === Tracing control command-line interface
1887 .The tracing control command-line interface.
1888 image::plumbing-lttng-cli.png[]
1890 The _man:lttng(1) command-line tool_ is the standard user interface to
1891 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1892 is part of LTTng-tools.
1894 The cmd:lttng tool is linked with
1895 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1896 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1898 The cmd:lttng tool has a Git-like interface:
1902 $ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1905 The <<controlling-tracing,Tracing control>> section explores the
1906 available features of LTTng using the cmd:lttng tool.
1909 [[liblttng-ctl-lttng]]
1910 === Tracing control library
1913 .The tracing control library.
1914 image::plumbing-liblttng-ctl.png[]
1916 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1917 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1918 underlying details of the protocol. `liblttng-ctl` is part of LTTng-tools.
1920 The <<lttng-cli,cmd:lttng command-line tool>>
1921 is linked with `liblttng-ctl`.
1923 Use `liblttng-ctl` in C or $$C++$$ source code by including its
1928 #include <lttng/lttng.h>
1931 Some objects are referenced by name (C string), such as tracing
1932 sessions, but most of them require to create a handle first using
1933 `lttng_create_handle()`.
1935 As of LTTng{nbsp}{revision}, the best available developer documentation for
1936 `liblttng-ctl` is its installed header files. Every function and structure is
1937 thoroughly documented.
1941 === User space tracing library
1944 .The user space tracing library.
1945 image::plumbing-liblttng-ust.png[]
1947 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1948 is the LTTng user space tracer. It receives commands from a
1949 <<lttng-sessiond,session daemon>>, for example to
1950 enable and disable specific instrumentation points, and writes event
1951 records to ring buffers shared with a
1952 <<lttng-consumerd,consumer daemon>>.
1953 `liblttng-ust` is part of LTTng-UST.
1955 Public C header files are installed beside `liblttng-ust` to
1956 instrument any <<c-application,C or $$C++$$ application>>.
1958 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1959 packages, use their own library providing tracepoints which is
1960 linked with `liblttng-ust`.
1962 An application or library doesn't have to initialize `liblttng-ust`
1963 manually: its constructor does the necessary tasks to properly register
1964 to a session daemon. The initialization phase also enables the
1965 instrumentation points matching the <<event,event rules>> that you
1969 [[lttng-ust-agents]]
1970 === User space tracing agents
1973 .The user space tracing agents.
1974 image::plumbing-lttng-ust-agents.png[]
1976 The _LTTng-UST Java and Python agents_ are regular Java and Python
1977 packages which add LTTng tracing capabilities to the
1978 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1980 In the case of Java, the
1981 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1982 core logging facilities] and
1983 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
1984 Note that Apache Log4{nbsp}2 isn't supported.
1986 In the case of Python, the standard
1987 https://docs.python.org/3/library/logging.html[`logging`] package
1988 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
1989 LTTng-UST Python agent package.
1991 The applications using the LTTng-UST agents are in the
1992 `java.util.logging` (JUL),
1993 log4j, and Python <<domain,tracing domains>>.
1995 Both agents use the same mechanism to trace the log statements. When an
1996 agent initializes, it creates a log handler that attaches to the root
1997 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1998 When the application executes a log statement, the root logger passes it
1999 to the log handler of the agent. The log handler of the agent calls a
2000 native function in a tracepoint provider package shared library linked
2001 with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
2002 other fields, like its logger name and its log level. This native
2003 function contains a user space instrumentation point, hence tracing the
2006 The log level condition of an
2007 <<event,event rule>> is considered when tracing
2008 a Java or a Python application, and it's compatible with the standard
2009 JUL, log4j, and Python log levels.
2013 === LTTng kernel modules
2016 .The LTTng kernel modules.
2017 image::plumbing-lttng-modules.png[]
2019 The _LTTng kernel modules_ are a set of Linux kernel modules
2020 which implement the kernel tracer of the LTTng project. The LTTng
2021 kernel modules are part of LTTng-modules.
2023 The LTTng kernel modules include:
2025 * A set of _probe_ modules.
2027 Each module attaches to a specific subsystem
2028 of the Linux kernel using its tracepoint instrument points. There are
2029 also modules to attach to the entry and return points of the Linux
2030 system call functions.
2032 * _Ring buffer_ modules.
2034 A ring buffer implementation is provided as kernel modules. The LTTng
2035 kernel tracer writes to the ring buffer; a
2036 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
2038 * The _LTTng kernel tracer_ module.
2039 * The _LTTng logger_ module.
2041 The LTTng logger module implements the special path:{/proc/lttng-logger}
2042 (and path:{/dev/lttng-logger} since LTTng{nbsp}2.11) files so that any
2043 executable can generate LTTng events by opening and writing to those
2046 See <<proc-lttng-logger-abi,LTTng logger>>.
2048 Generally, you don't have to load the LTTng kernel modules manually
2049 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
2050 daemon>> loads the necessary modules when starting. If you have extra
2051 probe modules, you can specify to load them to the session daemon on
2054 The LTTng kernel modules are installed in
2055 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
2056 the kernel release (see `uname --kernel-release`).
2063 .The session daemon.
2064 image::plumbing-sessiond.png[]
2066 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
2067 managing tracing sessions and for controlling the various components of
2068 LTTng. The session daemon is part of LTTng-tools.
2070 The session daemon sends control requests to and receives control
2073 * The <<lttng-ust,user space tracing library>>.
2075 Any instance of the user space tracing library first registers to
2076 a session daemon. Then, the session daemon can send requests to
2077 this instance, such as:
2080 ** Get the list of tracepoints.
2081 ** Share an <<event,event rule>> so that the user space tracing library
2082 can enable or disable tracepoints. Amongst the possible conditions
2083 of an event rule is a filter expression which `liblttng-ust` evalutes
2084 when an event occurs.
2085 ** Share <<channel,channel>> attributes and ring buffer locations.
2088 The session daemon and the user space tracing library use a Unix
2089 domain socket for their communication.
2091 * The <<lttng-ust-agents,user space tracing agents>>.
2093 Any instance of a user space tracing agent first registers to
2094 a session daemon. Then, the session daemon can send requests to
2095 this instance, such as:
2098 ** Get the list of loggers.
2099 ** Enable or disable a specific logger.
2102 The session daemon and the user space tracing agent use a TCP connection
2103 for their communication.
2105 * The <<lttng-modules,LTTng kernel tracer>>.
2106 * The <<lttng-consumerd,consumer daemon>>.
2108 The session daemon sends requests to the consumer daemon to instruct
2109 it where to send the trace data streams, amongst other information.
2111 * The <<lttng-relayd,relay daemon>>.
2113 The session daemon receives commands from the
2114 <<liblttng-ctl-lttng,tracing control library>>.
2116 The root session daemon loads the appropriate
2117 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2118 a <<lttng-consumerd,consumer daemon>> as soon as you create
2119 an <<event,event rule>>.
2121 The session daemon doesn't send and receive trace data: this is the
2122 role of the <<lttng-consumerd,consumer daemon>> and
2123 <<lttng-relayd,relay daemon>>. It does, however, generate the
2124 http://diamon.org/ctf/[CTF] metadata stream.
2126 Each Unix user can have its own session daemon instance. The
2127 tracing sessions which different session daemons manage are completely
2130 The root user's session daemon is the only one which is
2131 allowed to control the LTTng kernel tracer, and its spawned consumer
2132 daemon is the only one which is allowed to consume trace data from the
2133 LTTng kernel tracer. Note, however, that any Unix user which is a member
2134 of the <<tracing-group,tracing group>> is allowed
2135 to create <<channel,channels>> in the
2136 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
2139 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2140 session daemon when using its `create` command if none is currently
2141 running. You can also start the session daemon manually.
2148 .The consumer daemon.
2149 image::plumbing-consumerd.png[]
2151 The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
2152 ring buffers with user applications or with the LTTng kernel modules to
2153 collect trace data and send it to some location (on disk or to a
2154 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
2155 is part of LTTng-tools.
2157 You don't start a consumer daemon manually: a consumer daemon is always
2158 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
2159 <<event,event rule>>, that is, before you start tracing. When you kill
2160 its owner session daemon, the consumer daemon also exits because it is
2161 the child process of the session daemon. Command-line options of
2162 man:lttng-sessiond(8) target the consumer daemon process.
2164 There are up to two running consumer daemons per Unix user, whereas only
2165 one session daemon can run per user. This is because each process can be
2166 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2167 and 64-bit processes, it is more efficient to have separate
2168 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2169 exception: it can have up to _three_ running consumer daemons: 32-bit
2170 and 64-bit instances for its user applications, and one more
2171 reserved for collecting kernel trace data.
2179 image::plumbing-relayd.png[]
2181 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
2182 between remote session and consumer daemons, local trace files, and a
2183 remote live trace viewer. The relay daemon is part of LTTng-tools.
2185 The main purpose of the relay daemon is to implement a receiver of
2186 <<sending-trace-data-over-the-network,trace data over the network>>.
2187 This is useful when the target system doesn't have much file system
2188 space to record trace files locally.
2190 The relay daemon is also a server to which a
2191 <<lttng-live,live trace viewer>> can
2192 connect. The live trace viewer sends requests to the relay daemon to
2193 receive trace data as the target system emits events. The
2194 communication protocol is named _LTTng live_; it is used over TCP
2197 Note that you can start the relay daemon on the target system directly.
2198 This is the setup of choice when the use case is to view events as
2199 the target system emits them without the need of a remote system.
2203 == [[using-lttng]]Instrumentation
2205 There are many examples of tracing and monitoring in our everyday life:
2207 * You have access to real-time and historical weather reports and
2208 forecasts thanks to weather stations installed around the country.
2209 * You know your heart is safe thanks to an electrocardiogram.
2210 * You make sure not to drive your car too fast and to have enough fuel
2211 to reach your destination thanks to gauges visible on your dashboard.
2213 All the previous examples have something in common: they rely on
2214 **instruments**. Without the electrodes attached to the surface of your
2215 body skin, cardiac monitoring is futile.
2217 LTTng, as a tracer, is no different from those real life examples. If
2218 you're about to trace a software system or, in other words, record its
2219 history of execution, you better have **instrumentation points** in the
2220 subject you're tracing, that is, the actual software.
2222 Various ways were developed to instrument a piece of software for LTTng
2223 tracing. The most straightforward one is to manually place
2224 instrumentation points, called _tracepoints_, in the source code of the
2225 software. It is also possible to add instrumentation points dynamically
2226 in the Linux kernel <<domain,tracing domain>>.
2228 If you're only interested in tracing the Linux kernel, your
2229 instrumentation needs are probably already covered by the built-in
2230 <<lttng-modules,Linux kernel tracepoints>> of LTTng. You may also wish
2231 to trace a user application which is already instrumented for LTTng
2232 tracing. In such cases, skip this whole section and read the topics of
2233 the <<controlling-tracing,Tracing control>> section.
2235 Many methods are available to instrument a piece of software for LTTng
2238 * <<c-application,User space instrumentation for C and $$C++$$
2240 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
2241 * <<java-application,User space Java agent>>.
2242 * <<python-application,User space Python agent>>.
2243 * <<proc-lttng-logger-abi,LTTng logger>>.
2244 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
2248 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
2250 The procedure to instrument a C or $$C++$$ user application with
2251 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
2253 . <<tracepoint-provider,Create the source files of a tracepoint provider
2255 . <<probing-the-application-source-code,Add tracepoints to
2256 the source code of the application>>.
2257 . <<building-tracepoint-providers-and-user-application,Build and link
2258 a tracepoint provider package and the user application>>.
2260 If you need quick, man:printf(3)-like instrumentation, skip
2261 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
2264 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2265 instrument a user application with `liblttng-ust`.
2268 [[tracepoint-provider]]
2269 ==== Create the source files of a tracepoint provider package
2271 A _tracepoint provider_ is a set of compiled functions which provide
2272 **tracepoints** to an application, the type of instrumentation point
2273 supported by LTTng-UST. Those functions can emit events with
2274 user-defined fields and serialize those events as event records to one
2275 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
2276 macro, which you <<probing-the-application-source-code,insert in the
2277 source code of a user application>>, calls those functions.
2279 A _tracepoint provider package_ is an object file (`.o`) or a shared
2280 library (`.so`) which contains one or more tracepoint providers.
2281 Its source files are:
2283 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2284 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2286 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2287 the LTTng user space tracer, at run time.
2290 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2291 image::ust-app.png[]
2293 NOTE: If you need quick, man:printf(3)-like instrumentation,
2294 skip creating and using a tracepoint provider and use
2295 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
2299 ===== Create a tracepoint provider header file template
2301 A _tracepoint provider header file_ contains the tracepoint
2302 definitions of a tracepoint provider.
2304 To create a tracepoint provider header file:
2306 . Start from this template:
2310 .Tracepoint provider header file template (`.h` file extension).
2312 #undef TRACEPOINT_PROVIDER
2313 #define TRACEPOINT_PROVIDER provider_name
2315 #undef TRACEPOINT_INCLUDE
2316 #define TRACEPOINT_INCLUDE "./tp.h"
2318 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
2321 #include <lttng/tracepoint.h>
2324 * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
2325 * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
2330 #include <lttng/tracepoint-event.h>
2336 * `provider_name` with the name of your tracepoint provider.
2337 * `"tp.h"` with the name of your tracepoint provider header file.
2339 . Below the `#include <lttng/tracepoint.h>` line, put your
2340 <<defining-tracepoints,tracepoint definitions>>.
2342 Your tracepoint provider name must be unique amongst all the possible
2343 tracepoint provider names used on the same target system. We
2344 suggest to include the name of your project or company in the name,
2345 for example, `org_lttng_my_project_tpp`.
2347 TIP: [[lttng-gen-tp]]Use the man:lttng-gen-tp(1) tool to create
2348 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
2349 write are the <<defining-tracepoints,tracepoint definitions>>.
2352 [[defining-tracepoints]]
2353 ===== Create a tracepoint definition
2355 A _tracepoint definition_ defines, for a given tracepoint:
2357 * Its **input arguments**. They are the macro parameters that the
2358 `tracepoint()` macro accepts for this particular tracepoint
2359 in the source code of the user application.
2360 * Its **output event fields**. They are the sources of event fields
2361 that form the payload of any event that the execution of the
2362 `tracepoint()` macro emits for this particular tracepoint.
2364 Create a tracepoint definition by using the
2365 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2367 <<tpp-header,tracepoint provider header file template>>.
2369 The syntax of the `TRACEPOINT_EVENT()` macro is:
2372 .`TRACEPOINT_EVENT()` macro syntax.
2375 /* Tracepoint provider name */
2378 /* Tracepoint name */
2381 /* Input arguments */
2386 /* Output event fields */
2395 * `provider_name` with your tracepoint provider name.
2396 * `tracepoint_name` with your tracepoint name.
2397 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2398 * `fields` with the <<tpp-def-output-fields,output event field>>
2401 This tracepoint emits events named `provider_name:tracepoint_name`.
2404 .Event name length limitation
2406 The concatenation of the tracepoint provider name and the
2407 tracepoint name must not exceed **254{nbsp}characters**. If it does, the
2408 instrumented application compiles and runs, but LTTng throws multiple
2409 warnings and you could experience serious issues.
2412 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2415 .`TP_ARGS()` macro syntax.
2424 * `type` with the C type of the argument.
2425 * `arg_name` with the argument name.
2427 You can repeat `type` and `arg_name` up to 10{nbsp}times to have more
2430 .`TP_ARGS()` usage with three arguments.
2442 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2443 tracepoint definition with no input arguments.
2445 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2446 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2447 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2448 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2451 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2452 C expression that the tracer evalutes at the `tracepoint()` macro site
2453 in the source code of the application. This expression provides the
2454 source of data of a field. The argument expression can include input
2455 argument names listed in the `TP_ARGS()` macro.
2457 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2458 must be unique within a given tracepoint definition.
2460 Here's a complete tracepoint definition example:
2462 .Tracepoint definition.
2464 The following tracepoint definition defines a tracepoint which takes
2465 three input arguments and has four output event fields.
2469 #include "my-custom-structure.h"
2475 const struct my_custom_structure*, my_custom_structure,
2480 ctf_string(query_field, query)
2481 ctf_float(double, ratio_field, ratio)
2482 ctf_integer(int, recv_size, my_custom_structure->recv_size)
2483 ctf_integer(int, send_size, my_custom_structure->send_size)
2488 Refer to this tracepoint definition with the `tracepoint()` macro in
2489 the source code of your application like this:
2493 tracepoint(my_provider, my_tracepoint,
2494 my_structure, some_ratio, the_query);
2498 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2499 if they satisfy an enabled <<event,event rule>>.
2502 [[using-tracepoint-classes]]
2503 ===== Use a tracepoint class
2505 A _tracepoint class_ is a class of tracepoints which share the same
2506 output event field definitions. A _tracepoint instance_ is one
2507 instance of such a defined tracepoint class, with its own tracepoint
2510 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2511 shorthand which defines both a tracepoint class and a tracepoint
2512 instance at the same time.
2514 When you build a tracepoint provider package, the C or $$C++$$ compiler
2515 creates one serialization function for each **tracepoint class**. A
2516 serialization function is responsible for serializing the event fields
2517 of a tracepoint to a sub-buffer when tracing.
2519 For various performance reasons, when your situation requires multiple
2520 tracepoint definitions with different names, but with the same event
2521 fields, we recommend that you manually create a tracepoint class
2522 and instantiate as many tracepoint instances as needed. One positive
2523 effect of such a design, amongst other advantages, is that all
2524 tracepoint instances of the same tracepoint class reuse the same
2525 serialization function, thus reducing
2526 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2528 .Use a tracepoint class and tracepoint instances.
2530 Consider the following three tracepoint definitions:
2542 ctf_integer(int, userid, userid)
2543 ctf_integer(size_t, len, len)
2555 ctf_integer(int, userid, userid)
2556 ctf_integer(size_t, len, len)
2568 ctf_integer(int, userid, userid)
2569 ctf_integer(size_t, len, len)
2574 In this case, we create three tracepoint classes, with one implicit
2575 tracepoint instance for each of them: `get_account`, `get_settings`, and
2576 `get_transaction`. However, they all share the same event field names
2577 and types. Hence three identical, yet independent serialization
2578 functions are created when you build the tracepoint provider package.
2580 A better design choice is to define a single tracepoint class and three
2581 tracepoint instances:
2585 /* The tracepoint class */
2586 TRACEPOINT_EVENT_CLASS(
2587 /* Tracepoint provider name */
2590 /* Tracepoint class name */
2593 /* Input arguments */
2599 /* Output event fields */
2601 ctf_integer(int, userid, userid)
2602 ctf_integer(size_t, len, len)
2606 /* The tracepoint instances */
2607 TRACEPOINT_EVENT_INSTANCE(
2608 /* Tracepoint provider name */
2611 /* Tracepoint class name */
2614 /* Tracepoint name */
2617 /* Input arguments */
2623 TRACEPOINT_EVENT_INSTANCE(
2632 TRACEPOINT_EVENT_INSTANCE(
2645 [[assigning-log-levels]]
2646 ===== Assign a log level to a tracepoint definition
2648 Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
2649 with the `TRACEPOINT_LOGLEVEL()` macro.
2651 Assigning different levels of severity to tracepoint definitions can
2652 be useful: when you <<enabling-disabling-events,create an event rule>>,
2653 you can target tracepoints having a log level as severe as a specific
2656 The concept of LTTng-UST log levels is similar to the levels found
2657 in typical logging frameworks:
2659 * In a logging framework, the log level is given by the function
2660 or method name you use at the log statement site: `debug()`,
2661 `info()`, `warn()`, `error()`, and so on.
2662 * In LTTng-UST, you statically assign the log level to a tracepoint
2663 definition; any `tracepoint()` macro invocation which refers to
2664 this definition has this log level.
2666 You must use `TRACEPOINT_LOGLEVEL()` _after_ the
2667 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2668 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2671 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2674 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2676 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2681 * `provider_name` with the tracepoint provider name.
2682 * `tracepoint_name` with the tracepoint name.
2683 * `log_level` with the log level to assign to the tracepoint
2684 definition named `tracepoint_name` in the `provider_name`
2685 tracepoint provider.
2687 See man:lttng-ust(3) for a list of available log level names.
2689 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2693 /* Tracepoint definition */
2702 ctf_integer(int, userid, userid)
2703 ctf_integer(size_t, len, len)
2707 /* Log level assignment */
2708 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2714 ===== Create a tracepoint provider package source file
2716 A _tracepoint provider package source file_ is a C source file which
2717 includes a <<tpp-header,tracepoint provider header file>> to expand its
2718 macros into event serialization and other functions.
2720 Use the following tracepoint provider package source file template:
2723 .Tracepoint provider package source file template.
2725 #define TRACEPOINT_CREATE_PROBES
2730 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2731 header file>> name. You may also include more than one tracepoint
2732 provider header file here to create a tracepoint provider package
2733 holding more than one tracepoint providers.
2736 [[probing-the-application-source-code]]
2737 ==== Add tracepoints to the source code of an application
2739 Once you <<tpp-header,create a tracepoint provider header file>>, use
2740 the `tracepoint()` macro in the source code of your application to
2741 insert the tracepoints that this header
2742 <<defining-tracepoints,defines>>.
2744 The `tracepoint()` macro takes at least two parameters: the tracepoint
2745 provider name and the tracepoint name. The corresponding tracepoint
2746 definition defines the other parameters.
2748 .`tracepoint()` usage.
2750 The following <<defining-tracepoints,tracepoint definition>> defines a
2751 tracepoint which takes two input arguments and has two output event
2755 .Tracepoint provider header file.
2757 #include "my-custom-structure.h"
2764 const char*, cmd_name
2767 ctf_string(cmd_name, cmd_name)
2768 ctf_integer(int, number_of_args, argc)
2773 Refer to this tracepoint definition with the `tracepoint()` macro in
2774 the source code of your application like this:
2777 .Application source file.
2781 int main(int argc, char* argv[])
2783 tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2789 Note how the source code of the application includes
2790 the tracepoint provider header file containing the tracepoint
2791 definitions to use, path:{tp.h}.
2794 .`tracepoint()` usage with a complex tracepoint definition.
2796 Consider this complex tracepoint definition, where multiple event
2797 fields refer to the same input arguments in their argument expression
2801 .Tracepoint provider header file.
2803 /* For `struct stat` */
2804 #include <sys/types.h>
2805 #include <sys/stat.h>
2817 ctf_integer(int, my_constant_field, 23 + 17)
2818 ctf_integer(int, my_int_arg_field, my_int_arg)
2819 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2820 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2821 my_str_arg[2] + my_str_arg[3])
2822 ctf_string(my_str_arg_field, my_str_arg)
2823 ctf_integer_hex(off_t, size_field, st->st_size)
2824 ctf_float(double, size_dbl_field, (double) st->st_size)
2825 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2826 size_t, strlen(my_str_arg) / 2)
2831 Refer to this tracepoint definition with the `tracepoint()` macro in
2832 the source code of your application like this:
2835 .Application source file.
2837 #define TRACEPOINT_DEFINE
2844 stat("/etc/fstab", &s);
2845 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2851 If you look at the event record that LTTng writes when tracing this
2852 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2853 it should look like this:
2855 .Event record fields
2857 |Field name |Field value
2858 |`my_constant_field` |40
2859 |`my_int_arg_field` |23
2860 |`my_int_arg_field2` |529
2862 |`my_str_arg_field` |`Hello, World!`
2863 |`size_field` |0x12d
2864 |`size_dbl_field` |301.0
2865 |`half_my_str_arg_field` |`Hello,`
2869 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2870 compute--they use the call stack, for example. To avoid this computation
2871 when the tracepoint is disabled, use the `tracepoint_enabled()` and
2872 `do_tracepoint()` macros.
2874 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2878 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2880 tracepoint_enabled(provider_name, tracepoint_name)
2881 do_tracepoint(provider_name, tracepoint_name, ...)
2886 * `provider_name` with the tracepoint provider name.
2887 * `tracepoint_name` with the tracepoint name.
2889 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2890 `tracepoint_name` from the provider named `provider_name` is enabled
2893 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2894 if the tracepoint is enabled. Using `tracepoint()` with
2895 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2896 the `tracepoint_enabled()` check, thus a race condition is
2897 possible in this situation:
2900 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2902 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2903 stuff = prepare_stuff();
2906 tracepoint(my_provider, my_tracepoint, stuff);
2909 If the tracepoint is enabled after the condition, then `stuff` isn't
2910 prepared: the emitted event will either contain wrong data, or the whole
2911 application could crash (segmentation fault, for example).
2913 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2914 `STAP_PROBEV()` call. If you need it, you must emit
2918 [[building-tracepoint-providers-and-user-application]]
2919 ==== Build and link a tracepoint provider package and an application
2921 Once you have one or more <<tpp-header,tracepoint provider header
2922 files>> and a <<tpp-source,tracepoint provider package source file>>,
2923 create the tracepoint provider package by compiling its source
2924 file. From here, multiple build and run scenarios are possible. The
2925 following table shows common application and library configurations
2926 along with the required command lines to achieve them.
2928 In the following diagrams, we use the following file names:
2931 Executable application.
2934 Application object file.
2937 Tracepoint provider package object file.
2940 Tracepoint provider package archive file.
2943 Tracepoint provider package shared object file.
2946 User library object file.
2949 User library shared object file.
2951 We use the following symbols in the diagrams of table below:
2954 .Symbols used in the build scenario diagrams.
2955 image::ust-sit-symbols.png[]
2957 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2958 variable in the following instructions.
2960 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2961 .Common tracepoint provider package scenarios.
2963 |Scenario |Instructions
2966 The instrumented application is statically linked with
2967 the tracepoint provider package object.
2969 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2972 include::../common/ust-sit-step-tp-o.txt[]
2974 To build the instrumented application:
2976 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2981 #define TRACEPOINT_DEFINE
2985 . Compile the application source file:
2994 . Build the application:
2999 $ gcc -o app app.o tpp.o -llttng-ust -ldl
3003 To run the instrumented application:
3005 * Start the application:
3015 The instrumented application is statically linked with the
3016 tracepoint provider package archive file.
3018 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
3021 To create the tracepoint provider package archive file:
3023 . Compile the <<tpp-source,tracepoint provider package source file>>:
3032 . Create the tracepoint provider package archive file:
3037 $ ar rcs tpp.a tpp.o
3041 To build the instrumented application:
3043 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3048 #define TRACEPOINT_DEFINE
3052 . Compile the application source file:
3061 . Build the application:
3066 $ gcc -o app app.o tpp.a -llttng-ust -ldl
3070 To run the instrumented application:
3072 * Start the application:
3082 The instrumented application is linked with the tracepoint provider
3083 package shared object.
3085 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
3088 include::../common/ust-sit-step-tp-so.txt[]
3090 To build the instrumented application:
3092 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3097 #define TRACEPOINT_DEFINE
3101 . Compile the application source file:
3110 . Build the application:
3115 $ gcc -o app app.o -ldl -L. -ltpp
3119 To run the instrumented application:
3121 * Start the application:
3131 The tracepoint provider package shared object is preloaded before the
3132 instrumented application starts.
3134 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3137 include::../common/ust-sit-step-tp-so.txt[]
3139 To build the instrumented application:
3141 . In path:{app.c}, before including path:{tpp.h}, add the
3147 #define TRACEPOINT_DEFINE
3148 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3152 . Compile the application source file:
3161 . Build the application:
3166 $ gcc -o app app.o -ldl
3170 To run the instrumented application with tracing support:
3172 * Preload the tracepoint provider package shared object and
3173 start the application:
3178 $ LD_PRELOAD=./libtpp.so ./app
3182 To run the instrumented application without tracing support:
3184 * Start the application:
3194 The instrumented application dynamically loads the tracepoint provider
3195 package shared object.
3197 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3200 include::../common/ust-sit-step-tp-so.txt[]
3202 To build the instrumented application:
3204 . In path:{app.c}, before including path:{tpp.h}, add the
3210 #define TRACEPOINT_DEFINE
3211 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3215 . Compile the application source file:
3224 . Build the application:
3229 $ gcc -o app app.o -ldl
3233 To run the instrumented application:
3235 * Start the application:
3245 The application is linked with the instrumented user library.
3247 The instrumented user library is statically linked with the tracepoint
3248 provider package object file.
3250 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3253 include::../common/ust-sit-step-tp-o-fpic.txt[]
3255 To build the instrumented user library:
3257 . In path:{emon.c}, before including path:{tpp.h}, add the
3263 #define TRACEPOINT_DEFINE
3267 . Compile the user library source file:
3272 $ gcc -I. -fpic -c emon.c
3276 . Build the user library shared object:
3281 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3285 To build the application:
3287 . Compile the application source file:
3296 . Build the application:
3301 $ gcc -o app app.o -L. -lemon
3305 To run the application:
3307 * Start the application:
3317 The application is linked with the instrumented user library.
3319 The instrumented user library is linked with the tracepoint provider
3320 package shared object.
3322 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3325 include::../common/ust-sit-step-tp-so.txt[]
3327 To build the instrumented user library:
3329 . In path:{emon.c}, before including path:{tpp.h}, add the
3335 #define TRACEPOINT_DEFINE
3339 . Compile the user library source file:
3344 $ gcc -I. -fpic -c emon.c
3348 . Build the user library shared object:
3353 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3357 To build the application:
3359 . Compile the application source file:
3368 . Build the application:
3373 $ gcc -o app app.o -L. -lemon
3377 To run the application:
3379 * Start the application:
3389 The tracepoint provider package shared object is preloaded before the
3392 The application is linked with the instrumented user library.
3394 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3397 include::../common/ust-sit-step-tp-so.txt[]
3399 To build the instrumented user library:
3401 . In path:{emon.c}, before including path:{tpp.h}, add the
3407 #define TRACEPOINT_DEFINE
3408 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3412 . Compile the user library source file:
3417 $ gcc -I. -fpic -c emon.c
3421 . Build the user library shared object:
3426 $ gcc -shared -o libemon.so emon.o -ldl
3430 To build the application:
3432 . Compile the application source file:
3441 . Build the application:
3446 $ gcc -o app app.o -L. -lemon
3450 To run the application with tracing support:
3452 * Preload the tracepoint provider package shared object and
3453 start the application:
3458 $ LD_PRELOAD=./libtpp.so ./app
3462 To run the application without tracing support:
3464 * Start the application:
3474 The application is linked with the instrumented user library.
3476 The instrumented user library dynamically loads the tracepoint provider
3477 package shared object.
3479 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3482 include::../common/ust-sit-step-tp-so.txt[]
3484 To build the instrumented user library:
3486 . In path:{emon.c}, before including path:{tpp.h}, add the
3492 #define TRACEPOINT_DEFINE
3493 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3497 . Compile the user library source file:
3502 $ gcc -I. -fpic -c emon.c
3506 . Build the user library shared object:
3511 $ gcc -shared -o libemon.so emon.o -ldl
3515 To build the application:
3517 . Compile the application source file:
3526 . Build the application:
3531 $ gcc -o app app.o -L. -lemon
3535 To run the application:
3537 * Start the application:
3547 The application dynamically loads the instrumented user library.
3549 The instrumented user library is linked with the tracepoint provider
3550 package shared object.
3552 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3555 include::../common/ust-sit-step-tp-so.txt[]
3557 To build the instrumented user library:
3559 . In path:{emon.c}, before including path:{tpp.h}, add the
3565 #define TRACEPOINT_DEFINE
3569 . Compile the user library source file:
3574 $ gcc -I. -fpic -c emon.c
3578 . Build the user library shared object:
3583 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3587 To build the application:
3589 . Compile the application source file:
3598 . Build the application:
3603 $ gcc -o app app.o -ldl -L. -lemon
3607 To run the application:
3609 * Start the application:
3619 The application dynamically loads the instrumented user library.
3621 The instrumented user library dynamically loads the tracepoint provider
3622 package shared object.
3624 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3627 include::../common/ust-sit-step-tp-so.txt[]
3629 To build the instrumented user library:
3631 . In path:{emon.c}, before including path:{tpp.h}, add the
3637 #define TRACEPOINT_DEFINE
3638 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3642 . Compile the user library source file:
3647 $ gcc -I. -fpic -c emon.c
3651 . Build the user library shared object:
3656 $ gcc -shared -o libemon.so emon.o -ldl
3660 To build the application:
3662 . Compile the application source file:
3671 . Build the application:
3676 $ gcc -o app app.o -ldl -L. -lemon
3680 To run the application:
3682 * Start the application:
3692 The tracepoint provider package shared object is preloaded before the
3695 The application dynamically loads the instrumented user library.
3697 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3700 include::../common/ust-sit-step-tp-so.txt[]
3702 To build the instrumented user library:
3704 . In path:{emon.c}, before including path:{tpp.h}, add the
3710 #define TRACEPOINT_DEFINE
3711 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3715 . Compile the user library source file:
3720 $ gcc -I. -fpic -c emon.c
3724 . Build the user library shared object:
3729 $ gcc -shared -o libemon.so emon.o -ldl
3733 To build the application:
3735 . Compile the application source file:
3744 . Build the application:
3749 $ gcc -o app app.o -L. -lemon
3753 To run the application with tracing support:
3755 * Preload the tracepoint provider package shared object and
3756 start the application:
3761 $ LD_PRELOAD=./libtpp.so ./app
3765 To run the application without tracing support:
3767 * Start the application:
3777 The application is statically linked with the tracepoint provider
3778 package object file.
3780 The application is linked with the instrumented user library.
3782 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3785 include::../common/ust-sit-step-tp-o.txt[]
3787 To build the instrumented user library:
3789 . In path:{emon.c}, before including path:{tpp.h}, add the
3795 #define TRACEPOINT_DEFINE
3799 . Compile the user library source file:
3804 $ gcc -I. -fpic -c emon.c
3808 . Build the user library shared object:
3813 $ gcc -shared -o libemon.so emon.o
3817 To build the application:
3819 . Compile the application source file:
3828 . Build the application:
3833 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3837 To run the instrumented application:
3839 * Start the application:
3849 The application is statically linked with the tracepoint provider
3850 package object file.
3852 The application dynamically loads the instrumented user library.
3854 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3857 include::../common/ust-sit-step-tp-o.txt[]
3859 To build the application:
3861 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3866 #define TRACEPOINT_DEFINE
3870 . Compile the application source file:
3879 . Build the application:
3884 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3889 The `--export-dynamic` option passed to the linker is necessary for the
3890 dynamically loaded library to ``see'' the tracepoint symbols defined in
3893 To build the instrumented user library:
3895 . Compile the user library source file:
3900 $ gcc -I. -fpic -c emon.c
3904 . Build the user library shared object:
3909 $ gcc -shared -o libemon.so emon.o
3913 To run the application:
3915 * Start the application:
3926 [[using-lttng-ust-with-daemons]]
3927 ===== Use noch:{LTTng-UST} with daemons
3929 If your instrumented application calls man:fork(2), man:clone(2),
3930 or BSD's man:rfork(2), without a following man:exec(3)-family
3931 system call, you must preload the path:{liblttng-ust-fork.so} shared
3932 object when you start the application.
3936 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3939 If your tracepoint provider package is
3940 a shared library which you also preload, you must put both
3941 shared objects in env:LD_PRELOAD:
3945 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3951 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
3953 If your instrumented application closes one or more file descriptors
3954 which it did not open itself, you must preload the
3955 path:{liblttng-ust-fd.so} shared object when you start the application:
3959 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
3962 Typical use cases include closing all the file descriptors after
3963 man:fork(2) or man:rfork(2) and buggy applications doing
3967 [[lttng-ust-pkg-config]]
3968 ===== Use noch:{pkg-config}
3970 On some distributions, LTTng-UST ships with a
3971 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3972 metadata file. If this is your case, then use cmd:pkg-config to
3973 build an application on the command line:
3977 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3981 [[instrumenting-32-bit-app-on-64-bit-system]]
3982 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3984 In order to trace a 32-bit application running on a 64-bit system,
3985 LTTng must use a dedicated 32-bit
3986 <<lttng-consumerd,consumer daemon>>.
3988 The following steps show how to build and install a 32-bit consumer
3989 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3990 build and install the 32-bit LTTng-UST libraries, and how to build and
3991 link an instrumented 32-bit application in that context.
3993 To build a 32-bit instrumented application for a 64-bit target system,
3994 assuming you have a fresh target system with no installed Userspace RCU
3997 . Download, build, and install a 32-bit version of Userspace RCU:
4002 $ cd $(mktemp -d) &&
4003 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
4004 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
4005 cd userspace-rcu-0.9.* &&
4006 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
4008 sudo make install &&
4013 . Using the package manager of your distribution, or from source,
4014 install the following 32-bit versions of the following dependencies of
4015 LTTng-tools and LTTng-UST:
4018 * https://sourceforge.net/projects/libuuid/[libuuid]
4019 * http://directory.fsf.org/wiki/Popt[popt]
4020 * http://www.xmlsoft.org/[libxml2]
4023 . Download, build, and install a 32-bit version of the latest
4024 LTTng-UST{nbsp}{revision}:
4029 $ cd $(mktemp -d) &&
4030 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
4031 tar -xf lttng-ust-latest-2.12.tar.bz2 &&
4032 cd lttng-ust-2.12.* &&
4033 ./configure --libdir=/usr/local/lib32 \
4034 CFLAGS=-m32 CXXFLAGS=-m32 \
4035 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4037 sudo make install &&
4044 Depending on your distribution,
4045 32-bit libraries could be installed at a different location than
4046 `/usr/lib32`. For example, Debian is known to install
4047 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
4049 In this case, make sure to set `LDFLAGS` to all the
4050 relevant 32-bit library paths, for example:
4054 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
4058 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
4059 the 32-bit consumer daemon:
4064 $ cd $(mktemp -d) &&
4065 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
4066 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
4067 cd lttng-tools-2.12.* &&
4068 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
4069 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
4070 --disable-bin-lttng --disable-bin-lttng-crash \
4071 --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
4073 cd src/bin/lttng-consumerd &&
4074 sudo make install &&
4079 . From your distribution or from source,
4080 <<installing-lttng,install>> the 64-bit versions of
4081 LTTng-UST and Userspace RCU.
4082 . Download, build, and install the 64-bit version of the
4083 latest LTTng-tools{nbsp}{revision}:
4088 $ cd $(mktemp -d) &&
4089 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
4090 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
4091 cd lttng-tools-2.12.* &&
4092 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4093 --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4095 sudo make install &&
4100 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4101 when linking your 32-bit application:
4104 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4105 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4108 For example, let's rebuild the quick start example in
4109 <<tracing-your-own-user-application,Trace a user application>> as an
4110 instrumented 32-bit application:
4115 $ gcc -m32 -c -I. hello-tp.c
4116 $ gcc -m32 -c hello.c
4117 $ gcc -m32 -o hello hello.o hello-tp.o \
4118 -L/usr/lib32 -L/usr/local/lib32 \
4119 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4124 No special action is required to execute the 32-bit application and
4125 to trace it: use the command-line man:lttng(1) tool as usual.
4132 man:tracef(3) is a small LTTng-UST API designed for quick,
4133 man:printf(3)-like instrumentation without the burden of
4134 <<tracepoint-provider,creating>> and
4135 <<building-tracepoint-providers-and-user-application,building>>
4136 a tracepoint provider package.
4138 To use `tracef()` in your application:
4140 . In the C or C++ source files where you need to use `tracef()`,
4141 include `<lttng/tracef.h>`:
4146 #include <lttng/tracef.h>
4150 . In the source code of the application, use `tracef()` like you would
4158 tracef("my message: %d (%s)", my_integer, my_string);
4164 . Link your application with `liblttng-ust`:
4169 $ gcc -o app app.c -llttng-ust
4173 To trace the events that `tracef()` calls emit:
4175 * <<enabling-disabling-events,Create an event rule>> which matches the
4176 `lttng_ust_tracef:*` event name:
4181 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4186 .Limitations of `tracef()`
4188 The `tracef()` utility function was developed to make user space tracing
4189 super simple, albeit with notable disadvantages compared to
4190 <<defining-tracepoints,user-defined tracepoints>>:
4192 * All the emitted events have the same tracepoint provider and
4193 tracepoint names, respectively `lttng_ust_tracef` and `event`.
4194 * There is no static type checking.
4195 * The only event record field you actually get, named `msg`, is a string
4196 potentially containing the values you passed to `tracef()`
4197 using your own format string. This also means that you can't filter
4198 events with a custom expression at run time because there are no
4200 * Since `tracef()` uses the man:vasprintf(3) function of the
4201 C{nbsp}standard library behind the scenes to format the strings at run
4202 time, its expected performance is lower than with user-defined
4203 tracepoints, which don't require a conversion to a string.
4205 Taking this into consideration, `tracef()` is useful for some quick
4206 prototyping and debugging, but you shouldn't consider it for any
4207 permanent and serious applicative instrumentation.
4213 ==== Use `tracelog()`
4215 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
4216 the difference that it accepts an additional log level parameter.
4218 The goal of `tracelog()` is to ease the migration from logging to
4221 To use `tracelog()` in your application:
4223 . In the C or C++ source files where you need to use `tracelog()`,
4224 include `<lttng/tracelog.h>`:
4229 #include <lttng/tracelog.h>
4233 . In the source code of the application, use `tracelog()` like you would
4234 use man:printf(3), except for the first parameter which is the log
4242 tracelog(TRACE_WARNING, "my message: %d (%s)",
4243 my_integer, my_string);
4249 See man:lttng-ust(3) for a list of available log level names.
4251 . Link your application with `liblttng-ust`:
4256 $ gcc -o app app.c -llttng-ust
4260 To trace the events that `tracelog()` calls emit with a log level
4261 _as severe as_ a specific log level:
4263 * <<enabling-disabling-events,Create an event rule>> which matches the
4264 `lttng_ust_tracelog:*` event name and a minimum level
4270 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4271 --loglevel=TRACE_WARNING
4275 To trace the events that `tracelog()` calls emit with a
4276 _specific log level_:
4278 * Create an event rule which matches the `lttng_ust_tracelog:*`
4279 event name and a specific log level:
4284 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4285 --loglevel-only=TRACE_INFO
4290 [[prebuilt-ust-helpers]]
4291 === Prebuilt user space tracing helpers
4293 The LTTng-UST package provides a few helpers in the form of preloadable
4294 shared objects which automatically instrument system functions and
4297 The helper shared objects are normally found in dir:{/usr/lib}. If you
4298 built LTTng-UST <<building-from-source,from source>>, they are probably
4299 located in dir:{/usr/local/lib}.
4301 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4304 path:{liblttng-ust-libc-wrapper.so}::
4305 path:{liblttng-ust-pthread-wrapper.so}::
4306 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4307 memory and POSIX threads function tracing>>.
4309 path:{liblttng-ust-cyg-profile.so}::
4310 path:{liblttng-ust-cyg-profile-fast.so}::
4311 <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4313 path:{liblttng-ust-dl.so}::
4314 <<liblttng-ust-dl,Dynamic linker tracing>>.
4316 To use a user space tracing helper with any user application:
4318 * Preload the helper shared object when you start the application:
4323 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4327 You can preload more than one helper:
4332 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4338 [[liblttng-ust-libc-pthread-wrapper]]
4339 ==== Instrument C standard library memory and POSIX threads functions
4341 The path:{liblttng-ust-libc-wrapper.so} and
4342 path:{liblttng-ust-pthread-wrapper.so} helpers
4343 add instrumentation to some C standard library and POSIX
4347 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4349 |TP provider name |TP name |Instrumented function
4351 .6+|`lttng_ust_libc` |`malloc` |man:malloc(3)
4352 |`calloc` |man:calloc(3)
4353 |`realloc` |man:realloc(3)
4354 |`free` |man:free(3)
4355 |`memalign` |man:memalign(3)
4356 |`posix_memalign` |man:posix_memalign(3)
4360 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4362 |TP provider name |TP name |Instrumented function
4364 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4365 |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4366 |`pthread_mutex_trylock` |man:pthread_mutex_trylock(3p)
4367 |`pthread_mutex_unlock` |man:pthread_mutex_unlock(3p)
4370 When you preload the shared object, it replaces the functions listed
4371 in the previous tables by wrappers which contain tracepoints and call
4372 the replaced functions.
4375 [[liblttng-ust-cyg-profile]]
4376 ==== Instrument function entry and exit
4378 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4379 to the entry and exit points of functions.
4381 man:gcc(1) and man:clang(1) have an option named
4382 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4383 which generates instrumentation calls for entry and exit to functions.
4384 The LTTng-UST function tracing helpers,
4385 path:{liblttng-ust-cyg-profile.so} and
4386 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4387 to add tracepoints to the two generated functions (which contain
4388 `cyg_profile` in their names, hence the name of the helper).
4390 To use the LTTng-UST function tracing helper, the source files to
4391 instrument must be built using the `-finstrument-functions` compiler
4394 There are two versions of the LTTng-UST function tracing helper:
4396 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4397 that you should only use when it can be _guaranteed_ that the
4398 complete event stream is recorded without any lost event record.
4399 Any kind of duplicate information is left out.
4401 Assuming no event record is lost, having only the function addresses on
4402 entry is enough to create a call graph, since an event record always
4403 contains the ID of the CPU that generated it.
4405 Use a tool like man:addr2line(1) to convert function addresses back to
4406 source file names and line numbers.
4408 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4409 which also works in use cases where event records might get discarded or
4410 not recorded from application startup.
4411 In these cases, the trace analyzer needs more information to be
4412 able to reconstruct the program flow.
4414 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4415 points of this helper.
4417 All the tracepoints that this helper provides have the
4418 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4420 TIP: It's sometimes a good idea to limit the number of source files that
4421 you compile with the `-finstrument-functions` option to prevent LTTng
4422 from writing an excessive amount of trace data at run time. When using
4424 `-finstrument-functions-exclude-function-list` option to avoid
4425 instrument entries and exits of specific function names.
4430 ==== Instrument the dynamic linker
4432 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4433 man:dlopen(3) and man:dlclose(3) function calls.
4435 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4440 [[java-application]]
4441 === User space Java agent
4443 You can instrument any Java application which uses one of the following
4446 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4447 (JUL) core logging facilities.
4448 * http://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4449 LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4452 .LTTng-UST Java agent imported by a Java application.
4453 image::java-app.png[]
4455 Note that the methods described below are new in LTTng{nbsp}2.8.
4456 Previous LTTng versions use another technique.
4458 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4459 and https://ci.lttng.org/[continuous integration], thus this version is
4460 directly supported. However, the LTTng-UST Java agent is also tested
4461 with OpenJDK{nbsp}7.
4466 ==== Use the LTTng-UST Java agent for `java.util.logging`
4468 To use the LTTng-UST Java agent in a Java application which uses
4469 `java.util.logging` (JUL):
4471 . In the source code of the Java application, import the LTTng-UST log
4472 handler package for `java.util.logging`:
4477 import org.lttng.ust.agent.jul.LttngLogHandler;
4481 . Create an LTTng-UST JUL log handler:
4486 Handler lttngUstLogHandler = new LttngLogHandler();
4490 . Add this handler to the JUL loggers which should emit LTTng events:
4495 Logger myLogger = Logger.getLogger("some-logger");
4497 myLogger.addHandler(lttngUstLogHandler);
4501 . Use `java.util.logging` log statements and configuration as usual.
4502 The loggers with an attached LTTng-UST log handler can emit
4505 . Before exiting the application, remove the LTTng-UST log handler from
4506 the loggers attached to it and call its `close()` method:
4511 myLogger.removeHandler(lttngUstLogHandler);
4512 lttngUstLogHandler.close();
4516 This isn't strictly necessary, but it is recommended for a clean
4517 disposal of the resources of the handler.
4519 . Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
4520 path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4522 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4523 path] when you build the Java application.
4525 The JAR files are typically located in dir:{/usr/share/java}.
4527 IMPORTANT: The LTTng-UST Java agent must be
4528 <<installing-lttng,installed>> for the logging framework your
4531 .Use the LTTng-UST Java agent for `java.util.logging`.
4536 import java.io.IOException;
4537 import java.util.logging.Handler;
4538 import java.util.logging.Logger;
4539 import org.lttng.ust.agent.jul.LttngLogHandler;
4543 private static final int answer = 42;
4545 public static void main(String[] argv) throws Exception
4548 Logger logger = Logger.getLogger("jello");
4550 // Create an LTTng-UST log handler
4551 Handler lttngUstLogHandler = new LttngLogHandler();
4553 // Add the LTTng-UST log handler to our logger
4554 logger.addHandler(lttngUstLogHandler);
4557 logger.info("some info");
4558 logger.warning("some warning");
4560 logger.finer("finer information; the answer is " + answer);
4562 logger.severe("error!");
4564 // Not mandatory, but cleaner
4565 logger.removeHandler(lttngUstLogHandler);
4566 lttngUstLogHandler.close();
4575 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4578 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4579 <<enabling-disabling-events,create an event rule>> matching the
4580 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4585 $ lttng enable-event --jul jello
4589 Run the compiled class:
4593 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4596 <<basic-tracing-session-control,Stop tracing>> and inspect the
4606 In the resulting trace, an <<event,event record>> generated by a Java
4607 application using `java.util.logging` is named `lttng_jul:event` and
4608 has the following fields:
4617 Name of the class in which the log statement was executed.
4620 Name of the method in which the log statement was executed.
4623 Logging time (timestamp in milliseconds).
4626 Log level integer value.
4629 ID of the thread in which the log statement was executed.
4631 Use the opt:lttng-enable-event(1):--loglevel or
4632 opt:lttng-enable-event(1):--loglevel-only option of the
4633 man:lttng-enable-event(1) command to target a range of JUL log levels
4634 or a specific JUL log level.
4639 ==== Use the LTTng-UST Java agent for Apache log4j
4641 To use the LTTng-UST Java agent in a Java application which uses
4642 Apache log4j{nbsp}1.2:
4644 . In the source code of the Java application, import the LTTng-UST log
4645 appender package for Apache log4j:
4650 import org.lttng.ust.agent.log4j.LttngLogAppender;
4654 . Create an LTTng-UST log4j log appender:
4659 Appender lttngUstLogAppender = new LttngLogAppender();
4663 . Add this appender to the log4j loggers which should emit LTTng events:
4668 Logger myLogger = Logger.getLogger("some-logger");
4670 myLogger.addAppender(lttngUstLogAppender);
4674 . Use Apache log4j log statements and configuration as usual. The
4675 loggers with an attached LTTng-UST log appender can emit LTTng events.
4677 . Before exiting the application, remove the LTTng-UST log appender from
4678 the loggers attached to it and call its `close()` method:
4683 myLogger.removeAppender(lttngUstLogAppender);
4684 lttngUstLogAppender.close();
4688 This isn't strictly necessary, but it is recommended for a clean
4689 disposal of the resources of the appender.
4691 . Include the common and log4j-specific JAR
4692 files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
4693 path:{lttng-ust-agent-log4j.jar}, in the
4694 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4695 path] when you build the Java application.
4697 The JAR files are typically located in dir:{/usr/share/java}.
4699 IMPORTANT: The LTTng-UST Java agent must be
4700 <<installing-lttng,installed>> for the logging framework your
4703 .Use the LTTng-UST Java agent for Apache log4j.
4708 import org.apache.log4j.Appender;
4709 import org.apache.log4j.Logger;
4710 import org.lttng.ust.agent.log4j.LttngLogAppender;
4714 private static final int answer = 42;
4716 public static void main(String[] argv) throws Exception
4719 Logger logger = Logger.getLogger("jello");
4721 // Create an LTTng-UST log appender
4722 Appender lttngUstLogAppender = new LttngLogAppender();
4724 // Add the LTTng-UST log appender to our logger
4725 logger.addAppender(lttngUstLogAppender);
4728 logger.info("some info");
4729 logger.warn("some warning");
4731 logger.debug("debug information; the answer is " + answer);
4733 logger.fatal("error!");
4735 // Not mandatory, but cleaner
4736 logger.removeAppender(lttngUstLogAppender);
4737 lttngUstLogAppender.close();
4743 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4748 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4751 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4752 <<enabling-disabling-events,create an event rule>> matching the
4753 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4758 $ lttng enable-event --log4j jello
4762 Run the compiled class:
4766 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4769 <<basic-tracing-session-control,Stop tracing>> and inspect the
4779 In the resulting trace, an <<event,event record>> generated by a Java
4780 application using log4j is named `lttng_log4j:event` and
4781 has the following fields:
4790 Name of the class in which the log statement was executed.
4793 Name of the method in which the log statement was executed.
4796 Name of the file in which the executed log statement is located.
4799 Line number at which the log statement was executed.
4805 Log level integer value.
4808 Name of the Java thread in which the log statement was executed.
4810 Use the opt:lttng-enable-event(1):--loglevel or
4811 opt:lttng-enable-event(1):--loglevel-only option of the
4812 man:lttng-enable-event(1) command to target a range of Apache log4j
4813 log levels or a specific log4j log level.
4817 [[java-application-context]]
4818 ==== Provide application-specific context fields in a Java application
4820 A Java application-specific context field is a piece of state provided
4821 by the application which <<adding-context,you can add>>, using the
4822 man:lttng-add-context(1) command, to each <<event,event record>>
4823 produced by the log statements of this application.
4825 For example, a given object might have a current request ID variable.
4826 You can create a context information retriever for this object and
4827 assign a name to this current request ID. You can then, using the
4828 man:lttng-add-context(1) command, add this context field by name to
4829 the JUL or log4j <<channel,channel>>.
4831 To provide application-specific context fields in a Java application:
4833 . In the source code of the Java application, import the LTTng-UST
4834 Java agent context classes and interfaces:
4839 import org.lttng.ust.agent.context.ContextInfoManager;
4840 import org.lttng.ust.agent.context.IContextInfoRetriever;
4844 . Create a context information retriever class, that is, a class which
4845 implements the `IContextInfoRetriever` interface:
4850 class MyContextInfoRetriever implements IContextInfoRetriever
4853 public Object retrieveContextInfo(String key)
4855 if (key.equals("intCtx")) {
4857 } else if (key.equals("strContext")) {
4858 return "context value!";
4867 This `retrieveContextInfo()` method is the only member of the
4868 `IContextInfoRetriever` interface. Its role is to return the current
4869 value of a state by name to create a context field. The names of the
4870 context fields and which state variables they return depends on your
4873 All primitive types and objects are supported as context fields.
4874 When `retrieveContextInfo()` returns an object, the context field
4875 serializer calls its `toString()` method to add a string field to
4876 event records. The method can also return `null`, which means that
4877 no context field is available for the required name.
4879 . Register an instance of your context information retriever class to
4880 the context information manager singleton:
4885 IContextInfoRetriever cir = new MyContextInfoRetriever();
4886 ContextInfoManager cim = ContextInfoManager.getInstance();
4887 cim.registerContextInfoRetriever("retrieverName", cir);
4891 . Before exiting the application, remove your context information
4892 retriever from the context information manager singleton:
4897 ContextInfoManager cim = ContextInfoManager.getInstance();
4898 cim.unregisterContextInfoRetriever("retrieverName");
4902 This isn't strictly necessary, but it is recommended for a clean
4903 disposal of some resources of the manager.
4905 . Build your Java application with LTTng-UST Java agent support as
4906 usual, following the procedure for either the <<jul,JUL>> or
4907 <<log4j,Apache log4j>> framework.
4910 .Provide application-specific context fields in a Java application.
4915 import java.util.logging.Handler;
4916 import java.util.logging.Logger;
4917 import org.lttng.ust.agent.jul.LttngLogHandler;
4918 import org.lttng.ust.agent.context.ContextInfoManager;
4919 import org.lttng.ust.agent.context.IContextInfoRetriever;
4923 // Our context information retriever class
4924 private static class MyContextInfoRetriever
4925 implements IContextInfoRetriever
4928 public Object retrieveContextInfo(String key) {
4929 if (key.equals("intCtx")) {
4931 } else if (key.equals("strContext")) {
4932 return "context value!";
4939 private static final int answer = 42;
4941 public static void main(String args[]) throws Exception
4943 // Get the context information manager instance
4944 ContextInfoManager cim = ContextInfoManager.getInstance();
4946 // Create and register our context information retriever
4947 IContextInfoRetriever cir = new MyContextInfoRetriever();
4948 cim.registerContextInfoRetriever("myRetriever", cir);
4951 Logger logger = Logger.getLogger("jello");
4953 // Create an LTTng-UST log handler
4954 Handler lttngUstLogHandler = new LttngLogHandler();
4956 // Add the LTTng-UST log handler to our logger
4957 logger.addHandler(lttngUstLogHandler);
4960 logger.info("some info");
4961 logger.warning("some warning");
4963 logger.finer("finer information; the answer is " + answer);
4965 logger.severe("error!");
4967 // Not mandatory, but cleaner
4968 logger.removeHandler(lttngUstLogHandler);
4969 lttngUstLogHandler.close();
4970 cim.unregisterContextInfoRetriever("myRetriever");
4979 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4982 <<creating-destroying-tracing-sessions,Create a tracing session>>
4983 and <<enabling-disabling-events,create an event rule>> matching the
4989 $ lttng enable-event --jul jello
4992 <<adding-context,Add the application-specific context fields>> to the
4997 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
4998 $ lttng add-context --jul --type='$app.myRetriever:strContext'
5001 <<basic-tracing-session-control,Start tracing>>:
5008 Run the compiled class:
5012 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5015 <<basic-tracing-session-control,Stop tracing>> and inspect the
5027 [[python-application]]
5028 === User space Python agent
5030 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
5032 https://docs.python.org/3/library/logging.html[`logging`] package.
5034 Each log statement emits an LTTng event once the
5035 application module imports the
5036 <<lttng-ust-agents,LTTng-UST Python agent>> package.
5039 .A Python application importing the LTTng-UST Python agent.
5040 image::python-app.png[]
5042 To use the LTTng-UST Python agent:
5044 . In the source code of the Python application, import the LTTng-UST
5054 The LTTng-UST Python agent automatically adds its logging handler to the
5055 root logger at import time.
5057 Any log statement that the application executes before this import does
5058 not emit an LTTng event.
5060 IMPORTANT: The LTTng-UST Python agent must be
5061 <<installing-lttng,installed>>.
5063 . Use log statements and logging configuration as usual.
5064 Since the LTTng-UST Python agent adds a handler to the _root_
5065 logger, you can trace any log statement from any logger.
5067 .Use the LTTng-UST Python agent.
5078 logging.basicConfig()
5079 logger = logging.getLogger('my-logger')
5082 logger.debug('debug message')
5083 logger.info('info message')
5084 logger.warn('warn message')
5085 logger.error('error message')
5086 logger.critical('critical message')
5090 if __name__ == '__main__':
5094 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5095 logging handler which prints to the standard error stream, isn't
5096 strictly required for LTTng-UST tracing to work, but in versions of
5097 Python preceding{nbsp}3.2, you could see a warning message which indicates
5098 that no handler exists for the logger `my-logger`.
5100 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5101 <<enabling-disabling-events,create an event rule>> matching the
5102 `my-logger` Python logger, and <<basic-tracing-session-control,start
5108 $ lttng enable-event --python my-logger
5112 Run the Python script:
5119 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5129 In the resulting trace, an <<event,event record>> generated by a Python
5130 application is named `lttng_python:event` and has the following fields:
5133 Logging time (string).
5142 Name of the function in which the log statement was executed.
5145 Line number at which the log statement was executed.
5148 Log level integer value.
5151 ID of the Python thread in which the log statement was executed.
5154 Name of the Python thread in which the log statement was executed.
5156 Use the opt:lttng-enable-event(1):--loglevel or
5157 opt:lttng-enable-event(1):--loglevel-only option of the
5158 man:lttng-enable-event(1) command to target a range of Python log levels
5159 or a specific Python log level.
5161 When an application imports the LTTng-UST Python agent, the agent tries
5162 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5163 <<start-sessiond,start the session daemon>> _before_ you run the Python
5164 application. If a session daemon is found, the agent tries to register
5165 to it during five seconds, after which the application continues
5166 without LTTng tracing support. Override this timeout value with
5167 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5170 If the session daemon stops while a Python application with an imported
5171 LTTng-UST Python agent runs, the agent retries to connect and to
5172 register to a session daemon every three seconds. Override this
5173 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5178 [[proc-lttng-logger-abi]]
5181 The `lttng-tracer` Linux kernel module, part of
5182 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5183 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5184 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5185 to any of those files to emit an LTTng event.
5188 .An application writes to the LTTng logger file to emit an LTTng event.
5189 image::lttng-logger.png[]
5191 The LTTng logger is the quickest method--not the most efficient,
5192 however--to add instrumentation to an application. It is designed
5193 mostly to instrument shell scripts:
5197 $ echo "Some message, some $variable" > /dev/lttng-logger
5200 Any event that the LTTng logger emits is named `lttng_logger` and
5201 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5202 other instrumentation points in the kernel tracing domain, **any Unix
5203 user** can <<enabling-disabling-events,create an event rule>> which
5204 matches its event name, not only the root user or users in the
5205 <<tracing-group,tracing group>>.
5207 To use the LTTng logger:
5209 * From any application, write text data to the path:{/dev/lttng-logger}
5212 The `msg` field of `lttng_logger` event records contains the
5215 NOTE: The maximum message length of an LTTng logger event is
5216 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5217 than one event to contain the remaining data.
5219 You shouldn't use the LTTng logger to trace a user application which
5220 can be instrumented in a more efficient way, namely:
5222 * <<c-application,C and $$C++$$ applications>>.
5223 * <<java-application,Java applications>>.
5224 * <<python-application,Python applications>>.
5226 .Use the LTTng logger.
5231 echo 'Hello, World!' > /dev/lttng-logger
5233 df --human-readable --print-type / > /dev/lttng-logger
5236 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5237 <<enabling-disabling-events,create an event rule>> matching the
5238 `lttng_logger` Linux kernel tracepoint, and
5239 <<basic-tracing-session-control,start tracing>>:
5244 $ lttng enable-event --kernel lttng_logger
5248 Run the Bash script:
5255 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5266 [[instrumenting-linux-kernel]]
5267 === LTTng kernel tracepoints
5269 NOTE: This section shows how to _add_ instrumentation points to the
5270 Linux kernel. The subsystems of the kernel are already thoroughly
5271 instrumented at strategic places for LTTng when you
5272 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5276 There are two methods to instrument the Linux kernel:
5278 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
5279 tracepoint which uses the `TRACE_EVENT()` API.
5281 Choose this if you want to instrumentation a Linux kernel tree with an
5282 instrumentation point compatible with ftrace, perf, and SystemTap.
5284 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
5285 instrument an out-of-tree kernel module.
5287 Choose this if you don't need ftrace, perf, or SystemTap support.
5291 [[linux-add-lttng-layer]]
5292 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5294 This section shows how to add an LTTng layer to existing ftrace
5295 instrumentation using the `TRACE_EVENT()` API.
5297 This section doesn't document the `TRACE_EVENT()` macro. Read the
5298 following articles to learn more about this API:
5300 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5301 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5302 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5304 The following procedure assumes that your ftrace tracepoints are
5305 correctly defined in their own header and that they are created in
5306 one source file using the `CREATE_TRACE_POINTS` definition.
5308 To add an LTTng layer over an existing ftrace tracepoint:
5310 . Make sure the following kernel configuration options are
5316 * `CONFIG_HIGH_RES_TIMERS`
5317 * `CONFIG_TRACEPOINTS`
5320 . Build the Linux source tree with your custom ftrace tracepoints.
5321 . Boot the resulting Linux image on your target system.
5323 Confirm that the tracepoints exist by looking for their names in the
5324 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5325 is your subsystem name.
5327 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5332 $ cd $(mktemp -d) &&
5333 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
5334 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
5335 cd lttng-modules-2.12.*
5339 . In dir:{instrumentation/events/lttng-module}, relative to the root
5340 of the LTTng-modules source tree, create a header file named
5341 +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5342 LTTng-modules tracepoint definitions using the LTTng-modules
5345 Start with this template:
5349 .path:{instrumentation/events/lttng-module/my_subsys.h}
5352 #define TRACE_SYSTEM my_subsys
5354 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5355 #define _LTTNG_MY_SUBSYS_H
5357 #include "../../../probes/lttng-tracepoint-event.h"
5358 #include <linux/tracepoint.h>
5360 LTTNG_TRACEPOINT_EVENT(
5362 * Format is identical to the TRACE_EVENT() version for the three
5363 * following macro parameters:
5366 TP_PROTO(int my_int, const char *my_string),
5367 TP_ARGS(my_int, my_string),
5369 /* LTTng-modules specific macros */
5371 ctf_integer(int, my_int_field, my_int)
5372 ctf_string(my_bar_field, my_bar)
5376 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5378 #include "../../../probes/define_trace.h"
5382 The entries in the `TP_FIELDS()` section are the list of fields for the
5383 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5384 the `TRACE_EVENT()` ftrace macro.
5386 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5387 complete description of the available `ctf_*()` macros.
5389 . Create the kernel module C{nbsp}source file of the LTTng-modules
5390 probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5395 .path:{probes/lttng-probe-my-subsys.c}
5397 #include <linux/module.h>
5398 #include "../lttng-tracer.h"
5401 * Build-time verification of mismatch between mainline
5402 * TRACE_EVENT() arguments and the LTTng-modules adaptation
5403 * layer LTTNG_TRACEPOINT_EVENT() arguments.
5405 #include <trace/events/my_subsys.h>
5407 /* Create LTTng tracepoint probes */
5408 #define LTTNG_PACKAGE_BUILD
5409 #define CREATE_TRACE_POINTS
5410 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5412 #include "../instrumentation/events/lttng-module/my_subsys.h"
5414 MODULE_LICENSE("GPL and additional rights");
5415 MODULE_AUTHOR("Your name <your-email>");
5416 MODULE_DESCRIPTION("LTTng my_subsys probes");
5417 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5418 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5419 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5420 LTTNG_MODULES_EXTRAVERSION);
5424 . Edit path:{probes/KBuild} and add your new kernel module object
5425 next to the existing ones:
5429 .path:{probes/KBuild}
5433 obj-m += lttng-probe-module.o
5434 obj-m += lttng-probe-power.o
5436 obj-m += lttng-probe-my-subsys.o
5442 . Build and install the LTTng kernel modules:
5447 $ make KERNELDIR=/path/to/linux
5448 # make modules_install && depmod -a
5452 Replace `/path/to/linux` with the path to the Linux source tree where
5453 you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
5455 Note that you can also use the
5456 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5457 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5458 C code that need to be executed before the event fields are recorded.
5460 The best way to learn how to use the previous LTTng-modules macros is to
5461 inspect the existing LTTng-modules tracepoint definitions in the
5462 dir:{instrumentation/events/lttng-module} header files. Compare them
5463 with the Linux kernel mainline versions in the
5464 dir:{include/trace/events} directory of the Linux source tree.
5468 [[lttng-tracepoint-event-code]]
5469 ===== Use custom C code to access the data for tracepoint fields
5471 Although we recommended to always use the
5472 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5473 the arguments and fields of an LTTng-modules tracepoint when possible,
5474 sometimes you need a more complex process to access the data that the
5475 tracer records as event record fields. In other words, you need local
5476 variables and multiple C{nbsp}statements instead of simple
5477 argument-based expressions that you pass to the
5478 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5480 Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5481 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5482 a block of C{nbsp}code to be executed before LTTng records the fields.
5483 The structure of this macro is:
5486 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5488 LTTNG_TRACEPOINT_EVENT_CODE(
5490 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5491 * version for the following three macro parameters:
5494 TP_PROTO(int my_int, const char *my_string),
5495 TP_ARGS(my_int, my_string),
5497 /* Declarations of custom local variables */
5500 unsigned long b = 0;
5501 const char *name = "(undefined)";
5502 struct my_struct *my_struct;
5506 * Custom code which uses both tracepoint arguments
5507 * (in TP_ARGS()) and local variables (in TP_locvar()).
5509 * Local variables are actually members of a structure pointed
5510 * to by the special variable tp_locvar.
5514 tp_locvar->a = my_int + 17;
5515 tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5516 tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5517 tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5518 put_my_struct(tp_locvar->my_struct);
5527 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5528 * version for this, except that tp_locvar members can be
5529 * used in the argument expression parameters of
5530 * the ctf_*() macros.
5533 ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5534 ctf_integer(int, my_struct_a, tp_locvar->a)
5535 ctf_string(my_string_field, my_string)
5536 ctf_string(my_struct_name, tp_locvar->name)
5541 IMPORTANT: The C code defined in `TP_code()` must not have any side
5542 effects when executed. In particular, the code must not allocate
5543 memory or get resources without deallocating this memory or putting
5544 those resources afterwards.
5547 [[instrumenting-linux-kernel-tracing]]
5548 ==== Load and unload a custom probe kernel module
5550 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5551 kernel module>> in the kernel before it can emit LTTng events.
5553 To load the default probe kernel modules and a custom probe kernel
5556 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5557 probe modules to load when starting a root <<lttng-sessiond,session
5561 .Load the `my_subsys`, `usb`, and the default probe modules.
5565 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5570 You only need to pass the subsystem name, not the whole kernel module
5573 To load _only_ a given custom probe kernel module:
5575 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5576 modules to load when starting a root session daemon:
5579 .Load only the `my_subsys` and `usb` probe modules.
5583 # lttng-sessiond --kmod-probes=my_subsys,usb
5588 To confirm that a probe module is loaded:
5595 $ lsmod | grep lttng_probe_usb
5599 To unload the loaded probe modules:
5601 * Kill the session daemon with `SIGTERM`:
5606 # pkill lttng-sessiond
5610 You can also use the man:modprobe(8) `--remove` option if the session
5611 daemon terminates abnormally.
5614 [[controlling-tracing]]
5617 Once an application or a Linux kernel is
5618 <<instrumenting,instrumented>> for LTTng tracing,
5621 This section is divided in topics on how to use the various
5622 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5623 command-line tool>>, to _control_ the LTTng daemons and tracers.
5625 NOTE: In the following subsections, we refer to an man:lttng(1) command
5626 using its man page name. For example, instead of _Run the `create`
5627 command to..._, we use _Run the man:lttng-create(1) command to..._.
5631 === Start a session daemon
5633 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5634 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5637 You will see the following error when you run a command while no session
5641 Error: No session daemon is available
5644 The only command that automatically runs a session daemon is
5645 man:lttng-create(1), which you use to
5646 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5647 this is most of the time the first operation that you do, sometimes it's
5648 not. Some examples are:
5650 * <<list-instrumentation-points,List the available instrumentation points>>.
5651 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5653 [[tracing-group]] Each Unix user must have its own running session
5654 daemon to trace user applications. The session daemon that the root user
5655 starts is the only one allowed to control the LTTng kernel tracer. Users
5656 that are part of the _tracing group_ can control the root session
5657 daemon. The default tracing group name is `tracing`; set it to something
5658 else with the opt:lttng-sessiond(8):--group option when you start the
5659 root session daemon.
5661 To start a user session daemon:
5663 * Run man:lttng-sessiond(8):
5668 $ lttng-sessiond --daemonize
5672 To start the root session daemon:
5674 * Run man:lttng-sessiond(8) as the root user:
5679 # lttng-sessiond --daemonize
5683 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5684 start the session daemon in foreground.
5686 To stop a session daemon, use man:kill(1) on its process ID (standard
5689 Note that some Linux distributions could manage the LTTng session daemon
5690 as a service. In this case, you should use the service manager to
5691 start, restart, and stop session daemons.
5694 [[creating-destroying-tracing-sessions]]
5695 === Create and destroy a tracing session
5697 Almost all the LTTng control operations happen in the scope of
5698 a <<tracing-session,tracing session>>, which is the dialogue between the
5699 <<lttng-sessiond,session daemon>> and you.
5701 To create a tracing session with a generated name:
5703 * Use the man:lttng-create(1) command:
5712 The name of the created tracing session is `auto` followed by the
5715 To create a tracing session with a specific name:
5717 * Use the optional argument of the man:lttng-create(1) command:
5722 $ lttng create my-session
5726 Replace `my-session` with the specific tracing session name.
5728 LTTng appends the creation date to the name of the created tracing
5731 LTTng writes the traces of a tracing session in
5732 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5733 name of the tracing session. Note that the env:LTTNG_HOME environment
5734 variable defaults to `$HOME` if not set.
5736 To output LTTng traces to a non-default location:
5738 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5743 $ lttng create my-session --output=/tmp/some-directory
5747 You may create as many tracing sessions as you wish.
5749 To list all the existing tracing sessions for your Unix user:
5751 * Use the man:lttng-list(1) command:
5760 [[cur-tracing-session]]When you create a tracing session, it is set as
5761 the _current tracing session_. The following man:lttng(1) commands
5762 operate on the current tracing session when you don't specify one:
5764 [role="list-3-cols"]
5765 * man:lttng-add-context(1)
5766 * man:lttng-clear(1)
5767 * man:lttng-destroy(1)
5768 * man:lttng-disable-channel(1)
5769 * man:lttng-disable-event(1)
5770 * man:lttng-disable-rotation(1)
5771 * man:lttng-enable-channel(1)
5772 * man:lttng-enable-event(1)
5773 * man:lttng-enable-rotation(1)
5775 * man:lttng-regenerate(1)
5776 * man:lttng-rotate(1)
5778 * man:lttng-snapshot(1)
5779 * man:lttng-start(1)
5780 * man:lttng-status(1)
5782 * man:lttng-track(1)
5783 * man:lttng-untrack(1)
5786 To change the current tracing session:
5788 * Use the man:lttng-set-session(1) command:
5793 $ lttng set-session new-session
5797 Replace `new-session` by the name of the new current tracing session.
5799 When you're done tracing in a given tracing session, destroy it. This
5800 operation frees the resources taken by the tracing session to destroy;
5801 it doesn't destroy the trace data that LTTng wrote for this tracing
5802 session (see <<clear,Clear a tracing session>> for one way to do this).
5804 To destroy the current tracing session:
5806 * Use the man:lttng-destroy(1) command:
5815 The man:lttng-destroy(1) command also runs the man:lttng-stop(1)
5816 command implicitly (see <<basic-tracing-session-control,Start and stop a
5817 tracing session>>). You need to stop tracing to make LTTng flush the
5818 remaining trace data and make the trace readable.
5821 [[list-instrumentation-points]]
5822 === List the available instrumentation points
5824 The <<lttng-sessiond,session daemon>> can query the running instrumented
5825 user applications and the Linux kernel to get a list of available
5826 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5827 they are tracepoints and system calls. For the user space tracing
5828 domain, they are tracepoints. For the other tracing domains, they are
5831 To list the available instrumentation points:
5833 * Use the man:lttng-list(1) command with the option of the requested
5834 tracing domain amongst:
5837 opt:lttng-list(1):--kernel::
5838 Linux kernel tracepoints (your Unix user must be a root user, or it
5839 must be a member of the <<tracing-group,tracing group>>).
5841 opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
5842 Linux kernel system calls (your Unix user must be a root user, or it
5843 must be a member of the tracing group).
5845 opt:lttng-list(1):--userspace::
5846 User space tracepoints.
5848 opt:lttng-list(1):--jul::
5849 `java.util.logging` loggers.
5851 opt:lttng-list(1):--log4j::
5852 Apache log4j loggers.
5854 opt:lttng-list(1):--python::
5858 .List the available user space tracepoints.
5862 $ lttng list --userspace
5866 .List the available Linux kernel system call tracepoints.
5870 $ lttng list --kernel --syscall
5875 [[enabling-disabling-events]]
5876 === Create and enable an event rule
5878 Once you <<creating-destroying-tracing-sessions,create a tracing
5879 session>>, you can create <<event,event rules>> with the
5880 man:lttng-enable-event(1) command.
5882 You specify each condition with a command-line option. The available
5883 condition arguments are shown in the following table.
5885 [role="growable",cols="asciidoc,asciidoc,default"]
5886 .Condition command-line arguments for the man:lttng-enable-event(1) command.
5888 |Argument |Description |Applicable tracing domains
5894 . +--probe=__ADDR__+
5895 . +--function=__ADDR__+
5896 . +--userspace-probe=__PATH__:__SYMBOL__+
5897 . +--userspace-probe=sdt:__PATH__:__PROVIDER__:__NAME__+
5900 Instead of using the default _tracepoint_ instrumentation type, use:
5902 . A Linux system call (entry and exit).
5903 . A Linux https://lwn.net/Articles/132196/[kprobe] (symbol or address).
5904 . The entry and return points of a Linux function (symbol or address).
5905 . The entry point of a user application or library function (path to
5906 application/library and symbol).
5907 . A https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[SystemTap
5908 Statically Defined Tracing] (USDT) probe (path to application/library,
5909 provider and probe names).
5913 |First positional argument.
5916 Tracepoint or system call name.
5918 With the opt:lttng-enable-event(1):--probe,
5919 opt:lttng-enable-event(1):--function, and
5920 opt:lttng-enable-event(1):--userspace-probe options, this is a custom
5921 name given to the event rule. With the JUL, log4j, and Python domains,
5922 this is a logger name.
5924 With a tracepoint, logger, or system call name, use the special
5925 `*` globbing character to match anything (for example, `sched_*`,
5933 . +--loglevel=__LEVEL__+
5934 . +--loglevel-only=__LEVEL__+
5937 . Match only tracepoints or log statements with a logging level at
5938 least as severe as +__LEVEL__+.
5939 . Match only tracepoints or log statements with a logging level
5940 equal to +__LEVEL__+.
5942 See man:lttng-enable-event(1) for the list of available logging level
5945 |User space, JUL, log4j, and Python.
5947 |+--exclude=__EXCLUSIONS__+
5950 When you use a `*` character at the end of the tracepoint or logger
5951 name (first positional argument), exclude the specific names in the
5952 comma-delimited list +__EXCLUSIONS__+.
5955 User space, JUL, log4j, and Python.
5957 |+--filter=__EXPR__+
5960 Match only events which satisfy the expression +__EXPR__+.
5962 See man:lttng-enable-event(1) to learn more about the syntax of a
5969 You attach an event rule to a <<channel,channel>> on creation. If you do
5970 not specify the channel with the opt:lttng-enable-event(1):--channel
5971 option, and if the event rule to create is the first in its
5972 <<domain,tracing domain>> for a given tracing session, then LTTng
5973 creates a _default channel_ for you. This default channel is reused in
5974 subsequent invocations of the man:lttng-enable-event(1) command for the
5975 same tracing domain.
5977 An event rule is always enabled at creation time.
5979 The following examples show how to combine the previous
5980 command-line options to create simple to more complex event rules.
5982 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5986 $ lttng enable-event --kernel sched_switch
5990 .Create an event rule matching four Linux kernel system calls (default channel).
5994 $ lttng enable-event --kernel --syscall open,write,read,close
5998 .Create event rules matching tracepoints with filter expressions (default channel).
6002 $ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
6007 $ lttng enable-event --kernel --all \
6008 --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
6013 $ lttng enable-event --jul my_logger \
6014 --filter='$app.retriever:cur_msg_id > 3'
6017 IMPORTANT: Make sure to always quote the filter string when you
6018 use man:lttng(1) from a shell.
6020 See also <<pid-tracking,Track process attributes>> which offers another,
6021 more efficient filtering mechanism for process ID, user ID, and group
6025 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
6029 $ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
6032 IMPORTANT: Make sure to always quote the wildcard character when you
6033 use man:lttng(1) from a shell.
6036 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
6040 $ lttng enable-event --python my-app.'*' \
6041 --exclude='my-app.module,my-app.hello'
6045 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
6049 $ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
6053 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
6057 $ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
6061 .Create an event rule matching the `malloc` function entry in path:{/usr/lib/libc.so.6}:
6065 $ lttng enable-event --kernel --userspace-probe=/usr/lib/libc.so.6:malloc \
6070 .Create an event rule matching the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
6074 $ lttng enable-event --kernel --userspace-probe=sdt:serv:server:accept_request \
6075 server_accept_request
6079 The event rules of a given channel form a whitelist: as soon as an
6080 emitted event passes one of them, LTTng can record the event. For
6081 example, an event named `my_app:my_tracepoint` emitted from a user space
6082 tracepoint with a `TRACE_ERROR` log level passes both of the following
6087 $ lttng enable-event --userspace my_app:my_tracepoint
6088 $ lttng enable-event --userspace my_app:my_tracepoint \
6089 --loglevel=TRACE_INFO
6092 The second event rule is redundant: the first one includes
6096 [[disable-event-rule]]
6097 === Disable an event rule
6099 To disable an event rule that you <<enabling-disabling-events,created>>
6100 previously, use the man:lttng-disable-event(1) command. This command
6101 disables _all_ the event rules (of a given tracing domain and channel)
6102 which match an instrumentation point. The other conditions aren't
6103 supported as of LTTng{nbsp}{revision}.
6105 The LTTng tracer doesn't record an emitted event which passes
6106 a _disabled_ event rule.
6108 .Disable an event rule matching a Python logger (default channel).
6112 $ lttng disable-event --python my-logger
6116 .Disable an event rule matching all `java.util.logging` loggers (default channel).
6120 $ lttng disable-event --jul '*'
6124 .Disable _all_ the event rules of the default channel.
6126 The opt:lttng-disable-event(1):--all-events option isn't, like the
6127 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
6128 equivalent of the event name `*` (wildcard): it disables _all_ the event
6129 rules of a given channel.
6133 $ lttng disable-event --jul --all-events
6137 NOTE: You can't delete an event rule once you create it.
6141 === Get the status of a tracing session
6143 To get the status of the <<cur-tracing-session,current tracing
6144 session>>, that is, its parameters, its channels, event rules, and their
6147 * Use the man:lttng-status(1) command:
6156 To get the status of any tracing session:
6158 * Use the man:lttng-list(1) command with the name of the tracing
6164 $ lttng list my-session
6168 Replace `my-session` with the desired tracing session name.
6171 [[basic-tracing-session-control]]
6172 === Start and stop a tracing session
6174 Once you <<creating-destroying-tracing-sessions,create a tracing
6176 <<enabling-disabling-events,create one or more event rules>>,
6177 you can start and stop the tracers for this tracing session.
6179 To start tracing in the <<cur-tracing-session,current tracing session>>:
6181 * Use the man:lttng-start(1) command:
6190 LTTng is very flexible: you can launch user applications before
6191 or after the you start the tracers. The tracers only record the events
6192 if they pass enabled event rules and if they occur while the tracers are
6195 To stop tracing in the current tracing session:
6197 * Use the man:lttng-stop(1) command:
6206 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6207 records>> or lost sub-buffers since the last time you ran
6208 man:lttng-start(1), warnings are printed when you run the
6209 man:lttng-stop(1) command.
6211 IMPORTANT: You need to stop tracing to make LTTng flush the remaining
6212 trace data and make the trace readable. Note that the
6213 man:lttng-destroy(1) command (see
6214 <<creating-destroying-tracing-sessions,Create and destroy a tracing
6215 session>>) also runs the man:lttng-stop(1) command implicitly.
6219 === Clear a tracing session
6221 You might need to remove all the current tracing data of one or more
6222 <<tracing-session,tracing sessions>> between multiple attempts to
6223 reproduce a problem without interrupting the LTTng tracing activity.
6225 To clear the tracing data of the
6226 <<cur-tracing-session,current tracing session>>:
6228 * Use the man:lttng-clear(1) command:
6237 To clear the tracing data of all the tracing sessions:
6239 * Use the `lttng clear` command with the opt:lttng-clear(1):--all
6250 [[enabling-disabling-channels]]
6251 === Create a channel
6253 Once you create a tracing session, you can create a <<channel,channel>>
6254 with the man:lttng-enable-channel(1) command.
6256 Note that LTTng automatically creates a default channel when, for a
6257 given <<domain,tracing domain>>, no channels exist and you
6258 <<enabling-disabling-events,create>> the first event rule. This default
6259 channel is named `channel0` and its attributes are set to reasonable
6260 values. Therefore, you only need to create a channel when you need
6261 non-default attributes.
6263 You specify each non-default channel attribute with a command-line
6264 option when you use the man:lttng-enable-channel(1) command. The
6265 available command-line options are:
6267 [role="growable",cols="asciidoc,asciidoc"]
6268 .Command-line options for the man:lttng-enable-channel(1) command.
6270 |Option |Description
6276 <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> instead
6277 of the default _discard_ mode.
6279 |`--buffers-pid` (user space tracing domain only)
6282 Use the per-process <<channel-buffering-schemes,buffering scheme>>
6283 instead of the default per-user buffering scheme.
6285 |+--subbuf-size=__SIZE__+
6288 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
6289 either for each Unix user (default), or for each instrumented process.
6291 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6293 |+--num-subbuf=__COUNT__+
6296 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
6297 for each Unix user (default), or for each instrumented process.
6299 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6301 |+--tracefile-size=__SIZE__+
6304 Set the maximum size of each trace file that this channel writes within
6305 a stream to +__SIZE__+ bytes instead of no maximum.
6307 See <<tracefile-rotation,Trace file count and size>>.
6309 |+--tracefile-count=__COUNT__+
6312 Limit the number of trace files that this channel creates to
6313 +__COUNT__+ channels instead of no limit.
6315 See <<tracefile-rotation,Trace file count and size>>.
6317 |+--switch-timer=__PERIODUS__+
6320 Set the <<channel-switch-timer,switch timer period>>
6321 to +__PERIODUS__+{nbsp}µs.
6323 |+--read-timer=__PERIODUS__+
6326 Set the <<channel-read-timer,read timer period>>
6327 to +__PERIODUS__+{nbsp}µs.
6329 |[[opt-blocking-timeout]]+--blocking-timeout=__TIMEOUTUS__+
6332 Set the timeout of user space applications which load LTTng-UST
6333 in blocking mode to +__TIMEOUTUS__+:
6336 Never block (non-blocking mode).
6339 Block forever until space is available in a sub-buffer to record
6342 __n__, a positive value::
6343 Wait for at most __n__ µs when trying to write into a sub-buffer.
6345 Note that, for this option to have any effect on an instrumented
6346 user space application, you need to run the application with a set
6347 env:LTTNG_UST_ALLOW_BLOCKING environment variable.
6349 |+--output=__TYPE__+ (Linux kernel tracing domain only)
6352 Set the output type of the channel to +__TYPE__+, either `mmap` or
6357 You can only create a channel in the Linux kernel and user space
6358 <<domain,tracing domains>>: other tracing domains have their own channel
6359 created on the fly when <<enabling-disabling-events,creating event
6364 Because of a current LTTng limitation, you must create all channels
6365 _before_ you <<basic-tracing-session-control,start tracing>> in a given
6366 tracing session, that is, before the first time you run
6369 Since LTTng automatically creates a default channel when you use the
6370 man:lttng-enable-event(1) command with a specific tracing domain, you
6371 can't, for example, create a Linux kernel event rule, start tracing,
6372 and then create a user space event rule, because no user space channel
6373 exists yet and it's too late to create one.
6375 For this reason, make sure to configure your channels properly
6376 before starting the tracers for the first time!
6379 The following examples show how to combine the previous
6380 command-line options to create simple to more complex channels.
6382 .Create a Linux kernel channel with default attributes.
6386 $ lttng enable-channel --kernel my-channel
6390 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6394 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6395 --buffers-pid my-channel
6399 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6401 <<creating-destroying-tracing-sessions,Create a tracing-session>>,
6402 create the channel, <<enabling-disabling-events,create an event rule>>,
6403 and <<basic-tracing-session-control,start tracing>>:
6408 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
6409 $ lttng enable-event --userspace --channel=blocking-channel --all
6413 Run an application instrumented with LTTng-UST and allow it to block:
6417 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6421 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream
6425 $ lttng enable-channel --kernel --tracefile-count=8 \
6426 --tracefile-size=4194304 my-channel
6430 .Create a user space channel in overwrite (or _flight recorder_) mode.
6434 $ lttng enable-channel --userspace --overwrite my-channel
6438 <<enabling-disabling-events,Create>> the same event rule in
6439 two different channels:
6443 $ lttng enable-event --userspace --channel=my-channel app:tp
6444 $ lttng enable-event --userspace --channel=other-channel app:tp
6447 If both channels are enabled, when a tracepoint named `app:tp` is
6448 reached, LTTng records two events, one for each channel.
6452 === Disable a channel
6454 To disable a specific channel that you <<enabling-disabling-channels,created>>
6455 previously, use the man:lttng-disable-channel(1) command.
6457 .Disable a specific Linux kernel channel.
6461 $ lttng disable-channel --kernel my-channel
6465 The state of a channel precedes the individual states of event rules
6466 attached to it: event rules which belong to a disabled channel, even if
6467 they are enabled, are also considered disabled.
6471 === Add context fields to a channel
6473 Event record fields in trace files provide important information about
6474 events that occured previously, but sometimes some external context may
6475 help you solve a problem faster.
6477 Examples of context fields are:
6479 * The **process ID**, **thread ID**, **process name**, and
6480 **process priority** of the thread in which the event occurs.
6481 * The **hostname** of the system on which the event occurs.
6482 * The Linux kernel and user call stacks (since
6484 * The current values of many possible **performance counters** using
6486 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6488 ** Branch instructions, misses, and loads.
6490 * Any context defined at the application level (supported for the
6491 JUL and log4j <<domain,tracing domains>>).
6493 To get the full list of available context fields, see
6494 `lttng add-context --list`. Some context fields are reserved for a
6495 specific <<domain,tracing domain>> (Linux kernel or user space).
6497 You add context fields to <<channel,channels>>. All the events
6498 that a channel with added context fields records contain those fields.
6500 To add context fields to one or all the channels of a given tracing
6503 * Use the man:lttng-add-context(1) command.
6505 .Add context fields to all the channels of the current tracing session.
6507 The following command line adds the virtual process identifier and
6508 the per-thread CPU cycles count fields to all the user space channels
6510 <<cur-tracing-session,current tracing session>>.
6514 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6518 .Add performance counter context fields by raw ID
6520 See man:lttng-add-context(1) for the exact format of the context field
6521 type, which is partly compatible with the format used in
6526 $ lttng add-context --userspace --type=perf:thread:raw:r0110:test
6527 $ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6531 .Add context fields to a specific channel.
6533 The following command line adds the thread identifier and user call
6534 stack context fields to the Linux kernel channel named `my-channel` in
6535 the current tracing session.
6539 $ lttng add-context --kernel --channel=my-channel \
6540 --type=tid --type=callstack-user
6544 .Add an application-specific context field to a specific channel.
6546 The following command line adds the `cur_msg_id` context field of the
6547 `retriever` context retriever for all the instrumented
6548 <<java-application,Java applications>> recording <<event,event records>>
6549 in the channel named `my-channel`:
6553 $ lttng add-context --kernel --channel=my-channel \
6554 --type='$app:retriever:cur_msg_id'
6557 IMPORTANT: Make sure to always quote the `$` character when you
6558 use man:lttng-add-context(1) from a shell.
6561 NOTE: You can't remove context fields from a channel once you add it.
6566 === Track process attributes
6568 It's often useful to only allow processes with specific attributes to
6569 emit events. For example, you may wish to record all the system calls
6570 which a given process makes (à la
6571 http://linux.die.net/man/1/strace[strace]).
6573 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6574 purpose. Both commands operate on _inclusion sets_ of process attribute
6575 values. The available process attribute types are:
6577 Linux kernel <<domain,tracing domain>> only::
6581 * Virtual process ID (VPID).
6583 This is the PID as seen by the application.
6585 * Unix user ID (UID) (since LTTng{nbsp}2.12).
6587 * Virtual Unix user ID (VUID) (since LTTng{nbsp}2.12).
6589 This is the UID as seen by the application.
6591 * Unix group ID (GID) (since LTTng{nbsp}2.12).
6593 * Virtual Unix group ID (VGID) (since LTTng{nbsp}2.12).
6595 This is the GID as seen by the application.
6598 User space tracing domain::
6601 * VUID (since LTTng{nbsp}2.12).
6602 * VGID (since LTTng{nbsp}2.12).
6604 Each tracing domain has one inclusion set per process attribute type:
6605 the Linux kernel tracing domain has six while the user space tracing
6608 For a given event which passes an enabled <<event,event rule>> to be
6609 recorded, _all_ the attributes of its executing process must be part of
6610 the inclusion sets of the tracing domain of the event rule.
6612 Add entries to an inclusion set with the man:lttng-track(1) command and
6613 remove entries with the man:lttng-untrack(1) command. A process
6614 attribute is _tracked_ when it's part of an inclusion set and
6615 _untracked_ otherwise.
6619 The process attribute values are _numeric_.
6621 Should a process with a given tracked process ID, for example, exit, and
6622 then a new process be given this ID, then the latter would also be
6623 allowed to emit events.
6625 With the `lttng track` command, you can add Unix user and group _names_
6626 to the user and group inclusion sets: the <<lttng-sessiond,session
6627 daemon>> finds the corresponding UID, VUID, GID, or VGID once on
6628 _addition_ to the inclusion set. This means that if you rename the user
6629 or group after you run `lttng track`, its user/group ID remains tracked.
6632 .Track and untrack virtual process IDs.
6634 For the sake of the following example, assume the target system has
6635 16{nbsp}possible VPIDs.
6638 <<creating-destroying-tracing-sessions,create a tracing session>>,
6639 the user space VPID inclusion set contains _all_ the possible VPIDs:
6642 .All VPIDs are tracked.
6643 image::track-all.png[]
6645 When the inclusion set is full and you use the man:lttng-track(1)
6646 command to specify some VPIDs to track, LTTng first clears the inclusion
6647 set, and then it adds the specific VPIDs to track. After:
6651 $ lttng track --userspace --vpid=3,4,7,10,13
6654 the VPID inclusion set is:
6657 .VPIDs 3, 4, 7, 10, and 13 are tracked.
6658 image::track-3-4-7-10-13.png[]
6660 Add more VPIDs to the inclusion set afterwards:
6664 $ lttng track --userspace --vpid=1,15,16
6670 .VPIDs 1, 15, and 16 are added to the inclusion set.
6671 image::track-1-3-4-7-10-13-15-16.png[]
6673 The man:lttng-untrack(1) command removes entries from process attribute
6674 inclusion sets. Given the previous example, the following command:
6678 $ lttng untrack --userspace --vpid=3,7,10,13
6681 leads to this VPID inclusion set:
6684 .VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6685 image::track-1-4-15-16.png[]
6687 LTTng can track all the possible VPIDs again using the
6688 opt:lttng-track(1):--all option:
6692 $ lttng track --userspace --vpid --all
6695 The result is, again:
6698 .All VPIDs are tracked.
6699 image::track-all.png[]
6702 .Track only specific process attributes.
6704 A typical use case with process attribute tracking is to start with an
6705 empty inclusion set, then <<basic-tracing-session-control,start the
6706 tracers>>, and then add entries manually while the tracers are active.
6708 Use the opt:lttng-untrack(1):--all option of the
6709 man:lttng-untrack(1) command to clear the inclusion set after you
6710 <<creating-destroying-tracing-sessions,create a tracing session>>, for
6711 example (with UIDs):
6715 $ lttng untrack --kernel --uid --all
6721 .No UIDs are tracked.
6722 image::untrack-all.png[]
6724 If you trace with this inclusion set configuration, the LTTng kernel
6725 tracer records no events within the <<cur-tracing-session,current
6726 tracing session>> because it doesn't track any UID. Use the
6727 man:lttng-track(1) command as usual to track specific UIDs when you need
6732 $ lttng track --kernel --uid=http,11
6738 .UIDs 6 (`http`) and 11 are tracked.
6739 image::track-6-11.png[]
6744 [[saving-loading-tracing-session]]
6745 === Save and load tracing session configurations
6747 Configuring a <<tracing-session,tracing session>> can be long. Some of
6748 the tasks involved are:
6750 * <<enabling-disabling-channels,Create channels>> with
6751 specific attributes.
6752 * <<adding-context,Add context fields>> to specific channels.
6753 * <<enabling-disabling-events,Create event rules>> with specific log
6754 level and filter conditions.
6756 If you use LTTng to solve real world problems, chances are you have to
6757 record events using the same tracing session setup over and over,
6758 modifying a few variables each time in your instrumented program
6759 or environment. To avoid constant tracing session reconfiguration,
6760 the man:lttng(1) command-line tool can save and load tracing session
6761 configurations to/from XML files.
6763 To save a given tracing session configuration:
6765 * Use the man:lttng-save(1) command:
6770 $ lttng save my-session
6774 Replace `my-session` with the name of the tracing session to save.
6776 LTTng saves tracing session configurations to
6777 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6778 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6779 the opt:lttng-save(1):--output-path option to change this destination
6782 LTTng saves all configuration parameters, for example:
6784 * The tracing session name.
6785 * The trace data output path.
6786 * The channels with their state and all their attributes.
6787 * The context fields you added to channels.
6788 * The event rules with their state, log level and filter conditions.
6790 To load a tracing session:
6792 * Use the man:lttng-load(1) command:
6797 $ lttng load my-session
6801 Replace `my-session` with the name of the tracing session to load.
6803 When LTTng loads a configuration, it restores your saved tracing session
6804 as if you just configured it manually.
6806 See man:lttng-load(1) for the complete list of command-line options. You
6807 can also save and load many sessions at a time, and decide in which
6808 directory to output the XML files.
6811 [[sending-trace-data-over-the-network]]
6812 === Send trace data over the network
6814 LTTng can send the recorded trace data to a remote system over the
6815 network instead of writing it to the local file system.
6817 To send the trace data over the network:
6819 . On the _remote_ system (which can also be the target system),
6820 start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6829 . On the _target_ system, create a tracing session configured to
6830 send trace data over the network:
6835 $ lttng create my-session --set-url=net://remote-system
6839 Replace `remote-system` by the host name or IP address of the
6840 remote system. See man:lttng-create(1) for the exact URL format.
6842 . On the target system, use the man:lttng(1) command-line tool as usual.
6843 When tracing is active, the consumer daemon of the target sends
6844 sub-buffers to the relay daemon running on the remote system instead
6845 of flushing them to the local file system. The relay daemon writes the
6846 received packets to the local file system.
6848 The relay daemon writes trace files to
6849 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6850 +__hostname__+ is the host name of the target system and +__session__+
6851 is the tracing session name. Note that the env:LTTNG_HOME environment
6852 variable defaults to `$HOME` if not set. Use the
6853 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6854 trace files to another base directory.
6859 === View events as LTTng emits them (noch:{LTTng} live)
6861 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6862 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6863 display events as LTTng emits them on the target system while tracing is
6866 The relay daemon creates a _tee_: it forwards the trace data to both
6867 the local file system and to connected live viewers:
6870 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6875 . On the _target system_, create a <<tracing-session,tracing session>>
6881 $ lttng create my-session --live
6885 This spawns a local relay daemon.
6887 . Start the live viewer and configure it to connect to the relay
6888 daemon. For example, with
6889 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]:
6894 $ babeltrace2 net://localhost/host/hostname/my-session
6901 * `hostname` with the host name of the target system.
6902 * `my-session` with the name of the tracing session to view.
6905 . Configure the tracing session as usual with the man:lttng(1)
6906 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6908 List the available live tracing sessions with Babeltrace{nbsp}2:
6912 $ babeltrace2 net://localhost
6915 You can start the relay daemon on another system. In this case, you need
6916 to specify the URL of the relay daemon when you create the tracing
6917 session with the opt:lttng-create(1):--set-url option. You also need to
6918 replace `localhost` in the procedure above with the host name of the
6919 system on which the relay daemon is running.
6921 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6922 command-line options.
6926 [[taking-a-snapshot]]
6927 === Take a snapshot of the current sub-buffers of a tracing session
6929 The normal behavior of LTTng is to append full sub-buffers to growing
6930 trace data files. This is ideal to keep a full history of the events
6931 that occurred on the target system, but it can
6932 represent too much data in some situations. For example, you may wish
6933 to trace your application continuously until some critical situation
6934 happens, in which case you only need the latest few recorded
6935 events to perform the desired analysis, not multi-gigabyte trace files.
6937 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6938 current sub-buffers of a given <<tracing-session,tracing session>>.
6939 LTTng can write the snapshot to the local file system or send it over
6943 .A snapshot is a copy of the current sub-buffers, which aren't cleared after the operation.
6944 image::snapshot.png[]
6946 If you wish to create unmanaged, self-contained, non-overlapping
6947 trace chunk archives instead of a simple copy of the current
6948 sub-buffers, see the <<session-rotation,tracing session rotation>>
6949 feature (available since LTTng{nbsp}2.11).
6953 . Create a tracing session in _snapshot mode_:
6958 $ lttng create my-session --snapshot
6962 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
6963 <<channel,channels>> created in this mode is automatically set to
6964 _overwrite_ (flight recorder mode).
6966 . Configure the tracing session as usual with the man:lttng(1)
6967 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6969 . **Optional**: When you need to take a snapshot,
6970 <<basic-tracing-session-control,stop tracing>>.
6972 You can take a snapshot when the tracers are active, but if you stop
6973 them first, you're sure that the data in the sub-buffers doesn't
6974 change before you actually take the snapshot.
6981 $ lttng snapshot record --name=my-first-snapshot
6985 LTTng writes the current sub-buffers of all the channels of the
6986 <<cur-tracing-session,current tracing session>> to
6987 trace files on the local file system. Those trace files have
6988 `my-first-snapshot` in their name.
6990 There is no difference between the format of a normal trace file and the
6991 format of a snapshot: viewers of LTTng traces also support LTTng
6994 By default, LTTng writes snapshot files to the path shown by
6995 `lttng snapshot list-output`. You can change this path or decide to send
6996 snapshots over the network using either:
6998 . An output path or URL that you specify when you
6999 <<creating-destroying-tracing-sessions,create the tracing session>>.
7000 . A snapshot output path or URL that you add using
7001 `lttng snapshot add-output`.
7002 . An output path or URL that you provide directly to the
7003 `lttng snapshot record` command.
7005 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
7006 you specify a URL, a relay daemon must listen on a remote system (see
7007 <<sending-trace-data-over-the-network,Send trace data over the
7012 [[session-rotation]]
7013 === Archive the current trace chunk (rotate a tracing session)
7015 The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
7016 current sub-buffers of a tracing session to the file system or send them
7017 over the network. When you take a snapshot, LTTng doesn't clear the ring
7018 buffers of the tracing session: if you take another snapshot immediately
7019 after, both snapshots could contain overlapping trace data.
7021 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
7022 _tracing session rotation_ is a feature which appends the content of the
7023 ring buffers to what's already on the file system or sent over the
7024 network since the creation of the tracing session or since the last
7025 rotation, and then clears those ring buffers to avoid trace data
7028 What LTTng is about to write when performing a tracing session rotation
7029 is called the _current trace chunk_. When this current trace chunk is
7030 written to the file system or sent over the network, it becomes a _trace
7031 chunk archive_. Therefore, a tracing session rotation _archives_ the
7032 current trace chunk.
7035 .A tracing session rotation operation _archives_ the current trace chunk.
7036 image::rotation.png[]
7038 A trace chunk archive is a self-contained LTTng trace which LTTng
7039 doesn't manage anymore: you can read it, modify it, move it, or remove
7042 There are two methods to perform a tracing session rotation: immediately
7043 or with a rotation schedule.
7045 To perform an immediate tracing session rotation:
7047 . <<creating-destroying-tracing-sessions,Create a tracing session>>
7048 in _normal mode_ or _network streaming mode_
7049 (only those two creation modes support tracing session rotation):
7054 $ lttng create my-session
7058 . <<enabling-disabling-events,Create one or more event rules>>
7059 and <<basic-tracing-session-control,start tracing>>:
7064 $ lttng enable-event --kernel sched_'*'
7069 . When needed, immediately rotate the
7070 <<cur-tracing-session,current tracing session>>:
7079 The cmd:lttng-rotate command prints the path to the created trace
7080 chunk archive. See man:lttng-rotate(1) to learn about the format
7081 of trace chunk archive directory names.
7083 Perform other immediate rotations while the tracing session is
7084 active. It is guaranteed that all the trace chunk archives don't
7085 contain overlapping trace data. You can also perform an immediate
7086 rotation once you have <<basic-tracing-session-control,stopped>> the
7089 . When you're done tracing,
7090 <<creating-destroying-tracing-sessions,destroy the current tracing
7100 The tracing session destruction operation creates one last trace
7101 chunk archive from the current trace chunk.
7103 A tracing session rotation schedule is a planned rotation which LTTng
7104 performs automatically based on one of the following conditions:
7106 * A timer with a configured period times out.
7108 * The total size of the flushed part of the current trace chunk
7109 becomes greater than or equal to a configured value.
7111 To schedule a tracing session rotation, set a _rotation schedule_:
7113 . <<creating-destroying-tracing-sessions,Create a tracing session>>
7114 in _normal mode_ or _network streaming mode_
7115 (only those two creation modes support tracing session rotation):
7120 $ lttng create my-session
7124 . <<enabling-disabling-events,Create one or more event rules>>:
7129 $ lttng enable-event --kernel sched_'*'
7133 . Set a tracing session rotation schedule:
7138 $ lttng enable-rotation --timer=10s
7142 In this example, we set a rotation schedule so that LTTng performs a
7143 tracing session rotation every ten seconds.
7145 See man:lttng-enable-rotation(1) to learn more about other ways to set a
7148 . <<basic-tracing-session-control,Start tracing>>:
7157 LTTng performs tracing session rotations automatically while the tracing
7158 session is active thanks to the rotation schedule.
7160 . When you're done tracing,
7161 <<creating-destroying-tracing-sessions,destroy the current tracing
7171 The tracing session destruction operation creates one last trace chunk
7172 archive from the current trace chunk.
7174 Use man:lttng-disable-rotation(1) to unset a tracing session
7177 NOTE: man:lttng-rotate(1) and man:lttng-enable-rotation(1) list
7178 limitations regarding those two commands.
7183 === Use the machine interface
7185 With any command of the man:lttng(1) command-line tool, set the
7186 opt:lttng(1):--mi option to `xml` (before the command name) to get an
7187 XML machine interface output, for example:
7191 $ lttng --mi=xml enable-event --kernel --syscall open
7194 A schema definition (XSD) is
7195 https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/src/common/mi-lttng-4.0.xsd[available]
7196 to ease the integration with external tools as much as possible.
7200 [[metadata-regenerate]]
7201 === Regenerate the metadata of an LTTng trace
7203 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
7204 data stream files and a metadata file. This metadata file contains,
7205 amongst other things, information about the offset of the clock sources
7206 used to timestamp <<event,event records>> when tracing.
7208 If, once a <<tracing-session,tracing session>> is
7209 <<basic-tracing-session-control,started>>, a major
7210 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
7211 happens, the clock offset of the trace also needs to be updated. Use
7212 the `metadata` item of the man:lttng-regenerate(1) command to do so.
7214 The main use case of this command is to allow a system to boot with
7215 an incorrect wall time and trace it with LTTng before its wall time
7216 is corrected. Once the system is known to be in a state where its
7217 wall time is correct, it can run `lttng regenerate metadata`.
7219 To regenerate the metadata of an LTTng trace:
7221 * Use the `metadata` item of the man:lttng-regenerate(1) command:
7226 $ lttng regenerate metadata
7232 `lttng regenerate metadata` has the following limitations:
7234 * Tracing session <<creating-destroying-tracing-sessions,created>>
7236 * User space <<channel,channels>>, if any, are using
7237 <<channel-buffering-schemes,per-user buffering>>.
7242 [[regenerate-statedump]]
7243 === Regenerate the state dump of a tracing session
7245 The LTTng kernel and user space tracers generate state dump
7246 <<event,event records>> when the application starts or when you
7247 <<basic-tracing-session-control,start a tracing session>>. An analysis
7248 can use the state dump event records to set an initial state before it
7249 builds the rest of the state from the following event records.
7250 http://tracecompass.org/[Trace Compass] is a notable example of an
7251 application which uses the state dump of an LTTng trace.
7253 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7254 state dump event records aren't included in the snapshot because they
7255 were recorded to a sub-buffer that has been consumed or overwritten
7258 Use the `lttng regenerate statedump` command to emit the state
7259 dump event records again.
7261 To regenerate the state dump of the current tracing session, provided
7262 create it in snapshot mode, before you take a snapshot:
7264 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7269 $ lttng regenerate statedump
7273 . <<basic-tracing-session-control,Stop the tracing session>>:
7282 . <<taking-a-snapshot,Take a snapshot>>:
7287 $ lttng snapshot record --name=my-snapshot
7291 Depending on the event throughput, you should run steps 1 and 2
7292 as closely as possible.
7294 NOTE: To record the state dump events, you need to
7295 <<enabling-disabling-events,create event rules>> which enable them.
7296 LTTng-UST state dump tracepoints start with `lttng_ust_statedump:`.
7297 LTTng-modules state dump tracepoints start with `lttng_statedump_`.
7301 [[persistent-memory-file-systems]]
7302 === Record trace data on persistent memory file systems
7304 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
7305 (NVRAM) is random-access memory that retains its information when power
7306 is turned off (non-volatile). Systems with such memory can store data
7307 structures in RAM and retrieve them after a reboot, without flushing
7308 to typical _storage_.
7310 Linux supports NVRAM file systems thanks to either
7311 http://pramfs.sourceforge.net/[PRAMFS] or
7312 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
7313 (requires Linux{nbsp}4.1+).
7315 This section doesn't describe how to operate such file systems;
7316 we assume that you have a working persistent memory file system.
7318 When you create a <<tracing-session,tracing session>>, you can specify
7319 the path of the shared memory holding the sub-buffers. If you specify a
7320 location on an NVRAM file system, then you can retrieve the latest
7321 recorded trace data when the system reboots after a crash.
7323 To record trace data on a persistent memory file system and retrieve the
7324 trace data after a system crash:
7326 . Create a tracing session with a sub-buffer shared memory path located
7327 on an NVRAM file system:
7332 $ lttng create my-session --shm-path=/path/to/shm
7336 . Configure the tracing session as usual with the man:lttng(1)
7337 command-line tool, and <<basic-tracing-session-control,start tracing>>.
7339 . After a system crash, use the man:lttng-crash(1) command-line tool to
7340 view the trace data recorded on the NVRAM file system:
7345 $ lttng-crash /path/to/shm
7349 The binary layout of the ring buffer files isn't exactly the same as
7350 the trace files layout. This is why you need to use man:lttng-crash(1)
7351 instead of your preferred trace viewer directly.
7353 To convert the ring buffer files to LTTng trace files:
7355 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
7360 $ lttng-crash --extract=/path/to/trace /path/to/shm
7366 [[notif-trigger-api]]
7367 === Get notified when the buffer usage of a channel is too high or too low
7369 With the $$C/C++$$ notification and trigger API of LTTng, your user
7370 application can get notified when the buffer usage of one or more
7371 <<channel,channels>> becomes too low or too high. Use this API
7372 and enable or disable <<event,event rules>> during tracing to avoid
7373 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
7375 .Have a user application get notified when the buffer usage of an LTTng channel is too high.
7377 In this example, we create and build an application which gets notified
7378 when the buffer usage of a specific LTTng channel is higher than
7379 75{nbsp}%. We only print that it is the case in the example, but we
7380 could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
7381 disable event rules when this happens.
7383 . Create the C{nbsp}source file of application:
7391 #include <lttng/domain.h>
7392 #include <lttng/action/action.h>
7393 #include <lttng/action/notify.h>
7394 #include <lttng/condition/condition.h>
7395 #include <lttng/condition/buffer-usage.h>
7396 #include <lttng/condition/evaluation.h>
7397 #include <lttng/notification/channel.h>
7398 #include <lttng/notification/notification.h>
7399 #include <lttng/trigger/trigger.h>
7400 #include <lttng/endpoint.h>
7402 int main(int argc, char *argv[])
7404 int exit_status = 0;
7405 struct lttng_notification_channel *notification_channel;
7406 struct lttng_condition *condition;
7407 struct lttng_action *action;
7408 struct lttng_trigger *trigger;
7409 const char *tracing_session_name;
7410 const char *channel_name;
7413 tracing_session_name = argv[1];
7414 channel_name = argv[2];
7417 * Create a notification channel. A notification channel
7418 * connects the user application to the LTTng session daemon.
7419 * This notification channel can be used to listen to various
7420 * types of notifications.
7422 notification_channel = lttng_notification_channel_create(
7423 lttng_session_daemon_notification_endpoint);
7426 * Create a "high buffer usage" condition. In this case, the
7427 * condition is reached when the buffer usage is greater than or
7428 * equal to 75 %. We create the condition for a specific tracing
7429 * session name, channel name, and for the user space tracing
7432 * The "low buffer usage" condition type also exists.
7434 condition = lttng_condition_buffer_usage_high_create();
7435 lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
7436 lttng_condition_buffer_usage_set_session_name(
7437 condition, tracing_session_name);
7438 lttng_condition_buffer_usage_set_channel_name(condition,
7440 lttng_condition_buffer_usage_set_domain_type(condition,
7444 * Create an action (get a notification) to take when the
7445 * condition created above is reached.
7447 action = lttng_action_notify_create();
7450 * Create a trigger. A trigger associates a condition to an
7451 * action: the action is executed when the condition is reached.
7453 trigger = lttng_trigger_create(condition, action);
7455 /* Register the trigger to LTTng. */
7456 lttng_register_trigger(trigger);
7459 * Now that we have registered a trigger, a notification will be
7460 * emitted everytime its condition is met. To receive this
7461 * notification, we must subscribe to notifications that match
7462 * the same condition.
7464 lttng_notification_channel_subscribe(notification_channel,
7468 * Notification loop. Put this in a dedicated thread to avoid
7469 * blocking the main thread.
7472 struct lttng_notification *notification;
7473 enum lttng_notification_channel_status status;
7474 const struct lttng_evaluation *notification_evaluation;
7475 const struct lttng_condition *notification_condition;
7476 double buffer_usage;
7478 /* Receive the next notification. */
7479 status = lttng_notification_channel_get_next_notification(
7480 notification_channel, ¬ification);
7483 case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7485 case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7487 * The session daemon can drop notifications if a monitoring
7488 * application isn't consuming the notifications fast
7492 case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7494 * The notification channel has been closed by the
7495 * session daemon. This is typically caused by a session
7496 * daemon shutting down.
7500 /* Unhandled conditions or errors. */
7506 * A notification provides, amongst other things:
7508 * * The condition that caused this notification to be
7510 * * The condition evaluation, which provides more
7511 * specific information on the evaluation of the
7514 * The condition evaluation provides the buffer usage
7515 * value at the moment the condition was reached.
7517 notification_condition = lttng_notification_get_condition(
7519 notification_evaluation = lttng_notification_get_evaluation(
7522 /* We're subscribed to only one condition. */
7523 assert(lttng_condition_get_type(notification_condition) ==
7524 LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
7527 * Get the exact sampled buffer usage from the
7528 * condition evaluation.
7530 lttng_evaluation_buffer_usage_get_usage_ratio(
7531 notification_evaluation, &buffer_usage);
7534 * At this point, instead of printing a message, we
7535 * could do something to reduce the buffer usage of the channel,
7536 * like disable specific events.
7538 printf("Buffer usage is %f %% in tracing session \"%s\", "
7539 "user space channel \"%s\".\n", buffer_usage * 100,
7540 tracing_session_name, channel_name);
7541 lttng_notification_destroy(notification);
7545 lttng_action_destroy(action);
7546 lttng_condition_destroy(condition);
7547 lttng_trigger_destroy(trigger);
7548 lttng_notification_channel_destroy(notification_channel);
7554 . Build the `notif-app` application, linking it to `liblttng-ctl`:
7559 $ gcc -o notif-app notif-app.c -llttng-ctl
7563 . <<creating-destroying-tracing-sessions,Create a tracing session>>,
7564 <<enabling-disabling-events,create an event rule>> matching all the
7565 user space tracepoints, and
7566 <<basic-tracing-session-control,start tracing>>:
7571 $ lttng create my-session
7572 $ lttng enable-event --userspace --all
7577 If you create the channel manually with the man:lttng-enable-channel(1)
7578 command, control how frequently LTTng samples the current values of the
7579 channel properties to evaluate user conditions with the
7580 opt:lttng-enable-channel(1):--monitor-timer option.
7582 . Run the `notif-app` application. This program accepts the
7583 <<tracing-session,tracing session>> name and the user space channel
7584 name as its two first arguments. The channel which LTTng automatically
7585 creates with the man:lttng-enable-event(1) command above is named
7591 $ ./notif-app my-session channel0
7595 . In another terminal, run an application with a very high event
7596 throughput so that the 75{nbsp}% buffer usage condition is reached.
7598 In the first terminal, the application should print lines like this:
7601 Buffer usage is 81.45197 % in tracing session "my-session", user space
7605 If you don't see anything, try modifying the condition in
7606 path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
7607 (step{nbsp}2) and running it again (step{nbsp}4).
7614 [[lttng-modules-ref]]
7615 === noch:{LTTng-modules}
7619 [[lttng-tracepoint-enum]]
7620 ==== `LTTNG_TRACEPOINT_ENUM()` usage
7622 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
7626 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
7631 * `name` with the name of the enumeration (C identifier, unique
7632 amongst all the defined enumerations).
7633 * `entries` with a list of enumeration entries.
7635 The available enumeration entry macros are:
7637 +ctf_enum_value(__name__, __value__)+::
7638 Entry named +__name__+ mapped to the integral value +__value__+.
7640 +ctf_enum_range(__name__, __begin__, __end__)+::
7641 Entry named +__name__+ mapped to the range of integral values between
7642 +__begin__+ (included) and +__end__+ (included).
7644 +ctf_enum_auto(__name__)+::
7645 Entry named +__name__+ mapped to the integral value following the
7648 The last value of a `ctf_enum_value()` entry is its +__value__+
7651 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
7653 If `ctf_enum_auto()` is the first entry in the list, its integral
7656 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
7657 to use a defined enumeration as a tracepoint field.
7659 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
7663 LTTNG_TRACEPOINT_ENUM(
7666 ctf_enum_auto("AUTO: EXPECT 0")
7667 ctf_enum_value("VALUE: 23", 23)
7668 ctf_enum_value("VALUE: 27", 27)
7669 ctf_enum_auto("AUTO: EXPECT 28")
7670 ctf_enum_range("RANGE: 101 TO 303", 101, 303)
7671 ctf_enum_auto("AUTO: EXPECT 304")
7679 [[lttng-modules-tp-fields]]
7680 ==== Tracepoint fields macros (for `TP_FIELDS()`)
7682 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
7683 tracepoint fields, which must be listed within `TP_FIELDS()` in
7684 `LTTNG_TRACEPOINT_EVENT()`, are:
7686 [role="func-desc growable",cols="asciidoc,asciidoc"]
7687 .Available macros to define LTTng-modules tracepoint fields
7689 |Macro |Description and parameters
7692 +ctf_integer(__t__, __n__, __e__)+
7694 +ctf_integer_nowrite(__t__, __n__, __e__)+
7696 +ctf_user_integer(__t__, __n__, __e__)+
7698 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
7700 Standard integer, displayed in base{nbsp}10.
7703 Integer C type (`int`, `long`, `size_t`, ...).
7709 Argument expression.
7712 +ctf_integer_hex(__t__, __n__, __e__)+
7714 +ctf_user_integer_hex(__t__, __n__, __e__)+
7716 Standard integer, displayed in base{nbsp}16.
7725 Argument expression.
7727 |+ctf_integer_oct(__t__, __n__, __e__)+
7729 Standard integer, displayed in base{nbsp}8.
7738 Argument expression.
7741 +ctf_integer_network(__t__, __n__, __e__)+
7743 +ctf_user_integer_network(__t__, __n__, __e__)+
7745 Integer in network byte order (big-endian), displayed in base{nbsp}10.
7754 Argument expression.
7757 +ctf_integer_network_hex(__t__, __n__, __e__)+
7759 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
7761 Integer in network byte order, displayed in base{nbsp}16.
7770 Argument expression.
7773 +ctf_enum(__N__, __t__, __n__, __e__)+
7775 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
7777 +ctf_user_enum(__N__, __t__, __n__, __e__)+
7779 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
7784 Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
7787 Integer C type (`int`, `long`, `size_t`, ...).
7793 Argument expression.
7796 +ctf_string(__n__, __e__)+
7798 +ctf_string_nowrite(__n__, __e__)+
7800 +ctf_user_string(__n__, __e__)+
7802 +ctf_user_string_nowrite(__n__, __e__)+
7804 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
7810 Argument expression.
7813 +ctf_array(__t__, __n__, __e__, __s__)+
7815 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
7817 +ctf_user_array(__t__, __n__, __e__, __s__)+
7819 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
7821 Statically-sized array of integers.
7824 Array element C type.
7830 Argument expression.
7836 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
7838 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7840 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
7842 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7844 Statically-sized array of bits.
7846 The type of +__e__+ must be an integer type. +__s__+ is the number
7847 of elements of such type in +__e__+, not the number of bits.
7850 Array element C type.
7856 Argument expression.
7862 +ctf_array_text(__t__, __n__, __e__, __s__)+
7864 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
7866 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
7868 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
7870 Statically-sized array, printed as text.
7872 The string doesn't need to be null-terminated.
7875 Array element C type (always `char`).
7881 Argument expression.
7887 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
7889 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7891 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
7893 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7895 Dynamically-sized array of integers.
7897 The type of +__E__+ must be unsigned.
7900 Array element C type.
7906 Argument expression.
7909 Length expression C type.
7915 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7917 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7919 Dynamically-sized array of integers, displayed in base{nbsp}16.
7921 The type of +__E__+ must be unsigned.
7924 Array element C type.
7930 Argument expression.
7933 Length expression C type.
7938 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
7940 Dynamically-sized array of integers in network byte order (big-endian),
7941 displayed in base{nbsp}10.
7943 The type of +__E__+ must be unsigned.
7946 Array element C type.
7952 Argument expression.
7955 Length expression C type.
7961 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7963 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7965 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7967 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7969 Dynamically-sized array of bits.
7971 The type of +__e__+ must be an integer type. +__s__+ is the number
7972 of elements of such type in +__e__+, not the number of bits.
7974 The type of +__E__+ must be unsigned.
7977 Array element C type.
7983 Argument expression.
7986 Length expression C type.
7992 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7994 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7996 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7998 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8000 Dynamically-sized array, displayed as text.
8002 The string doesn't need to be null-terminated.
8004 The type of +__E__+ must be unsigned.
8006 The behaviour is undefined if +__e__+ is `NULL`.
8009 Sequence element C type (always `char`).
8015 Argument expression.
8018 Length expression C type.
8024 Use the `_user` versions when the argument expression, `e`, is
8025 a user space address. In the cases of `ctf_user_integer*()` and
8026 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
8029 The `_nowrite` versions omit themselves from the session trace, but are
8030 otherwise identical. This means the `_nowrite` fields won't be written
8031 in the recorded trace. Their primary purpose is to make some
8032 of the event context available to the
8033 <<enabling-disabling-events,event filters>> without having to
8034 commit the data to sub-buffers.
8040 Terms related to LTTng and to tracing in general:
8043 The http://diamon.org/babeltrace[Babeltrace] project, which includes:
8046 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
8047 command-line interface.
8048 * The libbabeltrace2 library which offers a
8049 https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
8050 * https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
8053 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
8054 A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
8056 [[def-channel]]<<channel,channel>>::
8057 An entity which is responsible for a set of
8058 <<def-ring-buffer,ring buffers>>.
8060 <<def-event-rule,Event rules>> are always attached to a specific
8064 A source of time for a <<def-tracer,tracer>>.
8066 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
8067 A process which is responsible for consuming the full
8068 <<def-sub-buffer,sub-buffers>> and write them to a file system or
8069 send them over the network.
8071 [[def-current-trace-chunk]]current trace chunk::
8072 A <<def-trace-chunk,trace chunk>> which includes the current content
8073 of all the <<def-sub-buffer,sub-buffers>> of the
8074 <<def-tracing-session-rotation,tracing session>> and the stream files
8075 produced since the latest event amongst:
8077 * The creation of the <<def-tracing-session,tracing session>>.
8078 * The last tracing session rotation, if any.
8080 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
8081 The <<def-event-record-loss-mode,event record loss mode>> in which
8082 the <<def-tracer,tracer>> _discards_ new event records when there's no
8083 <<def-sub-buffer,sub-buffer>> space left to store them.
8085 [[def-event]]event::
8086 The consequence of the execution of an
8087 <<def-instrumentation-point,instrumentation point>>, like a
8088 <<def-tracepoint,tracepoint>> that you manually place in some source
8089 code, or a Linux kernel kprobe.
8091 An event is said to _occur_ at a specific time. <<def-lttng,LTTng>> can
8092 take various actions upon the occurrence of an event, like record its
8093 payload to a <<def-sub-buffer,sub-buffer>>.
8095 [[def-event-name]]event name::
8096 The name of an <<def-event,event>>, which is also the name of the
8097 <<def-event-record,event record>>.
8099 This is also called the _instrumentation point name_.
8101 [[def-event-record]]event record::
8102 A record, in a <<def-trace,trace>>, of the payload of an
8103 <<def-event,event>> which occured.
8105 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
8106 The mechanism by which event records of a given
8107 <<def-channel,channel>> are lost (not recorded) when there is no
8108 <<def-sub-buffer,sub-buffer>> space left to store them.
8110 [[def-event-rule]]<<event,event rule>>::
8111 Set of conditions which must be satisfied for one or more occuring
8112 <<def-event,events>> to be recorded.
8114 [[def-incl-set]]inclusion set::
8115 In the <<pid-tracking,process attribute tracking>> context: a
8116 set of <<def-proc-attr,process attributes>> of a given type.
8118 <<instrumenting,instrumentation>>::
8119 The use of <<def-lttng,LTTng>> probes to make a piece of software
8122 [[def-instrumentation-point]]instrumentation point::
8123 A point in the execution path of a piece of software that, when
8124 reached by this execution, can emit an <<def-event,event>>.
8126 instrumentation point name::
8127 See _<<def-event-name,event name>>_.
8129 `java.util.logging`::
8131 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
8132 of the Java platform.
8135 A http://logging.apache.org/log4j/1.2/[logging library] for Java
8136 developed by the Apache Software Foundation.
8139 Level of severity of a log statement or user space
8140 <<def-instrumentation-point,instrumentation point>>.
8142 [[def-lttng]]LTTng::
8143 The _Linux Trace Toolkit: next generation_ project.
8145 <<lttng-cli,cmd:lttng>>::
8146 A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
8147 project which you can use to send and receive control messages to and
8148 from a <<def-session-daemon,session daemon>>.
8151 The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
8152 which is a set of analyzing programs that you can use to obtain a
8153 higher level view of an <<def-lttng,LTTng>> <<def-trace,trace>>.
8155 cmd:lttng-consumerd::
8156 The name of the <<def-consumer-daemon,consumer daemon>> program.
8159 A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8160 which can convert <<def-ring-buffer,ring buffer>> files (usually
8161 <<persistent-memory-file-systems,saved on a persistent memory file
8162 system>>) to <<def-trace,trace>> files.
8164 See man:lttng-crash(1).
8166 LTTng Documentation::
8169 <<lttng-live,LTTng live>>::
8170 A communication protocol between the <<lttng-relayd,relay daemon>> and
8171 live viewers which makes it possible to see <<def-event-record,event
8172 records>> ``live'', as they are received by the
8173 <<def-relay-daemon,relay daemon>>.
8175 <<lttng-modules,LTTng-modules>>::
8176 The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8177 which contains the Linux kernel modules to make the Linux kernel
8178 <<def-instrumentation-point,instrumentation points>> available for
8179 <<def-lttng,LTTng>> tracing.
8182 The name of the <<def-relay-daemon,relay daemon>> program.
8184 cmd:lttng-sessiond::
8185 The name of the <<def-session-daemon,session daemon>> program.
8187 [[def-lttng-tools]]LTTng-tools::
8188 The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8189 contains the various programs and libraries used to
8190 <<controlling-tracing,control tracing>>.
8192 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8193 The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8194 contains libraries to instrument
8195 <<def-user-application,user applications>>.
8197 <<lttng-ust-agents,LTTng-UST Java agent>>::
8198 A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8199 allow the LTTng instrumentation of `java.util.logging` and Apache
8200 log4j{nbsp}1.2 logging statements.
8202 <<lttng-ust-agents,LTTng-UST Python agent>>::
8203 A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8204 to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8207 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8208 The <<def-event-record-loss-mode,event record loss mode>> in which new
8209 <<def-event-record,event records>> _overwrite_ older event records
8210 when there's no <<def-sub-buffer,sub-buffer>> space left to store
8213 <<channel-buffering-schemes,per-process buffering>>::
8214 A <<def-buffering-scheme,buffering scheme>> in which each instrumented
8215 process has its own <<def-sub-buffer,sub-buffers>> for a given user
8216 space <<def-channel,channel>>.
8218 <<channel-buffering-schemes,per-user buffering>>::
8219 A <<def-buffering-scheme,buffering scheme>> in which all the processes
8220 of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
8221 given user space <<def-channel,channel>>.
8223 [[def-proc-attr]]process attribute::
8224 In the <<pid-tracking,process attribute tracking>> context:
8227 * A virtual process ID.
8229 * A virtual Unix user ID.
8231 * A virtual Unix group ID.
8233 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
8234 A process which is responsible for receiving the <<def-trace,trace>>
8235 data which a distant <<def-consumer-daemon,consumer daemon>> sends.
8237 [[def-ring-buffer]]ring buffer::
8238 A set of <<def-sub-buffer,sub-buffers>>.
8241 See _<<def-tracing-session-rotation,tracing session rotation>>_.
8243 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
8244 A process which receives control commands from you and orchestrates
8245 the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
8247 <<taking-a-snapshot,snapshot>>::
8248 A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
8249 of a given <<def-tracing-session,tracing session>>, saved as
8250 <<def-trace,trace>> files.
8252 [[def-sub-buffer]]sub-buffer::
8253 One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
8254 which contains <<def-event-record,event records>>.
8257 The time information attached to an <<def-event,event>> when it is
8260 [[def-trace]]trace (_noun_)::
8263 * One http://diamon.org/ctf/[CTF] metadata stream file.
8264 * One or more CTF data stream files which are the concatenations of one
8265 or more flushed <<def-sub-buffer,sub-buffers>>.
8267 [[def-trace-verb]]trace (_verb_)::
8268 The action of recording the <<def-event,events>> emitted by an
8269 application or by a system, or to initiate such recording by
8270 controlling a <<def-tracer,tracer>>.
8272 [[def-trace-chunk]]trace chunk::
8273 A self-contained <<def-trace,trace>> which is part of a
8274 <<def-tracing-session,tracing session>>. Each
8275 <<def-tracing-session-rotation, tracing session rotation>> produces a
8276 <<def-trace-chunk-archive,trace chunk archive>>.
8278 [[def-trace-chunk-archive]]trace chunk archive::
8279 The result of a <<def-tracing-session-rotation, tracing session rotation>>.
8281 <<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
8282 containing <<def-tracing-session,tracing session>> is still active: you
8283 are free to read it, modify it, move it, or remove it.
8286 The http://tracecompass.org[Trace Compass] project and application.
8288 [[def-tracepoint]]tracepoint::
8289 An instrumentation point using the tracepoint mechanism of the Linux
8290 kernel or of <<def-lttng-ust,LTTng-UST>>.
8292 tracepoint definition::
8293 The definition of a single <<def-tracepoint,tracepoint>>.
8296 The name of a <<def-tracepoint,tracepoint>>.
8298 [[def-tracepoint-provider]]tracepoint provider::
8299 A set of functions providing <<def-tracepoint,tracepoints>> to an
8300 instrumented <<def-user-application,user application>>.
8302 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
8303 provider package>>: many tracepoint providers can exist within a
8304 tracepoint provider package.
8306 [[def-tracepoint-provider-package]]tracepoint provider package::
8307 One or more <<def-tracepoint-provider,tracepoint providers>> compiled
8308 as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
8309 link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
8312 [[def-tracer]]tracer::
8313 A software which records emitted <<def-event,events>>.
8315 <<domain,tracing domain>>::
8316 A namespace for <<def-event,event>> sources.
8318 <<tracing-group,tracing group>>::
8319 The Unix group in which a Unix user can be to be allowed to
8320 <<def-trace-verb,trace>> the Linux kernel.
8322 [[def-tracing-session]]<<tracing-session,tracing session>>::
8323 A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
8325 [[def-tracing-session-rotation]]<<session-rotation,tracing session rotation>>::
8326 The action of archiving the
8327 <<def-current-trace-chunk,current trace chunk>> of a
8328 <<def-tracing-session,tracing session>>.
8330 tracked <<def-proc-attr,process attribute>>::
8331 A process attribute which is part of an <<def-incl-set,inclusion
8334 untracked process attribute::
8335 A process attribute which isn't part of an <<def-incl-set,inclusion
8338 [[def-user-application]]user application::
8339 An application running in user space, as opposed to a Linux kernel
8340 module, for example.