1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
7 include::../common/copyright.txt[]
10 include::../common/warning-not-maintained.txt[]
13 include::../common/welcome.txt[]
16 include::../common/audience.txt[]
20 === Chapter descriptions
22 What follows is a list of brief descriptions of this documentation's
23 chapters. The latter are ordered in such a way as to make the reading
24 as linear as possible.
26 . <<nuts-and-bolts,Nuts and bolts>> explains the
27 rudiments of software tracing and the rationale behind the
29 . <<installing-lttng,Installing LTTng>> is divided into
30 sections describing the steps needed to get a working installation
31 of LTTng packages for common Linux distributions and from its
33 . <<getting-started,Getting started>> is a very concise guide to
34 get started quickly with LTTng kernel and user space tracing. This
35 chapter is recommended if you're new to LTTng or software tracing
37 . <<understanding-lttng,Understanding LTTng>> deals with some
38 core concepts and components of the LTTng suite. Understanding
39 those is important since the next chapter assumes you're familiar
41 . <<using-lttng,Using LTTng>> is a complete user guide of the
42 LTTng project. It shows in great details how to instrument user
43 applications and the Linux kernel, how to control tracing sessions
44 using the `lttng` command line tool and miscellaneous practical use
46 . <<reference,Reference>> contains references of LTTng components,
47 like links to online manpages and various APIs.
49 We recommend that you read the above chapters in this order, although
50 some of them may be skipped depending on your situation. You may skip
51 <<nuts-and-bolts,Nuts and bolts>> if you're familiar with tracing
52 and LTTng. Also, you may jump over <<installing-lttng,Installing LTTng>>
53 if LTTng is already properly installed on your target system.
56 include::../common/convention.txt[]
59 include::../common/acknowledgements.txt[]
63 == What's new in LTTng {revision}?
65 The **LTTng {revision}** toolchain introduces many interesting features,
66 some of them which have been requested by users many times.
69 <<saving-loading-tracing-session,save and restore tracing sessions>>.
70 Sessions are saved to and loaded from XML files located by default in a
71 subdirectory of the user's home directory. LTTng daemons are also
72 configurable by configuration files as of LTTng-tools {revision}. This version
73 also makes it possible to load user-defined kernel probes with the new
74 session daemon's `--kmod-probes` option (or using the
75 `LTTNG_KMOD_PROBES` environment variable).
77 <<tracef,`tracef()`>> is a new instrumentation facility in LTTng-UST {revision}
78 which makes it possible to insert `printf()`-like tracepoints in C/$$C++$$
79 code for quick debugging. LTTng-UST {revision} also adds support for perf PMU
80 counters in user space on the x86 architecture
81 (see <<adding-context,Adding some context to channels>>).
83 As of LTTng-modules {revision}, a new
84 <<proc-lttng-logger-abi,LTTng logger ABI>>
85 is made available, making tracing Bash scripts, for example, much more
86 easier (just `echo` whatever you need to record to path:{/proc/lttng-logger}
87 while tracing is active). On the kernel side, some tracepoints are
88 added: state dumps of block devices, file descriptors, and file modes,
89 as well as http://en.wikipedia.org/wiki/Video4Linux[V4L2] events. Linux
90 3.15 is now officially supported, and system call tracing is now
91 possible on the MIPS32 architecture.
93 To learn more about the new features of LTTng {revision}, see
94 http://lttng.org/blog/2014/08/04/lttng-toolchain-2-5-0-is-out/[this
95 release announcement].
101 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
102 generation_ is a modern toolkit for tracing Linux systems and
103 applications. So your first question might rather be: **what is
106 As the history of software engineering progressed and led to what
107 we now take for granted--complex, numerous and
108 interdependent software applications running in parallel on
109 sophisticated operating systems like Linux--the authors of such
110 components, or software developers, began feeling a natural
111 urge of having tools to ensure the robustness and good performance
112 of their masterpieces.
114 One major achievement in this field is, inarguably, the
115 https://www.gnu.org/software/gdb/[GNU debugger (GDB)], which is an
116 essential tool for developers to find and fix bugs. But even the best
117 debugger won't help make your software run faster, and nowadays, faster
118 software means either more work done by the same hardware, or cheaper
119 hardware for the same work.
121 A _profiler_ is often the tool of choice to identify performance
122 bottlenecks. Profiling is suitable to identify _where_ performance is
123 lost in a given software; the profiler outputs a profile, a statistical
124 summary of observed events, which you may use to know which functions
125 took the most time to execute. However, a profiler won't report _why_
126 some identified functions are the bottleneck. Also, bottlenecks might
127 only occur when specific conditions are met. For a thorough
128 investigation of software performance issues, a history of execution,
129 with historical values of chosen variables, is essential. This is where
130 tracing comes in handy.
132 _Tracing_ is a technique used to understand what goes on in a running
133 software system. The software used for tracing is called a _tracer_,
134 which is conceptually similar to a tape recorder. When recording,
135 specific points placed in the software source code generate events that
136 are saved on a giant tape: a _trace_ file. Both user applications and
137 the operating system may be traced at the same time, opening the
138 possibility of resolving a wide range of problems that are otherwise
139 extremely challenging.
141 Tracing is often compared to _logging_. However, tracers and loggers are
142 two different types of tools, serving two different purposes. Tracers
143 are designed to record much lower-level events that occur much more
144 frequently than log messages, often in the thousands per second range,
145 with very little execution overhead. Logging is more appropriate for
146 very high-level analysis of less frequent events: user accesses,
147 exceptional conditions (e.g., errors, warnings), database transactions,
148 instant messaging communications, etc. More formally, logging is one of
149 several use cases that can be accomplished with tracing.
151 The list of recorded events inside a trace file may be read manually
152 like a log file for the maximum level of detail, but it is generally
153 much more interesting to perform application-specific analyses to
154 produce reduced statistics and graphs that are useful to resolve a given
155 problem. Trace viewers and analysers are specialized tools which achieve
158 So, in the end, this is what LTTng is: a powerful, open source set of
159 tools to trace the Linux kernel and user applications. LTTng is composed
160 of several components actively maintained and developed by its
161 http://lttng.org/community/#where[community].
163 Excluding proprietary solutions, a few competing software tracers exist
165 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace] is the
166 de facto function tracer of the Linux kernel.
167 http://linux.die.net/man/1/strace[strace] is able to record all system
168 calls made by a user process.
169 https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and user
170 space tracer which uses custom user scripts to produce plain text
171 traces. http://www.sysdig.org/[sysdig] also uses scripts, written in
172 Lua, to trace and analyze the Linux kernel.
174 The main distinctive features of LTTng is that it produces correlated
175 kernel and user space traces, as well as doing so with the lowest
176 overhead amongst other solutions. It produces trace files in the
177 http://www.efficios.com/ctf[CTF] format, an optimized file format for
178 production and analyses of multi-gigabyte data. LTTng is the result of
179 close to 10 years of active development by a community of passionate
180 developers. It is currently available on some major desktop, server, and
181 embedded Linux distributions.
183 The main interface for tracing control is a single command line tool
184 named `lttng`. The latter can create several tracing sessions,
185 enable/disable events on the fly, filter them efficiently with custom
186 user expressions, start/stop tracing and do much more. Traces can be
187 recorded on disk or sent over the network, kept totally or partially,
188 and viewed once tracing is inactive or in real-time.
190 <<installing-lttng,Install LTTng now>> and start tracing!
196 include::../common/warning-installation-outdated.txt[]
198 **LTTng** is a set of software components which interact to allow
199 instrumenting the Linux kernel and user applications and controlling
200 tracing sessions (starting/stopping tracing, enabling/disabling events,
201 etc.). Those components are bundled into the following packages:
204 Libraries and command line interface to control tracing sessions.
207 Linux kernel modules allowing Linux to be traced using LTTng.
210 User space tracing library.
212 Most distributions mark the LTTng-modules and LTTng-UST packages as
213 optional. In the following sections, we always provide the steps to
214 install all three, but be aware that LTTng-modules is only required if
215 you intend to trace the Linux kernel and LTTng-UST is only required if
216 you intend to trace user space applications.
218 This chapter shows how to install the above packages on a Linux system.
219 The easiest way is to use the package manager of the system's
220 distribution (<<desktop-distributions,desktop>> or
221 <<embedded-distributions,embedded>>). Support is also available for
222 <<enterprise-distributions,enterprise distributions>>, such as Red Hat
223 Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES).
225 <<building-from-source,build the LTTng packages from source>>.
228 [[desktop-distributions]]
229 === Desktop distributions
231 Official LTTng {revision} packages are available for <<ubuntu,Ubuntu>> and
234 More recent versions of LTTng are available for Fedora, openSUSE,
235 as well as Arch Linux.
237 Should any issue arise when following the procedures below, please
238 inform the http://lttng.org/community[community] about it.
244 LTTng {revision} is packaged in Ubuntu 15.04 _Vivid Vervet_. For other
245 releases of Ubuntu, you need to build and install LTTng
246 <<building-from-source,from source>>. Ubuntu 15.10 _Wily Werewolf_
247 ships with link:/docs/v2.6/[LTTng 2.6].
249 To install LTTng {revision} from the official Ubuntu repositories,
250 simply use `apt-get`:
254 sudo apt-get install lttng-tools
255 sudo apt-get install lttng-modules-dkms
256 sudo apt-get install liblttng-ust-dev
263 Debian "jessie" has official packages of LTTng {revision}:
267 sudo apt-get install lttng-tools
268 sudo apt-get install lttng-modules-dkms
269 sudo apt-get install liblttng-ust-dev
273 [[embedded-distributions]]
274 === Embedded distributions
276 Some developers may be interested in tracing the Linux kernel and user space
277 applications running on embedded systems. LTTng is packaged by two popular
278 embedded Linux distributions: <<buildroot,Buildroot>> and
279 <<oe-yocto,OpenEmbedded/Yocto>>.
285 LTTng {revision} packages in Buildroot 2014.11 and 2015.02 are named
286 `lttng-tools`, `lttng-modules`, and `lttng-libust`.
288 To enable them, start the Buildroot configuration menu as usual:
297 * _Kernel_: make sure _Linux kernel_ is enabled
298 * _Toolchain_: make sure the following options are enabled:
299 ** _Enable large file (files > 2GB) support_
300 ** _Enable WCHAR support_
302 In _Target packages_/_Debugging, profiling and benchmark_, enable
303 _lttng-modules_ and _lttng-tools_. In
304 _Target packages_/_Libraries_/_Other_, enable _lttng-libust_.
308 ==== OpenEmbedded/Yocto
310 LTTng {revision} recipes are available in the `openembedded-core` layer of
311 OpenEmbedded from August 15th, 2014 to February 8th, 2015 under the
318 Using BitBake, the simplest way to include LTTng recipes in your
319 target image is to add them to `IMAGE_INSTALL_append` in
320 path:{conf/local.conf}:
323 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
326 If you're using Hob, click _Edit image recipe_ once you have selected
327 a machine and an image recipe. Then, in the _All recipes_ tab, search
328 for `lttng` and you should find and be able to include the three LTTng
332 [[enterprise-distributions]]
333 === Enterprise distributions (RHEL, SLES)
335 To install LTTng on enterprise Linux distributions
336 (such as RHEL and SLES), please see
337 http://packages.efficios.com/[EfficiOS Enterprise Packages].
340 [[building-from-source]]
341 === Building from source
343 As <<installing-lttng,previously stated>>, LTTng is shipped as three
344 packages: LTTng-tools, LTTng-modules and LTTng-UST. LTTng-tools contains
345 everything needed to control tracing sessions, while LTTng-modules is
346 only needed for Linux kernel tracing and LTTng-UST is only needed for
349 The tarballs are available in the
350 http://lttng.org/download#build-from-source[Download section]
351 of the LTTng website.
353 Please refer to the path:{README.md} files provided by each package to
354 properly build and install them.
356 TIP: The aforementioned path:{README.md} files are rendered as
357 rich text when https://github.com/lttng[viewed on GitHub].
361 == Getting started with LTTng
363 This is a small guide to get started quickly with LTTng kernel and user
364 space tracing. For intermediate to advanced use cases and a more
365 thorough understanding of LTTng, see <<using-lttng,Using LTTng>> and
366 <<understanding-lttng,Understanding LTTng>>.
368 Before reading this guide, make sure LTTng
369 <<installing-lttng,is installed>>. You will at least need LTTng-tools.
370 Also install LTTng-modules for
371 <<tracing-the-linux-kernel,tracing the Linux kernel>>
372 and LTTng-UST for <<tracing-your-own-user-application,tracing your own
373 user space applications>>. When your traces are finally written and
375 <<viewing-and-analyzing-your-traces,Viewing and analyzing your traces>>
376 section of this chapter will help you analyze your tracepoint
377 events to investigate.
380 [[tracing-the-linux-kernel]]
381 === Tracing the Linux kernel
383 Make sure LTTng-tools and LTTng-modules packages
384 <<installing-lttng,are installed>>.
386 Since you're about to trace the Linux kernel itself, let's look at the
387 available kernel events using the `lttng` tool, which has a
388 Git-like command line structure:
395 Before tracing, you need to create a session:
399 sudo lttng create my-session
402 TIP: You can avoid using `sudo` in the previous and following commands
403 if your user is a member of the <<lttng-sessiond,tracing group>>.
405 `my-session` is the tracing session name and could be anything you
406 like. `auto` will be used if omitted.
408 Let's now enable some events for this session:
412 sudo lttng enable-event --kernel sched_switch,sched_process_fork
415 or you might want to simply enable all available kernel events (beware
416 that trace files will grow rapidly when doing this):
420 sudo lttng enable-event --kernel --all
430 By default, traces are saved in
431 +\~/lttng-traces/__name__-__date__-__time__+,
432 where +__name__+ is the session name.
434 When you're done tracing:
442 Although `destroy` looks scary here, it doesn't actually destroy the
443 outputted trace files: it only destroys the tracing session.
445 What's next? Have a look at
446 <<viewing-and-analyzing-your-traces,Viewing and analyzing your traces>>
447 to view and analyze the trace you just recorded.
450 [[tracing-your-own-user-application]]
451 === Tracing your own user application
453 The previous section helped you create a trace out of Linux kernel
454 events. This section steps you through a simple example showing you how
455 to trace a _Hello world_ program written in C.
457 Make sure LTTng-tools and LTTng-UST packages
458 <<installing-lttng,are installed>>.
460 Tracing is just like having `printf()` calls at specific locations of
461 your source code, albeit LTTng is much faster and more flexible than
462 `printf()`. In the LTTng realm, **`tracepoint()`** is analogous to
465 Unlike `printf()`, though, `tracepoint()` does not use a format string to
466 know the types of its arguments: the formats of all tracepoints must be
467 defined before using them. So before even writing our _Hello world_ program,
468 we need to define the format of our tracepoint. This is done by writing a
469 **template file**, with a name usually ending
470 with the `.tp` extension (for **t**race**p**oint),
471 which the `lttng-gen-tp` tool (shipped with LTTng-UST) will use to generate
472 an object file (along with a `.c` file) and a header to be
473 included in our application source code.
475 Here's the whole flow:
478 .Build workflow for LTTng application tracing.
479 image::lttng-lttng-gen-tp.png[]
481 The template file format is a list of tracepoint definitions
482 and other optional definition entries which we will skip for
483 this quickstart. Each tracepoint is defined using the
484 `TRACEPOINT_EVENT()` macro. For each tracepoint, you must provide:
486 * a **provider name**, which is the "scope" of this tracepoint (this usually
487 includes the company and project names)
488 * a **tracepoint name**
489 * a **list of arguments** for the eventual `tracepoint()` call,
491 ** the argument C type
493 * a **list of fields**, which will be the actual fields of the recorded events
496 Here's a simple tracepoint definition example with two arguments: an integer
509 ctf_string(my_string_field, my_string_arg)
510 ctf_integer(int, my_integer_field, my_integer_arg)
515 The exact syntax is well explained in the
516 <<c-application,C application>> instrumenting guide of the
517 <<using-lttng,Using LTTng>> chapter, as well as in man:lttng-ust(3).
519 Save the above snippet as path:{hello-tp.tp} and run:
523 lttng-gen-tp hello-tp.tp
526 The following files will be created next to path:{hello-tp.tp}:
532 path:{hello-tp.o} is the compiled object file of path:{hello-tp.c}.
534 Now, by including path:{hello-tp.h} in your own application, you may use the
535 tracepoint defined above by properly refering to it when calling
541 #include "hello-tp.h"
543 int main(int argc, char* argv[])
547 puts("Hello, World!\nPress Enter to continue...");
549 /* The following getchar() call is only placed here for the purpose
550 * of this demonstration, for pausing the application in order for
551 * you to have time to list its events. It's not needed otherwise.
555 /* A tracepoint() call. Arguments, as defined in hello-tp.tp:
557 * 1st: provider name (always)
558 * 2nd: tracepoint name (always)
559 * 3rd: my_integer_arg (first user-defined argument)
560 * 4th: my_string_arg (second user-defined argument)
562 * Notice the provider and tracepoint names are NOT strings;
563 * they are in fact parts of variables created by macros in
566 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
568 for (x = 0; x < argc; ++x) {
569 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
572 puts("Quitting now!");
574 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
580 Save this as path:{hello.c}, next to path:{hello-tp.tp}.
582 Notice path:{hello-tp.h}, the header file generated by path:{lttng-gen-tp} from
583 our template file path:{hello-tp.tp}, is included by path:{hello.c}.
585 You are now ready to compile the application with LTTng-UST support:
589 gcc -o hello hello.c hello-tp.o -llttng-ust -ldl
593 <<tracing-the-linux-kernel,Tracing the Linux kernel>> section, the
594 following steps will look familiar.
596 First, run the application with a few arguments:
600 ./hello world and beyond
607 Press Enter to continue...
610 Use the `lttng` tool to list all available user space events:
614 lttng list --userspace
617 You should see the `hello_world:my_first_tracepoint` tracepoint listed
618 under the `./hello` process.
620 Create a tracing session:
624 lttng create my-userspace-session
627 Enable the `hello_world:my_first_tracepoint` tracepoint:
631 lttng enable-event --userspace hello_world:my_first_tracepoint
641 Go back to the running path:{hello} application and press Enter. All
642 `tracepoint()` calls will be executed and the program will finally exit.
651 Done! You may use `lttng view` to list the recorded events. This command
653 http://www.efficios.com/babeltrace[`babeltrace`]
654 in the background, if it is installed:
661 should output something like:
664 [18:10:27.684304496] (+?.?????????) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "hi there!", my_integer_field = 23 }
665 [18:10:27.684338440] (+0.000033944) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "./hello", my_integer_field = 0 }
666 [18:10:27.684340692] (+0.000002252) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "world", my_integer_field = 1 }
667 [18:10:27.684342616] (+0.000001924) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "and", my_integer_field = 2 }
668 [18:10:27.684343518] (+0.000000902) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "beyond", my_integer_field = 3 }
669 [18:10:27.684357978] (+0.000014460) hostname hello_world:my_first_tracepoint: { cpu_id = 0 }, { my_string_field = "x^2", my_integer_field = 16 }
672 When you're done, you may destroy the tracing session, which does _not_
673 destroy the generated trace files, leaving them available for further
678 lttng destroy my-userspace-session
681 The next section presents other alternatives to view and analyze your
685 [[viewing-and-analyzing-your-traces]]
686 === Viewing and analyzing your traces
688 This section describes how to visualize the data gathered after tracing
689 the Linux kernel or a user space application.
691 Many ways exist to read your LTTng traces:
693 * **`babeltrace`** is a command line utility which converts trace formats;
694 it supports the format used by LTTng,
695 CTF, as well as a basic
696 text output which may be ++grep++ed. The `babeltrace` command is
697 part of the http://www.efficios.com/babeltrace[Babeltrace] project.
698 * Babeltrace also includes a **Python binding** so that you may
699 easily open and read an LTTng trace with your own script, benefiting
700 from the power of Python.
701 * **http://projects.eclipse.org/projects/tools.tracecompass[Trace Compass]**
702 is an Eclipse plugin used to visualize and analyze various types of
703 traces, including LTTng's. It also comes as a standalone application
704 and can be downloaded from
705 http://projects.eclipse.org/projects/tools.tracecompass/downloads[here].
707 LTTng trace files are usually recorded in the path:{~/lttng-traces} directory.
708 Let's now view the trace and perform a basic analysis using
711 The simplest way to list all the recorded events of a trace is to pass its
712 path to `babeltrace` with no options:
716 babeltrace ~/lttng-traces/my-session
719 `babeltrace` will find all traces within the given path recursively and
720 output all their events, merging them intelligently.
722 Listing all the system calls of a Linux kernel trace with their arguments is
723 easy with `babeltrace` and `grep`:
727 babeltrace ~/lttng-traces/my-kernel-session | grep sys_
730 Counting events is also straightforward:
734 babeltrace ~/lttng-traces/my-kernel-session | grep sys_read | wc --lines
737 The text output of `babeltrace` is useful for isolating events by simple
738 matching using `grep` and similar utilities. However, more elaborate filters
739 such as keeping only events with a field value falling within a specific range
740 are not trivial to write using a shell. Moreover, reductions and even the
741 most basic computations involving multiple events are virtually impossible
744 Fortunately, Babeltrace ships with a Python 3 binding which makes it
745 really easy to read the events of an LTTng trace sequentially and compute
746 the desired information.
748 Here's a simple example using the Babeltrace Python binding. The following
749 script accepts an LTTng Linux kernel trace path as its first argument and
750 outputs the short names of the top 5 running processes on CPU 0 during the
756 from collections import Counter
761 if len(sys.argv) != 2:
762 msg = 'Usage: python {} TRACEPATH'.format(sys.argv[0])
763 raise ValueError(msg)
765 # a trace collection holds one to many traces
766 col = babeltrace.TraceCollection()
768 # add the trace provided by the user
769 # (LTTng traces always have the 'ctf' format)
770 if col.add_trace(sys.argv[1], 'ctf') is None:
771 raise RuntimeError('Cannot add trace')
773 # this counter dict will hold execution times:
775 # task command name -> total execution time (ns)
776 exec_times = Counter()
778 # this holds the last `sched_switch` timestamp
782 for event in col.events:
783 # keep only `sched_switch` events
784 if event.name != 'sched_switch':
787 # keep only events which happened on CPU 0
788 if event['cpu_id'] != 0:
792 cur_ts = event.timestamp
798 # previous task command (short) name
799 prev_comm = event['prev_comm']
801 # initialize entry in our dict if not yet done
802 if prev_comm not in exec_times:
803 exec_times[prev_comm] = 0
805 # compute previous command execution time
806 diff = cur_ts - last_ts
808 # update execution time of this command
809 exec_times[prev_comm] += diff
811 # update last timestamp
815 for name, ns in exec_times.most_common(5):
817 print('{:20}{} s'.format(name, s))
820 if __name__ == '__main__':
824 Save this script as path:{top5proc.py} and run it with Python 3, providing the
825 path to an LTTng Linux kernel trace as the first argument:
829 python3 top5proc.py ~/lttng-sessions/my-session-.../kernel
832 Make sure the path you provide is the directory containing actual trace
833 files (path:{channel0_0}, path:{metadata}, etc.): the `babeltrace` utility
834 recurses directories, but the Python binding does not.
836 Here's an example of output:
839 swapper/0 48.607245889 s
840 chromium 7.192738188 s
841 pavucontrol 0.709894415 s
842 Compositor 0.660867933 s
843 Xorg.bin 0.616753786 s
846 Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we
847 weren't using the CPU that much when tracing, its first position in the list
851 [[understanding-lttng]]
852 == Understanding LTTng
854 If you're going to use LTTng in any serious way, it is fundamental that
855 you become familiar with its core concepts. Technical terms like
856 _tracing sessions_, _domains_, _channels_ and _events_ are used over
857 and over in the <<using-lttng,Using LTTng>> chapter,
858 and it is assumed that you understand what they mean when reading it.
860 LTTng, as you already know, is a _toolkit_. It would be wrong
861 to call it a simple _tool_ since it is composed of multiple interacting
862 components. This chapter also describes the latter, providing details
863 about their respective roles and how they connect together to form
864 the current LTTng ecosystem.
870 This section explains the various elementary concepts a user has to deal
871 with when using LTTng. They are:
873 * <<tracing-session,tracing session>>
875 * <<channel,channel>>
882 A _tracing session_ is--like any session--a container of
883 state. Anything that is done when tracing using LTTng happens in the
884 scope of a tracing session. In this regard, it is analogous to a bank
885 website's session: you can't interact online with your bank account
886 unless you are logged in a session, except for reading a few static
887 webpages (LTTng, too, can report some static information that does not
888 need a created tracing session).
890 A tracing session holds the following attributes and objects (some of
891 which are described in the following sections):
894 * the tracing state (tracing started or stopped)
895 * the trace data output path/URL (local path or sent over the network)
896 * a mode (normal, snapshot or live)
897 * the snapshot output paths/URLs (if applicable)
898 * for each <<domain,domain>>, a list of <<channel,channels>>
901 ** the channel state (enabled or disabled)
902 ** its parameters (event loss mode, sub-buffers size and count,
903 timer periods, output type, trace files size and count, etc.)
904 ** a list of added context information
905 ** a list of <<event,events>>
907 ** its state (enabled or disabled)
908 ** a list of instrumentation points (tracepoints, system calls,
909 dynamic probes, etc.)
910 ** associated log levels
911 ** a filter expression
913 All this information is completely isolated between tracing sessions.
915 Conceptually, a tracing session is a per-user object; the
916 <<plumbing,Plumbing>> section shows how this is actually
917 implemented. Any user may create as many concurrent tracing sessions
918 as desired. As you can see in the list above, even the tracing state
919 is a per-tracing session attribute, so that you may trace your target
920 system/application in a given tracing session with a specific
921 configuration while another one stays inactive.
923 The trace data generated in a tracing session may be either saved
924 to disk, sent over the network or not saved at all (in which case
925 snapshots may still be saved to disk or sent to a remote machine).
931 A tracing _domain_ is the official term the LTTng project uses to
932 designate a tracer category.
934 There are currently three known domains:
938 * `java.util.logging` (JUL)
940 Different tracers expose common features in their own interfaces, but,
941 from a user's perspective, you still need to target a specific type of
942 tracer to perform some actions. For example, since both kernel and user
943 space tracers support named tracepoints (probes manually inserted in
944 source code), you need to specify which one is concerned when enabling
945 an event because both domains could have existing events with the same
948 Some features are not available in all domains. Filtering enabled
949 events using custom expressions, for example, is currently not
950 supported in the kernel domain, but support could be added in the
957 A _channel_ is a set of events with specific parameters and potential
958 added context information. Channels have unique names per domain within
959 a tracing session. A given event is always registered to at least one
960 channel; having an enabled event in two channels will produce a trace
961 with this event recorded twice everytime it occurs.
963 Channels may be individually enabled or disabled. Occurring events of
964 a disabled channel will never make it to recorded events.
966 The fundamental role of a channel is to keep a shared ring buffer, where
967 events are eventually recorded by the tracer and consumed by a consumer
968 daemon. This internal ring buffer is divided into many sub-buffers of
971 Channels, when created, may be fine-tuned thanks to a few parameters,
972 many of them related to sub-buffers. The following subsections explain
973 what those parameters are and in which situations you should manually
977 [[channel-overwrite-mode-vs-discard-mode]]
978 ===== Overwrite and discard event loss modes
980 As previously mentioned, a channel's ring buffer is divided into many
981 equally sized sub-buffers.
983 As events occur, they are serialized as trace data into a specific
984 sub-buffer (yellow arc in the following animation) until it is full:
985 when this happens, the sub-buffer is marked as consumable (red) and
986 another, _empty_ (white) sub-buffer starts receiving the following
987 events. The marked sub-buffer will be consumed eventually by a consumer
988 daemon (returns to white).
991 [role="docsvg-channel-subbuf-anim"]
996 In an ideal world, sub-buffers are consumed faster than filled, like it
997 is the case above. In the real world, however, all sub-buffers could be
998 full at some point, leaving no space to record the following events. By
999 design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer
1000 exists, losing events is acceptable when the alternative would be to
1001 cause substantial delays in the instrumented application's execution.
1002 LTTng privileges performance over integrity, aiming at perturbing the
1003 traced system as little as possible in order to make tracing of subtle
1004 race conditions and rare interrupt cascades possible.
1006 When it comes to losing events because no empty sub-buffer is available,
1007 the channel's _event loss mode_ determines what to do amongst:
1010 Drop the newest events until a sub-buffer is released.
1013 Clear the sub-buffer containing the oldest recorded
1014 events and start recording the newest events there. This mode is
1015 sometimes called _flight recorder mode_ because it behaves like a
1016 flight recorder: always keep a fixed amount of the latest data.
1018 Which mechanism you should choose depends on your context: prioritize
1019 the newest or the oldest events in the ring buffer?
1021 Beware that, in overwrite mode, a whole sub-buffer is abandoned as soon
1022 as a new event doesn't find an empty sub-buffer, whereas in discard
1023 mode, only the event that doesn't fit is discarded.
1025 Also note that a count of lost events will be incremented and saved in
1026 the trace itself when an event is lost in discard mode, whereas no
1027 information is kept when a sub-buffer gets overwritten before being
1030 There are known ways to decrease your probability of losing events. The
1031 next section shows how tuning the sub-buffers count and size can be
1032 used to virtually stop losing events.
1035 [[channel-subbuf-size-vs-subbuf-count]]
1036 ===== Sub-buffers count and size
1038 For each channel, an LTTng user may set its number of sub-buffers and
1041 Note that there is a noticeable tracer's CPU overhead introduced when
1042 switching sub-buffers (marking a full one as consumable and switching
1043 to an empty one for the following events to be recorded). Knowing this,
1044 the following list presents a few practical situations along with how
1045 to configure sub-buffers for them:
1047 High event throughput::
1048 In general, prefer bigger sub-buffers to
1049 lower the risk of losing events. Having bigger sub-buffers will
1050 also ensure a lower sub-buffer switching frequency. The number of
1051 sub-buffers is only meaningful if the channel is in overwrite mode:
1052 in this case, if a sub-buffer overwrite happens, you will still have
1053 the other sub-buffers left unaltered.
1055 Low event throughput::
1056 In general, prefer smaller sub-buffers
1057 since the risk of losing events is already low. Since events
1058 happen less frequently, the sub-buffer switching frequency should
1059 remain low and thus the tracer's overhead should not be a problem.
1062 If your target system has a low memory
1063 limit, prefer fewer first, then smaller sub-buffers. Even if the
1064 system is limited in memory, you want to keep the sub-buffers as
1065 big as possible to avoid a high sub-buffer switching frequency.
1067 You should know that LTTng uses CTF as its trace format, which means
1068 event data is very compact. For example, the average LTTng Linux kernel
1069 event weights about 32{nbsp}bytes. A sub-buffer size of 1{nbsp}MiB is
1070 thus considered big.
1072 The previous situations highlight the major trade-off between a few big
1073 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1074 frequency vs. how much data is lost in overwrite mode. Assuming a
1075 constant event throughput and using the overwrite mode, the two
1076 following configurations have the same ring buffer total size:
1079 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1084 * **2 sub-buffers of 4 MiB each** lead to a very low sub-buffer
1085 switching frequency, but if a sub-buffer overwrite happens, half of
1086 the recorded events so far (4{nbsp}MiB) are definitely lost.
1087 * **8 sub-buffers of 1 MiB each** lead to 4{nbsp}times the tracer's
1088 overhead as the previous configuration, but if a sub-buffer
1089 overwrite happens, only the eighth of events recorded so far are
1092 In discard mode, the sub-buffers count parameter is pointless: use two
1093 sub-buffers and set their size according to the requirements of your
1097 [[channel-switch-timer]]
1100 The _switch timer_ period is another important configurable feature of
1101 channels to ensure periodic sub-buffer flushing.
1103 When the _switch timer_ fires, a sub-buffer switch happens. This timer
1104 may be used to ensure that event data is consumed and committed to
1105 trace files periodically in case of a low event throughput:
1108 [role="docsvg-channel-switch-timer"]
1113 It's also convenient when big sub-buffers are used to cope with
1114 sporadic high event throughput, even if the throughput is normally
1118 [[channel-buffering-schemes]]
1119 ===== Buffering schemes
1121 In the user space tracing domain, two **buffering schemes** are
1122 available when creating a channel:
1125 Keep one ring buffer per process.
1128 Keep one ring buffer for all processes of a single user.
1130 The per-PID buffering scheme will consume more memory than the per-UID
1131 option if more than one process is instrumented for LTTng-UST. However,
1132 per-PID buffering ensures that one process having a high event
1133 throughput won't fill all the shared sub-buffers, only its own.
1135 The Linux kernel tracing domain only has one available buffering scheme
1136 which is to use a single ring buffer for the whole system.
1142 An _event_, in LTTng's realm, is a term often used metonymically,
1143 having multiple definitions depending on the context:
1145 . When tracing, an event is a _point in space-time_. Space, in a
1146 tracing context, is the set of all executable positions of a
1147 compiled application by a logical processor. When a program is
1148 executed by a processor and some instrumentation point, or
1149 _probe_, is encountered, an event occurs. This event is accompanied
1150 by some contextual payload (values of specific variables at this
1151 point of execution) which may or may not be recorded.
1152 . In the context of a recorded trace file, the term _event_ implies
1154 . When configuring a tracing session, _enabled events_ refer to
1155 specific rules which could lead to the transfer of actual
1156 occurring events (1) to recorded events (2).
1158 The whole <<core-concepts,Core concepts>> section focuses on the
1159 third definition. An event is always registered to _one or more_
1160 channels and may be enabled or disabled at will per channel. A disabled
1161 event will never lead to a recorded event, even if its channel
1164 An event (3) is enabled with a few conditions that must _all_ be met
1165 when an event (1) happens in order to generate a recorded event (2):
1167 . A _probe_ or group of probes in the traced application must be
1169 . **Optionally**, the probe must have a log level matching a
1170 log level range specified when enabling the event.
1171 . **Optionally**, the occurring event must satisfy a custom
1172 expression, or _filter_, specified when enabling the event.
1174 The following illustration summarizes how tracing sessions, domains,
1175 channels and events are related:
1179 image::core-concepts.png[]
1181 This diagram also shows how events may be individually enabled/disabled
1182 (green/grey) and how a given event may be registered to more than one
1189 The previous section described the concepts at the heart of LTTng.
1190 This section summarizes LTTng's implementation: how those objects are
1191 managed by different applications and libraries working together to
1195 [[plumbing-overview]]
1198 As <<installing-lttng,mentioned previously>>, the whole LTTng suite
1199 is made of the following packages: LTTng-tools, LTTng-UST, and
1200 LTTng-modules. Together, they provide different daemons, libraries,
1201 kernel modules and command line interfaces. The following tree shows
1202 which usable component belongs to which package:
1205 ** session daemon (`lttng-sessiond`)
1206 ** consumer daemon (`lttng-consumerd`)
1207 ** relay daemon (`lttng-relayd`)
1208 ** tracing control library (`liblttng-ctl`)
1209 ** tracing control command line tool (`lttng`)
1211 ** user space tracing library (`liblttng-ust`) and its headers
1212 ** preloadable user space tracing helpers
1213 (`liblttng-ust-libc-wrapper`, `liblttng-ust-pthread-wrapper`,
1214 `liblttng-ust-cyg-profile`, `liblttng-ust-cyg-profile-fast`
1215 and `liblttng-ust-dl`)
1216 ** user space tracepoint code generator command line tool
1218 ** `java.util.logging` tracepoint provider (`liblttng-ust-jul-jni`)
1219 and JAR file (path:{liblttng-ust-jul.jar})
1220 * **LTTng-modules**:
1221 ** LTTng Linux kernel tracer module
1222 ** tracing ring buffer kernel modules
1223 ** many LTTng probe kernel modules
1225 The following diagram shows how the most important LTTng components
1226 interact. Plain black arrows represent trace data paths while dashed
1227 red arrows indicate control communications. The LTTng relay daemon is
1228 shown running on a remote system, although it could as well run on the
1229 target (monitored) system.
1233 image::plumbing.png[]
1235 Each component is described in the following subsections.
1241 At the heart of LTTng's plumbing is the _session daemon_, often called
1242 by its command name, `lttng-sessiond`.
1244 The session daemon is responsible for managing tracing sessions and
1245 what they logically contain (channel properties, enabled/disabled
1246 events, etc.). By communicating locally with instrumented applications
1247 (using LTTng-UST) and with the LTTng Linux kernel modules
1248 (LTTng-modules), it oversees all tracing activities.
1250 One of the many things that `lttng-sessiond` does is to keep
1251 track of the available event types. User space applications and
1252 libraries actively connect and register to the session daemon when they
1253 start. By contrast, `lttng-sessiond` seeks out and loads the appropriate
1254 LTTng kernel modules as part of its own initialization. Kernel event
1255 types are _pulled_ by `lttng-sessiond`, whereas user space event types
1256 are _pushed_ to it by the various user space tracepoint providers.
1258 Using a specific inter-process communication protocol with Linux kernel
1259 and user space tracers, the session daemon can send channel information
1260 so that they are initialized, enable/disable specific probes based on
1261 enabled/disabled events by the user, send event filters information to
1262 LTTng tracers so that filtering actually happens at the tracer site,
1263 start/stop tracing a specific application or the Linux kernel, etc.
1265 The session daemon is not useful without some user controlling it,
1266 because it's only a sophisticated control interchange and thus
1267 doesn't make any decision on its own. `lttng-sessiond` opens a local
1268 socket for controlling it, albeit the preferred way to control it is
1269 using `liblttng-ctl`, an installed C library hiding the communication
1270 protocol behind an easy-to-use API. The `lttng` tool makes use of
1271 `liblttng-ctl` to implement a user-friendly command line interface.
1273 `lttng-sessiond` does not receive any trace data from instrumented
1274 applications; the _consumer daemons_ are the programs responsible for
1275 collecting trace data using shared ring buffers. However, the session
1276 daemon is the one that must spawn a consumer daemon and establish
1277 a control communication with it.
1279 Session daemons run on a per-user basis. Knowing this, multiple
1280 instances of `lttng-sessiond` may run simultaneously, each belonging
1281 to a different user and each operating independently of the others.
1282 Only `root`'s session daemon, however, may control LTTng kernel modules
1283 (i.e. the kernel tracer). With that in mind, if a user has no root
1284 access on the target system, he cannot trace the system's kernel, but
1285 should still be able to trace its own instrumented applications.
1287 It has to be noted that, although only `root`'s session daemon may
1288 control the kernel tracer, the `lttng-sessiond` command has a `--group`
1289 option which may be used to specify the name of a special user group
1290 allowed to communicate with `root`'s session daemon and thus record
1291 kernel traces. By default, this group is named `tracing`.
1293 If not done yet, the `lttng` tool, by default, automatically starts a
1294 session daemon. `lttng-sessiond` may also be started manually:
1301 This will start the session daemon in foreground. Use
1305 lttng-sessiond --daemonize
1308 to start it as a true daemon.
1310 To kill the current user's session daemon, `pkill` may be used:
1314 pkill lttng-sessiond
1317 The default `SIGTERM` signal will terminate it cleanly.
1319 Several other options are available and described in
1320 man:lttng-sessiond(8) or by running `lttng-sessiond --help`.
1324 ==== Consumer daemon
1326 The _consumer daemon_, or `lttng-consumerd`, is a program sharing some
1327 ring buffers with user applications or the LTTng kernel modules to
1328 collect trace data and output it at some place (on disk or sent over
1329 the network to an LTTng relay daemon).
1331 Consumer daemons are created by a session daemon as soon as events are
1332 enabled within a tracing session, well before tracing is activated
1333 for the latter. Entirely managed by session daemons,
1334 consumer daemons survive session destruction to be reused later,
1335 should a new tracing session be created. Consumer daemons are always
1336 owned by the same user as their session daemon. When its owner session
1337 daemon is killed, the consumer daemon also exits. This is because
1338 the consumer daemon is always the child process of a session daemon.
1339 Consumer daemons should never be started manually. For this reason,
1340 they are not installed in one of the usual locations listed in the
1341 `PATH` environment variable. `lttng-sessiond` has, however, a
1342 bunch of options (see man:lttng-sessiond(8)) to
1343 specify custom consumer daemon paths if, for some reason, a consumer
1344 daemon other than the default installed one is needed.
1346 There are up to two running consumer daemons per user, whereas only one
1347 session daemon may run per user. This is because each process has
1348 independent bitness: if the target system runs a mixture of 32-bit and
1349 64-bit processes, it is more efficient to have separate corresponding
1350 32-bit and 64-bit consumer daemons. The `root` user is an exception: it
1351 may have up to _three_ running consumer daemons: 32-bit and 64-bit
1352 instances for its user space applications and one more reserved for
1353 collecting kernel trace data.
1355 As new tracing domains are added to LTTng, the development community's
1356 intent is to minimize the need for additionnal consumer daemon instances
1357 dedicated to them. For instance, the `java.util.logging` (JUL) domain
1358 events are in fact mapped to the user space domain, thus tracing this
1359 particular domain is handled by existing user space domain consumer
1366 When a tracing session is configured to send its trace data over the
1367 network, an LTTng _relay daemon_ must be used at the other end to
1368 receive trace packets and serialize them to trace files. This setup
1369 makes it possible to trace a target system without ever committing trace
1370 data to its local storage, a feature which is useful for embedded
1371 systems, amongst others. The command implementing the relay daemon
1374 The basic use case of `lttng-relayd` is to transfer trace data received
1375 over the network to trace files on the local file system. The relay
1376 daemon must listen on two TCP ports to achieve this: one control port,
1377 used by the target session daemon, and one data port, used by the
1378 target consumer daemon. The relay and session daemons agree on common
1379 default ports when custom ones are not specified.
1381 Since the communication transport protocol for both ports is standard
1382 TCP, the relay daemon may be started either remotely or locally (on the
1385 While two instances of consumer daemons (32-bit and 64-bit) may run
1386 concurrently for a given user, `lttng-relayd` needs only be of its
1387 host operating system's bitness.
1389 The other important feature of LTTng's relay daemon is the support of
1390 _LTTng live_. LTTng live is an application protocol to view events as
1391 they arrive. The relay daemon will still record events in trace files,
1392 but a _tee_ may be created to inspect incoming events. Using LTTng live
1393 locally thus requires to run a local relay daemon.
1396 [[liblttng-ctl-lttng]]
1397 ==== [[lttng-cli]]Control library and command line interface
1399 The LTTng control library, `liblttng-ctl`, can be used to communicate
1400 with the session daemon using a C API that hides the underlying
1401 protocol's details. `liblttng-ctl` is part of LTTng-tools.
1403 `liblttng-ctl` may be used by including its "master" header:
1407 #include <lttng/lttng.h>
1410 Some objects are referred by name (C string), such as tracing sessions,
1411 but most of them require creating a handle first using
1412 `lttng_create_handle()`. The best available developer documentation for
1413 `liblttng-ctl` is, for the moment, its installed header files as such.
1414 Every function/structure is thoroughly documented.
1416 The `lttng` program is the _de facto_ standard user interface to
1417 control LTTng tracing sessions. `lttng` uses `liblttng-ctl` to
1418 communicate with session daemons behind the scenes.
1419 Its man page, man:lttng(1), is exhaustive, as well as its command
1420 line help (+lttng _cmd_ --help+, where +_cmd_+ is the command name).
1422 The <<controlling-tracing,Controlling tracing>> section is a feature
1423 tour of the `lttng` tool.
1427 ==== User space tracing library
1429 The user space tracing part of LTTng is possible thanks to the user
1430 space tracing library, `liblttng-ust`, which is part of the LTTng-UST
1433 `liblttng-ust` provides header files containing macros used to define
1434 tracepoints and create tracepoint providers, as well as a shared object
1435 that must be linked to individual applications to connect to and
1436 communicate with a session daemon and a consumer daemon as soon as the
1439 The exact mechanism by which an application is registered to the
1440 session daemon is beyond the scope of this documentation. The only thing
1441 you need to know is that, since the library constructor does this job
1442 automatically, tracepoints may be safely inserted anywhere in the source
1443 code without prior manual initialization of `liblttng-ust`.
1445 The `liblttng-ust`-session daemon collaboration also provides an
1446 interesting feature: user space events may be enabled _before_
1447 applications actually start. By doing this and starting tracing before
1448 launching the instrumented application, you make sure that even the
1449 earliest occurring events can be recorded.
1451 The <<c-application,C application>> instrumenting guide of the
1452 <<using-lttng,Using LTTng>> chapter focuses on using `liblttng-ust`:
1453 instrumenting, building/linking and running a user application.
1457 ==== LTTng kernel modules
1459 The LTTng Linux kernel modules provide everything needed to trace the
1460 Linux kernel: various probes, a ring buffer implementation for a
1461 consumer daemon to read trace data and the tracer itself.
1463 Only in exceptional circumstances should you ever need to load the
1464 LTTng kernel modules manually: it is normally the responsability of
1465 `root`'s session daemon to do so. If you were to develop your own LTTng
1466 probe module, however--for tracing a custom kernel or some kernel
1467 module (this topic is covered in the
1468 <<instrumenting-linux-kernel,Linux kernel>> instrumenting guide of
1469 the <<using-lttng,Using LTTng>> chapter)--you should either
1470 load it manually, or use the `--kmod-probes` option of the session
1471 daemon to load a specific list of kernel probes (beware, however,
1472 that the `--kmod-probes` option specifies an _absolute_ list, which
1473 means you also have to specify the default probes you need). The
1474 session and consumer daemons of regular users do not interact with the
1475 LTTng kernel modules at all.
1477 LTTng kernel modules are installed, by default, in
1478 +/usr/lib/modules/_release_/extra+, where +_release_+ is the
1479 kernel release (see `uname --kernel-release`).
1485 Using LTTng involves two main activities: **instrumenting** and
1486 **controlling tracing**.
1488 _<<instrumenting,Instrumenting>>_ is the process of inserting probes
1489 into some source code. It can be done manually, by writing tracepoint
1490 calls at specific locations in the source code of the program to trace,
1491 or more automatically using dynamic probes (address in assembled code,
1492 symbol name, function entry/return, etc.).
1494 It has to be noted that, as an LTTng user, you may not have to worry
1495 about the instrumentation process. Indeed, you may want to trace a
1496 program already instrumented. As an example, the Linux kernel is
1497 thoroughly instrumented, which is why you can trace it without caring
1498 about adding probes.
1500 _<<controlling-tracing,Controlling tracing>>_ is everything
1501 that can be done by the LTTng session daemon, which is controlled using
1502 `liblttng-ctl` or its command line utility, `lttng`: creating tracing
1503 sessions, listing tracing sessions and events, enabling/disabling
1504 events, starting/stopping the tracers, taking snapshots, etc.
1506 This chapter is a complete user guide of both activities,
1507 with common use cases of LTTng exposed throughout the text. It is
1508 assumed that you are familiar with LTTng's concepts (events, channels,
1509 domains, tracing sessions) and that you understand the roles of its
1510 components (daemons, libraries, command line tools); if not, we invite
1511 you to read the <<understanding-lttng,Understanding LTTng>> chapter
1512 before you begin reading this one.
1514 If you're new to LTTng, we suggest that you rather start with the
1515 <<getting-started,Getting started>> small guide first, then come
1516 back here to broaden your knowledge.
1518 If you're only interested in tracing the Linux kernel with its current
1519 instrumentation, you may skip the
1520 <<instrumenting,Instrumenting>> section.
1526 There are many examples of tracing and monitoring in our everyday life.
1527 You have access to real-time and historical weather reports and forecasts
1528 thanks to weather stations installed around the country. You know your
1529 possibly hospitalized friends' and family's hearts are safe thanks to
1530 electrocardiography. You make sure not to drive your car too fast
1531 and have enough fuel to reach your destination thanks to gauges visible
1534 All the previous examples have something in common: they rely on
1535 **probes**. Without electrodes attached to the surface of a body's
1536 skin, cardiac monitoring would be futile.
1538 LTTng, as a tracer, is no different from the real life examples above.
1539 If you're about to trace a software system, i.e. record its history of
1540 execution, you better have probes in the subject you're
1541 tracing: the actual software. Various ways were developed to do this.
1542 The most straightforward one is to manually place probes, called
1543 _tracepoints_, in the software's source code. The Linux kernel tracing
1544 domain also allows probes added dynamically.
1546 If you're only interested in tracing the Linux kernel, it may very well
1547 be that your tracing needs are already appropriately covered by LTTng's
1548 built-in Linux kernel tracepoints and other probes. Or you may be in
1549 possession of a user space application which has already been
1550 instrumented. In such cases, the work will reside entirely in the design
1551 and execution of tracing sessions, allowing you to jump to
1552 <<controlling-tracing,Controlling tracing>> right now.
1554 This chapter focuses on the following use cases of instrumentation:
1556 * <<c-application,C>> and <<cxx-application,$$C++$$>> applications
1557 * <<prebuilt-ust-helpers,prebuilt user space tracing helpers>>
1558 * <<java-application,Java application>>
1559 * <<instrumenting-linux-kernel,Linux kernel>> module or the
1561 * the <<proc-lttng-logger-abi,path:{/proc/lttng-logger} ABI>>
1563 Some advanced techniques are also presented at the very end of this
1570 Instrumenting a C (or $$C++$$) application, be it an executable program or
1571 a library, implies using LTTng-UST, the
1572 user space tracing component of LTTng. For C/$$C++$$ applications, the
1573 LTTng-UST package includes a dynamically loaded library
1574 (`liblttng-ust`), C headers and the `lttng-gen-tp` command line utility.
1576 Since C and $$C++$$ are the base languages of virtually all other
1577 programming languages
1578 (Java virtual machine, Python, Perl, PHP and Node.js interpreters, etc.),
1579 implementing user space tracing for an unsupported language is just a
1580 matter of using the LTTng-UST C API at the right places.
1582 The usual work flow to instrument a user space C application with
1585 . Define tracepoints (actual probes)
1586 . Write tracepoint providers
1587 . Insert tracepoints into target source code
1588 . Package (build) tracepoint providers
1589 . Build user application and link it with tracepoint providers
1591 The steps above are discussed in greater detail in the following
1595 [[tracepoint-provider]]
1596 ===== Tracepoint provider
1598 Before jumping into defining tracepoints and inserting
1599 them into the application source code, you must understand what a
1600 _tracepoint provider_ is.
1602 For the sake of this guide, consider the following two files:
1607 #undef TRACEPOINT_PROVIDER
1608 #define TRACEPOINT_PROVIDER my_provider
1610 #undef TRACEPOINT_INCLUDE
1611 #define TRACEPOINT_INCLUDE "./tp.h"
1613 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1616 #include <lttng/tracepoint.h>
1620 my_first_tracepoint,
1622 int, my_integer_arg,
1623 char*, my_string_arg
1626 ctf_string(my_string_field, my_string_arg)
1627 ctf_integer(int, my_integer_field, my_integer_arg)
1633 my_other_tracepoint,
1638 ctf_integer(int, some_field, my_int)
1644 #include <lttng/tracepoint-event.h>
1650 #define TRACEPOINT_CREATE_PROBES
1655 The two files above are defining a _tracepoint provider_. A tracepoint
1656 provider is some sort of namespace for _tracepoint definitions_. Tracepoint
1657 definitions are written above with the `TRACEPOINT_EVENT()` macro, and allow
1658 eventual `tracepoint()` calls respecting their definitions to be inserted
1659 into the user application's C source code (we explore this in a
1662 Many tracepoint definitions may be part of the same tracepoint provider
1663 and many tracepoint providers may coexist in a user space application. A
1664 tracepoint provider is packaged either:
1666 * directly into an existing user application's C source file
1668 * as a static library
1669 * as a shared library
1671 The two files above, path:{tp.h} and path:{tp.c}, show a typical template for
1672 writing a tracepoint provider. LTTng-UST was designed so that two
1673 tracepoint providers should not be defined in the same header file.
1675 We will now go through the various parts of the above files and
1676 give them a meaning. As you may have noticed, the LTTng-UST API for
1677 C/$$C++$$ applications is some preprocessor sorcery. The LTTng-UST macros
1678 used in your application and those in the LTTng-UST headers are
1679 combined to produce actual source code needed to make tracing possible
1682 Let's start with the header file, path:{tp.h}. It begins with
1686 #undef TRACEPOINT_PROVIDER
1687 #define TRACEPOINT_PROVIDER my_provider
1690 `TRACEPOINT_PROVIDER` defines the name of the provider to which the
1691 following tracepoint definitions will belong. It is used internally by
1692 LTTng-UST headers and _must_ be defined. Since `TRACEPOINT_PROVIDER`
1693 could have been defined by another header file also included by the same
1694 C source file, the best practice is to undefine it first.
1696 NOTE: Names in LTTng-UST follow the C
1697 _identifier_ syntax (starting with a letter and containing either
1698 letters, numbers or underscores); they are _not_ C strings
1699 (not surrounded by double quotes). This is because LTTng-UST macros
1700 use those identifier-like strings to create symbols (named types and
1703 The tracepoint provider is a group of tracepoint definitions; its chosen
1704 name should reflect this. A hierarchy like Java packages is recommended,
1705 using underscores instead of dots, e.g., `org_company_project_component`.
1707 Next is `TRACEPOINT_INCLUDE`:
1711 #undef TRACEPOINT_INCLUDE
1712 #define TRACEPOINT_INCLUDE "./tp.h"
1715 This little bit of instrospection is needed by LTTng-UST to include
1716 your header at various predefined places.
1718 Include guard follows:
1722 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1726 Add these precompiler conditionals to ensure the tracepoint event
1727 generation can include this file more than once.
1729 The `TRACEPOINT_EVENT()` macro is defined in a LTTng-UST header file which
1734 #include <lttng/tracepoint.h>
1737 This will also allow the application to use the `tracepoint()` macro.
1739 Next is a list of `TRACEPOINT_EVENT()` macro calls which create the
1740 actual tracepoint definitions. We will skip this for the moment and
1741 come back to how to use `TRACEPOINT_EVENT()`
1742 <<defining-tracepoints,in a later section>>. Just pay attention to
1743 the first argument: it's always the name of the tracepoint provider
1744 being defined in this header file.
1746 End of include guard:
1753 Finally, include `<lttng/tracepoint-event.h>` to expand the macros:
1757 #include <lttng/tracepoint-event.h>
1760 That's it for path:{tp.h}. Of course, this is only a header file; it must be
1761 included in some C source file to actually use it. This is the job of
1766 #define TRACEPOINT_CREATE_PROBES
1771 When `TRACEPOINT_CREATE_PROBES` is defined, the macros used in path:{tp.h},
1772 which is included just after, will actually create the source code for
1773 LTTng-UST probes (global data structures and functions) out of your
1774 tracepoint definitions. How exactly this is done is out of this text's scope.
1775 `TRACEPOINT_CREATE_PROBES` is discussed further
1777 <<building-tracepoint-providers-and-user-application,Building/linking
1778 tracepoint providers and the user application>>.
1780 You could include other header files like path:{tp.h} here to create the probes
1781 of different tracepoint providers, e.g.:
1785 #define TRACEPOINT_CREATE_PROBES
1791 The rule is: probes of a given tracepoint provider
1792 must be created in exactly one source file. This source file could be one
1793 of your project's; it doesn't have to be on its own like
1794 path:{tp.c}, although
1795 <<building-tracepoint-providers-and-user-application,a later section>>
1796 shows that doing so allows packaging the tracepoint providers
1797 independently and keep them out of your application, also making it
1798 possible to reuse them between projects.
1800 The following sections explain how to define tracepoints, how to use the
1801 `tracepoint()` macro to instrument your user space C application and how
1802 to build/link tracepoint providers and your application with LTTng-UST
1807 ===== Using `lttng-gen-tp`
1809 LTTng-UST ships with `lttng-gen-tp`, a handy command line utility for
1810 generating most of the stuff discussed above. It takes a _template file_,
1811 with a name usually ending with the `.tp` extension, containing only
1812 tracepoint definitions, and outputs a tracepoint provider (either a C
1813 source file or a precompiled object file) with its header file.
1815 `lttng-gen-tp` should suffice in <<static-linking,static linking>>
1816 situations. When using it, write a template file containing a list of
1817 `TRACEPOINT_EVENT()` macro calls. The tool will find the provider names
1818 used and generate the appropriate files which are going to look a lot
1819 like path:{tp.h} and path:{tp.c} above.
1821 Just call `lttng-gen-tp` like this:
1825 lttng-gen-tp my-template.tp
1828 path:{my-template.c}, path:{my-template.o} and path:{my-template.h}
1829 will be created in the same directory.
1831 You may specify custom C flags passed to the compiler invoked by
1832 `lttng-gen-tp` using the `CFLAGS` environment variable:
1836 CFLAGS=-I/custom/include/path lttng-gen-tp my-template.tp
1839 For more information on `lttng-gen-tp`, see man:lttng-gen-tp(1).
1842 [[defining-tracepoints]]
1843 ===== Defining tracepoints
1845 As written in <<tracepoint-provider,Tracepoint provider>>,
1846 tracepoints are defined using the
1847 `TRACEPOINT_EVENT()` macro. Each tracepoint, when called using the
1848 `tracepoint()` macro in the actual application's source code, generates
1849 a specific event type with its own fields.
1851 Let's have another look at the example above, with a few added comments:
1856 /* tracepoint provider name */
1859 /* tracepoint/event name */
1860 my_first_tracepoint,
1862 /* list of tracepoint arguments */
1864 int, my_integer_arg,
1865 char*, my_string_arg
1868 /* list of fields of eventual event */
1870 ctf_string(my_string_field, my_string_arg)
1871 ctf_integer(int, my_integer_field, my_integer_arg)
1876 The tracepoint provider name must match the name of the tracepoint
1877 provider in which this tracepoint is defined
1878 (see <<tracepoint-provider,Tracepoint provider>>). In other words,
1879 always use the same string as the value of `TRACEPOINT_PROVIDER` above.
1881 The tracepoint name will become the event name once events are recorded
1882 by the LTTng-UST tracer. It must follow the tracepoint provider name
1883 syntax: start with a letter and contain either letters, numbers or
1884 underscores. Two tracepoints under the same provider cannot have the
1885 same name, i.e. you cannot overload a tracepoint like you would
1886 overload functions and methods in $$C++$$/Java.
1888 NOTE: The concatenation of the tracepoint
1889 provider name and the tracepoint name cannot exceed 254 characters. If
1890 it does, the instrumented application will compile and run, but LTTng
1891 will issue multiple warnings and you could experience serious problems.
1893 The list of tracepoint arguments gives this tracepoint its signature:
1894 see it like the declaration of a C function. The format of `TP_ARGS()`
1895 arguments is: C type, then argument name; repeat as needed, up to ten
1896 times. For example, if we were to replicate the signature of C standard
1897 library's `fseek()`, the `TP_ARGS()` part would look like:
1908 Of course, you will need to include appropriate header files before
1909 the `TRACEPOINT_EVENT()` macro calls if any argument has a complex type.
1911 `TP_ARGS()` may not be omitted, but may be empty. `TP_ARGS(void)` is
1914 The list of fields is where the fun really begins. The fields defined
1915 in this list will be the fields of the events generated by the execution
1916 of this tracepoint. Each tracepoint field definition has a C
1917 _argument expression_ which will be evaluated when the execution reaches
1918 the tracepoint. Tracepoint arguments _may be_ used freely in those
1919 argument expressions, but they _don't_ have to.
1921 There are several types of tracepoint fields available. The macros to
1922 define them are given and explained in the
1923 <<liblttng-ust-tp-fields,LTTng-UST library reference>> section.
1925 Field names must follow the standard C identifier syntax: letter, then
1926 optional sequence of letters, numbers or underscores. Each field must have
1929 Those `ctf_*()` macros are added to the `TP_FIELDS()` part of
1930 `TRACEPOINT_EVENT()`. Note that they are not delimited by commas.
1931 `TP_FIELDS()` may be empty, but the `TP_FIELDS(void)` form is _not_
1934 The following snippet shows how argument expressions may be used in
1935 tracepoint fields and how they may refer freely to tracepoint arguments.
1939 /* for struct stat */
1940 #include <sys/types.h>
1941 #include <sys/stat.h>
1953 /* simple integer field with constant value */
1955 int, /* field C type */
1956 my_constant_field, /* field name */
1957 23 + 17 /* argument expression */
1960 /* my_int_arg tracepoint argument */
1967 /* my_int_arg squared */
1971 my_int_arg * my_int_arg
1974 /* sum of first 4 characters of my_str_arg */
1978 my_str_arg[0] + my_str_arg[1] +
1979 my_str_arg[2] + my_str_arg[3]
1982 /* my_str_arg as string field */
1984 my_str_arg_field, /* field name */
1985 my_str_arg /* argument expression */
1988 /* st_size member of st tracepoint argument, hexadecimal */
1990 off_t, /* field C type */
1991 size_field, /* field name */
1992 st->st_size /* argument expression */
1995 /* st_size member of st tracepoint argument, as double */
1997 double, /* field C type */
1998 size_dbl_field, /* field name */
1999 (double) st->st_size /* argument expression */
2002 /* half of my_str_arg string as text sequence */
2004 char, /* element C type */
2005 half_my_str_arg_field, /* field name */
2006 my_str_arg, /* argument expression */
2007 size_t, /* length expression C type */
2008 strlen(my_str_arg) / 2 /* length expression */
2014 As you can see, having a custom argument expression for each field
2015 makes tracepoints very flexible for tracing a user space C application.
2016 This tracepoint definition is reused later in this guide, when
2017 actually using tracepoints in a user space application.
2020 [[using-tracepoint-classes]]
2021 ===== Using tracepoint classes
2023 In LTTng-UST, a _tracepoint class_ is a class of tracepoints sharing the
2024 same field types and names. A _tracepoint instance_ is one instance of
2025 such a declared tracepoint class, with its own event name and tracepoint
2028 What is documented in <<defining-tracepoints,Defining tracepoints>>
2029 is actually how to declare a _tracepoint class_ and define a
2030 _tracepoint instance_ at the same time. Without revealing the internals
2031 of LTTng-UST too much, it has to be noted that one serialization
2032 function is created for each tracepoint class. A serialization
2033 function is responsible for serializing the fields of a tracepoint
2034 into a sub-buffer when tracing. For various performance reasons, when
2035 your situation requires multiple tracepoints with different names, but
2036 with the same fields layout, the best practice is to manually create
2037 a tracepoint class and instantiate as many tracepoint instances as
2038 needed. One positive effect of such a design, amongst other advantages,
2039 is that all tracepoint instances of the same tracepoint class will
2040 reuse the same serialization function, thus reducing cache pollution.
2042 As an example, here are three tracepoint definitions as we know them:
2054 ctf_integer(int, userid, userid)
2055 ctf_integer(size_t, len, len)
2067 ctf_integer(int, userid, userid)
2068 ctf_integer(size_t, len, len)
2080 ctf_integer(int, userid, userid)
2081 ctf_integer(size_t, len, len)
2086 In this case, three tracepoint classes are created, with one tracepoint
2087 instance for each of them: `get_account`, `get_settings` and
2088 `get_transaction`. However, they all share the same field names and
2089 types. Declaring one tracepoint class and three tracepoint instances of
2090 the latter is a better design choice:
2094 /* the tracepoint class */
2095 TRACEPOINT_EVENT_CLASS(
2096 /* tracepoint provider name */
2099 /* tracepoint class name */
2110 ctf_integer(int, userid, userid)
2111 ctf_integer(size_t, len, len)
2115 /* the tracepoint instances */
2116 TRACEPOINT_EVENT_INSTANCE(
2117 /* tracepoint provider name */
2120 /* tracepoint class name */
2123 /* tracepoint/event name */
2132 TRACEPOINT_EVENT_INSTANCE(
2141 TRACEPOINT_EVENT_INSTANCE(
2152 Of course, all those names and `TP_ARGS()` invocations are redundant,
2153 but some C preprocessor magic can solve this:
2157 #define MY_TRACEPOINT_ARGS \
2163 TRACEPOINT_EVENT_CLASS(
2168 ctf_integer(int, userid, userid)
2169 ctf_integer(size_t, len, len)
2173 #define MY_APP_TRACEPOINT_INSTANCE(name) \
2174 TRACEPOINT_EVENT_INSTANCE( \
2178 MY_TRACEPOINT_ARGS \
2181 MY_APP_TRACEPOINT_INSTANCE(get_account)
2182 MY_APP_TRACEPOINT_INSTANCE(get_settings)
2183 MY_APP_TRACEPOINT_INSTANCE(get_transaction)
2187 [[assigning-log-levels]]
2188 ===== Assigning log levels to tracepoints
2190 Optionally, a log level can be assigned to a defined tracepoint.
2191 Assigning different levels of importance to tracepoints can be useful;
2192 when controlling tracing sessions,
2193 <<controlling-tracing,you can choose>> to only enable tracepoints
2194 falling into a specific log level range.
2196 Log levels are assigned to defined tracepoints using the
2197 `TRACEPOINT_LOGLEVEL()` macro. The latter must be used _after_ having
2198 used `TRACEPOINT_EVENT()` for a given tracepoint. The
2199 `TRACEPOINT_LOGLEVEL()` macro has the following construct:
2203 TRACEPOINT_LOGLEVEL(PROVIDER_NAME, TRACEPOINT_NAME, LOG_LEVEL)
2206 where the first two arguments are the same as the first two arguments
2207 of `TRACEPOINT_EVENT()` and `LOG_LEVEL` is one
2208 of the values given in the
2209 <<liblttng-ust-tracepoint-loglevel,LTTng-UST library reference>>
2212 As an example, let's assign a `TRACE_DEBUG_UNIT` log level to our
2213 previous tracepoint definition:
2217 TRACEPOINT_LOGLEVEL(my_provider, my_tracepoint, TRACE_DEBUG_UNIT)
2221 [[probing-the-application-source-code]]
2222 ===== Probing the application's source code
2224 Once tracepoints are properly defined within a tracepoint provider,
2225 they may be inserted into the user application to be instrumented
2226 using the `tracepoint()` macro. Its first argument is the tracepoint
2227 provider name and its second is the tracepoint name. The next, optional
2228 arguments are defined by the `TP_ARGS()` part of the definition of
2229 the tracepoint to use.
2231 As an example, let us again take the following tracepoint definition:
2236 /* tracepoint provider name */
2239 /* tracepoint/event name */
2240 my_first_tracepoint,
2242 /* list of tracepoint arguments */
2244 int, my_integer_arg,
2245 char*, my_string_arg
2248 /* list of fields of eventual event */
2250 ctf_string(my_string_field, my_string_arg)
2251 ctf_integer(int, my_integer_field, my_integer_arg)
2256 Assuming this is part of a file named path:{tp.h} which defines the tracepoint
2257 provider and which is included by path:{tp.c}, here's a complete C application
2258 calling this tracepoint (multiple times):
2262 #define TRACEPOINT_DEFINE
2265 int main(int argc, char* argv[])
2269 tracepoint(my_provider, my_first_tracepoint, 23, "Hello, World!");
2271 for (i = 0; i < argc; ++i) {
2272 tracepoint(my_provider, my_first_tracepoint, i, argv[i]);
2279 For each tracepoint provider, `TRACEPOINT_DEFINE` must be defined into
2280 exactly one translation unit (C source file) of the user application,
2281 before including the tracepoint provider header file. In other words,
2282 for a given tracepoint provider, you cannot define `TRACEPOINT_DEFINE`,
2283 and then include its header file in two separate C source files of
2284 the same application. `TRACEPOINT_DEFINE` is discussed further in
2285 <<building-tracepoint-providers-and-user-application,Building/linking
2286 tracepoint providers and the user application>>.
2288 As another example, remember this definition we wrote in a previous
2289 section (comments are stripped):
2293 /* for struct stat */
2294 #include <sys/types.h>
2295 #include <sys/stat.h>
2307 ctf_integer(int, my_constant_field, 23 + 17)
2308 ctf_integer(int, my_int_arg_field, my_int_arg)
2309 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2310 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2311 my_str_arg[2] + my_str_arg[3])
2312 ctf_string(my_str_arg_field, my_str_arg)
2313 ctf_integer_hex(off_t, size_field, st->st_size)
2314 ctf_float(double, size_dbl_field, (double) st->st_size)
2315 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2316 size_t, strlen(my_str_arg) / 2)
2321 Here's an example of calling it:
2325 #define TRACEPOINT_DEFINE
2332 stat("/etc/fstab", &s);
2334 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2340 When viewing the trace, assuming the file size of path:{/etc/fstab} is
2341 301{nbsp}bytes, the event generated by the execution of this tracepoint
2342 should have the following fields, in this order:
2345 my_constant_field 40
2347 my_int_arg_field2 529
2349 my_str_arg_field "Hello, World!"
2351 size_dbl_field 301.0
2352 half_my_str_arg_field "Hello,"
2356 [[building-tracepoint-providers-and-user-application]]
2357 ===== Building/linking tracepoint providers and the user application
2359 The final step of using LTTng-UST for tracing a user space C application
2360 (beside running the application) is building and linking tracepoint
2361 providers and the application itself.
2363 As discussed above, the macros used by the user-written tracepoint provider
2364 header file are useless until actually used to create probes code
2365 (global data structures and functions) in a translation unit (C source file).
2366 This is accomplished by defining `TRACEPOINT_CREATE_PROBES` in a translation
2367 unit and then including the tracepoint provider header file.
2368 When `TRACEPOINT_CREATE_PROBES` is defined, macros used and included by
2369 the tracepoint provider header will output actual source code needed by any
2370 application using the defined tracepoints. Defining
2371 `TRACEPOINT_CREATE_PROBES` produces code used when registering
2372 tracepoint providers when the tracepoint provider package loads.
2374 The other important definition is `TRACEPOINT_DEFINE`. This one creates
2375 global, per-tracepoint structures referencing the tracepoint providers
2376 data. Those structures are required by the actual functions inserted
2377 where `tracepoint()` macros are placed and need to be defined by the
2378 instrumented application.
2380 Both `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` need to be defined
2381 at some places in order to trace a user space C application using LTTng.
2382 Although explaining their exact mechanism is beyond the scope of this
2383 document, the reason they both exist separately is to allow the trace
2384 providers to be packaged as a shared object (dynamically loaded library).
2386 There are two ways to compile and link the tracepoint providers
2387 with the application: _<<static-linking,statically>>_ or
2388 _<<dynamic-linking,dynamically>>_. Both methods are covered in the
2389 following subsections.
2393 ===== Static linking the tracepoint providers to the application
2395 With the static linking method, compiled tracepoint providers are copied
2396 into the target application. There are three ways to do this:
2398 . Use one of your **existing C source files** to create probes.
2399 . Create probes in a separate C source file and build it as an
2400 **object file** to be linked with the application (more decoupled).
2401 . Create probes in a separate C source file, build it as an
2402 object file and archive it to create a **static library**
2403 (more decoupled, more portable).
2405 The first approach is to define `TRACEPOINT_CREATE_PROBES` and include
2406 your tracepoint provider(s) header file(s) directly into an existing C
2407 source file. Here's an example:
2415 #define TRACEPOINT_CREATE_PROBES
2416 #define TRACEPOINT_DEFINE
2421 int my_func(int a, const char* b)
2425 tracepoint(my_provider, my_tracepoint, buf, sz, limit, &tt)
2433 Again, before including a given tracepoint provider header file,
2434 `TRACEPOINT_CREATE_PROBES` and `TRACEPOINT_DEFINE` must be defined in
2435 one, **and only one**, translation unit. Other C source files of the
2436 same application may include path:{tp.h} to use tracepoints with
2437 the `tracepoint()` macro, but must not define
2438 `TRACEPOINT_CREATE_PROBES`/`TRACEPOINT_DEFINE` again.
2440 This translation unit may be built as an object file by making sure to
2441 add `.` to the include path:
2448 The second approach is to isolate the tracepoint provider code into a
2449 separate object file by using a dedicated C source file to create probes:
2453 #define TRACEPOINT_CREATE_PROBES
2458 `TRACEPOINT_DEFINE` must be defined by a translation unit of the
2459 application. Since we're talking about static linking here, it could as
2460 well be defined directly in the file above, before `#include "tp.h"`:
2464 #define TRACEPOINT_CREATE_PROBES
2465 #define TRACEPOINT_DEFINE
2470 This is actually what <<lttng-gen-tp,`lttng-gen-tp`>> does, and is
2471 the recommended practice.
2473 Build the tracepoint provider:
2480 Finally, the resulting object file may be archived to create a
2481 more portable tracepoint provider static library:
2488 Using a static library does have the advantage of centralising the
2489 tracepoint providers objects so they can be shared between multiple
2490 applications. This way, when the tracepoint provider is modified, the
2491 source code changes don't have to be patched into each application's source
2492 code tree. The applications need to be relinked after each change, but need
2493 not to be otherwise recompiled (unless the tracepoint provider's API
2496 Regardless of which method you choose, you end up with an object file
2497 (potentially archived) containing the trace providers assembled code.
2498 To link this code with the rest of your application, you must also link
2499 with `liblttng-ust` and `libdl`:
2503 gcc -o app tp.o other.o files.o of.o your.o app.o -llttng-ust -ldl
2510 gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -ldl
2513 If you're using a BSD
2514 system, replace `-ldl` with `-lc`:
2518 gcc -o app tp.a other.o files.o of.o your.o app.o -llttng-ust -lc
2521 The application can be started as usual, e.g.:
2528 The `lttng` command line tool can be used to
2529 <<controlling-tracing,control tracing>>.
2533 ===== Dynamic linking the tracepoint providers to the application
2535 The second approach to package the tracepoint providers is to use
2536 dynamic linking: the library and its member functions are explicitly
2537 sought, loaded and unloaded at runtime using `libdl`.
2539 It has to be noted that, for a variety of reasons, the created shared
2540 library will be dynamically _loaded_, as opposed to dynamically
2541 _linked_. The tracepoint provider shared object is, however, linked
2542 with `liblttng-ust`, so that `liblttng-ust` is guaranteed to be loaded
2543 as soon as the tracepoint provider is. If the tracepoint provider is
2544 not loaded, since the application itself is not linked with
2545 `liblttng-ust`, the latter is not loaded at all and the tracepoint calls
2548 The process to create the tracepoint provider shared object is pretty
2549 much the same as the static library method, except that:
2551 * since the tracepoint provider is not part of the application
2552 anymore, `TRACEPOINT_DEFINE` _must_ be defined, for each tracepoint
2553 provider, in exactly one translation unit (C source file) of the
2555 * `TRACEPOINT_PROBE_DYNAMIC_LINKAGE` must be defined next to
2556 `TRACEPOINT_DEFINE`.
2558 Regarding `TRACEPOINT_DEFINE` and `TRACEPOINT_PROBE_DYNAMIC_LINKAGE`,
2559 the recommended practice is to use a separate C source file in your
2560 application to define them, and then include the tracepoint provider
2561 header files afterwards, e.g.:
2565 #define TRACEPOINT_DEFINE
2566 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2568 /* include the header files of one or more tracepoint providers below */
2574 `TRACEPOINT_PROBE_DYNAMIC_LINKAGE` makes the macros included afterwards
2575 (by including the tracepoint provider header, which itself includes
2576 LTTng-UST headers) aware that the tracepoint provider is to be loaded
2577 dynamically and not part of the application's executable.
2579 The tracepoint provider object file used to create the shared library
2580 is built like it is using the static library method, only with the
2581 `-fpic` option added:
2585 gcc -c -fpic -I. tp.c
2588 It is then linked as a shared library like this:
2592 gcc -shared -Wl,--no-as-needed -o tp.so -llttng-ust tp.o
2595 As previously stated, this tracepoint provider shared object isn't
2596 linked with the user application: it will be loaded manually. This is
2597 why the application is built with no mention of this tracepoint
2598 provider, but still needs `libdl`:
2602 gcc -o app other.o files.o of.o your.o app.o -ldl
2605 Now, to make LTTng-UST tracing available to the application, the
2606 `LD_PRELOAD` environment variable is used to preload the tracepoint
2607 provider shared library _before_ the application actually starts:
2611 LD_PRELOAD=/path/to/tp.so ./app
2616 It is not safe to use
2617 `dlclose()` on a tracepoint provider shared object that
2618 is being actively used for tracing, due to a lack of reference
2619 counting from LTTng-UST to the shared object.
2621 For example, statically linking a tracepoint provider to a
2622 shared object which is to be dynamically loaded by an application
2623 (e.g., a plugin) is not safe: the shared object, which contains the
2624 tracepoint provider, could be dynamically closed
2625 (`dlclose()`) at any time by the application.
2627 To instrument a shared object, either:
2629 * Statically link the tracepoint provider to the _application_, or
2630 * Build the tracepoint provider as a shared object (following
2631 the procedure shown in this section), and preload it when
2632 tracing is needed using the `LD_PRELOAD`
2633 environment variable.
2636 Your application will still work without this preloading, albeit without
2637 LTTng-UST tracing support:
2645 [[using-lttng-ust-with-daemons]]
2646 ===== Using LTTng-UST with daemons
2648 Some extra care is needed when using `liblttng-ust` with daemon
2649 applications that call `fork()`, `clone()` or BSD's `rfork()` without
2650 a following `exec()` family system call. The `liblttng-ust-fork`
2651 library must be preloaded for the application.
2657 LD_PRELOAD=liblttng-ust-fork.so ./app
2660 Or, if you're using a tracepoint provider shared library:
2664 LD_PRELOAD="liblttng-ust-fork.so /path/to/tp.so" ./app
2668 [[lttng-ust-pkg-config]]
2669 ===== Using pkg-config
2671 On some distributions, LTTng-UST is shipped with a pkg-config metadata
2672 file, so that you may use the `pkg-config` tool:
2676 pkg-config --libs lttng-ust
2679 This will return `-llttng-ust -ldl` on Linux systems.
2681 You may also check the LTTng-UST version using `pkg-config`:
2685 pkg-config --modversion lttng-ust
2688 For more information about pkg-config, see
2689 http://linux.die.net/man/1/pkg-config[its manpage].
2693 ===== Using `tracef()`
2695 `tracef()` is a small LTTng-UST API to avoid defining your own
2696 tracepoints and tracepoint providers. The signature of `tracef()` is
2697 the same as `printf()`'s.
2699 The `tracef()` utility function was developed to make user space tracing
2700 super simple, albeit with notable disadvantages compared to custom,
2701 full-fledged tracepoint providers:
2703 * All generated events have the same provider/event names, respectively
2704 `lttng_ust_tracef` and `event`.
2705 * There's no static type checking.
2706 * The only event field you actually get, named `msg`, is a string
2707 potentially containing the values you passed to the function
2708 using your own format. This also means that you cannot use filtering
2709 using a custom expression at runtime because there are no isolated
2711 * Since `tracef()` uses C standard library's `vasprintf()` function
2712 in the background to format the strings at runtime, its
2713 expected performance is lower than using custom tracepoint providers
2714 with typed fields, which do not require a conversion to a string.
2716 Thus, `tracef()` is useful for quick prototyping and debugging, but
2717 should not be considered for any permanent/serious application
2720 To use `tracef()`, first include `<lttng/tracef.h>` in the C source file
2721 where you need to insert probes:
2725 #include <lttng/tracef.h>
2728 Use `tracef()` like you would use `printf()` in your source code, e.g.:
2734 tracef("my message, my integer: %d", my_integer);
2739 Link your application with `liblttng-ust`:
2743 gcc -o app app.c -llttng-ust
2746 Execute the application as usual:
2753 Voilà ! Use the `lttng` command line tool to
2754 <<controlling-tracing,control tracing>>. You can enable `tracef()`
2759 lttng enable-event --userspace 'lttng_ust_tracef:*'
2763 [[lttng-ust-environment-variables-compiler-flags]]
2764 ===== LTTng-UST environment variables and special compilation flags
2766 A few special environment variables and compile flags may affect the
2767 behavior of LTTng-UST.
2769 LTTng-UST's debugging can be activated by setting the environment
2770 variable `LTTNG_UST_DEBUG` to `1` when launching the application. It
2771 can also be enabled at compile time by defining `LTTNG_UST_DEBUG` when
2772 compiling LTTng-UST (using the `-DLTTNG_UST_DEBUG` compiler option).
2774 The environment variable `LTTNG_UST_REGISTER_TIMEOUT` can be used to
2775 specify how long the application should wait for the
2776 <<lttng-sessiond,session daemon>>'s _registration done_ command
2777 before proceeding to execute the main program. The timeout value is
2778 specified in milliseconds. 0 means _don't wait_. -1 means
2779 _wait forever_. Setting this environment variable to 0 is recommended
2780 for applications with time contraints on the process startup time.
2782 The default value of `LTTNG_UST_REGISTER_TIMEOUT` (when not defined)
2783 is **3000{nbsp}ms**.
2785 The compilation definition `LTTNG_UST_DEBUG_VALGRIND` should be enabled
2786 at build time (`-DLTTNG_UST_DEBUG_VALGRIND`) to allow `liblttng-ust`
2787 to be used with http://valgrind.org/[Valgrind].
2788 The side effect of defining `LTTNG_UST_DEBUG_VALGRIND` is that per-CPU
2789 buffering is disabled.
2793 ==== $$C++$$ application
2795 Because of $$C++$$'s cross-compatibility with the C language, $$C++$$
2796 applications can be readily instrumented with the LTTng-UST C API.
2798 Follow the <<c-application,C application>> user guide above. It
2799 should be noted that, in this case, tracepoint providers should have
2800 the typical `.cpp`, `.cxx` or `.cc` extension and be built with `g++`
2801 instead of `gcc`. This is the easiest way of avoiding linking errors
2802 due to symbol name mangling incompatibilities between both languages.
2805 [[prebuilt-ust-helpers]]
2806 ==== Prebuilt user space tracing helpers
2808 The LTTng-UST package provides a few helpers that one may find
2809 useful in some situations. They all work the same way: you must
2810 preload the appropriate shared object before running the user
2811 application (using the `LD_PRELOAD` environment variable).
2813 The shared objects are normally found in dir:{/usr/lib}.
2815 The current installed helpers are:
2817 path:{liblttng-ust-libc-wrapper.so} and path:{liblttng-ust-pthread-wrapper.so}::
2818 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
2819 and POSIX threads tracing>>.
2821 path:{liblttng-ust-cyg-profile.so} and path:{liblttng-ust-cyg-profile-fast.so}::
2822 <<liblttng-ust-cyg-profile,Function tracing>>.
2824 path:{liblttng-ust-dl.so}::
2825 <<liblttng-ust-dl,Dynamic linker tracing>>.
2827 The following subsections document what helpers instrument exactly
2828 and how to use them.
2831 [[liblttng-ust-libc-pthread-wrapper]]
2832 ===== C standard library and POSIX threads tracing
2834 path:{liblttng-ust-libc-wrapper.so} and path:{liblttng-ust-pthread-wrapper.so}
2835 can add instrumentation to respectively some C standard library and
2836 POSIX threads functions.
2838 The following functions are traceable by path:{liblttng-ust-libc-wrapper.so}:
2841 .Functions instrumented by path:{liblttng-ust-libc-wrapper.so}
2843 |TP provider name |TP name |Instrumented function
2845 .6+|`ust_libc` |`malloc` |`malloc()`
2846 |`calloc` |`calloc()`
2847 |`realloc` |`realloc()`
2849 |`memalign` |`memalign()`
2850 |`posix_memalign` |`posix_memalign()`
2853 The following functions are traceable by
2854 path:{liblttng-ust-pthread-wrapper.so}:
2857 .Functions instrumented by path:{liblttng-ust-pthread-wrapper.so}
2859 |TP provider name |TP name |Instrumented function
2861 .4+|`ust_pthread` |`pthread_mutex_lock_req` |`pthread_mutex_lock()` (request time)
2862 |`pthread_mutex_lock_acq` |`pthread_mutex_lock()` (acquire time)
2863 |`pthread_mutex_trylock` |`pthread_mutex_trylock()`
2864 |`pthread_mutex_unlock` |`pthread_mutex_unlock()`
2867 All tracepoints have fields corresponding to the arguments of the
2868 function they instrument.
2870 To use one or the other with any user application, independently of
2871 how the latter is built, do:
2875 LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
2882 LD_PRELOAD=liblttng-ust-pthread-wrapper.so my-app
2889 LD_PRELOAD="liblttng-ust-libc-wrapper.so liblttng-ust-pthread-wrapper.so" my-app
2892 When the shared object is preloaded, it effectively replaces the
2893 functions listed in the above tables by wrappers which add tracepoints
2894 and call the replaced functions.
2896 Of course, like any other tracepoint, the ones above need to be enabled
2897 in order for LTTng-UST to generate events. This is done using the
2898 `lttng` command line tool
2899 (see <<controlling-tracing,Controlling tracing>>).
2902 [[liblttng-ust-cyg-profile]]
2903 ===== Function tracing
2905 Function tracing is the recording of which functions are entered and
2906 left during the execution of an application. Like with any LTTng event,
2907 the precise time at which this happens is also kept.
2909 GCC and clang have an option named
2910 https://gcc.gnu.org/onlinedocs/gcc-4.9.1/gcc/Code-Gen-Options.html[`-finstrument-functions`]
2911 which generates instrumentation calls for entry and exit to functions.
2912 The LTTng-UST function tracing helpers, path:{liblttng-ust-cyg-profile.so}
2913 and path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
2914 to add instrumentation to the two generated functions (which contain
2915 `cyg_profile` in their names, hence the shared object's name).
2917 In order to use LTTng-UST function tracing, the translation units to
2918 instrument must be built using the `-finstrument-functions` compiler
2921 LTTng-UST function tracing comes in two flavors, each providing
2922 different trade-offs: path:{liblttng-ust-cyg-profile-fast.so} and
2923 path:{liblttng-ust-cyg-profile.so}.
2925 **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant that
2926 should only be used where it can be _guaranteed_ that the complete event
2927 stream is recorded without any missing events. Any kind of duplicate
2928 information is left out. This version registers the following
2931 [role="growable",options="header,autowidth"]
2932 .Functions instrumented by path:{liblttng-ust-cyg-profile-fast.so}
2934 |TP provider name |TP name |Instrumented function
2936 .2+|`lttng_ust_cyg_profile_fast`
2942 Address of called function.
2948 Assuming no event is lost, having only the function addresses on entry
2949 is enough for creating a call graph (remember that a recorded event
2950 always contains the ID of the CPU that generated it). A tool like
2951 https://sourceware.org/binutils/docs/binutils/addr2line.html[`addr2line`]
2952 may be used to convert function addresses back to source files names
2956 **path:{liblttng-ust-cyg-profile.so}**,
2957 is a more robust variant which also works for use cases where
2958 events might get discarded or not recorded from application startup.
2959 In these cases, the trace analyzer needs extra information to be
2960 able to reconstruct the program flow. This version registers the
2961 following tracepoints:
2963 [role="growable",options="header,autowidth"]
2964 .Functions instrumented by path:{liblttng-ust-cyg-profile.so}
2966 |TP provider name |TP name |Instrumented function
2968 .2+|`lttng_ust_cyg_profile`
2974 Address of called function.
2983 Address of called function.
2989 To use one or the other variant with any user application, assuming at
2990 least one translation unit of the latter is compiled with the
2991 `-finstrument-functions` option, do:
2995 LD_PRELOAD=liblttng-ust-cyg-profile-fast.so my-app
3002 LD_PRELOAD=liblttng-ust-cyg-profile.so my-app
3005 It might be necessary to limit the number of source files where
3006 `-finstrument-functions` is used to prevent excessive amount of trace
3007 data to be generated at runtime.
3009 TIP: When using GCC, at least, you can use
3010 the `-finstrument-functions-exclude-function-list`
3011 option to avoid instrumenting entries and exits of specific
3014 All events generated from LTTng-UST function tracing are provided on
3015 log level `TRACE_DEBUG_FUNCTION`, which is useful to easily enable
3016 function tracing events in your tracing session using the
3017 `--loglevel-only` option of `lttng enable-event`
3018 (see <<controlling-tracing,Controlling tracing>>).
3022 ===== Dynamic linker tracing
3024 This LTTng-UST helper causes all calls to `dlopen()` and `dlclose()`
3025 in the target application to be traced with LTTng.
3027 The helper's shared object, path:{liblttng-ust-dl.so}, registers the
3028 following tracepoints when preloaded:
3030 [role="growable",options="header,autowidth"]
3031 .Functions instrumented by path:{liblttng-ust-dl.so}
3033 |TP provider name |TP name |Instrumented function
3041 Memory base address (where the dynamic linker placed the shared
3045 File system path to the loaded shared object.
3048 File size of the the loaded shared object.
3051 Last modification time (seconds since Epoch time) of the loaded shared
3058 Memory base address (where the dynamic linker placed the shared
3062 To use this LTTng-UST helper with any user application, independently of
3063 how the latter is built, do:
3067 LD_PRELOAD=liblttng-ust-dl.so my-app
3070 Of course, like any other tracepoint, the ones above need to be enabled
3071 in order for LTTng-UST to generate events. This is done using the
3072 `lttng` command line tool
3073 (see <<controlling-tracing,Controlling tracing>>).
3076 [[java-application]]
3077 ==== Java application
3079 LTTng-UST provides a _logging_ back-end for Java applications using
3080 http://docs.oracle.com/javase/7/docs/api/java/util/logging/Logger.html[`java.util.logging`]
3081 (JUL). This back-end is called the _LTTng-UST JUL agent_ and is
3082 responsible for communications with an LTTng session daemon.
3084 From the user's point of view, once the LTTng-UST JUL agent has been
3085 initialized, JUL loggers may be created and used as usual. The agent
3086 adds its own handler to the _root logger_, so that all loggers may
3087 generate LTTng events with no effort.
3089 Common JUL features are supported using the `lttng` tool
3090 (see <<controlling-tracing,Controlling tracing>>):
3092 * listing all logger names
3093 * enabling/disabling events per logger name
3100 import java.util.logging.Logger;
3101 import org.lttng.ust.jul.LTTngAgent;
3105 public static void main(String[] argv) throws Exception
3108 Logger logger = Logger.getLogger("jello");
3110 // call this as soon as possible (before logging)
3111 LTTngAgent lttngAgent = LTTngAgent.getLTTngAgent();
3114 logger.info("some info");
3115 logger.warning("some warning");
3117 logger.finer("finer information...");
3119 logger.severe("error!");
3121 // not mandatory, but cleaner
3122 lttngAgent.dispose();
3127 The LTTng-UST JUL agent Java classes are packaged in a JAR file named
3128 path:{liblttng-ust-jul.jar}. It is typically located in
3129 dir:{/usr/lib/lttng/java}. To compile the snippet above
3130 (saved as path:{Test.java}), do:
3134 javac -cp /usr/lib/lttng/java/liblttng-ust-jul.jar Test.java
3137 You can run the resulting compiled class:
3141 java -cp /usr/lib/lttng/java/liblttng-ust-jul.jar:. Test
3144 NOTE: http://openjdk.java.net/[OpenJDK] 7 is used for development and
3145 continuous integration, thus this version is directly supported.
3146 However, the LTTng-UST JUL agent has also been tested with OpenJDK 6.
3149 [[instrumenting-linux-kernel]]
3152 The Linux kernel can be instrumented for LTTng tracing, either its core
3153 source code or a kernel module. It has to be noted that Linux is
3154 readily traceable using LTTng since many parts of its source code are
3155 already instrumented: this is the job of the upstream
3156 http://git.lttng.org/?p=lttng-modules.git[LTTng-modules]
3157 package. This section presents how to add LTTng instrumentation where it
3158 does not currently exist and how to instrument custom kernel modules.
3160 All LTTng instrumentation in the Linux kernel is based on an existing
3161 infrastructure which bears the name of its main macro, `TRACE_EVENT()`.
3162 This macro is used to define tracepoints,
3163 each tracepoint having a name, usually with the
3164 +__subsys_____name__+ format,
3165 +_subsys_+ being the subsystem name and
3166 +_name_+ the specific event name.
3168 Tracepoints defined with `TRACE_EVENT()` may be inserted anywhere in
3169 the Linux kernel source code, after what callbacks, called _probes_,
3170 may be registered to execute some action when a tracepoint is
3171 executed. This mechanism is directly used by ftrace and perf,
3172 but cannot be used as is by LTTng: an adaptation layer is added to
3173 satisfy LTTng's specific needs.
3175 With that in mind, this documentation does not cover the `TRACE_EVENT()`
3176 format and how to use it, but it is mandatory to understand it and use
3177 it to instrument Linux for LTTng. A series of
3178 LWN articles explain
3179 `TRACE_EVENT()` in details:
3180 http://lwn.net/Articles/379903/[part 1],
3181 http://lwn.net/Articles/381064/[part 2], and
3182 http://lwn.net/Articles/383362/[part 3].
3183 Once you master `TRACE_EVENT()` enough for your use case, continue
3184 reading this section so that you can add the LTTng adaptation layer of
3187 This section first discusses the general method of instrumenting the
3188 Linux kernel for LTTng. This method is then reused for the specific
3189 case of instrumenting a kernel module.
3192 [[instrumenting-linux-kernel-itself]]
3193 ===== Instrumenting the Linux kernel for LTTng
3195 The following subsections explain strictly how to add custom LTTng
3196 instrumentation to the Linux kernel. They do not explain how the
3197 macros actually work and the internal mechanics of the tracer.
3199 You should have a Linux kernel source code tree to work with.
3200 Throughout this section, all file paths are relative to the root of
3201 this tree unless otherwise stated.
3203 You will need a copy of the LTTng-modules Git repository:
3207 git clone git://git.lttng.org/lttng-modules.git
3210 The steps to add custom LTTng instrumentation to a Linux kernel
3211 involves defining and using the mainline `TRACE_EVENT()` tracepoints
3212 first, then writing and using the LTTng adaptation layer.
3215 [[mainline-trace-event]]
3216 ===== Defining/using tracepoints with mainline `TRACE_EVENT()` infrastructure
3218 The first step is to define tracepoints using the mainline Linux
3219 `TRACE_EVENT()` macro and insert tracepoints where you want them.
3220 Your tracepoint definitions reside in a header file in
3221 dir:{include/trace/events}. If you're adding tracepoints to an existing
3222 subsystem, edit its appropriate header file.
3224 As an example, the following header file (let's call it
3225 path:{include/trace/events/hello.h}) defines one tracepoint using
3230 /* subsystem name is "hello" */
3232 #define TRACE_SYSTEM hello
3234 #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ)
3235 #define _TRACE_HELLO_H
3237 #include <linux/tracepoint.h>
3240 /* "hello" is the subsystem name, "world" is the event name */
3243 /* tracepoint function prototype */
3244 TP_PROTO(int foo, const char* bar),
3246 /* arguments for this tracepoint */
3249 /* LTTng doesn't need those */
3257 /* this part must be outside protection */
3258 #include <trace/define_trace.h>
3261 Notice that we don't use any of the last three arguments: they
3262 are left empty here because LTTng doesn't need them. You would only fill
3263 `TP_STRUCT__entry()`, `TP_fast_assign()` and `TP_printk()` if you were
3264 to also use this tracepoint for ftrace/perf.
3266 Once this is done, you may place calls to `trace_hello_world()`
3267 wherever you want in the Linux source code. As an example, let us place
3268 such a tracepoint in the `usb_probe_device()` static function
3269 (path:{drivers/usb/core/driver.c}):
3273 /* called from driver core with dev locked */
3274 static int usb_probe_device(struct device *dev)
3276 struct usb_device_driver *udriver = to_usb_device_driver(dev->driver);
3277 struct usb_device *udev = to_usb_device(dev);
3280 trace_hello_world(udev->devnum, udev->product);
3286 This tracepoint should fire every time a USB device is plugged in.
3288 At the top of path:{driver.c}, we need to include our actual tracepoint
3289 definition and, in this case (one place per subsystem), define
3290 `CREATE_TRACE_POINTS`, which will create our tracepoint:
3298 #define CREATE_TRACE_POINTS
3299 #include <trace/events/hello.h>
3304 Build your custom Linux kernel. In order to use LTTng, make sure the
3305 following kernel configuration options are enabled:
3307 * `CONFIG_MODULES` (loadable module support)
3308 * `CONFIG_KALLSYMS` (load all symbols for debugging/kksymoops)
3309 * `CONFIG_HIGH_RES_TIMERS` (high resolution timer support)
3310 * `CONFIG_TRACEPOINTS` (kernel tracepoint instrumentation)
3312 Boot the custom kernel. The directory
3313 dir:{/sys/kernel/debug/tracing/events/hello} should exist if everything
3314 went right, with a dir:{hello_world} subdirectory.
3317 [[lttng-adaptation-layer]]
3318 ===== Adding the LTTng adaptation layer
3320 The steps to write the LTTng adaptation layer are, in your
3321 LTTng-modules copy's source code tree:
3323 . In dir:{instrumentation/events/lttng-module},
3324 add a header +__subsys__.h+ for your custom
3325 subsystem +__subsys__+ and write your
3326 tracepoint definitions using LTTng-modules macros in it.
3327 Those macros look like the mainline kernel equivalents,
3328 but they present subtle, yet important differences.
3329 . In dir:{probes}, create the C source file of the LTTng probe kernel
3330 module for your subsystem. It should be named
3331 +lttng-probe-__subsys__.c+.
3332 . Edit path:{probes/Makefile} so that the LTTng-modules project
3333 builds your custom LTTng probe kernel module.
3334 . Build and install LTTng kernel modules.
3336 Following our `hello_world` event example, here's the content of
3337 path:{instrumentation/events/lttng-module/hello.h}:
3342 #define TRACE_SYSTEM hello
3344 #if !defined(_TRACE_HELLO_H) || defined(TRACE_HEADER_MULTI_READ)
3345 #define _TRACE_HELLO_H
3347 #include <linux/tracepoint.h>
3349 LTTNG_TRACEPOINT_EVENT(
3350 /* format identical to mainline version for those */
3352 TP_PROTO(int foo, const char* bar),
3355 /* possible differences */
3357 __field(int, my_int)
3358 __field(char, char0)
3359 __field(char, char1)
3360 __string(product, bar)
3363 /* notice the use of tp_assign()/tp_strcpy() and no semicolons */
3365 tp_assign(my_int, foo)
3366 tp_assign(char0, bar[0])
3367 tp_assign(char1, bar[1])
3368 tp_strcpy(product, bar)
3371 /* This one is actually not used by LTTng either, but must be
3372 * present for the moment.
3376 /* no semicolon after this either */
3381 /* other difference: do NOT include <trace/define_trace.h> */
3382 #include "../../../probes/define_trace.h"
3385 Some possible entries for `TP_STRUCT__entry()` and `TP_fast_assign()`,
3386 in the case of LTTng-modules, are shown in the
3387 <<lttng-modules-ref,LTTng-modules reference>> section.
3389 The best way to learn how to use the above macros is to inspect
3390 existing LTTng tracepoint definitions in
3391 dir:{instrumentation/events/lttng-module} header files. Compare
3392 them with the Linux kernel mainline versions in
3393 dir:{include/trace/events}.
3395 The next step is writing the LTTng probe kernel module C source file.
3396 This one is named +lttng-probe-__subsys__.c+
3397 in dir:{probes}. You may always use the following template:
3401 #include <linux/module.h>
3402 #include "../lttng-tracer.h"
3404 /* Build time verification of mismatch between mainline TRACE_EVENT()
3405 * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments.
3407 #include <trace/events/hello.h>
3409 /* create LTTng tracepoint probes */
3410 #define LTTNG_PACKAGE_BUILD
3411 #define CREATE_TRACE_POINTS
3412 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
3414 #include "../instrumentation/events/lttng-module/hello.h"
3416 MODULE_LICENSE("GPL and additional rights");
3417 MODULE_AUTHOR("Your name <your-email>");
3418 MODULE_DESCRIPTION("LTTng hello probes");
3419 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
3420 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
3421 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
3422 LTTNG_MODULES_EXTRAVERSION);
3425 Just replace `hello` with your subsystem name. In this example,
3426 `<trace/events/hello.h>`, which is the original mainline tracepoint
3427 definition header, is included for verification purposes: the
3428 LTTng-modules build system is able to emit an error at build time when
3429 the arguments of the mainline `TRACE_EVENT()` definitions do not match
3430 the ones of the LTTng-modules adaptation layer
3431 (`LTTNG_TRACEPOINT_EVENT()`).
3433 Edit path:{probes/Makefile} and add your new kernel module object
3434 next to existing ones:
3440 obj-m += lttng-probe-module.o
3441 obj-m += lttng-probe-power.o
3443 obj-m += lttng-probe-hello.o
3448 Time to build! Point to your custom Linux kernel source tree using
3449 the `KERNELDIR` variable:
3453 make KERNELDIR=/path/to/custom/linux
3456 Finally, install modules:
3460 sudo make modules_install
3464 [[instrumenting-linux-kernel-tracing]]
3467 The <<controlling-tracing,Controlling tracing>> section explains
3468 how to use the `lttng` tool to create and control tracing sessions.
3469 Although the `lttng` tool will load the appropriate _known_ LTTng kernel
3470 modules when needed (by launching `root`'s session daemon), it won't
3471 load your custom `lttng-probe-hello` module by default. You need to
3472 manually load the `lttng-probe-hello` module, and start an LTTng session
3477 sudo pkill -u root lttng-sessiond
3478 sudo modprobe lttng_probe_hello
3482 The first command makes sure any existing instance is killed. If
3483 you're not interested in using the default probes, or if you only
3484 want to use a few of them, you can use the `--kmod-probes` option
3485 of `lttng-sessiond` instead, which specifies an absolute list of
3486 probes to load (without the `lttng-probe-` prefix):
3490 sudo lttng-sessiond --kmod-probes=hello,ext4,net,block,signal,sched
3493 Confirm the custom probe module is loaded:
3497 lsmod | grep lttng_probe_hello
3500 The `hello_world` event should appear in the list when doing
3504 lttng list --kernel | grep hello
3507 You may now create an LTTng tracing session, enable the `hello_world`
3508 kernel event (and others if you wish) and start tracing:
3512 sudo lttng create my-session
3513 sudo lttng enable-event --kernel hello_world
3517 Plug a few USB devices, then stop tracing and inspect the trace (if
3518 http://diamon.org/babeltrace[Babeltrace]
3527 Here's a sample output:
3530 [15:30:34.835895035] (+?.?????????) hostname hello_world: { cpu_id = 1 }, { my_int = 8, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
3531 [15:30:42.262781421] (+7.426886386) hostname hello_world: { cpu_id = 1 }, { my_int = 9, char0 = 80, char1 = 97, product = "Patriot Memory" }
3532 [15:30:48.175621778] (+5.912840357) hostname hello_world: { cpu_id = 1 }, { my_int = 10, char0 = 68, char1 = 97, product = "DataTraveler 2.0" }
3535 Two USB flash drives were used for this test.
3537 You may change your LTTng custom probe, rebuild it and reload it at
3538 any time when not tracing. Make sure you remove the old module
3539 (either by killing the root LTTng session daemon which loaded the
3540 module in the first place (if you used `--kmod-probes`), or by
3541 using `modprobe --remove` directly) before loading the updated one.
3544 [[instrumenting-out-of-tree-linux-kernel]]
3545 ===== Advanced: Instrumenting an out-of-tree Linux kernel module for LTTng
3547 Instrumenting a custom Linux kernel module for LTTng follows the exact
3549 <<instrumenting-linux-kernel-itself,adding instrumentation
3550 to the Linux kernel itself>>,
3551 the only difference being that your mainline tracepoint definition
3552 header doesn't reside in the mainline source tree, but in your
3553 kernel module source tree.
3555 The only reference to this mainline header is in the LTTng custom
3556 probe's source code (path:{probes/lttng-probe-hello.c} in our example),
3557 for build time verification:
3563 /* Build time verification of mismatch between mainline TRACE_EVENT()
3564 * arguments and LTTng adaptation layer LTTNG_TRACEPOINT_EVENT() arguments.
3566 #include <trace/events/hello.h>
3571 The preferred, flexible way to include your module's mainline
3572 tracepoint definition header is to put it in a specific directory
3573 relative to your module's root, e.g., dir:{tracepoints}, and include it
3574 relative to your module's root directory in the LTTng custom probe's
3579 #include <tracepoints/hello.h>
3582 You may then build LTTng-modules by adding your module's root
3583 directory as an include path to the extra C flags:
3587 make ccflags-y=-I/path/to/kernel/module KERNELDIR=/path/to/custom/linux
3590 Using `ccflags-y` allows you to move your kernel module to another
3591 directory and rebuild the LTTng-modules project with no change to
3595 [[proc-lttng-logger-abi]]
3596 ==== LTTng logger ABI
3598 The `lttng-tracer` Linux kernel module, installed by the LTTng-modules
3599 package, creates a special LTTng logger ABI file path:{/proc/lttng-logger}
3600 when loaded. Writing text data to this file generates an LTTng kernel
3601 domain event named `lttng_logger`.
3603 Unlike other kernel domain events, `lttng_logger` may be enabled by
3604 any user, not only root users or members of the tracing group.
3606 To use the LTTng logger ABI, simply write a string to
3607 path:{/proc/lttng-logger}:
3611 echo -n 'Hello, World!' > /proc/lttng-logger
3614 The `msg` field of the `lttng_logger` event contains the recorded
3617 NOTE: Messages are split in chunks of 1024{nbsp}bytes.
3619 The LTTng logger ABI is a quick and easy way to trace some events from
3620 user space through the kernel tracer. However, it is much more basic
3621 than LTTng-UST: it's slower (involves system call round-trip to the
3622 kernel and only supports logging strings). The LTTng logger ABI is
3623 particularly useful for recording logs as LTTng traces from shell
3624 scripts, potentially combining them with other Linux kernel/user space
3628 [[instrumenting-32-bit-app-on-64-bit-system]]
3629 ==== Advanced: Instrumenting a 32-bit application on a 64-bit system
3631 [[advanced-instrumenting-techniques]]In order to trace a 32-bit
3632 application running on a 64-bit system,
3633 LTTng must use a dedicated 32-bit
3634 <<lttng-consumerd,consumer daemon>>. This section discusses how to
3635 build that daemon (which is _not_ part of the default 64-bit LTTng
3636 build) and the LTTng 32-bit tracing libraries, and how to instrument
3637 a 32-bit application in that context.
3639 Make sure you install all 32-bit versions of LTTng dependencies.
3640 Their names can be found in the path:{README.md} files of each LTTng package
3641 source. How to find and install them will vary depending on your target
3642 Linux distribution. `gcc-multilib` is a common package name for the
3643 multilib version of GCC, which you will also need.
3645 The following packages will be built for 32-bit support on a 64-bit
3646 system: http://urcu.so/[Userspace RCU],
3647 LTTng-UST and LTTng-tools.
3650 [[building-32-bit-userspace-rcu]]
3651 ===== Building 32-bit Userspace RCU
3657 git clone git://git.urcu.so/urcu.git
3660 ./configure --libdir=/usr/lib32 CFLAGS=-m32
3666 The `-m32` C compiler flag creates 32-bit object files and `--libdir`
3667 indicates where to install the resulting libraries.
3670 [[building-32-bit-lttng-ust]]
3671 ===== Building 32-bit LTTng-UST
3677 git clone http://git.lttng.org/lttng-ust.git
3680 ./configure --prefix=/usr \
3681 --libdir=/usr/lib32 \
3682 CFLAGS=-m32 CXXFLAGS=-m32 \
3683 LDFLAGS=-L/usr/lib32
3689 `-L/usr/lib32` is required for the build to find the 32-bit versions
3690 of Userspace RCU and other dependencies.
3694 Depending on your Linux distribution,
3695 32-bit libraries could be installed at a different location than
3696 dir:{/usr/lib32}. For example, Debian is known to install
3697 some 32-bit libraries in dir:{/usr/lib/i386-linux-gnu}.
3699 In this case, make sure to set `LDFLAGS` to all the
3700 relevant 32-bit library paths, e.g.,
3701 `LDFLAGS="-L/usr/lib32 -L/usr/lib/i386-linux-gnu"`.
3704 NOTE: You may add options to path:{./configure} if you need them, e.g., for
3705 Java and SystemTap support. Look at `./configure --help` for more
3709 [[building-32-bit-lttng-tools]]
3710 ===== Building 32-bit LTTng-tools
3712 Since the host is a 64-bit system, most 32-bit binaries and libraries of
3713 LTTng-tools are not needed; the host will use their 64-bit counterparts.
3714 The required step here is building and installing a 32-bit consumer
3721 git clone http://git.lttng.org/lttng-tools.git
3724 ./configure --prefix=/usr \
3725 --libdir=/usr/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3726 LDFLAGS=-L/usr/lib32
3728 cd src/bin/lttng-consumerd
3733 The above commands build all the LTTng-tools project as 32-bit
3734 applications, but only installs the 32-bit consumer daemon.
3737 [[building-64-bit-lttng-tools]]
3738 ===== Building 64-bit LTTng-tools
3740 Finally, you need to build a 64-bit version of LTTng-tools which is
3741 aware of the 32-bit consumer daemon previously built and installed:
3747 ./configure --prefix=/usr \
3748 --with-consumerd32-libdir=/usr/lib32 \
3749 --with-consumerd32-bin=/usr/lib32/lttng/libexec/lttng-consumerd
3755 Henceforth, the 64-bit session daemon will automatically find the
3756 32-bit consumer daemon if required.
3759 [[building-instrumented-32-bit-c-application]]
3760 ===== Building an instrumented 32-bit C application
3762 Let us reuse the _Hello world_ example of
3763 <<tracing-your-own-user-application,Tracing your own user application>>
3764 (<<getting-started,Getting started>> chapter).
3766 The instrumentation process is unaltered.
3768 First, a typical 64-bit build (assuming you're running a 64-bit system):
3772 gcc -o hello64 -I. hello.c hello-tp.c -ldl -llttng-ust
3775 Now, a 32-bit build:
3779 gcc -o hello32 -I. -m32 hello.c hello-tp.c -L/usr/lib32 \
3780 -ldl -llttng-ust -Wl,-rpath,/usr/lib32
3783 The `-rpath` option, passed to the linker, will make the dynamic loader
3784 check for libraries in dir:{/usr/lib32} before looking in its default paths,
3785 where it should find the 32-bit version of `liblttng-ust`.
3788 [[running-32-bit-and-64-bit-c-applications]]
3789 ===== Running 32-bit and 64-bit versions of an instrumented C application
3791 Now, both 32-bit and 64-bit versions of the _Hello world_ example above
3792 can be traced in the same tracing session. Use the `lttng` tool as usual
3793 to create a tracing session and start tracing:
3797 lttng create session-3264
3798 lttng enable-event -u -a
3804 Use `lttng view` to verify both processes were
3805 successfully traced.
3808 [[controlling-tracing]]
3809 === Controlling tracing
3811 Once you're in possession of a software that is properly
3812 <<instrumenting,instrumented>> for LTTng tracing, be it thanks to
3813 the built-in LTTng probes for the Linux kernel, a custom user
3814 application or a custom Linux kernel, all that is left is actually
3815 tracing it. As a user, you control LTTng tracing using a single command
3816 line interface: the `lttng` tool. This tool uses `liblttng-ctl` behind
3817 the scene to connect to and communicate with session daemons. LTTng
3818 session daemons may either be started manually (`lttng-sessiond`) or
3819 automatically by the `lttng` command when needed. Trace data may
3820 be forwarded to the network and used elsewhere using an LTTng relay
3821 daemon (`lttng-relayd`).
3823 The manpages of `lttng`, `lttng-sessiond` and `lttng-relayd` are pretty
3824 complete, thus this section is not an online copy of the latter (we
3825 leave this contents for the
3826 <<online-lttng-manpages,Online LTTng manpages>> section).
3827 This section is rather a tour of LTTng
3828 features through practical examples and tips.
3830 If not already done, make sure you understand the core concepts
3831 and how LTTng components connect together by reading the
3832 <<understanding-lttng,Understanding LTTng>> chapter; this section
3833 assumes you are familiar with them.
3836 [[creating-destroying-tracing-sessions]]
3837 ==== Creating and destroying tracing sessions
3839 Whatever you want to do with `lttng`, it has to happen inside a
3840 **tracing session**, created beforehand. A session, in general, is a
3841 per-user container of state. A tracing session is no different; it
3842 keeps a specific state of stuff like:
3845 * enabled/disabled channels with associated parameters
3846 * enabled/disabled events with associated log levels and filters
3847 * context information added to channels
3848 * tracing activity (started or stopped)
3852 A single user may have many active tracing sessions. LTTng session
3853 daemons are the ultimate owners and managers of tracing sessions. For
3854 user space tracing, each user has its own session daemon. Since Linux
3855 kernel tracing requires root privileges, only `root`'s session daemon
3856 may enable and trace kernel events. However, `lttng` has a `--group`
3857 option (which is passed to `lttng-sessiond` when starting it) to
3858 specify the name of a _tracing group_ which selected users may be part
3859 of to be allowed to communicate with `root`'s session daemon. By
3860 default, the tracing group name is `tracing`.
3862 To create a tracing session, do:
3866 lttng create my-session
3869 This will create a new tracing session named `my-session` and make it
3870 the current one. If you don't specify any name (calling only
3871 `lttng create`), your tracing session will be named `auto`. Traces
3872 are written in +\~/lttng-traces/__session__-+ followed
3873 by the tracing session's creation date/time by default, where
3874 +__session__+ is the tracing session name. To save them
3875 at a different location, use the `--output` option:
3879 lttng create --output /tmp/some-directory my-session
3882 You may create as many tracing sessions as you wish:
3886 lttng create other-session
3887 lttng create yet-another-session
3890 You may view all existing tracing sessions using the `list` command:
3897 The state of a _current tracing session_ is kept in path:{~/.lttngrc}. Each
3898 invocation of `lttng` reads this file to set its current tracing
3899 session name so that you don't have to specify a session name for each
3900 command. You could edit this file manually, but the preferred way to
3901 set the current tracing session is to use the `set-session` command:
3905 lttng set-session other-session
3908 Most `lttng` commands accept a `--session` option to specify the name
3909 of the target tracing session.
3911 Any existing tracing session may be destroyed using the `destroy`
3916 lttng destroy my-session
3919 Providing no argument to `lttng destroy` will destroy the current
3920 tracing session. Destroying a tracing session will stop any tracing
3921 running within the latter. Destroying a tracing session frees resources
3922 acquired by the session daemon and tracer side, making sure to flush
3925 You can't do much with LTTng using only the `create`, `set-session`
3926 and `destroy` commands of `lttng`, but it is essential to know them in
3927 order to control LTTng tracing, which always happen within the scope of
3931 [[enabling-disabling-events]]
3932 ==== Enabling and disabling events
3934 Inside a tracing session, individual events may be enabled or disabled
3935 so that tracing them may or may not generate trace data.
3937 We sometimes use the term _event_ metonymically throughout this text to
3938 refer to a specific condition, or _rule_, that could lead, when
3939 satisfied, to an actual occurring event (a point at a specific position
3940 in source code/binary program, logical processor and time capturing
3941 some payload) being recorded as trace data. This specific condition is
3944 . A **domain** (kernel, user space or `java.util.logging`) (required).
3945 . One or many **instrumentation points** in source code or binary
3946 program (tracepoint name, address, symbol name, function name,
3947 logger name, etc.) to be executed (required).
3948 . A **log level** (each instrumentation point declares its own log
3949 level) or log level range to match (optional; only valid for user
3951 . A **custom user expression**, or **filter**, that must evaluate to
3952 _true_ when a tracepoint is executed (optional; only valid for user
3955 All conditions are specified using arguments passed to the
3956 `enable-event` command of the `lttng` tool.
3958 Condition 1 is specified using either `--kernel/-k` (kernel),
3959 `--userspace/-u` (user space) or `--jul/-j`
3960 (JUL). Exactly one of those
3961 three arguments must be specified.
3963 Condition 2 is specified using one of:
3969 Dynamic probe (address, symbol name or combination
3970 of both in binary program; only valid for kernel domain).
3973 function entry/exit (address, symbol name or
3974 combination of both in binary program; only valid for kernel domain).
3977 System call entry/exit (only valid for kernel domain).
3979 When none of the above is specified, `enable-event` defaults to
3980 using `--tracepoint`.
3982 Condition 3 is specified using one of:
3985 Log level range from the specified level to the most severe
3991 See `lttng enable-event --help` for the complete list of log level
3994 Condition 4 is specified using the `--filter` option. This filter is
3995 a C-like expression, potentially reading real-time values of event
3996 fields, that has to evaluate to _true_ for the condition to be satisfied.
3997 Event fields are read using plain identifiers while context fields
3998 must be prefixed with `$ctx.`. See `lttng enable-event --help` for
4001 The aforementioned arguments are combined to create and enable events.
4002 Each unique combination of arguments leads to a different
4003 _enabled event_. The log level and filter arguments are optional, their
4004 default values being respectively all log levels and a filter which
4005 always returns _true_.
4007 Here are a few examples (you must
4008 <<creating-destroying-tracing-sessions,create a tracing session>>
4013 lttng enable-event -u --tracepoint my_app:hello_world
4014 lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_WARNING
4015 lttng enable-event -u --tracepoint 'my_other_app:*'
4016 lttng enable-event -u --tracepoint my_app:foo_bar \
4017 --filter 'some_field <= 23 && !other_field'
4018 lttng enable-event -k --tracepoint sched_switch
4019 lttng enable-event -k --tracepoint gpio_value
4020 lttng enable-event -k --function usb_probe_device usb_probe_device
4021 lttng enable-event -k --syscall --all
4024 The wildcard symbol, `*`, matches _anything_ and may only be used at
4025 the end of the string when specifying a _tracepoint_. Make sure to
4026 use it between single quotes in your favorite shell to avoid
4027 undesired shell expansion.
4029 You can see a list of events (enabled or disabled) using
4033 lttng list some-session
4036 where `some-session` is the name of the desired tracing session.
4038 What you're actually doing when enabling events with specific conditions
4039 is creating a **whitelist** of traceable events for a given channel.
4040 Thus, the following case presents redundancy:
4044 lttng enable-event -u --tracepoint my_app:hello_you
4045 lttng enable-event -u --tracepoint my_app:hello_you --loglevel TRACE_DEBUG
4048 The second command, matching a log level range, is useless since the first
4049 command enables all tracepoints matching the same name,
4052 Disabling an event is simpler: you only need to provide the event
4053 name to the `disable-event` command:
4057 lttng disable-event --userspace my_app:hello_you
4060 This name has to match a name previously given to `enable-event` (it
4061 has to be listed in the output of `lttng list some-session`).
4062 The `*` wildcard is supported, as long as you also used it in a
4063 previous `enable-event` invocation.
4065 Disabling an event does not add it to some blacklist: it simply removes
4066 it from its channel's whitelist. This is why you cannot disable an event
4067 which wasn't previously enabled.
4069 A disabled event will not generate any trace data, even if all its
4070 specified conditions are met.
4072 Events may be enabled and disabled at will, either when LTTng tracers
4073 are active or not. Events may be enabled before a user space application
4077 [[basic-tracing-session-control]]
4078 ==== Basic tracing session control
4081 <<creating-destroying-tracing-sessions,created a tracing session>>
4082 and <<enabling-disabling-events,enabled one or more events>>,
4083 you may activate the LTTng tracers for the current tracing session at
4091 Subsequently, you may stop the tracers:
4098 LTTng is very flexible: user space applications may be launched before
4099 or after the tracers are started. Events will only be recorded if they
4100 are properly enabled and if they occur while tracers are started.
4102 A tracing session name may be passed to both the `start` and `stop`
4103 commands to start/stop tracing a session other than the current one.
4106 [[enabling-disabling-channels]]
4107 ==== Enabling and disabling channels
4109 <<event,As mentioned>> in the
4110 <<understanding-lttng,Understanding LTTng>> chapter, enabled
4111 events are contained in a specific channel, itself contained in a
4112 specific tracing session. A channel is a group of events with
4113 tunable parameters (event loss mode, sub-buffer size, number of
4114 sub-buffers, trace file sizes and count, etc.). A given channel may
4115 only be responsible for enabled events belonging to one domain: either
4116 kernel or user space.
4118 If you only used the `create`, `enable-event` and `start`/`stop`
4119 commands of the `lttng` tool so far, one or two channels were
4120 automatically created for you (one for the kernel domain and/or one
4121 for the user space domain). The default channels are both named
4122 `channel0`; channels from different domains may have the same name.
4124 The current channels of a given tracing session can be viewed with
4128 lttng list some-session
4131 where `some-session` is the name of the desired tracing session.
4133 To create and enable a channel, use the `enable-channel` command:
4137 lttng enable-channel --kernel my-channel
4140 This will create a kernel domain channel named `my-channel` with
4141 default parameters in the current tracing session.
4145 Because of a current limitation, all
4146 channels must be _created_ prior to beginning tracing in a
4147 given tracing session, i.e. before the first time you do
4150 Since a channel is automatically created by
4151 `enable-event` only for the specified domain, you cannot,
4152 for example, enable a kernel domain event, start tracing and then
4153 enable a user space domain event because no user space channel
4154 exists yet and it's too late to create one.
4156 For this reason, make sure to configure your channels properly
4157 before starting the tracers for the first time!
4160 Here's another example:
4164 lttng enable-channel --userspace --session other-session --overwrite \
4165 --tracefile-size 1048576 1mib-channel
4168 This will create a user space domain channel named `1mib-channel` in
4169 the tracing session named `other-session` that loses new events by
4170 overwriting previously recorded events (instead of the default mode of
4171 discarding newer ones) and saves trace files with a maximum size of
4174 Note that channels may also be created using the `--channel` option of
4175 the `enable-event` command when the provided channel name doesn't exist
4176 for the specified domain:
4180 lttng enable-event --kernel --channel some-channel sched_switch
4183 If no kernel domain channel named `some-channel` existed before calling
4184 the above command, it would be created with default parameters.
4186 You may enable the same event in two different channels:
4190 lttng enable-event --userspace --channel my-channel app:tp
4191 lttng enable-event --userspace --channel other-channel app:tp
4194 If both channels are enabled, the occurring `app:tp` event will
4195 generate two recorded events, one for each channel.
4197 Disabling a channel is done with the `disable-event` command:
4201 lttng disable-event --kernel some-channel
4204 The state of a channel precedes the individual states of events within
4205 it: events belonging to a disabled channel, even if they are
4206 enabled, won't be recorded.
4210 [[fine-tuning-channels]]
4211 ===== Fine-tuning channels
4213 There are various parameters that may be fine-tuned with the
4214 `enable-channel` command. The latter are well documented in
4215 man:lttng(1) and in the <<channel,Channel>> section of the
4216 <<understanding-lttng,Understanding LTTng>> chapter. For basic
4217 tracing needs, their default values should be just fine, but here are a
4218 few examples to break the ice.
4220 As the frequency of recorded events increases--either because the
4221 event throughput is actually higher or because you enabled more events
4222 than usual—__event loss__ might be experienced. Since LTTng never
4223 waits, by design, for sub-buffer space availability (non-blocking
4224 tracer), when a sub-buffer is full and no empty sub-buffers are left,
4225 there are two possible outcomes: either the new events that do not fit
4226 are rejected, or they start replacing the oldest recorded events.
4227 The choice of which algorithm to use is a per-channel parameter, the
4228 default being discarding the newest events until there is some space
4229 left. If your situation always needs the latest events at the expense
4230 of writing over the oldest ones, create a channel with the `--overwrite`
4235 lttng enable-channel --kernel --overwrite my-channel
4238 When an event is lost, it means no space was available in any
4239 sub-buffer to accommodate it. Thus, if you want to cope with sporadic
4240 high event throughput situations and avoid losing events, you need to
4241 allocate more room for storing them in memory. This can be done by
4242 either increasing the size of sub-buffers or by adding sub-buffers.
4243 The following example creates a user space domain channel with
4244 16{nbsp}sub-buffers of 512{nbsp}kiB each:
4248 lttng enable-channel --userspace --num-subbuf 16 --subbuf-size 512k big-channel
4251 Both values need to be powers of two, otherwise they are rounded up
4254 Two other interesting available parameters of `enable-channel` are
4255 `--tracefile-size` and `--tracefile-count`, which respectively limit
4256 the size of each trace file and the their count for a given channel.
4257 When the number of written trace files reaches its limit for a given
4258 channel-CPU pair, the next trace file will overwrite the very first
4259 one. The following example creates a kernel domain channel with a
4260 maximum of three trace files of 1{nbsp}MiB each:
4264 lttng enable-channel --kernel --tracefile-size 1M --tracefile-count 3 my-channel
4267 An efficient way to make sure lots of events are generated is enabling
4268 all kernel events in this channel and starting the tracer:
4272 lttng enable-event --kernel --all --channel my-channel
4276 After a few seconds, look at trace files in your tracing session
4277 output directory. For two CPUs, it should look like:
4280 my-channel_0_0 my-channel_1_0
4281 my-channel_0_1 my-channel_1_1
4282 my-channel_0_2 my-channel_1_2
4285 Amongst the files above, you might see one in each group with a size
4286 lower than 1{nbsp}MiB: they are the files currently being written.
4288 Since all those small files are valid LTTng trace files, LTTng trace
4289 viewers may read them. It is the viewer's responsibility to properly
4290 merge the streams so as to present an ordered list to the user.
4291 http://diamon.org/babeltrace[Babeltrace]
4292 merges LTTng trace files correctly and is fast at doing it.
4296 ==== Adding some context to channels
4298 If you read all the sections of
4299 <<controlling-tracing,Controlling tracing>> so far, you should be
4300 able to create tracing sessions, create and enable channels and events
4301 within them and start/stop the LTTng tracers. Event fields recorded in
4302 trace files provide important information about occurring events, but
4303 sometimes external context may help you solve a problem faster. This
4304 section discusses how to add context information to events of a
4305 specific channel using the `lttng` tool.
4307 There are various available context values which can accompany events
4308 recorded by LTTng, for example:
4310 * **process information**:
4314 ** scheduling priority (niceness)
4315 ** thread identifier (TID)
4316 * the **hostname** of the system on which the event occurred
4317 * plenty of **performance counters** using perf:
4318 ** CPU cycles, stalled cycles, idle cycles, etc.
4320 ** branch instructions, misses, loads, etc.
4324 The full list is available in the output of `lttng add-context --help`.
4325 Some of them are reserved for a specific domain (kernel or
4326 user space) while others are available for both.
4328 To add context information to one or all channels of a given tracing
4329 session, use the `add-context` command:
4333 lttng add-context --userspace --type vpid --type perf:thread:cpu-cycles
4336 The above example adds the virtual process identifier and per-thread
4337 CPU cycles count values to all recorded user space domain events of the
4338 current tracing session. Use the `--channel` option to select a specific
4343 lttng add-context --kernel --channel my-channel --type tid
4346 adds the thread identifier value to all recorded kernel domain events
4347 in the channel `my-channel` of the current tracing session.
4349 Beware that context information cannot be removed from channels once
4350 it's added for a given tracing session.
4353 [[saving-loading-tracing-session]]
4354 ==== Saving and loading tracing session configurations
4356 Configuring a tracing session may be long: creating and enabling
4357 channels with specific parameters, enabling kernel and user space
4358 domain events with specific log levels and filters, adding context
4359 to some channels, etc. If you're going to use LTTng to solve real
4360 world problems, chances are you're going to have to record events using
4361 the same tracing session setup over and over, modifying a few variables
4362 each time in your instrumented program or environment. To avoid
4363 constant tracing session reconfiguration, the `lttng` tool is able to
4364 save and load tracing session configurations to/from XML files.
4366 To save a given tracing session configuration, do:
4370 lttng save my-session
4373 where `my-session` is the name of the tracing session to save. Tracing
4374 session configurations are saved to dir:{~/.lttng/sessions} by default;
4375 use the `--output-path` option to change this destination directory.
4377 All configuration parameters are saved:
4379 * tracing session name
4380 * trace data output path
4381 * channels with their state and all their parameters
4382 * context information added to channels
4383 * events with their state, log level and filter
4384 * tracing activity (started or stopped)
4386 To load a tracing session, simply do:
4390 lttng load my-session
4393 or, if you used a custom path:
4397 lttng load --input-path /path/to/my-session.lttng
4400 Your saved tracing session will be restored as if you just configured
4404 [[sending-trace-data-over-the-network]]
4405 ==== Sending trace data over the network
4407 The possibility of sending trace data over the network comes as a
4408 built-in feature of LTTng-tools. For this to be possible, an LTTng
4409 _relay daemon_ must be executed and listening on the machine where
4410 trace data is to be received, and the user must create a tracing
4411 session using appropriate options to forward trace data to the remote
4414 The relay daemon listens on two different TCP ports: one for control
4415 information and the other for actual trace data.
4417 Starting the relay daemon on the remote machine is as easy as:
4424 This will make it listen to its default ports: 5342 for control and
4425 5343 for trace data. The `--control-port` and `--data-port` options may
4426 be used to specify different ports.
4428 Traces written by `lttng-relayd` are written to
4429 +\~/lttng-traces/__hostname__/__session__+ by
4430 default, where +__hostname__+ is the host name of the
4431 traced (monitored) system and +__session__+ is the
4432 tracing session name. Use the `--output` option to write trace data
4433 outside dir:{~/lttng-traces}.
4435 On the sending side, a tracing session must be created using the
4436 `lttng` tool with the `--set-url` option to connect to the distant
4441 lttng create my-session --set-url net://distant-host
4444 The URL format is described in the output of `lttng create --help`.
4445 The above example will use the default ports; the `--ctrl-url` and
4446 `--data-url` options may be used to set the control and data URLs
4449 Once this basic setup is completed and the connection is established,
4450 you may use the `lttng` tool on the target machine as usual; everything
4451 you do will be transparently forwarded to the remote machine if needed.
4452 For example, a parameter changing the maximum size of trace files will
4453 have an effect on the distant relay daemon actually writing the trace.
4457 ==== Viewing events as they arrive
4459 We have seen how trace files may be produced by LTTng out of generated
4460 application and Linux kernel events. We have seen that those trace files
4461 may be either recorded locally by consumer daemons or remotely using
4462 a relay daemon. And we have seen that the maximum size and count of
4463 trace files is configurable for each channel. With all those features,
4464 it's still not possible to read a trace file as it is being written
4465 because it could be incomplete and appear corrupted to the viewer.
4466 There is a way to view events as they arrive, however: using
4469 LTTng live is implemented, in LTTng, solely on the relay daemon side.
4470 As trace data is sent over the network to a relay daemon by a (possibly
4471 remote) consumer daemon, a _tee_ may be created: trace data will be
4472 recorded to trace files _as well as_ being transmitted to a
4473 connected live viewer:
4476 .LTTng live and the relay daemon.
4477 image::lttng-live-relayd.png[]
4479 In order to use this feature, a tracing session must created in live
4480 mode on the target system:
4487 An optional parameter may be passed to `--live` to set the interval
4488 of time (in microseconds) between flushes to the network
4489 (1{nbsp}second is the default):
4493 lttng create --live 100000
4496 will flush every 100{nbsp}ms.
4498 If no network output is specified to the `create` command, a local
4499 relay daemon will be spawned. In this very common case, viewing a live
4500 trace is easy: enable events and start tracing as usual, then use
4501 `lttng view` to start the default live viewer:
4508 The correct arguments will be passed to the live viewer so that it
4509 may connect to the local relay daemon and start reading live events.
4511 You may also wish to use a live viewer not running on the target
4512 system. In this case, you should specify a network output when using
4513 the `create` command (`--set-url` or `--ctrl-url`/`--data-url` options).
4514 A distant LTTng relay daemon should also be started to receive control
4515 and trace data. By default, `lttng-relayd` listens on 127.0.0.1:5344
4516 for an LTTng live connection. Otherwise, the desired URL may be
4517 specified using its `--live-port` option.
4520 http://diamon.org/babeltrace[`babeltrace`]
4521 viewer supports LTTng live as one of its input formats. `babeltrace` is
4522 the default viewer when using `lttng view`. To use it manually, first
4523 list active tracing sessions by doing the following (assuming the relay
4524 daemon to connect to runs on the same host):
4528 babeltrace --input-format lttng-live net://localhost
4531 Then, choose a tracing session and start viewing events as they arrive
4532 using LTTng live, e.g.:
4536 babeltrace --input-format lttng-live net://localhost/host/hostname/my-session
4540 [[taking-a-snapshot]]
4541 ==== Taking a snapshot
4543 The normal behavior of LTTng is to record trace data as trace files.
4544 This is ideal for keeping a long history of events that occurred on
4545 the target system and applications, but may be too much data in some
4546 situations. For example, you may wish to trace your application
4547 continuously until some critical situation happens, in which case you
4548 would only need the latest few recorded events to perform the desired
4549 analysis, not multi-gigabyte trace files.
4551 LTTng has an interesting feature called _snapshots_. When creating
4552 a tracing session in snapshot mode, no trace files are written; the
4553 tracers' sub-buffers are constantly overwriting the oldest recorded
4554 events with the newest. At any time, either when the tracers are started
4555 or stopped, you may take a snapshot of those sub-buffers.
4557 There is no difference between the format of a normal trace file and the
4558 format of a snapshot: viewers of LTTng traces will also support LTTng
4559 snapshots. By default, snapshots are written to disk, but they may also
4560 be sent over the network.
4562 To create a tracing session in snapshot mode, do:
4566 lttng create --snapshot my-snapshot-session
4569 Next, enable channels, events and add context to channels as usual.
4570 Once a tracing session is created in snapshot mode, channels will be
4572 <<channel-overwrite-mode-vs-discard-mode,overwrite>> mode
4573 (`--overwrite` option of the `enable-channel` command; also called
4574 _flight recorder mode_) and have an `mmap()` channel type
4577 Start tracing. When you're ready to take a snapshot, do:
4581 lttng snapshot record --name my-snapshot
4584 This will record a snapshot named `my-snapshot` of all channels of
4585 all domains of the current tracing session. By default, snapshots files
4586 are recorded in the path returned by `lttng snapshot list-output`. You
4587 may change this path or decide to send snapshots over the network
4590 . an output path/URL specified when creating the tracing session
4592 . an added snapshot output path/URL using
4593 `lttng snapshot add-output`
4594 . an output path/URL provided directly to the
4595 `lttng snapshot record` command
4597 Method 3 overrides method 2 which overrides method 1. When specifying
4598 a URL, a relay daemon must be listening on some machine (see
4599 <<sending-trace-data-over-the-network,Sending trace data over the network>>).
4601 If you need to make absolutely sure that the output file won't be
4602 larger than a certain limit, you can set a maximum snapshot size when
4603 taking it with the `--max-size` option:
4607 lttng snapshot record --name my-snapshot --max-size 2M
4610 Older recorded events will be discarded in order to respect this
4617 This chapter presents various references for LTTng packages such as links
4618 to online manpages, tables needed by the rest of the text, descriptions
4619 of library functions, etc.
4622 [[online-lttng-manpages]]
4623 === Online LTTng manpages
4625 LTTng packages currently install the following manpages, available
4626 online using the links below:
4630 ** man:lttng-sessiond(8)
4631 ** man:lttng-relayd(8)
4633 ** man:lttng-gen-tp(1)
4635 ** man:lttng-ust-cyg-profile(3)
4636 ** man:lttng-ust-dl(3)
4642 This section presents references of the LTTng-UST package.
4646 ==== LTTng-UST library (+liblttng‑ust+)
4648 The LTTng-UST library, or `liblttng-ust`, is the main shared object
4649 against which user applications are linked to make LTTng user space
4652 The <<c-application,C application>> guide shows the complete
4653 process to instrument, build and run a C/$$C++$$ application using
4654 LTTng-UST, while this section contains a few important tables.
4657 [[liblttng-ust-tp-fields]]
4658 ===== Tracepoint fields macros (for `TP_FIELDS()`)
4660 The available macros to define tracepoint fields, which should be listed
4661 within `TP_FIELDS()` in `TRACEPOINT_EVENT()`, are:
4663 [role="growable func-desc",cols="asciidoc,asciidoc"]
4664 .Available macros to define LTTng-UST tracepoint fields
4666 |Macro |Description and parameters
4669 +ctf_integer(__t__, __n__, __e__)+
4671 +ctf_integer_nowrite(__t__, __n__, __e__)+
4673 Standard integer, displayed in base 10.
4676 Integer C type (`int`, `long`, `size_t`, etc.).
4682 Argument expression.
4684 |+ctf_integer_hex(__t__, __n__, __e__)+
4686 Standard integer, displayed in base 16.
4695 Argument expression.
4697 |+ctf_integer_network(__t__, __n__, __e__)+
4699 Integer in network byte order (big endian), displayed in base 10.
4708 Argument expression.
4710 |+ctf_integer_network_hex(__t__, __n__, __e__)+
4712 Integer in network byte order, displayed in base 16.
4721 Argument expression.
4724 +ctf_float(__t__, __n__, __e__)+
4726 +ctf_float_nowrite(__t__, __n__, __e__)+
4728 Floating point number.
4731 Floating point number C type (`float` or `double`).
4737 Argument expression.
4740 +ctf_string(__n__, __e__)+
4742 +ctf_string_nowrite(__n__, __e__)+
4744 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
4750 Argument expression.
4753 +ctf_array(__t__, __n__, __e__, __s__)+
4755 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
4757 Statically-sized array of integers
4760 Array element C type.
4766 Argument expression.
4772 +ctf_array_text(__t__, __n__, __e__, __s__)+
4774 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
4776 Statically-sized array, printed as text.
4778 The string does not need to be null-terminated.
4781 Array element C type (always `char`).
4787 Argument expression.
4793 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
4795 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
4797 Dynamically-sized array of integers.
4799 The type of +__E__+ needs to be unsigned.
4802 Array element C type.
4808 Argument expression.
4811 Length expression C type.
4817 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
4819 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
4821 Dynamically-sized array, displayed as text.
4823 The string does not need to be null-terminated.
4825 The type of +__E__+ needs to be unsigned.
4827 The behaviour is undefined if +__e__+ is `NULL`.
4830 Sequence element C type (always `char`).
4836 Argument expression.
4839 Length expression C type.
4845 The `_nowrite` versions omit themselves from the session trace, but are
4846 otherwise identical. This means the `_nowrite` fields won't be written
4847 in the recorded trace. Their primary purpose is to make some
4848 of the event context available to the
4849 <<enabling-disabling-events,event filters>> without having to
4850 commit the data to sub-buffers.
4853 [[liblttng-ust-tracepoint-loglevel]]
4854 ===== Tracepoint log levels (for `TRACEPOINT_LOGLEVEL()`)
4856 The following table shows the available log level values for the
4857 `TRACEPOINT_LOGLEVEL()` macro:
4863 Action must be taken immediately.
4866 Critical conditions.
4875 Normal, but significant, condition.
4878 Informational message.
4880 `TRACE_DEBUG_SYSTEM`::
4881 Debug information with system-level scope (set of programs).
4883 `TRACE_DEBUG_PROGRAM`::
4884 Debug information with program-level scope (set of processes).
4886 `TRACE_DEBUG_PROCESS`::
4887 Debug information with process-level scope (set of modules).
4889 `TRACE_DEBUG_MODULE`::
4890 Debug information with module (executable/library) scope (set of units).
4892 `TRACE_DEBUG_UNIT`::
4893 Debug information with compilation unit scope (set of functions).
4895 `TRACE_DEBUG_FUNCTION`::
4896 Debug information with function-level scope.
4898 `TRACE_DEBUG_LINE`::
4899 Debug information with line-level scope (TRACEPOINT_EVENT default).
4902 Debug-level message.
4904 Log levels `TRACE_EMERG` through `TRACE_INFO` and `TRACE_DEBUG` match
4905 http://man7.org/linux/man-pages/man3/syslog.3.html[syslog]
4906 level semantics. Log levels `TRACE_DEBUG_SYSTEM` through `TRACE_DEBUG`
4907 offer more fine-grained selection of debug information.
4910 [[lttng-modules-ref]]
4913 This section presents references of the LTTng-modules package.
4916 [[lttng-modules-tp-struct-entry]]
4917 ==== Tracepoint fields macros (for `TP_STRUCT__entry()`)
4919 This table describes possible entries for the `TP_STRUCT__entry()` part
4920 of `LTTNG_TRACEPOINT_EVENT()`:
4922 [role="growable func-desc",cols="asciidoc,asciidoc"]
4923 .Available entries for `TP_STRUCT__entry()` (in `LTTNG_TRACEPOINT_EVENT()`)
4925 |Macro |Description and parameters
4927 |+\__field(__t__, __n__)+
4929 Standard integer, displayed in base 10.
4932 Integer C type (`int`, `unsigned char`, `size_t`, etc.).
4937 |+\__field_hex(__t__, __n__)+
4939 Standard integer, displayed in base 16.
4947 |+\__field_oct(__t__, __n__)+
4949 Standard integer, displayed in base 8.
4957 |+\__field_network(__t__, __n__)+
4959 Integer in network byte order (big endian), displayed in base 10.
4967 |+\__field_network_hex(__t__, __n__)+
4969 Integer in network byte order (big endian), displayed in base 16.
4977 |+\__array(__t__, __n__, __s__)+
4979 Statically-sized array, elements displayed in base 10.
4982 Array element C type.
4990 |+\__array_hex(__t__, __n__, __s__)+
4992 Statically-sized array, elements displayed in base 16.
4995 array element C type.
5001 |+\__array_text(__t__, __n__, __s__)+
5003 Statically-sized array, displayed as text.
5006 Array element C type (always char).
5014 |+\__dynamic_array(__t__, __n__, __s__)+
5016 Dynamically-sized array, displayed in base 10.
5019 Array element C type.
5025 Length C expression.
5027 |+\__dynamic_array_hex(__t__, __n__, __s__)+
5029 Dynamically-sized array, displayed in base 16.
5032 Array element C type.
5038 Length C expression.
5040 |+\__dynamic_array_text(__t__, __n__, __s__)+
5042 Dynamically-sized array, displayed as text.
5045 Array element C type (always char).
5051 Length C expression.
5053 |+\__string(n, __s__)+
5055 Null-terminated string.
5057 The behaviour is undefined behavior if +__s__+ is `NULL`.
5063 String source (pointer).
5066 The above macros should cover the majority of cases. For advanced items,
5067 see path:{probes/lttng-events.h}.
5070 [[lttng-modules-tp-fast-assign]]
5071 ==== Tracepoint assignment macros (for `TP_fast_assign()`)
5073 This table describes possible entries for the `TP_fast_assign()` part
5074 of `LTTNG_TRACEPOINT_EVENT()`:
5076 .Available entries for `TP_fast_assign()` (in `LTTNG_TRACEPOINT_EVENT()`)
5077 [role="growable func-desc",cols="asciidoc,asciidoc"]
5079 |Macro |Description and parameters
5081 |+tp_assign(__d__, __s__)+
5083 Assignment of C expression +__s__+ to tracepoint field +__d__+.
5086 Name of destination tracepoint field.
5089 Source C expression (may refer to tracepoint arguments).
5091 |+tp_memcpy(__d__, __s__, __l__)+
5093 Memory copy of +__l__+ bytes from +__s__+ to tracepoint field
5094 +__d__+ (use with array fields).
5097 Name of destination tracepoint field.
5100 Source C expression (may refer to tracepoint arguments).
5103 Number of bytes to copy.
5105 |+tp_memcpy_from_user(__d__, __s__, __l__)+
5107 Memory copy of +__l__+ bytes from user space +__s__+ to tracepoint
5108 field +__d__+ (use with array fields).
5111 Name of destination tracepoint field.
5114 Source C expression (may refer to tracepoint arguments).
5117 Number of bytes to copy.
5119 |+tp_memcpy_dyn(__d__, __s__)+
5121 Memory copy of dynamically-sized array from +__s__+ to tracepoint field
5124 The number of bytes is known from the field's length expression
5125 (use with dynamically-sized array fields).
5128 Name of destination tracepoint field.
5131 Source C expression (may refer to tracepoint arguments).
5134 Number of bytes to copy.
5136 |+tp_strcpy(__d__, __s__)+
5138 String copy of +__s__+ to tracepoint field +__d__+ (use with string
5142 Name of destination tracepoint field.
5145 Source C expression (may refer to tracepoint arguments).