2.12: use the "of" possessive for inanimate things and abstract ideas
[lttng-docs.git] / 2.12 / lttng-docs-2.12.txt
CommitLineData
96172a59
PP
1The LTTng Documentation
2=======================
3Philippe Proulx <pproulx@efficios.com>
05bbb80b 4v2.12, 14 January 2021
96172a59
PP
5
6
7include::../common/copyright.txt[]
8
9
10include::../common/welcome.txt[]
11
12
13include::../common/audience.txt[]
14
15
16[[chapters]]
17=== What's in this documentation?
18
19The LTTng Documentation is divided into the following sections:
20
21* **<<nuts-and-bolts,Nuts and bolts>>** explains the
22 rudiments of software tracing and the rationale behind the
23 LTTng project.
24+
25Skip this section if you’re familiar with software tracing and with the
26LTTng project.
27
28* **<<installing-lttng,Installation>>** describes the steps to
29 install the LTTng packages on common Linux distributions and from
30 their sources.
31+
32Skip this section if you already properly installed LTTng on your target
33system.
34
35* **<<getting-started,Quick start>>** is a concise guide to
36 getting started quickly with LTTng kernel and user space tracing.
37+
38We recommend this section if you're new to LTTng or to software tracing
39in general.
40+
41Skip this section if you're not new to LTTng.
42
43* **<<core-concepts,Core concepts>>** explains the concepts at
44 the heart of LTTng.
45+
46It's a good idea to become familiar with the core concepts
47before attempting to use the toolkit.
48
49* **<<plumbing,Components of LTTng>>** describes the various components
50 of the LTTng machinery, like the daemons, the libraries, and the
51 command-line interface.
52* **<<instrumenting,Instrumentation>>** shows different ways to
53 instrument user applications and the Linux kernel.
54+
55Instrumenting source code is essential to provide a meaningful
56source of events.
57+
58Skip this section if you don't have a programming background.
59
60* **<<controlling-tracing,Tracing control>>** is divided into topics
61 which demonstrate how to use the vast array of features that
62 LTTng{nbsp}{revision} offers.
63* **<<reference,Reference>>** contains reference tables.
64* **<<glossary,Glossary>>** is a specialized dictionary of terms related
65 to LTTng or to the field of software tracing.
66
67
68include::../common/convention.txt[]
69
70
71include::../common/acknowledgements.txt[]
72
73
74[[whats-new]]
75== What's new in LTTng{nbsp}{revision}?
76
77LTTng{nbsp}{revision} bears the name _Ta Meilleure_, a Northeast IPA
78beer brewed by https://lagabiere.com/[Lagabière]. Translating to ``Your
79best one'', this beer gives out strong aromas of passion fruit, lemon,
80and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a
81smooth lingering hop bitterness.
82
83New features and changes in LTTng{nbsp}{revision}:
84
85Tracing control::
86+
87* Clear the contents of one or more <<tracing-session,tracing sessions>>
88 without having to destroy and reconfigure them
89 with the new man:lttng-clear(1) command.
90+
05bbb80b 91This is especially useful to clear the tracing data of a tracing session
96172a59
PP
92between attempts to reproduce a problem.
93+
94See <<clear,Clear a tracing session>>.
95
96* Before LTTng{nbsp}{revision}, the man:lttng-track(1) and
97 man:lttng-untrack(1) commands used to add and remove process IDs
98 (PIDs) to a whitelist so that LTTng would only trace processes with
99 specific PIDs.
100+
101LTTng{nbsp}{revision} adds Unix user IDs (UIDs) and Unix group IDs
102(GIDs) to the available <<pid-tracking,process attributes to track>>.
103You can specify numeric user/group IDs and user/group names to track,
104for example:
105+
106[role="term"]
107----
108$ lttng track --userspace --vuid=http,999 --vgid=mysql,9
109----
110+
111While you can also track UIDs and GIDs with the
112opt:lttng-enable-event(1):--filter option of the `enable-event` command,
113this dedicated process attribute tracking approach reduces tracing
114overhead and prevents the creation of <<def-sub-buffer,sub-buffers>> for
115the users and groups which LTTng doesn't track.
116+
117In the command manual pages, the term ``whitelist'' is renamed to
118``inclusion set'' to clarify the concept.
119
120* The <<lttng-relayd,relay daemon>> can now maintain many files
121 virtually opened without using as many file descriptors (FD). It does
122 so by closing and reopening FDs as needed.
123+
124This feature is meant as a workaround for users who can't bump the
125system limit because of permission restrictions.
126+
127The new opt:lttng-relayd(8):--fd-pool-size relay daemon option
128sets the maximum number of simultaneously opened file descriptors
129(using the soft `RLIMIT_NOFILE` resource limit of the process by
130default; see man:getrlimit(2)).
131
132* By default, the relay daemon writes its traces under a predefined
133 directory hierarchy,
134 +$LTTNG_HOME/lttng-traces/__host__/__session__/__domain__+, with:
135+
136--
137+__host__+::
138 Remote hostname.
139
140+__session__+::
141 <<tracing-session,Tracing session>> name.
142
143+__domain__+::
144 <<domain,Tracing domain>> name (`ust` or `kernel`).
145--
146+
147Change this hierarchy to group traces by tracing session name rather
148than by hostname
149(+$LTTNG_HOME/lttng-traces/__session__/__host__/__domain__+) with the
05bbb80b
PP
150new opt:lttng-relayd(8):--group-output-by-session option of the
151relay daemon.
96172a59
PP
152+
153This feature is especially useful if you're tracing two or more hosts,
154having different hostnames, which share the same tracing session name as
155part of their configuration. In this scenario, you can use a descriptive
156tracing session name (for example, `connection-hang`) across a fleet of
157machines streaming to a single relay daemon.
158
159* The relay daemon has a new opt:lttng-relayd(8):--working-directory
160 option to override its working directory.
161
162Linux kernel tracing::
163+
164* New instrumentation hooks to trace the entry and exit tracepoints of
165 the network reception code paths of the Linux kernel.
166+
167Use the resulting event records to identify the bounds of a network
168reception and link the events that occur in the interim (for example,
169wake-ups) to a specific network reception instance. You can also
05bbb80b 170analyze the latency of the network stack thanks to those event records.
96172a59 171
05bbb80b 172* The `thread` field of the `irqaction` structure, which specifies the
96172a59
PP
173 process to wake up when a threaded interrupt request (IRQ) occurs, is
174 now part of the `lttng_statedump_interrupt` event record.
175+
176Use this information to discover which processes handle the various
177IRQs. You can also associate the events occurring in the context of
178those processes with their respective IRQ.
179
180* New `lttng_statedump_cpu_topology` tracepoint to record the active
181 CPU/NUMA topology.
182+
183Use this information to discover which CPUs are SMT siblings or part of
184the same socket. As of LTTng{nbsp}{revision}, only the x86 architecture
185is supported since all architectures describe their topologies
186differently.
187+
05bbb80b
PP
188The `architecture` field of the tracepoint is statically defined and
189exists for all architecture implementations. Analysis tools can
190therefore anticipate the layout of the event record.
96172a59
PP
191+
192Event record example:
193+
194[source,yaml]
195----
196lttng_statedump_cpu_topology:
197 architecture: x86
198 cpu_id: 0
199 vendor: GenuineIntel
200 family: 6
201 model: 142
202 model_name: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
203 physical_id: 0
204 core_id: 0
205 cores: 2
206----
207
208* New product UUID metadata environment field, `product_uuid`,
209 which LTTng copies from the
210 https://en.wikipedia.org/wiki/Desktop_Management_Interface[Desktop
211 Management Interface] (DMI).
212+
213Use this environment field to uniquely identify a machine (virtual or
214physical) in order to correlate traces from multiple virtual machines.
215
216
217[[nuts-and-bolts]]
218== Nuts and bolts
219
220What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
221generation_ is a modern toolkit for tracing Linux systems and
222applications. So your first question might be:
223**what is tracing?**
224
225
226[[what-is-tracing]]
227=== What is tracing?
228
229As the history of software engineering progressed and led to what
230we now take for granted--complex, numerous and
231interdependent software applications running in parallel on
232sophisticated operating systems like Linux--the authors of such
233components, software developers, began feeling a natural
234urge to have tools that would ensure the robustness and good performance
235of their masterpieces.
236
237One major achievement in this field is, inarguably, the
238https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
239an essential tool for developers to find and fix bugs. But even the best
240debugger won't help make your software run faster, and nowadays, faster
241software means either more work done by the same hardware, or cheaper
242hardware for the same work.
243
244A _profiler_ is often the tool of choice to identify performance
245bottlenecks. Profiling is suitable to identify _where_ performance is
246lost in a given software. The profiler outputs a profile, a statistical
247summary of observed events, which you may use to discover which
248functions took the most time to execute. However, a profiler won't
249report _why_ some identified functions are the bottleneck. Bottlenecks
250might only occur when specific conditions are met, conditions that are
251sometimes impossible to capture by a statistical profiler, or impossible
252to reproduce with an application altered by the overhead of an
253event-based profiler. For a thorough investigation of software
254performance issues, a history of execution is essential, with the
255recorded values of variables and context fields you choose, and
256with as little influence as possible on the instrumented software. This
257is where tracing comes in handy.
258
259_Tracing_ is a technique used to understand what goes on in a running
260software system. The software used for tracing is called a _tracer_,
261which is conceptually similar to a tape recorder. When recording,
262specific instrumentation points placed in the software source code
263generate events that are saved on a giant tape: a _trace_ file. You
264can trace user applications and the operating system at the same time,
265opening the possibility of resolving a wide range of problems that would
266otherwise be extremely challenging.
267
268Tracing is often compared to _logging_. However, tracers and loggers are
269two different tools, serving two different purposes. Tracers are
270designed to record much lower-level events that occur much more
271frequently than log messages, often in the range of thousands per
272second, with very little execution overhead. Logging is more appropriate
273for a very high-level analysis of less frequent events: user accesses,
274exceptional conditions (errors and warnings, for example), database
275transactions, instant messaging communications, and such. Simply put,
276logging is one of the many use cases that can be satisfied with tracing.
277
278The list of recorded events inside a trace file can be read manually
279like a log file for the maximum level of detail, but it is generally
280much more interesting to perform application-specific analyses to
281produce reduced statistics and graphs that are useful to resolve a
282given problem. Trace viewers and analyzers are specialized tools
283designed to do this.
284
285In the end, this is what LTTng is: a powerful, open source set of
286tools to trace the Linux kernel and user applications at the same time.
287LTTng is composed of several components actively maintained and
288developed by its link:/community/#where[community].
289
290
291[[lttng-alternatives]]
292=== Alternatives to noch:{LTTng}
293
294Excluding proprietary solutions, a few competing software tracers
295exist for Linux:
296
297https://github.com/dtrace4linux/linux[dtrace4linux]::
05bbb80b 298 A port of Sun Microsystems' DTrace to Linux.
96172a59
PP
299+
300The cmd:dtrace tool interprets user scripts and is responsible for
301loading code into the Linux kernel for further execution and collecting
302the outputted data.
303
304https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
305 A subsystem in the Linux kernel in which a virtual machine can
306 execute programs passed from the user space to the kernel.
307+
308You can attach such programs to tracepoints and kprobes thanks to a
309system call, and they can output data to the user space when executed
310thanks to different mechanisms (pipe, VM register values, and eBPF maps,
311to name a few).
312
313https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
314 The de facto function tracer of the Linux kernel.
315+
316Its user interface is a set of special files in sysfs.
317
318https://perf.wiki.kernel.org/[perf]::
319 A performance analysis tool for Linux which supports hardware
320 performance counters, tracepoints, as well as other counters and
321 types of probes.
322+
05bbb80b
PP
323The controlling utility of perf is the cmd:perf command line/text UI
324tool.
96172a59
PP
325
326http://linux.die.net/man/1/strace[strace]::
327 A command-line utility which records system calls made by a
328 user process, as well as signal deliveries and changes of process
329 state.
330+
331strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
332fulfill its function.
333
334http://www.sysdig.org/[sysdig]::
335 Like SystemTap, uses scripts to analyze Linux kernel events.
336+
05bbb80b
PP
337You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
338sysdig executes them while it traces the system or afterwards. The
339interface of sysdig is the cmd:sysdig command-line tool as well as the
340text UI-based cmd:csysdig tool.
96172a59
PP
341
342https://sourceware.org/systemtap/[SystemTap]::
343 A Linux kernel and user space tracer which uses custom user scripts
344 to produce plain text traces.
345+
346SystemTap converts the scripts to the C language, and then compiles them
05bbb80b
PP
347as Linux kernel modules which are loaded to produce trace data. The
348primary user interface of SystemTap is the cmd:stap command-line tool.
96172a59
PP
349
350The main distinctive features of LTTng is that it produces correlated
351kernel and user space traces, as well as doing so with the lowest
352overhead amongst other solutions. It produces trace files in the
353http://diamon.org/ctf[CTF] format, a file format optimized
354for the production and analyses of multi-gigabyte data.
355
356LTTng is the result of more than 10{nbsp}years of active open source
357development by a community of passionate developers. LTTng is currently
358available on major desktop and server Linux distributions.
359
360The main interface for tracing control is a single command-line tool
361named cmd:lttng. The latter can create several tracing sessions, enable
362and disable events on the fly, filter events efficiently with custom
363user expressions, start and stop tracing, and much more. LTTng can
364record the traces on the file system or send them over the network, and
365keep them totally or partially. You can view the traces once tracing
366becomes inactive or in real-time.
367
368<<installing-lttng,Install LTTng now>> and
369<<getting-started,start tracing>>!
370
371
372[[installing-lttng]]
373== Installation
374
375**LTTng** is a set of software <<plumbing,components>> which interact to
376<<instrumenting,instrument>> the Linux kernel and user applications, and
377to <<controlling-tracing,control tracing>> (start and stop
378tracing, enable and disable event rules, and the rest). Those
379components are bundled into the following packages:
380
381LTTng-tools::
382 Libraries and command-line interface to control tracing.
383
384LTTng-modules::
385 Linux kernel modules to instrument and trace the kernel.
386
387LTTng-UST::
388 Libraries and Java/Python packages to instrument and trace user
389 applications.
390
391Most distributions mark the LTTng-modules and LTTng-UST packages as
392optional when installing LTTng-tools (which is always required). In the
393following sections, we always provide the steps to install all three,
394but note that:
395
396* You only need to install LTTng-modules if you intend to trace the
397 Linux kernel.
398* You only need to install LTTng-UST if you intend to trace user
399 applications.
400
76a080e3 401[role="growable"]
33211adb 402.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 5{nbsp}August{nbsp}2020.
76a080e3
PP
403|====
404|Distribution |Available in releases
96172a59 405
76a080e3
PP
406|https://www.ubuntu.com/[Ubuntu]
407|Ubuntu{nbsp}16.04 _Xenial Xerus_, Ubuntu{nbsp}18.04 _Bionic Beaver_,
408and Ubuntu{nbsp}20.04 _Focal Fossa_:
409<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
410
411|https://www.debian.org/[Debian]
412|<<debian,Debian "bullseye" (testing)>>.
413
414|https://www.archlinux.org/[Arch Linux]
415|<<arch-linux,_Community_ repository and AUR>>.
416
417|https://alpinelinux.org/[Alpine Linux]
418|<<alpine-linux,Alpine Linux{nbsp}3.12>>.
419
420|https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
421|See http://packages.efficios.com/[EfficiOS Enterprise Packages].
422|====
423
424
425[[ubuntu-ppa]]
426=== Ubuntu: noch:{LTTng} Stable {revision} PPA
427
428The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
429Stable{nbsp}{revision} PPA] offers the latest stable
430LTTng{nbsp}{revision} packages for Ubuntu{nbsp}16.04 _Xenial Xerus_,
431Ubuntu{nbsp}18.04 _Bionic Beaver_, and Ubuntu{nbsp}20.04 _Focal Fossa_.
432
433To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
434PPA:
435
436. Add the LTTng Stable{nbsp}{revision} PPA repository and update the
437 list of packages:
438+
439--
440[role="term"]
441----
442# apt-add-repository ppa:lttng/stable-2.12
443# apt-get update
444----
445--
446
447. Install the main LTTng{nbsp}{revision} packages:
448+
449--
450[role="term"]
451----
452# apt-get install lttng-tools
453# apt-get install lttng-modules-dkms
454# apt-get install liblttng-ust-dev
455----
456--
457
458. **If you need to instrument and trace
459 <<java-application,Java applications>>**, install the LTTng-UST
460 Java agent:
461+
462--
463[role="term"]
464----
465# apt-get install liblttng-ust-agent-java
466----
467--
468
469. **If you need to instrument and trace
470 <<python-application,Python{nbsp}3 applications>>**, install the
471 LTTng-UST Python agent:
472+
473--
474[role="term"]
475----
476# apt-get install python3-lttngust
477----
478--
479
480
481[[debian]]
482=== Debian
483
484To install LTTng{nbsp}{revision} on Debian "bullseye" (testing):
485
486. Install the main LTTng{nbsp}{revision} packages:
487+
488--
489[role="term"]
490----
491# apt-get install lttng-modules-dkms
492# apt-get install liblttng-ust-dev
493# apt-get install lttng-tools
494----
495--
496
497. **If you need to instrument and trace <<java-application,Java
498 applications>>**, install the LTTng-UST Java agent:
499+
500--
501[role="term"]
502----
503# apt-get install liblttng-ust-agent-java
504----
505--
506
507. **If you need to instrument and trace <<python-application,Python
508 applications>>**, install the LTTng-UST Python agent:
509+
510--
511[role="term"]
512----
513# apt-get install python3-lttngust
514----
515--
96172a59
PP
516
517
518[[arch-linux]]
519=== Arch Linux
520
05bbb80b
PP
521LTTng-UST{nbsp}{revision} is available in the _community_
522repository of Arch Linux, while LTTng-tools{nbsp}{revision} and
96172a59
PP
523LTTng-modules{nbsp}{revision} are available in the
524https://aur.archlinux.org/[AUR].
525
526To install LTTng{nbsp}{revision} on Arch Linux, using
527https://github.com/Jguer/yay[yay] for the AUR packages:
528
529. Install the main LTTng{nbsp}{revision} packages:
530+
531--
532[role="term"]
533----
534# pacman -Sy lttng-ust
535$ yay -Sy lttng-tools
536$ yay -Sy lttng-modules
537----
538--
539
540. **If you need to instrument and trace <<python-application,Python
541 applications>>**, install the LTTng-UST Python agent:
542+
543--
544[role="term"]
545----
546# pacman -Sy python-lttngust
547# pacman -Sy python2-lttngust
548----
549--
550
551
76a080e3
PP
552[[alpine-linux]]
553=== Alpine Linux
554
555To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
556Alpine Linux{nbsp}3.12:
557
558. Add the LTTng packages:
559+
560--
561[role="term"]
562----
563# apk add lttng-tools
564# apk add lttng-ust-dev
565----
566--
567
568. Download, build, and install the latest LTTng-modules{nbsp}{revision}:
569+
570--
571[role="term"]
572----
573$ cd $(mktemp -d) &&
574wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
575tar -xf lttng-modules-latest-2.12.tar.bz2 &&
576cd lttng-modules-2.12.* &&
577make &&
578sudo make modules_install &&
579sudo depmod -a
580----
581--
582
583
96172a59
PP
584[[building-from-source]]
585=== Build from source
586
587To build and install LTTng{nbsp}{revision} from source:
588
05bbb80b
PP
589. Using the package manager of your distribution, or from source,
590 install the following dependencies of LTTng-tools and LTTng-UST:
96172a59
PP
591+
592--
593* https://sourceforge.net/projects/libuuid/[libuuid]
594* http://directory.fsf.org/wiki/Popt[popt]
595* http://liburcu.org/[Userspace RCU]
596* http://www.xmlsoft.org/[libxml2]
597* **Optional**: https://github.com/numactl/numactl[numactl]
598--
599
600. Download, build, and install the latest LTTng-modules{nbsp}{revision}:
601+
602--
603[role="term"]
604----
605$ cd $(mktemp -d) &&
606wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
607tar -xf lttng-modules-latest-2.12.tar.bz2 &&
608cd lttng-modules-2.12.* &&
609make &&
610sudo make modules_install &&
611sudo depmod -a
612----
613--
614
615. Download, build, and install the latest LTTng-UST{nbsp}{revision}:
616+
617--
618[role="term"]
619----
620$ cd $(mktemp -d) &&
621wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
622tar -xf lttng-ust-latest-2.12.tar.bz2 &&
623cd lttng-ust-2.12.* &&
624./configure &&
625make &&
626sudo make install &&
627sudo ldconfig
628----
629--
630+
631Add `--disable-numa` to `./configure` if you don't have
632https://github.com/numactl/numactl[numactl].
633+
634--
635[IMPORTANT]
636.Java and Python application tracing
637====
638If you need to instrument and trace <<java-application,Java
639applications>>, pass the `--enable-java-agent-jul`,
640`--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
641`configure` script, depending on which Java logging framework you use.
642
643If you need to instrument and trace <<python-application,Python
644applications>>, pass the `--enable-python-agent` option to the
645`configure` script. You can set the `PYTHON` environment variable to the
646path to the Python interpreter for which to install the LTTng-UST Python
647agent package.
648====
649--
650+
651--
652[NOTE]
653====
654By default, LTTng-UST libraries are installed to
655dir:{/usr/local/lib}, which is the de facto directory in which to
656keep self-compiled and third-party libraries.
657
658When <<building-tracepoint-providers-and-user-application,linking an
659instrumented user application with `liblttng-ust`>>:
660
661* Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
662 variable.
663* Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
664 man:gcc(1), man:g++(1), or man:clang(1).
665====
666--
667
668. Download, build, and install the latest LTTng-tools{nbsp}{revision}:
669+
670--
671[role="term"]
672----
673$ cd $(mktemp -d) &&
674wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
675tar -xf lttng-tools-latest-2.12.tar.bz2 &&
676cd lttng-tools-2.12.* &&
677./configure &&
678make &&
679sudo make install &&
680sudo ldconfig
681----
682--
683
684TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
685previous steps automatically for a given version of LTTng and confine
686the installed files in a specific directory. This can be useful to test
687LTTng without installing it on your system.
688
689
690[[getting-started]]
691== Quick start
692
693This is a short guide to get started quickly with LTTng kernel and user
694space tracing.
695
696Before you follow this guide, make sure to <<installing-lttng,install>>
697LTTng.
698
699This tutorial walks you through the steps to:
700
701. <<tracing-the-linux-kernel,Trace the Linux kernel>>.
702. <<tracing-your-own-user-application,Trace a user application>> written
703 in C.
704. <<viewing-and-analyzing-your-traces,View and analyze the
705 recorded events>>.
706
707
708[[tracing-the-linux-kernel]]
709=== Trace the Linux kernel
710
711The following command lines start with the `#` prompt because you need
712root privileges to trace the Linux kernel. You can also trace the kernel
713as a regular user if your Unix user is a member of the
714<<tracing-group,tracing group>>.
715
716. Create a <<tracing-session,tracing session>> which writes its traces
717 to dir:{/tmp/my-kernel-trace}:
718+
719--
720[role="term"]
721----
722# lttng create my-kernel-session --output=/tmp/my-kernel-trace
723----
724--
725
726. List the available kernel tracepoints and system calls:
727+
728--
729[role="term"]
730----
731# lttng list --kernel
732# lttng list --kernel --syscall
733----
734--
735
736. Create <<event,event rules>> which match the desired instrumentation
737 point names, for example the `sched_switch` and `sched_process_fork`
738 tracepoints, and the man:open(2) and man:close(2) system calls:
739+
740--
741[role="term"]
742----
743# lttng enable-event --kernel sched_switch,sched_process_fork
744# lttng enable-event --kernel --syscall open,close
745----
746--
747+
748Create an event rule which matches _all_ the Linux kernel
749tracepoints with the opt:lttng-enable-event(1):--all option
750(this will generate a lot of data when tracing):
751+
752--
753[role="term"]
754----
755# lttng enable-event --kernel --all
756----
757--
758
759. <<basic-tracing-session-control,Start tracing>>:
760+
761--
762[role="term"]
763----
764# lttng start
765----
766--
767
768. Do some operation on your system for a few seconds. For example,
769 load a website, or list the files of a directory.
770. <<creating-destroying-tracing-sessions,Destroy>> the current
771 tracing session:
772+
773--
774[role="term"]
775----
776# lttng destroy
777----
778--
779+
780The man:lttng-destroy(1) command doesn't destroy the trace data; it
781only destroys the state of the tracing session.
782+
783The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
784implicitly (see <<basic-tracing-session-control,Start and stop a tracing
785session>>). You need to stop tracing to make LTTng flush the remaining
786trace data and make the trace readable.
787
788. For the sake of this example, make the recorded trace accessible to
789 the non-root users:
790+
791--
792[role="term"]
793----
794# chown -R $(whoami) /tmp/my-kernel-trace
795----
796--
797
798See <<viewing-and-analyzing-your-traces,View and analyze the
799recorded events>> to view the recorded events.
800
801
802[[tracing-your-own-user-application]]
803=== Trace a user application
804
805This section steps you through a simple example to trace a
806_Hello world_ program written in C.
807
808To create the traceable user application:
809
810. Create the tracepoint provider header file, which defines the
811 tracepoints and the events they can generate:
812+
813--
814[source,c]
815.path:{hello-tp.h}
816----
817#undef TRACEPOINT_PROVIDER
818#define TRACEPOINT_PROVIDER hello_world
819
820#undef TRACEPOINT_INCLUDE
821#define TRACEPOINT_INCLUDE "./hello-tp.h"
822
823#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
824#define _HELLO_TP_H
825
826#include <lttng/tracepoint.h>
827
828TRACEPOINT_EVENT(
829 hello_world,
830 my_first_tracepoint,
831 TP_ARGS(
832 int, my_integer_arg,
833 char*, my_string_arg
834 ),
835 TP_FIELDS(
836 ctf_string(my_string_field, my_string_arg)
837 ctf_integer(int, my_integer_field, my_integer_arg)
838 )
839)
840
841#endif /* _HELLO_TP_H */
842
843#include <lttng/tracepoint-event.h>
844----
845--
846
847. Create the tracepoint provider package source file:
848+
849--
850[source,c]
851.path:{hello-tp.c}
852----
853#define TRACEPOINT_CREATE_PROBES
854#define TRACEPOINT_DEFINE
855
856#include "hello-tp.h"
857----
858--
859
860. Build the tracepoint provider package:
861+
862--
863[role="term"]
864----
865$ gcc -c -I. hello-tp.c
866----
867--
868
869. Create the _Hello World_ application source file:
870+
871--
872[source,c]
873.path:{hello.c}
874----
875#include <stdio.h>
876#include "hello-tp.h"
877
878int main(int argc, char *argv[])
879{
880 int x;
881
882 puts("Hello, World!\nPress Enter to continue...");
883
884 /*
885 * The following getchar() call is only placed here for the purpose
886 * of this demonstration, to pause the application in order for
887 * you to have time to list its tracepoints. It's not needed
888 * otherwise.
889 */
890 getchar();
891
892 /*
893 * A tracepoint() call.
894 *
895 * Arguments, as defined in hello-tp.h:
896 *
897 * 1. Tracepoint provider name (required)
898 * 2. Tracepoint name (required)
899 * 3. my_integer_arg (first user-defined argument)
900 * 4. my_string_arg (second user-defined argument)
901 *
902 * Notice the tracepoint provider and tracepoint names are
903 * NOT strings: they are in fact parts of variables that the
904 * macros in hello-tp.h create.
905 */
906 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
907
908 for (x = 0; x < argc; ++x) {
909 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
910 }
911
912 puts("Quitting now!");
913 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
914
915 return 0;
916}
917----
918--
919
920. Build the application:
921+
922--
923[role="term"]
924----
925$ gcc -c hello.c
926----
927--
928
929. Link the application with the tracepoint provider package,
930 `liblttng-ust`, and `libdl`:
931+
932--
933[role="term"]
934----
935$ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
936----
937--
938
939Here's the whole build process:
940
941[role="img-100"]
05bbb80b 942.Build steps of the user space tracing tutorial.
96172a59
PP
943image::ust-flow.png[]
944
945To trace the user application:
946
947. Run the application with a few arguments:
948+
949--
950[role="term"]
951----
952$ ./hello world and beyond
953----
954--
955+
956You see:
957+
958--
959----
960Hello, World!
961Press Enter to continue...
962----
963--
964
965. Start an LTTng <<lttng-sessiond,session daemon>>:
966+
967--
968[role="term"]
969----
970$ lttng-sessiond --daemonize
971----
972--
973+
05bbb80b
PP
974Note that a session daemon might already be running, for example as a
975service that the service manager of the distribution started.
96172a59
PP
976
977. List the available user space tracepoints:
978+
979--
980[role="term"]
981----
982$ lttng list --userspace
983----
984--
985+
986You see the `hello_world:my_first_tracepoint` tracepoint listed
987under the `./hello` process.
988
989. Create a <<tracing-session,tracing session>>:
990+
991--
992[role="term"]
993----
994$ lttng create my-user-space-session
995----
996--
997
998. Create an <<event,event rule>> which matches the
999 `hello_world:my_first_tracepoint` event name:
1000+
1001--
1002[role="term"]
1003----
1004$ lttng enable-event --userspace hello_world:my_first_tracepoint
1005----
1006--
1007
1008. <<basic-tracing-session-control,Start tracing>>:
1009+
1010--
1011[role="term"]
1012----
1013$ lttng start
1014----
1015--
1016
1017. Go back to the running `hello` application and press Enter. The
1018 program executes all `tracepoint()` instrumentation points and exits.
1019. <<creating-destroying-tracing-sessions,Destroy>> the current
1020 tracing session:
1021+
1022--
1023[role="term"]
1024----
1025$ lttng destroy
1026----
1027--
1028+
1029The man:lttng-destroy(1) command doesn't destroy the trace data; it
1030only destroys the state of the tracing session.
1031+
1032The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1033implicitly (see <<basic-tracing-session-control,Start and stop a tracing
1034session>>). You need to stop tracing to make LTTng flush the remaining
1035trace data and make the trace readable.
1036
1037By default, LTTng saves the traces in
1038+$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
1039where +__name__+ is the tracing session name. The
1040env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
1041
1042See <<viewing-and-analyzing-your-traces,View and analyze the
1043recorded events>> to view the recorded events.
1044
1045
1046[[viewing-and-analyzing-your-traces]]
1047=== View and analyze the recorded events
1048
1049Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
1050kernel>> and <<tracing-your-own-user-application,Trace a user
1051application>> tutorials, you can inspect the recorded events.
1052
1053There are many tools you can use to read LTTng traces:
1054
1055https://babeltrace.org/[Babeltrace{nbsp}2]::
1056 A rich, flexible trace manipulation toolkit which includes
1057 a versatile command-line interface
1058 (https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]),
1059 a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C library],
1060 and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1061 so that you can easily process or convert an LTTng trace with
1062 your own script.
1063+
1064The Babeltrace{nbsp}2 project ships with a
1065https://babeltrace.org/docs/v2.0/man7/babeltrace2-plugin-ctf.7/[plugin]
1066which supports the format of the traces which LTTng produces,
1067https://diamon.org/ctf/[CTF].
1068
1069http://tracecompass.org/[Trace Compass]::
1070 A graphical user interface for viewing and analyzing any type of
05bbb80b 1071 logs or traces, including those of LTTng.
96172a59
PP
1072
1073https://github.com/lttng/lttng-analyses[**LTTng analyses**]::
1074 An experimental project which includes many high-level analyses of
1075 LTTng kernel traces, like scheduling statistics, interrupt
1076 frequency distribution, top CPU usage, and more.
1077
1078NOTE: This section assumes that LTTng saved the traces it recorded
1079during the previous tutorials to their default location, in the
1080dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1081environment variable defaults to `$HOME` if not set.
1082
1083
1084[[viewing-and-analyzing-your-traces-bt]]
1085==== Use the cmd:babeltrace2 command-line tool
1086
1087The simplest way to list all the recorded events of an LTTng trace is to
1088pass its path to
1089https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
1090without options:
1091
1092[role="term"]
1093----
1094$ babeltrace2 ~/lttng-traces/my-user-space-session*
1095----
1096
1097cmd:babeltrace2 finds all traces recursively within the given path and
1098prints all their events, sorting them chronologically.
1099
1100Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1101further filtering:
1102
1103[role="term"]
1104----
1105$ babeltrace2 /tmp/my-kernel-trace | grep _switch
1106----
1107
1108Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1109the recorded events:
1110
1111[role="term"]
1112----
1113$ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1114----
1115
1116
1117[[viewing-and-analyzing-your-traces-bt-python]]
1118==== Use the Babeltrace{nbsp}2 Python bindings
1119
1120The <<viewing-and-analyzing-your-traces-bt,text output of
1121cmd:babeltrace2>> is useful to isolate events by simple matching using
1122man:grep(1) and similar utilities. However, more elaborate filters, such
1123as keeping only event records with a field value falling within a
1124specific range, are not trivial to write using a shell. Moreover,
1125reductions and even the most basic computations involving multiple event
1126records are virtually impossible to implement.
1127
1128Fortunately, Babeltrace{nbsp}2 ships with
1129https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1130which make it easy to read the event records of an LTTng trace
1131sequentially and compute the desired information.
1132
1133The following script accepts an LTTng Linux kernel trace path as its
1134first argument and prints the short names of the top five running
1135processes on CPU{nbsp}0 during the whole trace:
1136
1137[source,python]
1138.path:{top5proc.py}
1139----
1140import bt2
1141import sys
1142import collections
1143
1144def top5proc():
1145 # Get the trace path from the first command-line argument.
1146 it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1147
1148 # This counter dictionary will hold execution times:
1149 #
1150 # Task command name -> Total execution time (ns)
1151 exec_times = collections.Counter()
1152
1153 # This holds the last `sched_switch` timestamp.
1154 last_ts = None
1155
1156 for msg in it:
1157 # We only care about event messages.
1158 if type(msg) is not bt2._EventMessageConst:
1159 continue
1160
05bbb80b 1161 # Event of the event message.
96172a59
PP
1162 event = msg.event
1163
1164 # Keep only `sched_switch` events.
1165 if event.cls.name != 'sched_switch':
1166 continue
1167
1168 # Keep only events which occurred on CPU 0.
1169 if event.packet.context_field['cpu_id'] != 0:
1170 continue
1171
1172 # Event timestamp (ns).
1173 cur_ts = msg.default_clock_snapshot.ns_from_origin
1174
1175 if last_ts is None:
1176 # We start here.
1177 last_ts = cur_ts
1178
05bbb80b 1179 # (Short) name of the previous task command.
96172a59
PP
1180 prev_comm = str(event.payload_field['prev_comm'])
1181
1182 # Initialize an entry in our dictionary if not yet done.
1183 if prev_comm not in exec_times:
1184 exec_times[prev_comm] = 0
1185
1186 # Compute previous command execution time.
1187 diff = cur_ts - last_ts
1188
1189 # Update execution time of this command.
1190 exec_times[prev_comm] += diff
1191
1192 # Update last timestamp.
1193 last_ts = cur_ts
1194
1195 # Print top 5.
1196 for name, ns in exec_times.most_common(5):
1197 print('{:20}{} s'.format(name, ns / 1e9))
1198
1199
1200if __name__ == '__main__':
1201 top5proc()
1202----
1203
1204Run this script:
1205
1206[role="term"]
1207----
1208$ python3 top5proc.py /tmp/my-kernel-trace/kernel
1209----
1210
1211Output example:
1212
1213----
1214swapper/0 48.607245889 s
1215chromium 7.192738188 s
1216pavucontrol 0.709894415 s
1217Compositor 0.660867933 s
1218Xorg.bin 0.616753786 s
1219----
1220
1221Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1222since we weren't using the CPU that much when tracing, its first
1223position in the list makes sense.
1224
1225
1226[[core-concepts]]
1227== [[understanding-lttng]]Core concepts
1228
1229From a user's perspective, the LTTng system is built on a few concepts,
1230or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1231operates by sending commands to the <<lttng-sessiond,session daemon>>.
1232Understanding how those objects relate to eachother is key in mastering
1233the toolkit.
1234
1235The core concepts are:
1236
1237* <<tracing-session,Tracing session>>
1238* <<domain,Tracing domain>>
1239* <<channel,Channel and ring buffer>>
1240* <<"event","Instrumentation point, event rule, event, and event record">>
1241
1242
1243[[tracing-session]]
1244=== Tracing session
1245
1246A _tracing session_ is a stateful dialogue between you and
1247a <<lttng-sessiond,session daemon>>. You can
1248<<creating-destroying-tracing-sessions,create a new tracing
1249session>> with the `lttng create` command.
1250
1251Most of what you do when you control LTTng tracers happens within a
1252tracing session. In particular, a tracing session:
1253
1254* Has its own name.
1255* Has its own set of trace files.
1256* Has its own state of activity (started or stopped).
1257* Has its own <<tracing-session-mode,mode>> (local, network streaming,
1258 snapshot, or live).
1259* Has its own <<channel,channels>> to which are associated their own
1260 <<event,event rules>>.
1261* Has its own <<pid-tracking,process attribute tracking>> inclusion
1262 sets.
1263
1264[role="img-100"]
1265.A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1266image::concepts.png[]
1267
1268Those attributes and objects are completely isolated between different
1269tracing sessions.
1270
1271A tracing session is analogous to a cash machine session:
1272the operations you do on the banking system through the cash machine do
1273not alter the data of other users of the same system. In the case of
1274the cash machine, a session lasts as long as your bank card is inside.
1275In the case of LTTng, a tracing session lasts from the `lttng create`
1276command to the `lttng destroy` command.
1277
1278[role="img-100"]
1279.Each Unix user has its own set of tracing sessions.
1280image::many-sessions.png[]
1281
1282
1283[[tracing-session-mode]]
1284==== Tracing session mode
1285
1286LTTng can send the generated trace data to different locations. The
1287_tracing session mode_ dictates where to send it. The following modes
1288are available in LTTng{nbsp}{revision}:
1289
1290Local mode::
1291 LTTng writes the traces to the file system of the machine it traces
1292 (target system).
1293
1294Network streaming mode::
1295 LTTng sends the traces over the network to a
1296 <<lttng-relayd,relay daemon>> running on a remote system.
1297
1298Snapshot mode::
1299 LTTng doesn't write the traces by default.
1300+
1301Instead, you can request LTTng to <<taking-a-snapshot,take a snapshot>>,
05bbb80b
PP
1302that is, a copy of the current sub-buffers of the tracing session, and
1303to write it to the file system of the target or to send it over the
1304network to a <<lttng-relayd,relay daemon>> running on a remote system.
96172a59
PP
1305
1306Live mode::
1307 This mode is similar to the network streaming mode, but a live
1308 trace viewer can connect to the distant relay daemon to
1309 <<lttng-live,view event records as LTTng generates them>>.
1310
1311
1312[[domain]]
1313=== Tracing domain
1314
1315A _tracing domain_ is a namespace for event sources. A tracing domain
1316has its own properties and features.
1317
1318There are currently five available tracing domains:
1319
1320* Linux kernel
1321* User space
1322* `java.util.logging` (JUL)
1323* log4j
1324* Python
1325
1326You must specify a tracing domain when using some commands to avoid
1327ambiguity. For example, since all the domains support named tracepoints
1328as event sources (instrumentation points that you manually insert in the
1329source code), you need to specify a tracing domain when
1330<<enabling-disabling-events,creating an event rule>> because all the
1331tracing domains could have tracepoints with the same names.
1332
1333You can create <<channel,channels>> in the Linux kernel and user space
1334tracing domains. The other tracing domains have a single default
1335channel.
1336
1337
1338[[channel]]
1339=== Channel and ring buffer
1340
1341A _channel_ is an object which is responsible for a set of ring buffers.
1342Each ring buffer is divided into multiple sub-buffers. When an LTTng
1343tracer emits an event, it can record it to one or more
1344sub-buffers. The attributes of a channel determine what to do when
1345there's no space left for a new event record because all sub-buffers
1346are full, where to send a full sub-buffer, and other behaviours.
1347
1348A channel is always associated to a <<domain,tracing domain>>. The
1349`java.util.logging` (JUL), log4j, and Python tracing domains each have
1350a default channel which you can't configure.
1351
1352A channel also owns <<event,event rules>>. When an LTTng tracer emits
1353an event, it records it to the sub-buffers of all
1354the enabled channels with a satisfied event rule, as long as those
1355channels are part of active <<tracing-session,tracing sessions>>.
1356
1357
1358[[channel-buffering-schemes]]
1359==== Per-user vs. per-process buffering schemes
1360
1361A channel has at least one ring buffer _per CPU_. LTTng always
1362records an event to the ring buffer associated to the CPU on which it
1363occurs.
1364
1365Two _buffering schemes_ are available when you
1366<<enabling-disabling-channels,create a channel>> in the
1367user space <<domain,tracing domain>>:
1368
1369Per-user buffering::
1370 Allocate one set of ring buffers--one per CPU--shared by all the
1371 instrumented processes of each Unix user.
1372+
1373--
1374[role="img-100"]
1375.Per-user buffering scheme.
1376image::per-user-buffering.png[]
1377--
1378
1379Per-process buffering::
1380 Allocate one set of ring buffers--one per CPU--for each
1381 instrumented process.
1382+
1383--
1384[role="img-100"]
1385.Per-process buffering scheme.
1386image::per-process-buffering.png[]
1387--
1388+
1389The per-process buffering scheme tends to consume more memory than the
1390per-user option because systems generally have more instrumented
1391processes than Unix users running instrumented processes. However, the
1392per-process buffering scheme ensures that one process having a high
1393event throughput won't fill all the shared sub-buffers of the same
1394user, only its own.
1395
1396The Linux kernel tracing domain has only one available buffering scheme
1397which is to allocate a single set of ring buffers for the whole system.
1398This scheme is similar to the per-user option, but with a single, global
1399user ``running'' the kernel.
1400
1401
1402[[channel-overwrite-mode-vs-discard-mode]]
1403==== Overwrite vs. discard event record loss modes
1404
1405When an event occurs, LTTng records it to a specific sub-buffer (yellow
05bbb80b
PP
1406arc in the following animations) of the ring buffer of a specific
1407channel. When there's no space left in a sub-buffer, the tracer marks it
1408as consumable (red) and another, empty sub-buffer starts receiving the
96172a59
PP
1409following event records. A <<lttng-consumerd,consumer daemon>>
1410eventually consumes the marked sub-buffer (returns to white).
1411
1412[NOTE]
1413[role="docsvg-channel-subbuf-anim"]
1414====
1415{note-no-anim}
1416====
1417
1418In an ideal world, sub-buffers are consumed faster than they are filled,
1419as it is the case in the previous animation. In the real world,
1420however, all sub-buffers can be full at some point, leaving no space to
1421record the following events.
1422
1423By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when
1424no empty sub-buffer is available, it is acceptable to lose event records
1425when the alternative would be to cause substantial delays in the
05bbb80b
PP
1426execution of the instrumented application. LTTng privileges performance
1427over integrity; it aims at perturbing the target system as little as
1428possible in order to make tracing of subtle race conditions and rare
1429interrupt cascades possible.
96172a59
PP
1430
1431Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1432a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1433example>> to learn how to use the blocking mode.
1434
1435When it comes to losing event records because no empty sub-buffer is
1436available, or because the <<opt-blocking-timeout,blocking timeout>> is
05bbb80b
PP
1437reached, the _event record loss mode_ of the channel determines what to
1438do. The available event record loss modes are:
96172a59
PP
1439
1440Discard mode::
1441 Drop the newest event records until the tracer releases a sub-buffer.
1442+
1443This is the only available mode when you specify a
1444<<opt-blocking-timeout,blocking timeout>>.
1445
1446Overwrite mode::
1447 Clear the sub-buffer containing the oldest event records and start
1448 writing the newest event records there.
1449+
1450This mode is sometimes called _flight recorder mode_ because it's
1451similar to a
1452https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1453always keep a fixed amount of the latest data.
1454
1455Which mechanism you should choose depends on your context: prioritize
1456the newest or the oldest event records in the ring buffer?
1457
1458Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1459as soon as a there's no space left for a new event record, whereas in
1460discard mode, the tracer only discards the event record that doesn't
1461fit.
1462
1463In discard mode, LTTng increments a count of lost event records when an
1464event record is lost and saves this count to the trace. In overwrite
1465mode, since LTTng{nbsp}2.8, LTTng increments a count of lost sub-buffers
1466when a sub-buffer is lost and saves this count to the trace. In this
1467mode, LTTng doesn't write to the trace the exact number of lost event
05bbb80b
PP
1468records in those lost sub-buffers. Trace analyses can use saved
1469discarded event record and sub-buffer counts of the trace to decide
1470whether or not to perform the analyses even if trace data is known to be
1471missing.
96172a59
PP
1472
1473There are a few ways to decrease your probability of losing event
1474records.
1475<<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1476how to fine-tune the sub-buffer count and size of a channel to virtually
1477stop losing event records, though at the cost of greater memory usage.
1478
1479
1480[[channel-subbuf-size-vs-subbuf-count]]
1481==== Sub-buffer count and size
1482
1483When you <<enabling-disabling-channels,create a channel>>, you can
1484set its number of sub-buffers and their size.
1485
1486Note that there is noticeable CPU overhead introduced when
1487switching sub-buffers (marking a full one as consumable and switching
1488to an empty one for the following events to be recorded). Knowing this,
1489the following list presents a few practical situations along with how
1490to configure the sub-buffer count and size for them:
1491
1492* **High event throughput**: In general, prefer bigger sub-buffers to
1493 lower the risk of losing event records.
1494+
1495Having bigger sub-buffers also ensures a lower
1496<<channel-switch-timer,sub-buffer switching frequency>>.
1497+
1498The number of sub-buffers is only meaningful if you create the channel
1499in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1500other sub-buffers are left unaltered.
1501
1502* **Low event throughput**: In general, prefer smaller sub-buffers
1503 since the risk of losing event records is low.
1504+
1505Because events occur less frequently, the sub-buffer switching frequency
05bbb80b 1506should remain low and thus the overhead of the tracer shouldn't be a
96172a59
PP
1507problem.
1508
1509* **Low memory system**: If your target system has a low memory
1510 limit, prefer fewer first, then smaller sub-buffers.
1511+
1512Even if the system is limited in memory, you want to keep the
1513sub-buffers as big as possible to avoid a high sub-buffer switching
1514frequency.
1515
1516Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1517which means event data is very compact. For example, the average
1518LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1519sub-buffer size of 1{nbsp}MiB is considered big.
1520
1521The previous situations highlight the major trade-off between a few big
1522sub-buffers and more, smaller sub-buffers: sub-buffer switching
1523frequency vs. how much data is lost in overwrite mode. Assuming a
1524constant event throughput and using the overwrite mode, the two
1525following configurations have the same ring buffer total size:
1526
1527[NOTE]
1528[role="docsvg-channel-subbuf-size-vs-count-anim"]
1529====
1530{note-no-anim}
1531====
1532
1533* **Two sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1534 switching frequency, but if a sub-buffer overwrite happens, half of
1535 the event records so far (4{nbsp}MiB) are definitely lost.
05bbb80b
PP
1536* **Eight sub-buffers of 1{nbsp}MiB each**: Expect four times the
1537 overhead of the tracer as the previous configuration, but if a
1538 sub-buffer overwrite happens, only the eighth of event records so far
1539 are definitely lost.
96172a59
PP
1540
1541In discard mode, the sub-buffers count parameter is pointless: use two
1542sub-buffers and set their size according to the requirements of your
1543situation.
1544
1545
1546[[channel-switch-timer]]
1547==== Switch timer period
1548
1549The _switch timer period_ is an important configurable attribute of
1550a channel to ensure periodic sub-buffer flushing.
1551
1552When the _switch timer_ expires, a sub-buffer switch happens. Set
1553the switch timer period attribute when you
1554<<enabling-disabling-channels,create a channel>> to ensure that LTTng
1555consumes and commits trace data to trace files or to a distant relay
1556daemon periodically in case of a low event throughput.
1557
1558[NOTE]
1559[role="docsvg-channel-switch-timer"]
1560====
1561{note-no-anim}
1562====
1563
1564This attribute is also convenient when you use big sub-buffers to cope
1565with a sporadic high event throughput, even if the throughput is
1566normally low.
1567
1568
1569[[channel-read-timer]]
1570==== Read timer period
1571
1572By default, the LTTng tracers use a notification mechanism to signal a
1573full sub-buffer so that a consumer daemon can consume it. When such
1574notifications must be avoided, for example in real-time applications,
05bbb80b
PP
1575use the _read timer_ of the channel instead. When the read timer fires,
1576the <<lttng-consumerd,consumer daemon>> checks for full, consumable
96172a59
PP
1577sub-buffers.
1578
1579
1580[[tracefile-rotation]]
1581==== Trace file count and size
1582
05bbb80b
PP
1583By default, trace files can grow as large as needed. Set the maximum
1584size of each trace file that a channel writes when you
1585<<enabling-disabling-channels,create a channel>>. When the size of a
1586trace file reaches the fixed maximum size of the channel, LTTng creates
96172a59
PP
1587another file to contain the next event records. LTTng appends a file
1588count to each trace file name in this case.
1589
1590If you set the trace file size attribute when you create a channel, the
1591maximum number of trace files that LTTng creates is _unlimited_ by
1592default. To limit them, set a maximum number of trace files. When the
05bbb80b
PP
1593number of trace files reaches the fixed maximum count of the channel,
1594the oldest trace file is overwritten. This mechanism is called _trace
1595file rotation_.
96172a59
PP
1596
1597[IMPORTANT]
1598====
1599Even if you don't limit the trace file count, you can't assume that
1600LTTng doesn't manage any trace file.
1601
1602In other words, there is no safe way to know if LTTng still holds a
1603given trace file open with the trace file rotation feature.
1604
1605The only way to obtain an unmanaged, self-contained LTTng trace before
1606you <<creating-destroying-tracing-sessions,destroy>> the tracing session
1607is with the <<session-rotation,tracing session rotation>> feature
1608(available since LTTng{nbsp}2.11).
1609====
1610
1611
1612[[event]]
1613=== Instrumentation point, event rule, event, and event record
1614
1615An _event rule_ is a set of conditions which must be **all** satisfied
1616for LTTng to record an occuring event.
1617
1618You set the conditions when you <<enabling-disabling-events,create
1619an event rule>>.
1620
1621You always attach an event rule to a <<channel,channel>> when you create
1622it.
1623
1624When an event passes the conditions of an event rule, LTTng records it
05bbb80b 1625in one of the sub-buffers of the attached channel.
96172a59
PP
1626
1627The available conditions, as of LTTng{nbsp}{revision}, are:
1628
1629* The event rule _is enabled_.
05bbb80b
PP
1630* The type of the instrumentation point _is{nbsp}T_.
1631* The name of the instrumentation point (sometimes called _event name_)
96172a59 1632 _matches{nbsp}N_, but _isn't{nbsp}E_.
05bbb80b 1633* The log level of the instrumentation point _is as severe as{nbsp}L_, or
96172a59 1634 _is exactly{nbsp}L_.
05bbb80b 1635* The fields of the payload of the event _satisfy_ a filter
96172a59
PP
1636 expression{nbsp}__F__.
1637
1638As you can see, all the conditions but the dynamic filter are related to
05bbb80b 1639the status of the event rule or to the instrumentation point, not to the
96172a59
PP
1640occurring events. This is why, without a filter, checking if an event
1641passes an event rule isn't a dynamic task: when you create or modify an
1642event rule, all the tracers of its tracing domain enable or disable the
1643instrumentation points themselves once. This is possible because the
1644attributes of an instrumentation point (type, name, and log level) are
1645defined statically. In other words, without a dynamic filter, the tracer
1646_doesn't evaluate_ the arguments of an instrumentation point unless it
1647matches an enabled event rule.
1648
1649Note that, for LTTng to record an event, the <<channel,channel>> to
1650which a matching event rule is attached must also be enabled, and the
1651<<tracing-session,tracing session>> owning this channel must be active
1652(started).
1653
1654[role="img-100"]
1655.Logical path from an instrumentation point to an event record.
1656image::event-rule.png[]
1657
1658.Event, event record, or event rule?
1659****
1660With so many similar terms, it's easy to get confused.
1661
1662An **event** is the consequence of the execution of an _instrumentation
1663point_, like a tracepoint that you manually place in some source code,
1664or a Linux kernel kprobe. An event is said to _occur_ at a specific
1665time. Different actions can be taken upon the occurrence of an event,
05bbb80b 1666like record the payload of the event to a buffer.
96172a59
PP
1667
1668An **event record** is the representation of an event in a sub-buffer. A
1669tracer is responsible for capturing the payload of an event, current
05bbb80b 1670context variables, the ID of the event, and its timestamp. LTTng
96172a59
PP
1671can append this sub-buffer to a trace file.
1672
1673An **event rule** is a set of conditions which must _all_ be satisfied
1674for LTTng to record an occuring event. Events still occur without
1675satisfying event rules, but LTTng doesn't record them.
1676****
1677
1678
1679[[plumbing]]
1680== Components of noch:{LTTng}
1681
1682The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1683to call LTTng a simple _tool_ since it is composed of multiple
1684interacting components. This section describes those components,
1685explains their respective roles, and shows how they connect together to
1686form the LTTng ecosystem.
1687
1688The following diagram shows how the most important components of LTTng
1689interact with user applications, the Linux kernel, and you:
1690
1691[role="img-100"]
1692.Control and trace data paths between LTTng components.
1693image::plumbing.png[]
1694
1695The LTTng project incorporates:
1696
1697* **LTTng-tools**: Libraries and command-line interface to
1698 control tracing sessions.
1699** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1700** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1701** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1702** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1703** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1704* **LTTng-UST**: Libraries and Java/Python packages to trace user
1705 applications.
1706** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1707 headers to instrument and trace any native user application.
1708** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1709*** `liblttng-ust-libc-wrapper`
1710*** `liblttng-ust-pthread-wrapper`
1711*** `liblttng-ust-cyg-profile`
1712*** `liblttng-ust-cyg-profile-fast`
1713*** `liblttng-ust-dl`
1714** User space tracepoint provider source files generator command-line
1715 tool (man:lttng-gen-tp(1)).
1716** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1717 Java applications using `java.util.logging` or
1718 Apache log4j{nbsp}1.2 logging.
1719** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1720 Python applications using the standard `logging` package.
1721* **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1722 the kernel.
1723** LTTng kernel tracer module.
1724** Tracing ring buffer kernel modules.
1725** Probe kernel modules.
1726** LTTng logger kernel module.
1727
1728
1729[[lttng-cli]]
1730=== Tracing control command-line interface
1731
1732[role="img-100"]
1733.The tracing control command-line interface.
1734image::plumbing-lttng-cli.png[]
1735
1736The _man:lttng(1) command-line tool_ is the standard user interface to
1737control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1738is part of LTTng-tools.
1739
1740The cmd:lttng tool is linked with
1741<<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1742one or more <<lttng-sessiond,session daemons>> behind the scenes.
1743
1744The cmd:lttng tool has a Git-like interface:
1745
1746[role="term"]
1747----
1748$ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1749----
1750
1751The <<controlling-tracing,Tracing control>> section explores the
1752available features of LTTng using the cmd:lttng tool.
1753
1754
1755[[liblttng-ctl-lttng]]
1756=== Tracing control library
1757
1758[role="img-100"]
1759.The tracing control library.
1760image::plumbing-liblttng-ctl.png[]
1761
1762The _LTTng control library_, `liblttng-ctl`, is used to communicate
1763with a <<lttng-sessiond,session daemon>> using a C API that hides the
05bbb80b 1764underlying details of the protocol. `liblttng-ctl` is part of LTTng-tools.
96172a59
PP
1765
1766The <<lttng-cli,cmd:lttng command-line tool>>
1767is linked with `liblttng-ctl`.
1768
1769Use `liblttng-ctl` in C or $$C++$$ source code by including its
1770``master'' header:
1771
1772[source,c]
1773----
1774#include <lttng/lttng.h>
1775----
1776
1777Some objects are referenced by name (C string), such as tracing
1778sessions, but most of them require to create a handle first using
1779`lttng_create_handle()`.
1780
1781As of LTTng{nbsp}{revision}, the best available developer documentation for
1782`liblttng-ctl` is its installed header files. Every function and structure is
1783thoroughly documented.
1784
1785
1786[[lttng-ust]]
1787=== User space tracing library
1788
1789[role="img-100"]
1790.The user space tracing library.
1791image::plumbing-liblttng-ust.png[]
1792
1793The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1794is the LTTng user space tracer. It receives commands from a
1795<<lttng-sessiond,session daemon>>, for example to
1796enable and disable specific instrumentation points, and writes event
1797records to ring buffers shared with a
1798<<lttng-consumerd,consumer daemon>>.
1799`liblttng-ust` is part of LTTng-UST.
1800
1801Public C header files are installed beside `liblttng-ust` to
1802instrument any <<c-application,C or $$C++$$ application>>.
1803
1804<<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1805packages, use their own library providing tracepoints which is
1806linked with `liblttng-ust`.
1807
1808An application or library doesn't have to initialize `liblttng-ust`
1809manually: its constructor does the necessary tasks to properly register
1810to a session daemon. The initialization phase also enables the
1811instrumentation points matching the <<event,event rules>> that you
1812already created.
1813
1814
1815[[lttng-ust-agents]]
1816=== User space tracing agents
1817
1818[role="img-100"]
1819.The user space tracing agents.
1820image::plumbing-lttng-ust-agents.png[]
1821
1822The _LTTng-UST Java and Python agents_ are regular Java and Python
1823packages which add LTTng tracing capabilities to the
1824native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1825
1826In the case of Java, the
1827https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1828core logging facilities] and
1829https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
1830Note that Apache Log4{nbsp}2 isn't supported.
1831
1832In the case of Python, the standard
1833https://docs.python.org/3/library/logging.html[`logging`] package
1834is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
1835LTTng-UST Python agent package.
1836
1837The applications using the LTTng-UST agents are in the
1838`java.util.logging` (JUL),
1839log4j, and Python <<domain,tracing domains>>.
1840
1841Both agents use the same mechanism to trace the log statements. When an
1842agent initializes, it creates a log handler that attaches to the root
1843logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1844When the application executes a log statement, the root logger passes it
05bbb80b
PP
1845to the log handler of the agent. The log handler of the agent calls a
1846native function in a tracepoint provider package shared library linked
1847with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
96172a59
PP
1848other fields, like its logger name and its log level. This native
1849function contains a user space instrumentation point, hence tracing the
1850log statement.
1851
1852The log level condition of an
1853<<event,event rule>> is considered when tracing
1854a Java or a Python application, and it's compatible with the standard
1855JUL, log4j, and Python log levels.
1856
1857
1858[[lttng-modules]]
1859=== LTTng kernel modules
1860
1861[role="img-100"]
1862.The LTTng kernel modules.
1863image::plumbing-lttng-modules.png[]
1864
1865The _LTTng kernel modules_ are a set of Linux kernel modules
1866which implement the kernel tracer of the LTTng project. The LTTng
1867kernel modules are part of LTTng-modules.
1868
1869The LTTng kernel modules include:
1870
1871* A set of _probe_ modules.
1872+
1873Each module attaches to a specific subsystem
1874of the Linux kernel using its tracepoint instrument points. There are
1875also modules to attach to the entry and return points of the Linux
1876system call functions.
1877
1878* _Ring buffer_ modules.
1879+
1880A ring buffer implementation is provided as kernel modules. The LTTng
1881kernel tracer writes to the ring buffer; a
1882<<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1883
1884* The _LTTng kernel tracer_ module.
1885* The _LTTng logger_ module.
1886+
1887The LTTng logger module implements the special path:{/proc/lttng-logger}
1888(and path:{/dev/lttng-logger} since LTTng{nbsp}2.11) files so that any
1889executable can generate LTTng events by opening and writing to those
1890files.
1891+
1892See <<proc-lttng-logger-abi,LTTng logger>>.
1893
1894Generally, you don't have to load the LTTng kernel modules manually
1895(using man:modprobe(8), for example): a root <<lttng-sessiond,session
1896daemon>> loads the necessary modules when starting. If you have extra
1897probe modules, you can specify to load them to the session daemon on
1898the command line.
1899
1900The LTTng kernel modules are installed in
1901+/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
1902the kernel release (see `uname --kernel-release`).
1903
1904
1905[[lttng-sessiond]]
1906=== Session daemon
1907
1908[role="img-100"]
1909.The session daemon.
1910image::plumbing-sessiond.png[]
1911
1912The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
1913managing tracing sessions and for controlling the various components of
1914LTTng. The session daemon is part of LTTng-tools.
1915
1916The session daemon sends control requests to and receives control
1917responses from:
1918
1919* The <<lttng-ust,user space tracing library>>.
1920+
1921Any instance of the user space tracing library first registers to
1922a session daemon. Then, the session daemon can send requests to
1923this instance, such as:
1924+
1925--
1926** Get the list of tracepoints.
1927** Share an <<event,event rule>> so that the user space tracing library
1928 can enable or disable tracepoints. Amongst the possible conditions
1929 of an event rule is a filter expression which `liblttng-ust` evalutes
1930 when an event occurs.
1931** Share <<channel,channel>> attributes and ring buffer locations.
1932--
1933+
1934The session daemon and the user space tracing library use a Unix
1935domain socket for their communication.
1936
1937* The <<lttng-ust-agents,user space tracing agents>>.
1938+
1939Any instance of a user space tracing agent first registers to
1940a session daemon. Then, the session daemon can send requests to
1941this instance, such as:
1942+
1943--
1944** Get the list of loggers.
1945** Enable or disable a specific logger.
1946--
1947+
1948The session daemon and the user space tracing agent use a TCP connection
1949for their communication.
1950
1951* The <<lttng-modules,LTTng kernel tracer>>.
1952* The <<lttng-consumerd,consumer daemon>>.
1953+
1954The session daemon sends requests to the consumer daemon to instruct
1955it where to send the trace data streams, amongst other information.
1956
1957* The <<lttng-relayd,relay daemon>>.
1958
1959The session daemon receives commands from the
1960<<liblttng-ctl-lttng,tracing control library>>.
1961
1962The root session daemon loads the appropriate
1963<<lttng-modules,LTTng kernel modules>> on startup. It also spawns
1964a <<lttng-consumerd,consumer daemon>> as soon as you create
1965an <<event,event rule>>.
1966
1967The session daemon doesn't send and receive trace data: this is the
1968role of the <<lttng-consumerd,consumer daemon>> and
1969<<lttng-relayd,relay daemon>>. It does, however, generate the
1970http://diamon.org/ctf/[CTF] metadata stream.
1971
1972Each Unix user can have its own session daemon instance. The
1973tracing sessions which different session daemons manage are completely
1974independent.
1975
1976The root user's session daemon is the only one which is
1977allowed to control the LTTng kernel tracer, and its spawned consumer
1978daemon is the only one which is allowed to consume trace data from the
1979LTTng kernel tracer. Note, however, that any Unix user which is a member
1980of the <<tracing-group,tracing group>> is allowed
1981to create <<channel,channels>> in the
1982Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
1983kernel.
1984
1985The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
1986session daemon when using its `create` command if none is currently
1987running. You can also start the session daemon manually.
1988
1989
1990[[lttng-consumerd]]
1991=== Consumer daemon
1992
1993[role="img-100"]
1994.The consumer daemon.
1995image::plumbing-consumerd.png[]
1996
1997The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
1998ring buffers with user applications or with the LTTng kernel modules to
1999collect trace data and send it to some location (on disk or to a
2000<<lttng-relayd,relay daemon>> over the network). The consumer daemon
2001is part of LTTng-tools.
2002
2003You don't start a consumer daemon manually: a consumer daemon is always
2004spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
2005<<event,event rule>>, that is, before you start tracing. When you kill
2006its owner session daemon, the consumer daemon also exits because it is
05bbb80b 2007the child process of the session daemon. Command-line options of
96172a59
PP
2008man:lttng-sessiond(8) target the consumer daemon process.
2009
2010There are up to two running consumer daemons per Unix user, whereas only
2011one session daemon can run per user. This is because each process can be
2012either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2013and 64-bit processes, it is more efficient to have separate
2014corresponding 32-bit and 64-bit consumer daemons. The root user is an
2015exception: it can have up to _three_ running consumer daemons: 32-bit
2016and 64-bit instances for its user applications, and one more
2017reserved for collecting kernel trace data.
2018
2019
2020[[lttng-relayd]]
2021=== Relay daemon
2022
2023[role="img-100"]
2024.The relay daemon.
2025image::plumbing-relayd.png[]
2026
2027The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
2028between remote session and consumer daemons, local trace files, and a
2029remote live trace viewer. The relay daemon is part of LTTng-tools.
2030
2031The main purpose of the relay daemon is to implement a receiver of
2032<<sending-trace-data-over-the-network,trace data over the network>>.
2033This is useful when the target system doesn't have much file system
2034space to record trace files locally.
2035
2036The relay daemon is also a server to which a
2037<<lttng-live,live trace viewer>> can
2038connect. The live trace viewer sends requests to the relay daemon to
2039receive trace data as the target system emits events. The
2040communication protocol is named _LTTng live_; it is used over TCP
2041connections.
2042
2043Note that you can start the relay daemon on the target system directly.
2044This is the setup of choice when the use case is to view events as
2045the target system emits them without the need of a remote system.
2046
2047
2048[[instrumenting]]
2049== [[using-lttng]]Instrumentation
2050
2051There are many examples of tracing and monitoring in our everyday life:
2052
2053* You have access to real-time and historical weather reports and
2054 forecasts thanks to weather stations installed around the country.
2055* You know your heart is safe thanks to an electrocardiogram.
2056* You make sure not to drive your car too fast and to have enough fuel
2057 to reach your destination thanks to gauges visible on your dashboard.
2058
2059All the previous examples have something in common: they rely on
2060**instruments**. Without the electrodes attached to the surface of your
05bbb80b 2061body skin, cardiac monitoring is futile.
96172a59
PP
2062
2063LTTng, as a tracer, is no different from those real life examples. If
2064you're about to trace a software system or, in other words, record its
2065history of execution, you better have **instrumentation points** in the
2066subject you're tracing, that is, the actual software.
2067
2068Various ways were developed to instrument a piece of software for LTTng
2069tracing. The most straightforward one is to manually place
05bbb80b
PP
2070instrumentation points, called _tracepoints_, in the source code of the
2071software. It is also possible to add instrumentation points dynamically
2072in the Linux kernel <<domain,tracing domain>>.
96172a59
PP
2073
2074If you're only interested in tracing the Linux kernel, your
05bbb80b
PP
2075instrumentation needs are probably already covered by the built-in
2076<<lttng-modules,Linux kernel tracepoints>> of LTTng. You may also wish
2077to trace a user application which is already instrumented for LTTng
2078tracing. In such cases, skip this whole section and read the topics of
96172a59
PP
2079the <<controlling-tracing,Tracing control>> section.
2080
2081Many methods are available to instrument a piece of software for LTTng
2082tracing. They are:
2083
2084* <<c-application,User space instrumentation for C and $$C++$$
2085 applications>>.
2086* <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
2087* <<java-application,User space Java agent>>.
2088* <<python-application,User space Python agent>>.
2089* <<proc-lttng-logger-abi,LTTng logger>>.
2090* <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
2091
2092
2093[[c-application]]
2094=== [[cxx-application]]User space instrumentation for C and $$C++$$ applications
2095
2096The procedure to instrument a C or $$C++$$ user application with
2097the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
2098
2099. <<tracepoint-provider,Create the source files of a tracepoint provider
2100 package>>.
2101. <<probing-the-application-source-code,Add tracepoints to
05bbb80b 2102 the source code of the application>>.
96172a59
PP
2103. <<building-tracepoint-providers-and-user-application,Build and link
2104 a tracepoint provider package and the user application>>.
2105
2106If you need quick, man:printf(3)-like instrumentation, skip
2107those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
2108instead.
2109
2110IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2111instrument a user application with `liblttng-ust`.
2112
2113
2114[[tracepoint-provider]]
2115==== Create the source files of a tracepoint provider package
2116
2117A _tracepoint provider_ is a set of compiled functions which provide
2118**tracepoints** to an application, the type of instrumentation point
2119supported by LTTng-UST. Those functions can emit events with
2120user-defined fields and serialize those events as event records to one
2121or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
05bbb80b
PP
2122macro, which you <<probing-the-application-source-code,insert in the
2123source code of a user application>>, calls those functions.
96172a59
PP
2124
2125A _tracepoint provider package_ is an object file (`.o`) or a shared
2126library (`.so`) which contains one or more tracepoint providers.
2127Its source files are:
2128
2129* One or more <<tpp-header,tracepoint provider header>> (`.h`).
2130* A <<tpp-source,tracepoint provider package source>> (`.c`).
2131
2132A tracepoint provider package is dynamically linked with `liblttng-ust`,
2133the LTTng user space tracer, at run time.
2134
2135[role="img-100"]
2136.User application linked with `liblttng-ust` and containing a tracepoint provider.
2137image::ust-app.png[]
2138
2139NOTE: If you need quick, man:printf(3)-like instrumentation,
2140skip creating and using a tracepoint provider and use
2141<<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
2142
2143
2144[[tpp-header]]
2145===== Create a tracepoint provider header file template
2146
2147A _tracepoint provider header file_ contains the tracepoint
2148definitions of a tracepoint provider.
2149
2150To create a tracepoint provider header file:
2151
2152. Start from this template:
2153+
2154--
2155[source,c]
2156.Tracepoint provider header file template (`.h` file extension).
2157----
2158#undef TRACEPOINT_PROVIDER
2159#define TRACEPOINT_PROVIDER provider_name
2160
2161#undef TRACEPOINT_INCLUDE
2162#define TRACEPOINT_INCLUDE "./tp.h"
2163
2164#if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
2165#define _TP_H
2166
2167#include <lttng/tracepoint.h>
2168
2169/*
2170 * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
2171 * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
2172 */
2173
2174#endif /* _TP_H */
2175
2176#include <lttng/tracepoint-event.h>
2177----
2178--
2179
2180. Replace:
2181+
2182* `provider_name` with the name of your tracepoint provider.
2183* `"tp.h"` with the name of your tracepoint provider header file.
2184
2185. Below the `#include <lttng/tracepoint.h>` line, put your
2186 <<defining-tracepoints,tracepoint definitions>>.
2187
2188Your tracepoint provider name must be unique amongst all the possible
2189tracepoint provider names used on the same target system. We
2190suggest to include the name of your project or company in the name,
2191for example, `org_lttng_my_project_tpp`.
2192
2193TIP: [[lttng-gen-tp]]Use the man:lttng-gen-tp(1) tool to create
2194this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
2195write are the <<defining-tracepoints,tracepoint definitions>>.
2196
2197
2198[[defining-tracepoints]]
2199===== Create a tracepoint definition
2200
2201A _tracepoint definition_ defines, for a given tracepoint:
2202
2203* Its **input arguments**. They are the macro parameters that the
2204 `tracepoint()` macro accepts for this particular tracepoint
05bbb80b 2205 in the source code of the user application.
96172a59
PP
2206* Its **output event fields**. They are the sources of event fields
2207 that form the payload of any event that the execution of the
2208 `tracepoint()` macro emits for this particular tracepoint.
2209
2210Create a tracepoint definition by using the
2211`TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2212line in the
2213<<tpp-header,tracepoint provider header file template>>.
2214
2215The syntax of the `TRACEPOINT_EVENT()` macro is:
2216
2217[source,c]
2218.`TRACEPOINT_EVENT()` macro syntax.
2219----
2220TRACEPOINT_EVENT(
2221 /* Tracepoint provider name */
2222 provider_name,
2223
2224 /* Tracepoint name */
2225 tracepoint_name,
2226
2227 /* Input arguments */
2228 TP_ARGS(
2229 arguments
2230 ),
2231
2232 /* Output event fields */
2233 TP_FIELDS(
2234 fields
2235 )
2236)
2237----
2238
2239Replace:
2240
2241* `provider_name` with your tracepoint provider name.
2242* `tracepoint_name` with your tracepoint name.
2243* `arguments` with the <<tpp-def-input-args,input arguments>>.
2244* `fields` with the <<tpp-def-output-fields,output event field>>
2245 definitions.
2246
2247This tracepoint emits events named `provider_name:tracepoint_name`.
2248
2249[IMPORTANT]
05bbb80b 2250.Event name length limitation
96172a59
PP
2251====
2252The concatenation of the tracepoint provider name and the
2253tracepoint name must not exceed **254{nbsp}characters**. If it does, the
2254instrumented application compiles and runs, but LTTng throws multiple
2255warnings and you could experience serious issues.
2256====
2257
2258[[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2259
2260[source,c]
2261.`TP_ARGS()` macro syntax.
2262----
2263TP_ARGS(
2264 type, arg_name
2265)
2266----
2267
2268Replace:
2269
2270* `type` with the C type of the argument.
2271* `arg_name` with the argument name.
2272
2273You can repeat `type` and `arg_name` up to 10{nbsp}times to have more
2274than one argument.
2275
2276.`TP_ARGS()` usage with three arguments.
2277====
2278[source,c]
2279----
2280TP_ARGS(
2281 int, count,
2282 float, ratio,
2283 const char*, query
2284)
2285----
2286====
2287
2288The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2289tracepoint definition with no input arguments.
2290
2291[[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2292`ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2293man:lttng-ust(3) for a complete description of the available `ctf_*()`
2294macros. A `ctf_*()` macro specifies the type, size, and byte order of
2295one event field.
2296
2297Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2298C expression that the tracer evalutes at the `tracepoint()` macro site
05bbb80b
PP
2299in the source code of the application. This expression provides the
2300source of data of a field. The argument expression can include input
2301argument names listed in the `TP_ARGS()` macro.
96172a59
PP
2302
2303Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2304must be unique within a given tracepoint definition.
2305
2306Here's a complete tracepoint definition example:
2307
2308.Tracepoint definition.
2309====
2310The following tracepoint definition defines a tracepoint which takes
2311three input arguments and has four output event fields.
2312
2313[source,c]
2314----
2315#include "my-custom-structure.h"
2316
2317TRACEPOINT_EVENT(
2318 my_provider,
2319 my_tracepoint,
2320 TP_ARGS(
2321 const struct my_custom_structure*, my_custom_structure,
2322 float, ratio,
2323 const char*, query
2324 ),
2325 TP_FIELDS(
2326 ctf_string(query_field, query)
2327 ctf_float(double, ratio_field, ratio)
2328 ctf_integer(int, recv_size, my_custom_structure->recv_size)
2329 ctf_integer(int, send_size, my_custom_structure->send_size)
2330 )
2331)
2332----
2333
2334Refer to this tracepoint definition with the `tracepoint()` macro in
05bbb80b 2335the source code of your application like this:
96172a59
PP
2336
2337[source,c]
2338----
2339tracepoint(my_provider, my_tracepoint,
2340 my_structure, some_ratio, the_query);
2341----
2342====
2343
2344NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2345if they satisfy an enabled <<event,event rule>>.
2346
2347
2348[[using-tracepoint-classes]]
2349===== Use a tracepoint class
2350
2351A _tracepoint class_ is a class of tracepoints which share the same
2352output event field definitions. A _tracepoint instance_ is one
2353instance of such a defined tracepoint class, with its own tracepoint
2354name.
2355
2356The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2357shorthand which defines both a tracepoint class and a tracepoint
2358instance at the same time.
2359
2360When you build a tracepoint provider package, the C or $$C++$$ compiler
2361creates one serialization function for each **tracepoint class**. A
2362serialization function is responsible for serializing the event fields
2363of a tracepoint to a sub-buffer when tracing.
2364
2365For various performance reasons, when your situation requires multiple
2366tracepoint definitions with different names, but with the same event
2367fields, we recommend that you manually create a tracepoint class
2368and instantiate as many tracepoint instances as needed. One positive
2369effect of such a design, amongst other advantages, is that all
2370tracepoint instances of the same tracepoint class reuse the same
2371serialization function, thus reducing
2372https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2373
2374.Use a tracepoint class and tracepoint instances.
2375====
2376Consider the following three tracepoint definitions:
2377
2378[source,c]
2379----
2380TRACEPOINT_EVENT(
2381 my_app,
2382 get_account,
2383 TP_ARGS(
2384 int, userid,
2385 size_t, len
2386 ),
2387 TP_FIELDS(
2388 ctf_integer(int, userid, userid)
2389 ctf_integer(size_t, len, len)
2390 )
2391)
2392
2393TRACEPOINT_EVENT(
2394 my_app,
2395 get_settings,
2396 TP_ARGS(
2397 int, userid,
2398 size_t, len
2399 ),
2400 TP_FIELDS(
2401 ctf_integer(int, userid, userid)
2402 ctf_integer(size_t, len, len)
2403 )
2404)
2405
2406TRACEPOINT_EVENT(
2407 my_app,
2408 get_transaction,
2409 TP_ARGS(
2410 int, userid,
2411 size_t, len
2412 ),
2413 TP_FIELDS(
2414 ctf_integer(int, userid, userid)
2415 ctf_integer(size_t, len, len)
2416 )
2417)
2418----
2419
2420In this case, we create three tracepoint classes, with one implicit
2421tracepoint instance for each of them: `get_account`, `get_settings`, and
2422`get_transaction`. However, they all share the same event field names
2423and types. Hence three identical, yet independent serialization
2424functions are created when you build the tracepoint provider package.
2425
2426A better design choice is to define a single tracepoint class and three
2427tracepoint instances:
2428
2429[source,c]
2430----
2431/* The tracepoint class */
2432TRACEPOINT_EVENT_CLASS(
2433 /* Tracepoint provider name */
2434 my_app,
2435
2436 /* Tracepoint class name */
2437 my_class,
2438
2439 /* Input arguments */
2440 TP_ARGS(
2441 int, userid,
2442 size_t, len
2443 ),
2444
2445 /* Output event fields */
2446 TP_FIELDS(
2447 ctf_integer(int, userid, userid)
2448 ctf_integer(size_t, len, len)
2449 )
2450)
2451
2452/* The tracepoint instances */
2453TRACEPOINT_EVENT_INSTANCE(
2454 /* Tracepoint provider name */
2455 my_app,
2456
2457 /* Tracepoint class name */
2458 my_class,
2459
2460 /* Tracepoint name */
2461 get_account,
2462
2463 /* Input arguments */
2464 TP_ARGS(
2465 int, userid,
2466 size_t, len
2467 )
2468)
2469TRACEPOINT_EVENT_INSTANCE(
2470 my_app,
2471 my_class,
2472 get_settings,
2473 TP_ARGS(
2474 int, userid,
2475 size_t, len
2476 )
2477)
2478TRACEPOINT_EVENT_INSTANCE(
2479 my_app,
2480 my_class,
2481 get_transaction,
2482 TP_ARGS(
2483 int, userid,
2484 size_t, len
2485 )
2486)
2487----
2488====
2489
2490
2491[[assigning-log-levels]]
2492===== Assign a log level to a tracepoint definition
2493
2494Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
2495with the `TRACEPOINT_LOGLEVEL()` macro.
2496
2497Assigning different levels of severity to tracepoint definitions can
2498be useful: when you <<enabling-disabling-events,create an event rule>>,
2499you can target tracepoints having a log level as severe as a specific
2500value.
2501
2502The concept of LTTng-UST log levels is similar to the levels found
2503in typical logging frameworks:
2504
2505* In a logging framework, the log level is given by the function
2506 or method name you use at the log statement site: `debug()`,
2507 `info()`, `warn()`, `error()`, and so on.
2508* In LTTng-UST, you statically assign the log level to a tracepoint
2509 definition; any `tracepoint()` macro invocation which refers to
2510 this definition has this log level.
2511
2512You must use `TRACEPOINT_LOGLEVEL()` _after_ the
2513<<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2514<<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2515tracepoint.
2516
2517The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2518
2519[source,c]
2520.`TRACEPOINT_LOGLEVEL()` macro syntax.
2521----
2522TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2523----
2524
2525Replace:
2526
2527* `provider_name` with the tracepoint provider name.
2528* `tracepoint_name` with the tracepoint name.
2529* `log_level` with the log level to assign to the tracepoint
2530 definition named `tracepoint_name` in the `provider_name`
2531 tracepoint provider.
2532+
2533See man:lttng-ust(3) for a list of available log level names.
2534
2535.Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2536====
2537[source,c]
2538----
2539/* Tracepoint definition */
2540TRACEPOINT_EVENT(
2541 my_app,
2542 get_transaction,
2543 TP_ARGS(
2544 int, userid,
2545 size_t, len
2546 ),
2547 TP_FIELDS(
2548 ctf_integer(int, userid, userid)
2549 ctf_integer(size_t, len, len)
2550 )
2551)
2552
2553/* Log level assignment */
2554TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2555----
2556====
2557
2558
2559[[tpp-source]]
2560===== Create a tracepoint provider package source file
2561
2562A _tracepoint provider package source file_ is a C source file which
2563includes a <<tpp-header,tracepoint provider header file>> to expand its
2564macros into event serialization and other functions.
2565
2566Use the following tracepoint provider package source file template:
2567
2568[source,c]
2569.Tracepoint provider package source file template.
2570----
2571#define TRACEPOINT_CREATE_PROBES
2572
2573#include "tp.h"
2574----
2575
2576Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2577header file>> name. You may also include more than one tracepoint
2578provider header file here to create a tracepoint provider package
2579holding more than one tracepoint providers.
2580
2581
2582[[probing-the-application-source-code]]
05bbb80b 2583==== Add tracepoints to the source code of an application
96172a59 2584
05bbb80b
PP
2585Once you <<tpp-header,create a tracepoint provider header file>>, use
2586the `tracepoint()` macro in the source code of your application to
2587insert the tracepoints that this header
2588<<defining-tracepoints,defines>>.
96172a59
PP
2589
2590The `tracepoint()` macro takes at least two parameters: the tracepoint
2591provider name and the tracepoint name. The corresponding tracepoint
2592definition defines the other parameters.
2593
2594.`tracepoint()` usage.
2595====
2596The following <<defining-tracepoints,tracepoint definition>> defines a
2597tracepoint which takes two input arguments and has two output event
2598fields.
2599
2600[source,c]
2601.Tracepoint provider header file.
2602----
2603#include "my-custom-structure.h"
2604
2605TRACEPOINT_EVENT(
2606 my_provider,
2607 my_tracepoint,
2608 TP_ARGS(
2609 int, argc,
2610 const char*, cmd_name
2611 ),
2612 TP_FIELDS(
2613 ctf_string(cmd_name, cmd_name)
2614 ctf_integer(int, number_of_args, argc)
2615 )
2616)
2617----
2618
2619Refer to this tracepoint definition with the `tracepoint()` macro in
05bbb80b 2620the source code of your application like this:
96172a59
PP
2621
2622[source,c]
05bbb80b 2623.Application source file.
96172a59
PP
2624----
2625#include "tp.h"
2626
2627int main(int argc, char* argv[])
2628{
2629 tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2630
2631 return 0;
2632}
2633----
2634
05bbb80b 2635Note how the source code of the application includes
96172a59
PP
2636the tracepoint provider header file containing the tracepoint
2637definitions to use, path:{tp.h}.
2638====
2639
2640.`tracepoint()` usage with a complex tracepoint definition.
2641====
2642Consider this complex tracepoint definition, where multiple event
2643fields refer to the same input arguments in their argument expression
2644parameter:
2645
2646[source,c]
2647.Tracepoint provider header file.
2648----
2649/* For `struct stat` */
2650#include <sys/types.h>
2651#include <sys/stat.h>
2652#include <unistd.h>
2653
2654TRACEPOINT_EVENT(
2655 my_provider,
2656 my_tracepoint,
2657 TP_ARGS(
2658 int, my_int_arg,
2659 char*, my_str_arg,
2660 struct stat*, st
2661 ),
2662 TP_FIELDS(
2663 ctf_integer(int, my_constant_field, 23 + 17)
2664 ctf_integer(int, my_int_arg_field, my_int_arg)
2665 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2666 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2667 my_str_arg[2] + my_str_arg[3])
2668 ctf_string(my_str_arg_field, my_str_arg)
2669 ctf_integer_hex(off_t, size_field, st->st_size)
2670 ctf_float(double, size_dbl_field, (double) st->st_size)
2671 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2672 size_t, strlen(my_str_arg) / 2)
2673 )
2674)
2675----
2676
2677Refer to this tracepoint definition with the `tracepoint()` macro in
05bbb80b 2678the source code of your application like this:
96172a59
PP
2679
2680[source,c]
05bbb80b 2681.Application source file.
96172a59
PP
2682----
2683#define TRACEPOINT_DEFINE
2684#include "tp.h"
2685
2686int main(void)
2687{
2688 struct stat s;
2689
2690 stat("/etc/fstab", &s);
2691 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2692
2693 return 0;
2694}
2695----
2696
2697If you look at the event record that LTTng writes when tracing this
2698program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2699it should look like this:
2700
2701.Event record fields
2702|====
05bbb80b 2703|Field name |Field value
96172a59
PP
2704|`my_constant_field` |40
2705|`my_int_arg_field` |23
2706|`my_int_arg_field2` |529
2707|`sum4_field` |389
2708|`my_str_arg_field` |`Hello, World!`
2709|`size_field` |0x12d
2710|`size_dbl_field` |301.0
2711|`half_my_str_arg_field` |`Hello,`
2712|====
2713====
2714
2715Sometimes, the arguments you pass to `tracepoint()` are expensive to
2716compute--they use the call stack, for example. To avoid this computation
2717when the tracepoint is disabled, use the `tracepoint_enabled()` and
2718`do_tracepoint()` macros.
2719
2720The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2721is:
2722
2723[source,c]
2724.`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2725----
2726tracepoint_enabled(provider_name, tracepoint_name)
2727do_tracepoint(provider_name, tracepoint_name, ...)
2728----
2729
2730Replace:
2731
2732* `provider_name` with the tracepoint provider name.
2733* `tracepoint_name` with the tracepoint name.
2734
2735`tracepoint_enabled()` returns a non-zero value if the tracepoint named
2736`tracepoint_name` from the provider named `provider_name` is enabled
2737**at run time**.
2738
2739`do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2740if the tracepoint is enabled. Using `tracepoint()` with
2741`tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2742the `tracepoint_enabled()` check, thus a race condition is
2743possible in this situation:
2744
2745[source,c]
2746.Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2747----
2748if (tracepoint_enabled(my_provider, my_tracepoint)) {
2749 stuff = prepare_stuff();
2750}
2751
2752tracepoint(my_provider, my_tracepoint, stuff);
2753----
2754
2755If the tracepoint is enabled after the condition, then `stuff` isn't
2756prepared: the emitted event will either contain wrong data, or the whole
2757application could crash (segmentation fault, for example).
2758
2759NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2760`STAP_PROBEV()` call. If you need it, you must emit
2761this call yourself.
2762
2763
2764[[building-tracepoint-providers-and-user-application]]
2765==== Build and link a tracepoint provider package and an application
2766
2767Once you have one or more <<tpp-header,tracepoint provider header
2768files>> and a <<tpp-source,tracepoint provider package source file>>,
2769create the tracepoint provider package by compiling its source
2770file. From here, multiple build and run scenarios are possible. The
2771following table shows common application and library configurations
2772along with the required command lines to achieve them.
2773
2774In the following diagrams, we use the following file names:
2775
2776`app`::
2777 Executable application.
2778
2779`app.o`::
05bbb80b 2780 Application object file.
96172a59
PP
2781
2782`tpp.o`::
2783 Tracepoint provider package object file.
2784
2785`tpp.a`::
2786 Tracepoint provider package archive file.
2787
2788`libtpp.so`::
2789 Tracepoint provider package shared object file.
2790
2791`emon.o`::
2792 User library object file.
2793
2794`libemon.so`::
2795 User library shared object file.
2796
2797We use the following symbols in the diagrams of table below:
2798
2799[role="img-100"]
2800.Symbols used in the build scenario diagrams.
2801image::ust-sit-symbols.png[]
2802
2803We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2804variable in the following instructions.
2805
2806[role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2807.Common tracepoint provider package scenarios.
2808|====
2809|Scenario |Instructions
2810
2811|
2812The instrumented application is statically linked with
2813the tracepoint provider package object.
2814
2815image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2816
2817|
2818include::../common/ust-sit-step-tp-o.txt[]
2819
2820To build the instrumented application:
2821
2822. In path:{app.c}, before including path:{tpp.h}, add the following line:
2823+
2824--
2825[source,c]
2826----
2827#define TRACEPOINT_DEFINE
2828----
2829--
2830
2831. Compile the application source file:
2832+
2833--
2834[role="term"]
2835----
2836$ gcc -c app.c
2837----
2838--
2839
2840. Build the application:
2841+
2842--
2843[role="term"]
2844----
2845$ gcc -o app app.o tpp.o -llttng-ust -ldl
2846----
2847--
2848
2849To run the instrumented application:
2850
2851* Start the application:
2852+
2853--
2854[role="term"]
2855----
2856$ ./app
2857----
2858--
2859
2860|
2861The instrumented application is statically linked with the
2862tracepoint provider package archive file.
2863
2864image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2865
2866|
2867To create the tracepoint provider package archive file:
2868
2869. Compile the <<tpp-source,tracepoint provider package source file>>:
2870+
2871--
2872[role="term"]
2873----
2874$ gcc -I. -c tpp.c
2875----
2876--
2877
2878. Create the tracepoint provider package archive file:
2879+
2880--
2881[role="term"]
2882----
2883$ ar rcs tpp.a tpp.o
2884----
2885--
2886
2887To build the instrumented application:
2888
2889. In path:{app.c}, before including path:{tpp.h}, add the following line:
2890+
2891--
2892[source,c]
2893----
2894#define TRACEPOINT_DEFINE
2895----
2896--
2897
2898. Compile the application source file:
2899+
2900--
2901[role="term"]
2902----
2903$ gcc -c app.c
2904----
2905--
2906
2907. Build the application:
2908+
2909--
2910[role="term"]
2911----
2912$ gcc -o app app.o tpp.a -llttng-ust -ldl
2913----
2914--
2915
2916To run the instrumented application:
2917
2918* Start the application:
2919+
2920--
2921[role="term"]
2922----
2923$ ./app
2924----
2925--
2926
2927|
2928The instrumented application is linked with the tracepoint provider
2929package shared object.
2930
2931image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
2932
2933|
2934include::../common/ust-sit-step-tp-so.txt[]
2935
2936To build the instrumented application:
2937
2938. In path:{app.c}, before including path:{tpp.h}, add the following line:
2939+
2940--
2941[source,c]
2942----
2943#define TRACEPOINT_DEFINE
2944----
2945--
2946
2947. Compile the application source file:
2948+
2949--
2950[role="term"]
2951----
2952$ gcc -c app.c
2953----
2954--
2955
2956. Build the application:
2957+
2958--
2959[role="term"]
2960----
2961$ gcc -o app app.o -ldl -L. -ltpp
2962----
2963--
2964
2965To run the instrumented application:
2966
2967* Start the application:
2968+
2969--
2970[role="term"]
2971----
2972$ ./app
2973----
2974--
2975
2976|
2977The tracepoint provider package shared object is preloaded before the
2978instrumented application starts.
2979
2980image::ust-sit+tp-so-preloaded+app-instrumented.png[]
2981
2982|
2983include::../common/ust-sit-step-tp-so.txt[]
2984
2985To build the instrumented application:
2986
2987. In path:{app.c}, before including path:{tpp.h}, add the
2988 following lines:
2989+
2990--
2991[source,c]
2992----
2993#define TRACEPOINT_DEFINE
2994#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2995----
2996--
2997
2998. Compile the application source file:
2999+
3000--
3001[role="term"]
3002----
3003$ gcc -c app.c
3004----
3005--
3006
3007. Build the application:
3008+
3009--
3010[role="term"]
3011----
3012$ gcc -o app app.o -ldl
3013----
3014--
3015
3016To run the instrumented application with tracing support:
3017
3018* Preload the tracepoint provider package shared object and
3019 start the application:
3020+
3021--
3022[role="term"]
3023----
3024$ LD_PRELOAD=./libtpp.so ./app
3025----
3026--
3027
3028To run the instrumented application without tracing support:
3029
3030* Start the application:
3031+
3032--
3033[role="term"]
3034----
3035$ ./app
3036----
3037--
3038
3039|
3040The instrumented application dynamically loads the tracepoint provider
3041package shared object.
3042
3043image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3044
3045|
3046include::../common/ust-sit-step-tp-so.txt[]
3047
3048To build the instrumented application:
3049
3050. In path:{app.c}, before including path:{tpp.h}, add the
3051 following lines:
3052+
3053--
3054[source,c]
3055----
3056#define TRACEPOINT_DEFINE
3057#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3058----
3059--
3060
3061. Compile the application source file:
3062+
3063--
3064[role="term"]
3065----
3066$ gcc -c app.c
3067----
3068--
3069
3070. Build the application:
3071+
3072--
3073[role="term"]
3074----
3075$ gcc -o app app.o -ldl
3076----
3077--
3078
3079To run the instrumented application:
3080
3081* Start the application:
3082+
3083--
3084[role="term"]
3085----
3086$ ./app
3087----
3088--
3089
3090|
3091The application is linked with the instrumented user library.
3092
3093The instrumented user library is statically linked with the tracepoint
3094provider package object file.
3095
3096image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3097
3098|
3099include::../common/ust-sit-step-tp-o-fpic.txt[]
3100
3101To build the instrumented user library:
3102
3103. In path:{emon.c}, before including path:{tpp.h}, add the
3104 following line:
3105+
3106--
3107[source,c]
3108----
3109#define TRACEPOINT_DEFINE
3110----
3111--
3112
3113. Compile the user library source file:
3114+
3115--
3116[role="term"]
3117----
3118$ gcc -I. -fpic -c emon.c
3119----
3120--
3121
3122. Build the user library shared object:
3123+
3124--
3125[role="term"]
3126----
3127$ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3128----
3129--
3130
3131To build the application:
3132
3133. Compile the application source file:
3134+
3135--
3136[role="term"]
3137----
3138$ gcc -c app.c
3139----
3140--
3141
3142. Build the application:
3143+
3144--
3145[role="term"]
3146----
3147$ gcc -o app app.o -L. -lemon
3148----
3149--
3150
3151To run the application:
3152
3153* Start the application:
3154+
3155--
3156[role="term"]
3157----
3158$ ./app
3159----
3160--
3161
3162|
3163The application is linked with the instrumented user library.
3164
3165The instrumented user library is linked with the tracepoint provider
3166package shared object.
3167
3168image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3169
3170|
3171include::../common/ust-sit-step-tp-so.txt[]
3172
3173To build the instrumented user library:
3174
3175. In path:{emon.c}, before including path:{tpp.h}, add the
3176 following line:
3177+
3178--
3179[source,c]
3180----
3181#define TRACEPOINT_DEFINE
3182----
3183--
3184
3185. Compile the user library source file:
3186+
3187--
3188[role="term"]
3189----
3190$ gcc -I. -fpic -c emon.c
3191----
3192--
3193
3194. Build the user library shared object:
3195+
3196--
3197[role="term"]
3198----
3199$ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3200----
3201--
3202
3203To build the application:
3204
3205. Compile the application source file:
3206+
3207--
3208[role="term"]
3209----
3210$ gcc -c app.c
3211----
3212--
3213
3214. Build the application:
3215+
3216--
3217[role="term"]
3218----
3219$ gcc -o app app.o -L. -lemon
3220----
3221--
3222
3223To run the application:
3224
3225* Start the application:
3226+
3227--
3228[role="term"]
3229----
3230$ ./app
3231----
3232--
3233
3234|
3235The tracepoint provider package shared object is preloaded before the
3236application starts.
3237
3238The application is linked with the instrumented user library.
3239
3240image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3241
3242|
3243include::../common/ust-sit-step-tp-so.txt[]
3244
3245To build the instrumented user library:
3246
3247. In path:{emon.c}, before including path:{tpp.h}, add the
3248 following lines:
3249+
3250--
3251[source,c]
3252----
3253#define TRACEPOINT_DEFINE
3254#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3255----
3256--
3257
3258. Compile the user library source file:
3259+
3260--
3261[role="term"]
3262----
3263$ gcc -I. -fpic -c emon.c
3264----
3265--
3266
3267. Build the user library shared object:
3268+
3269--
3270[role="term"]
3271----
3272$ gcc -shared -o libemon.so emon.o -ldl
3273----
3274--
3275
3276To build the application:
3277
3278. Compile the application source file:
3279+
3280--
3281[role="term"]
3282----
3283$ gcc -c app.c
3284----
3285--
3286
3287. Build the application:
3288+
3289--
3290[role="term"]
3291----
3292$ gcc -o app app.o -L. -lemon
3293----
3294--
3295
3296To run the application with tracing support:
3297
3298* Preload the tracepoint provider package shared object and
3299 start the application:
3300+
3301--
3302[role="term"]
3303----
3304$ LD_PRELOAD=./libtpp.so ./app
3305----
3306--
3307
3308To run the application without tracing support:
3309
3310* Start the application:
3311+
3312--
3313[role="term"]
3314----
3315$ ./app
3316----
3317--
3318
3319|
3320The application is linked with the instrumented user library.
3321
3322The instrumented user library dynamically loads the tracepoint provider
3323package shared object.
3324
3325image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3326
3327|
3328include::../common/ust-sit-step-tp-so.txt[]
3329
3330To build the instrumented user library:
3331
3332. In path:{emon.c}, before including path:{tpp.h}, add the
3333 following lines:
3334+
3335--
3336[source,c]
3337----
3338#define TRACEPOINT_DEFINE
3339#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3340----
3341--
3342
3343. Compile the user library source file:
3344+
3345--
3346[role="term"]
3347----
3348$ gcc -I. -fpic -c emon.c
3349----
3350--
3351
3352. Build the user library shared object:
3353+
3354--
3355[role="term"]
3356----
3357$ gcc -shared -o libemon.so emon.o -ldl
3358----
3359--
3360
3361To build the application:
3362
3363. Compile the application source file:
3364+
3365--
3366[role="term"]
3367----
3368$ gcc -c app.c
3369----
3370--
3371
3372. Build the application:
3373+
3374--
3375[role="term"]
3376----
3377$ gcc -o app app.o -L. -lemon
3378----
3379--
3380
3381To run the application:
3382
3383* Start the application:
3384+
3385--
3386[role="term"]
3387----
3388$ ./app
3389----
3390--
3391
3392|
3393The application dynamically loads the instrumented user library.
3394
3395The instrumented user library is linked with the tracepoint provider
3396package shared object.
3397
3398image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3399
3400|
3401include::../common/ust-sit-step-tp-so.txt[]
3402
3403To build the instrumented user library:
3404
3405. In path:{emon.c}, before including path:{tpp.h}, add the
3406 following line:
3407+
3408--
3409[source,c]
3410----
3411#define TRACEPOINT_DEFINE
3412----
3413--
3414
3415. Compile the user library source file:
3416+
3417--
3418[role="term"]
3419----
3420$ gcc -I. -fpic -c emon.c
3421----
3422--
3423
3424. Build the user library shared object:
3425+
3426--
3427[role="term"]
3428----
3429$ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3430----
3431--
3432
3433To build the application:
3434
3435. Compile the application source file:
3436+
3437--
3438[role="term"]
3439----
3440$ gcc -c app.c
3441----
3442--
3443
3444. Build the application:
3445+
3446--
3447[role="term"]
3448----
3449$ gcc -o app app.o -ldl -L. -lemon
3450----
3451--
3452
3453To run the application:
3454
3455* Start the application:
3456+
3457--
3458[role="term"]
3459----
3460$ ./app
3461----
3462--
3463
3464|
3465The application dynamically loads the instrumented user library.
3466
3467The instrumented user library dynamically loads the tracepoint provider
3468package shared object.
3469
3470image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3471
3472|
3473include::../common/ust-sit-step-tp-so.txt[]
3474
3475To build the instrumented user library:
3476
3477. In path:{emon.c}, before including path:{tpp.h}, add the
3478 following lines:
3479+
3480--
3481[source,c]
3482----
3483#define TRACEPOINT_DEFINE
3484#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3485----
3486--
3487
3488. Compile the user library source file:
3489+
3490--
3491[role="term"]
3492----
3493$ gcc -I. -fpic -c emon.c
3494----
3495--
3496
3497. Build the user library shared object:
3498+
3499--
3500[role="term"]
3501----
3502$ gcc -shared -o libemon.so emon.o -ldl
3503----
3504--
3505
3506To build the application:
3507
3508. Compile the application source file:
3509+
3510--
3511[role="term"]
3512----
3513$ gcc -c app.c
3514----
3515--
3516
3517. Build the application:
3518+
3519--
3520[role="term"]
3521----
3522$ gcc -o app app.o -ldl -L. -lemon
3523----
3524--
3525
3526To run the application:
3527
3528* Start the application:
3529+
3530--
3531[role="term"]
3532----
3533$ ./app
3534----
3535--
3536
3537|
3538The tracepoint provider package shared object is preloaded before the
3539application starts.
3540
3541The application dynamically loads the instrumented user library.
3542
3543image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3544
3545|
3546include::../common/ust-sit-step-tp-so.txt[]
3547
3548To build the instrumented user library:
3549
3550. In path:{emon.c}, before including path:{tpp.h}, add the
3551 following lines:
3552+
3553--
3554[source,c]
3555----
3556#define TRACEPOINT_DEFINE
3557#define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3558----
3559--
3560
3561. Compile the user library source file:
3562+
3563--
3564[role="term"]
3565----
3566$ gcc -I. -fpic -c emon.c
3567----
3568--
3569
3570. Build the user library shared object:
3571+
3572--
3573[role="term"]
3574----
3575$ gcc -shared -o libemon.so emon.o -ldl
3576----
3577--
3578
3579To build the application:
3580
3581. Compile the application source file:
3582+
3583--
3584[role="term"]
3585----
3586$ gcc -c app.c
3587----
3588--
3589
3590. Build the application:
3591+
3592--
3593[role="term"]
3594----
3595$ gcc -o app app.o -L. -lemon
3596----
3597--
3598
3599To run the application with tracing support:
3600
3601* Preload the tracepoint provider package shared object and
3602 start the application:
3603+
3604--
3605[role="term"]
3606----
3607$ LD_PRELOAD=./libtpp.so ./app
3608----
3609--
3610
3611To run the application without tracing support:
3612
3613* Start the application:
3614+
3615--
3616[role="term"]
3617----
3618$ ./app
3619----
3620--
3621
3622|
3623The application is statically linked with the tracepoint provider
3624package object file.
3625
3626The application is linked with the instrumented user library.
3627
3628image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3629
3630|
3631include::../common/ust-sit-step-tp-o.txt[]
3632
3633To build the instrumented user library:
3634
3635. In path:{emon.c}, before including path:{tpp.h}, add the
3636 following line:
3637+
3638--
3639[source,c]
3640----
3641#define TRACEPOINT_DEFINE
3642----
3643--
3644
3645. Compile the user library source file:
3646+
3647--
3648[role="term"]
3649----
3650$ gcc -I. -fpic -c emon.c
3651----
3652--
3653
3654. Build the user library shared object:
3655+
3656--
3657[role="term"]
3658----
3659$ gcc -shared -o libemon.so emon.o
3660----
3661--
3662
3663To build the application:
3664
3665. Compile the application source file:
3666+
3667--
3668[role="term"]
3669----
3670$ gcc -c app.c
3671----
3672--
3673
3674. Build the application:
3675+
3676--
3677[role="term"]
3678----
3679$ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3680----
3681--
3682
3683To run the instrumented application:
3684
3685* Start the application:
3686+
3687--
3688[role="term"]
3689----
3690$ ./app
3691----
3692--
3693
3694|
3695The application is statically linked with the tracepoint provider
3696package object file.
3697
3698The application dynamically loads the instrumented user library.
3699
3700image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3701
3702|
3703include::../common/ust-sit-step-tp-o.txt[]
3704
3705To build the application:
3706
3707. In path:{app.c}, before including path:{tpp.h}, add the following line:
3708+
3709--
3710[source,c]
3711----
3712#define TRACEPOINT_DEFINE
3713----
3714--
3715
3716. Compile the application source file:
3717+
3718--
3719[role="term"]
3720----
3721$ gcc -c app.c
3722----
3723--
3724
3725. Build the application:
3726+
3727--
3728[role="term"]
3729----
3730$ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3731 -llttng-ust -ldl
3732----
3733--
3734+
3735The `--export-dynamic` option passed to the linker is necessary for the
3736dynamically loaded library to ``see'' the tracepoint symbols defined in
3737the application.
3738
3739To build the instrumented user library:
3740
3741. Compile the user library source file:
3742+
3743--
3744[role="term"]
3745----
3746$ gcc -I. -fpic -c emon.c
3747----
3748--
3749
3750. Build the user library shared object:
3751+
3752--
3753[role="term"]
3754----
3755$ gcc -shared -o libemon.so emon.o
3756----
3757--
3758
3759To run the application:
3760
3761* Start the application:
3762+
3763--
3764[role="term"]
3765----
3766$ ./app
3767----
3768--
3769|====
3770
3771
3772[[using-lttng-ust-with-daemons]]
3773===== Use noch:{LTTng-UST} with daemons
3774
3775If your instrumented application calls man:fork(2), man:clone(2),
3776or BSD's man:rfork(2), without a following man:exec(3)-family
3777system call, you must preload the path:{liblttng-ust-fork.so} shared
3778object when you start the application.
3779
3780[role="term"]
3781----
3782$ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3783----
3784
3785If your tracepoint provider package is
3786a shared library which you also preload, you must put both
3787shared objects in env:LD_PRELOAD:
3788
3789[role="term"]
3790----
3791$ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3792----
3793
3794
3795[role="since-2.9"]
3796[[liblttng-ust-fd]]
3797===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
3798
3799If your instrumented application closes one or more file descriptors
3800which it did not open itself, you must preload the
3801path:{liblttng-ust-fd.so} shared object when you start the application:
3802
3803[role="term"]
3804----
3805$ LD_PRELOAD=liblttng-ust-fd.so ./my-app
3806----
3807
3808Typical use cases include closing all the file descriptors after
3809man:fork(2) or man:rfork(2) and buggy applications doing
3810``double closes''.
3811
3812
3813[[lttng-ust-pkg-config]]
3814===== Use noch:{pkg-config}
3815
3816On some distributions, LTTng-UST ships with a
3817https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3818metadata file. If this is your case, then use cmd:pkg-config to
3819build an application on the command line:
3820
3821[role="term"]
3822----
3823$ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3824----
3825
3826
3827[[instrumenting-32-bit-app-on-64-bit-system]]
3828===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3829
3830In order to trace a 32-bit application running on a 64-bit system,
3831LTTng must use a dedicated 32-bit
3832<<lttng-consumerd,consumer daemon>>.
3833
3834The following steps show how to build and install a 32-bit consumer
3835daemon, which is _not_ part of the default 64-bit LTTng build, how to
3836build and install the 32-bit LTTng-UST libraries, and how to build and
3837link an instrumented 32-bit application in that context.
3838
3839To build a 32-bit instrumented application for a 64-bit target system,
3840assuming you have a fresh target system with no installed Userspace RCU
3841or LTTng packages:
3842
3843. Download, build, and install a 32-bit version of Userspace RCU:
3844+
3845--
3846[role="term"]
3847----
3848$ cd $(mktemp -d) &&
3849wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3850tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3851cd userspace-rcu-0.9.* &&
3852./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3853make &&
3854sudo make install &&
3855sudo ldconfig
3856----
3857--
3858
05bbb80b
PP
3859. Using the package manager of your distribution, or from source,
3860 install the following 32-bit versions of the following dependencies of
96172a59
PP
3861 LTTng-tools and LTTng-UST:
3862+
3863--
3864* https://sourceforge.net/projects/libuuid/[libuuid]
3865* http://directory.fsf.org/wiki/Popt[popt]
3866* http://www.xmlsoft.org/[libxml2]
3867--
3868
3869. Download, build, and install a 32-bit version of the latest
3870 LTTng-UST{nbsp}{revision}:
3871+
3872--
3873[role="term"]
3874----
3875$ cd $(mktemp -d) &&
3876wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
3877tar -xf lttng-ust-latest-2.12.tar.bz2 &&
3878cd lttng-ust-2.12.* &&
3879./configure --libdir=/usr/local/lib32 \
3880 CFLAGS=-m32 CXXFLAGS=-m32 \
3881 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
3882make &&
3883sudo make install &&
3884sudo ldconfig
3885----
3886--
3887+
3888[NOTE]
3889====
3890Depending on your distribution,
389132-bit libraries could be installed at a different location than
3892`/usr/lib32`. For example, Debian is known to install
3893some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
3894
3895In this case, make sure to set `LDFLAGS` to all the
3896relevant 32-bit library paths, for example:
3897
3898[role="term"]
3899----
3900$ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
3901----
3902====
3903
3904. Download the latest LTTng-tools{nbsp}{revision}, build, and install
3905 the 32-bit consumer daemon:
3906+
3907--
3908[role="term"]
3909----
3910$ cd $(mktemp -d) &&
3911wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
3912tar -xf lttng-tools-latest-2.12.tar.bz2 &&
3913cd lttng-tools-2.12.* &&
3914./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3915 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
3916 --disable-bin-lttng --disable-bin-lttng-crash \
3917 --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
3918make &&
3919cd src/bin/lttng-consumerd &&
3920sudo make install &&
3921sudo ldconfig
3922----
3923--
3924
3925. From your distribution or from source,
3926 <<installing-lttng,install>> the 64-bit versions of
3927 LTTng-UST and Userspace RCU.
3928. Download, build, and install the 64-bit version of the
3929 latest LTTng-tools{nbsp}{revision}:
3930+
3931--
3932[role="term"]
3933----
3934$ cd $(mktemp -d) &&
3935wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
3936tar -xf lttng-tools-latest-2.12.tar.bz2 &&
3937cd lttng-tools-2.12.* &&
3938./configure --with-consumerd32-libdir=/usr/local/lib32 \
3939 --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
3940make &&
3941sudo make install &&
3942sudo ldconfig
3943----
3944--
3945
3946. Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
3947 when linking your 32-bit application:
3948+
3949----
3950-m32 -L/usr/lib32 -L/usr/local/lib32 \
3951-Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
3952----
3953+
3954For example, let's rebuild the quick start example in
3955<<tracing-your-own-user-application,Trace a user application>> as an
3956instrumented 32-bit application:
3957+
3958--
3959[role="term"]
3960----
3961$ gcc -m32 -c -I. hello-tp.c
3962$ gcc -m32 -c hello.c
3963$ gcc -m32 -o hello hello.o hello-tp.o \
3964 -L/usr/lib32 -L/usr/local/lib32 \
3965 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
3966 -llttng-ust -ldl
3967----
3968--
3969
3970No special action is required to execute the 32-bit application and
3971to trace it: use the command-line man:lttng(1) tool as usual.
3972
3973
3974[role="since-2.5"]
3975[[tracef]]
3976==== Use `tracef()`
3977
3978man:tracef(3) is a small LTTng-UST API designed for quick,
3979man:printf(3)-like instrumentation without the burden of
3980<<tracepoint-provider,creating>> and
3981<<building-tracepoint-providers-and-user-application,building>>
3982a tracepoint provider package.
3983
3984To use `tracef()` in your application:
3985
3986. In the C or C++ source files where you need to use `tracef()`,
3987 include `<lttng/tracef.h>`:
3988+
3989--
3990[source,c]
3991----
3992#include <lttng/tracef.h>
3993----
3994--
3995
05bbb80b
PP
3996. In the source code of the application, use `tracef()` like you would
3997 use man:printf(3):
96172a59
PP
3998+
3999--
4000[source,c]
4001----
4002 /* ... */
4003
4004 tracef("my message: %d (%s)", my_integer, my_string);
4005
4006 /* ... */
4007----
4008--
4009
4010. Link your application with `liblttng-ust`:
4011+
4012--
4013[role="term"]
4014----
4015$ gcc -o app app.c -llttng-ust
4016----
4017--
4018
4019To trace the events that `tracef()` calls emit:
4020
4021* <<enabling-disabling-events,Create an event rule>> which matches the
4022 `lttng_ust_tracef:*` event name:
4023+
4024--
4025[role="term"]
4026----
4027$ lttng enable-event --userspace 'lttng_ust_tracef:*'
4028----
4029--
4030
4031[IMPORTANT]
4032.Limitations of `tracef()`
4033====
4034The `tracef()` utility function was developed to make user space tracing
4035super simple, albeit with notable disadvantages compared to
4036<<defining-tracepoints,user-defined tracepoints>>:
4037
4038* All the emitted events have the same tracepoint provider and
4039 tracepoint names, respectively `lttng_ust_tracef` and `event`.
4040* There is no static type checking.
4041* The only event record field you actually get, named `msg`, is a string
4042 potentially containing the values you passed to `tracef()`
4043 using your own format string. This also means that you can't filter
4044 events with a custom expression at run time because there are no
4045 isolated fields.
05bbb80b
PP
4046* Since `tracef()` uses the man:vasprintf(3) function of the
4047 C{nbsp}standard library behind the scenes to format the strings at run
4048 time, its expected performance is lower than with user-defined
4049 tracepoints, which don't require a conversion to a string.
96172a59
PP
4050
4051Taking this into consideration, `tracef()` is useful for some quick
4052prototyping and debugging, but you shouldn't consider it for any
4053permanent and serious applicative instrumentation.
4054====
4055
4056
4057[role="since-2.7"]
4058[[tracelog]]
4059==== Use `tracelog()`
4060
4061The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
4062the difference that it accepts an additional log level parameter.
4063
4064The goal of `tracelog()` is to ease the migration from logging to
4065tracing.
4066
4067To use `tracelog()` in your application:
4068
4069. In the C or C++ source files where you need to use `tracelog()`,
4070 include `<lttng/tracelog.h>`:
4071+
4072--
4073[source,c]
4074----
4075#include <lttng/tracelog.h>
4076----
4077--
4078
05bbb80b
PP
4079. In the source code of the application, use `tracelog()` like you would
4080 use man:printf(3), except for the first parameter which is the log
96172a59
PP
4081 level:
4082+
4083--
4084[source,c]
4085----
4086 /* ... */
4087
4088 tracelog(TRACE_WARNING, "my message: %d (%s)",
4089 my_integer, my_string);
4090
4091 /* ... */
4092----
4093--
4094+
4095See man:lttng-ust(3) for a list of available log level names.
4096
4097. Link your application with `liblttng-ust`:
4098+
4099--
4100[role="term"]
4101----
4102$ gcc -o app app.c -llttng-ust
4103----
4104--
4105
4106To trace the events that `tracelog()` calls emit with a log level
4107_as severe as_ a specific log level:
4108
4109* <<enabling-disabling-events,Create an event rule>> which matches the
4110 `lttng_ust_tracelog:*` event name and a minimum level
4111 of severity:
4112+
4113--
4114[role="term"]
4115----
4116$ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4117 --loglevel=TRACE_WARNING
4118----
4119--
4120
4121To trace the events that `tracelog()` calls emit with a
4122_specific log level_:
4123
4124* Create an event rule which matches the `lttng_ust_tracelog:*`
4125 event name and a specific log level:
4126+
4127--
4128[role="term"]
4129----
4130$ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4131 --loglevel-only=TRACE_INFO
4132----
4133--
4134
4135
4136[[prebuilt-ust-helpers]]
4137=== Prebuilt user space tracing helpers
4138
e9acbade 4139The LTTng-UST package provides a few helpers in the form of preloadable
96172a59
PP
4140shared objects which automatically instrument system functions and
4141calls.
4142
4143The helper shared objects are normally found in dir:{/usr/lib}. If you
4144built LTTng-UST <<building-from-source,from source>>, they are probably
4145located in dir:{/usr/local/lib}.
4146
4147The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4148are:
4149
4150path:{liblttng-ust-libc-wrapper.so}::
4151path:{liblttng-ust-pthread-wrapper.so}::
4152 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4153 memory and POSIX threads function tracing>>.
4154
4155path:{liblttng-ust-cyg-profile.so}::
4156path:{liblttng-ust-cyg-profile-fast.so}::
4157 <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4158
4159path:{liblttng-ust-dl.so}::
4160 <<liblttng-ust-dl,Dynamic linker tracing>>.
4161
4162To use a user space tracing helper with any user application:
4163
4164* Preload the helper shared object when you start the application:
4165+
4166--
4167[role="term"]
4168----
4169$ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4170----
4171--
4172+
4173You can preload more than one helper:
4174+
4175--
4176[role="term"]
4177----
4178$ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4179----
4180--
4181
4182
4183[role="since-2.3"]
4184[[liblttng-ust-libc-pthread-wrapper]]
4185==== Instrument C standard library memory and POSIX threads functions
4186
4187The path:{liblttng-ust-libc-wrapper.so} and
4188path:{liblttng-ust-pthread-wrapper.so} helpers
4189add instrumentation to some C standard library and POSIX
4190threads functions.
4191
4192[role="growable"]
4193.Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4194|====
4195|TP provider name |TP name |Instrumented function
4196
4197.6+|`lttng_ust_libc` |`malloc` |man:malloc(3)
4198 |`calloc` |man:calloc(3)
4199 |`realloc` |man:realloc(3)
4200 |`free` |man:free(3)
4201 |`memalign` |man:memalign(3)
4202 |`posix_memalign` |man:posix_memalign(3)
4203|====
4204
4205[role="growable"]
4206.Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4207|====
4208|TP provider name |TP name |Instrumented function
4209
4210.4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4211 |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4212 |`pthread_mutex_trylock` |man:pthread_mutex_trylock(3p)
4213 |`pthread_mutex_unlock` |man:pthread_mutex_unlock(3p)
4214|====
4215
4216When you preload the shared object, it replaces the functions listed
4217in the previous tables by wrappers which contain tracepoints and call
4218the replaced functions.
4219
4220
4221[[liblttng-ust-cyg-profile]]
4222==== Instrument function entry and exit
4223
4224The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4225to the entry and exit points of functions.
4226
4227man:gcc(1) and man:clang(1) have an option named
4228https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4229which generates instrumentation calls for entry and exit to functions.
4230The LTTng-UST function tracing helpers,
4231path:{liblttng-ust-cyg-profile.so} and
4232path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4233to add tracepoints to the two generated functions (which contain
05bbb80b 4234`cyg_profile` in their names, hence the name of the helper).
96172a59
PP
4235
4236To use the LTTng-UST function tracing helper, the source files to
4237instrument must be built using the `-finstrument-functions` compiler
4238flag.
4239
4240There are two versions of the LTTng-UST function tracing helper:
4241
4242* **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4243 that you should only use when it can be _guaranteed_ that the
4244 complete event stream is recorded without any lost event record.
4245 Any kind of duplicate information is left out.
4246+
4247Assuming no event record is lost, having only the function addresses on
4248entry is enough to create a call graph, since an event record always
4249contains the ID of the CPU that generated it.
4250+
4251Use a tool like man:addr2line(1) to convert function addresses back to
4252source file names and line numbers.
4253
4254* **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4255which also works in use cases where event records might get discarded or
4256not recorded from application startup.
4257In these cases, the trace analyzer needs more information to be
4258able to reconstruct the program flow.
4259
4260See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4261points of this helper.
4262
4263All the tracepoints that this helper provides have the
4264log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4265
4266TIP: It's sometimes a good idea to limit the number of source files that
4267you compile with the `-finstrument-functions` option to prevent LTTng
4268from writing an excessive amount of trace data at run time. When using
4269man:gcc(1), use the
4270`-finstrument-functions-exclude-function-list` option to avoid
4271instrument entries and exits of specific function names.
4272
4273
4274[role="since-2.4"]
4275[[liblttng-ust-dl]]
4276==== Instrument the dynamic linker
4277
4278The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4279man:dlopen(3) and man:dlclose(3) function calls.
4280
4281See man:lttng-ust-dl(3) to learn more about the instrumentation points
4282of this helper.
4283
4284
4285[role="since-2.4"]
4286[[java-application]]
4287=== User space Java agent
4288
4289You can instrument any Java application which uses one of the following
4290logging frameworks:
4291
4292* The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4293 (JUL) core logging facilities.
4294* http://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4295 LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4296
4297[role="img-100"]
4298.LTTng-UST Java agent imported by a Java application.
4299image::java-app.png[]
4300
4301Note that the methods described below are new in LTTng{nbsp}2.8.
4302Previous LTTng versions use another technique.
4303
4304NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4305and https://ci.lttng.org/[continuous integration], thus this version is
4306directly supported. However, the LTTng-UST Java agent is also tested
4307with OpenJDK{nbsp}7.
4308
4309
4310[role="since-2.8"]
4311[[jul]]
4312==== Use the LTTng-UST Java agent for `java.util.logging`
4313
4314To use the LTTng-UST Java agent in a Java application which uses
4315`java.util.logging` (JUL):
4316
05bbb80b
PP
4317. In the source code of the Java application, import the LTTng-UST log
4318 handler package for `java.util.logging`:
96172a59
PP
4319+
4320--
4321[source,java]
4322----
4323import org.lttng.ust.agent.jul.LttngLogHandler;
4324----
4325--
4326
4327. Create an LTTng-UST JUL log handler:
4328+
4329--
4330[source,java]
4331----
4332Handler lttngUstLogHandler = new LttngLogHandler();
4333----
4334--
4335
4336. Add this handler to the JUL loggers which should emit LTTng events:
4337+
4338--
4339[source,java]
4340----
4341Logger myLogger = Logger.getLogger("some-logger");
4342
4343myLogger.addHandler(lttngUstLogHandler);
4344----
4345--
4346
4347. Use `java.util.logging` log statements and configuration as usual.
4348 The loggers with an attached LTTng-UST log handler can emit
4349 LTTng events.
4350
4351. Before exiting the application, remove the LTTng-UST log handler from
4352 the loggers attached to it and call its `close()` method:
4353+
4354--
4355[source,java]
4356----
4357myLogger.removeHandler(lttngUstLogHandler);
4358lttngUstLogHandler.close();
4359----
4360--
4361+
4362This isn't strictly necessary, but it is recommended for a clean
05bbb80b 4363disposal of the resources of the handler.
96172a59 4364
05bbb80b 4365. Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
96172a59
PP
4366 path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4367 in the
4368 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4369 path] when you build the Java application.
4370+
4371The JAR files are typically located in dir:{/usr/share/java}.
4372+
4373IMPORTANT: The LTTng-UST Java agent must be
4374<<installing-lttng,installed>> for the logging framework your
4375application uses.
4376
4377.Use the LTTng-UST Java agent for `java.util.logging`.
4378====
4379[source,java]
4380.path:{Test.java}
4381----
4382import java.io.IOException;
4383import java.util.logging.Handler;
4384import java.util.logging.Logger;
4385import org.lttng.ust.agent.jul.LttngLogHandler;
4386
4387public class Test
4388{
4389 private static final int answer = 42;
4390
4391 public static void main(String[] argv) throws Exception
4392 {
4393 // Create a logger
4394 Logger logger = Logger.getLogger("jello");
4395
4396 // Create an LTTng-UST log handler
4397 Handler lttngUstLogHandler = new LttngLogHandler();
4398
4399 // Add the LTTng-UST log handler to our logger
4400 logger.addHandler(lttngUstLogHandler);
4401
4402 // Log at will!
4403 logger.info("some info");
4404 logger.warning("some warning");
4405 Thread.sleep(500);
4406 logger.finer("finer information; the answer is " + answer);
4407 Thread.sleep(123);
4408 logger.severe("error!");
4409
4410 // Not mandatory, but cleaner
4411 logger.removeHandler(lttngUstLogHandler);
4412 lttngUstLogHandler.close();
4413 }
4414}
4415----
4416
4417Build this example:
4418
4419[role="term"]
4420----
4421$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4422----
4423
4424<<creating-destroying-tracing-sessions,Create a tracing session>>,
4425<<enabling-disabling-events,create an event rule>> matching the
4426`jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4427
4428[role="term"]
4429----
4430$ lttng create
4431$ lttng enable-event --jul jello
4432$ lttng start
4433----
4434
4435Run the compiled class:
4436
4437[role="term"]
4438----
4439$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4440----
4441
4442<<basic-tracing-session-control,Stop tracing>> and inspect the
4443recorded events:
4444
4445[role="term"]
4446----
4447$ lttng stop
4448$ lttng view
4449----
4450====
4451
4452In the resulting trace, an <<event,event record>> generated by a Java
4453application using `java.util.logging` is named `lttng_jul:event` and
4454has the following fields:
4455
4456`msg`::
05bbb80b 4457 Log record message.
96172a59
PP
4458
4459`logger_name`::
4460 Logger name.
4461
4462`class_name`::
4463 Name of the class in which the log statement was executed.
4464
4465`method_name`::
4466 Name of the method in which the log statement was executed.
4467
4468`long_millis`::
4469 Logging time (timestamp in milliseconds).
4470
4471`int_loglevel`::
4472 Log level integer value.
4473
4474`int_threadid`::
4475 ID of the thread in which the log statement was executed.
4476
4477Use the opt:lttng-enable-event(1):--loglevel or
4478opt:lttng-enable-event(1):--loglevel-only option of the
4479man:lttng-enable-event(1) command to target a range of JUL log levels
4480or a specific JUL log level.
4481
4482
4483[role="since-2.8"]
4484[[log4j]]
4485==== Use the LTTng-UST Java agent for Apache log4j
4486
4487To use the LTTng-UST Java agent in a Java application which uses
4488Apache log4j{nbsp}1.2:
4489
05bbb80b
PP
4490. In the source code of the Java application, import the LTTng-UST log
4491 appender package for Apache log4j:
96172a59
PP
4492+
4493--
4494[source,java]
4495----
4496import org.lttng.ust.agent.log4j.LttngLogAppender;
4497----
4498--
4499
4500. Create an LTTng-UST log4j log appender:
4501+
4502--
4503[source,java]
4504----
4505Appender lttngUstLogAppender = new LttngLogAppender();
4506----
4507--
4508
4509. Add this appender to the log4j loggers which should emit LTTng events:
4510+
4511--
4512[source,java]
4513----
4514Logger myLogger = Logger.getLogger("some-logger");
4515
4516myLogger.addAppender(lttngUstLogAppender);
4517----
4518--
4519
4520. Use Apache log4j log statements and configuration as usual. The
4521 loggers with an attached LTTng-UST log appender can emit LTTng events.
4522
4523. Before exiting the application, remove the LTTng-UST log appender from
4524 the loggers attached to it and call its `close()` method:
4525+
4526--
4527[source,java]
4528----
4529myLogger.removeAppender(lttngUstLogAppender);
4530lttngUstLogAppender.close();
4531----
4532--
4533+
4534This isn't strictly necessary, but it is recommended for a clean
05bbb80b 4535disposal of the resources of the appender.
96172a59 4536
05bbb80b
PP
4537. Include the common and log4j-specific JAR
4538 files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
96172a59
PP
4539 path:{lttng-ust-agent-log4j.jar}, in the
4540 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4541 path] when you build the Java application.
4542+
4543The JAR files are typically located in dir:{/usr/share/java}.
4544+
4545IMPORTANT: The LTTng-UST Java agent must be
4546<<installing-lttng,installed>> for the logging framework your
4547application uses.
4548
4549.Use the LTTng-UST Java agent for Apache log4j.
4550====
4551[source,java]
4552.path:{Test.java}
4553----
4554import org.apache.log4j.Appender;
4555import org.apache.log4j.Logger;
4556import org.lttng.ust.agent.log4j.LttngLogAppender;
4557
4558public class Test
4559{
4560 private static final int answer = 42;
4561
4562 public static void main(String[] argv) throws Exception
4563 {
4564 // Create a logger
4565 Logger logger = Logger.getLogger("jello");
4566
4567 // Create an LTTng-UST log appender
4568 Appender lttngUstLogAppender = new LttngLogAppender();
4569
4570 // Add the LTTng-UST log appender to our logger
4571 logger.addAppender(lttngUstLogAppender);
4572
4573 // Log at will!
4574 logger.info("some info");
4575 logger.warn("some warning");
4576 Thread.sleep(500);
4577 logger.debug("debug information; the answer is " + answer);
4578 Thread.sleep(123);
4579 logger.fatal("error!");
4580
4581 // Not mandatory, but cleaner
4582 logger.removeAppender(lttngUstLogAppender);
4583 lttngUstLogAppender.close();
4584 }
4585}
4586
4587----
4588
4589Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4590file):
4591
4592[role="term"]
4593----
4594$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4595----
4596
4597<<creating-destroying-tracing-sessions,Create a tracing session>>,
4598<<enabling-disabling-events,create an event rule>> matching the
4599`jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4600
4601[role="term"]
4602----
4603$ lttng create
4604$ lttng enable-event --log4j jello
4605$ lttng start
4606----
4607
4608Run the compiled class:
4609
4610[role="term"]
4611----
4612$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4613----
4614
4615<<basic-tracing-session-control,Stop tracing>> and inspect the
4616recorded events:
4617
4618[role="term"]
4619----
4620$ lttng stop
4621$ lttng view
4622----
4623====
4624
4625In the resulting trace, an <<event,event record>> generated by a Java
4626application using log4j is named `lttng_log4j:event` and
4627has the following fields:
4628
4629`msg`::
05bbb80b 4630 Log record message.
96172a59
PP
4631
4632`logger_name`::
4633 Logger name.
4634
4635`class_name`::
4636 Name of the class in which the log statement was executed.
4637
4638`method_name`::
4639 Name of the method in which the log statement was executed.
4640
4641`filename`::
4642 Name of the file in which the executed log statement is located.
4643
4644`line_number`::
4645 Line number at which the log statement was executed.
4646
4647`timestamp`::
4648 Logging timestamp.
4649
4650`int_loglevel`::
4651 Log level integer value.
4652
4653`thread_name`::
4654 Name of the Java thread in which the log statement was executed.
4655
4656Use the opt:lttng-enable-event(1):--loglevel or
4657opt:lttng-enable-event(1):--loglevel-only option of the
4658man:lttng-enable-event(1) command to target a range of Apache log4j
4659log levels or a specific log4j log level.
4660
4661
4662[role="since-2.8"]
4663[[java-application-context]]
4664==== Provide application-specific context fields in a Java application
4665
4666A Java application-specific context field is a piece of state provided
4667by the application which <<adding-context,you can add>>, using the
4668man:lttng-add-context(1) command, to each <<event,event record>>
4669produced by the log statements of this application.
4670
4671For example, a given object might have a current request ID variable.
4672You can create a context information retriever for this object and
4673assign a name to this current request ID. You can then, using the
4674man:lttng-add-context(1) command, add this context field by name to
4675the JUL or log4j <<channel,channel>>.
4676
4677To provide application-specific context fields in a Java application:
4678
05bbb80b 4679. In the source code of the Java application, import the LTTng-UST
96172a59
PP
4680 Java agent context classes and interfaces:
4681+
4682--
4683[source,java]
4684----
4685import org.lttng.ust.agent.context.ContextInfoManager;
4686import org.lttng.ust.agent.context.IContextInfoRetriever;
4687----
4688--
4689
4690. Create a context information retriever class, that is, a class which
4691 implements the `IContextInfoRetriever` interface:
4692+
4693--
4694[source,java]
4695----
4696class MyContextInfoRetriever implements IContextInfoRetriever
4697{
4698 @Override
4699 public Object retrieveContextInfo(String key)
4700 {
4701 if (key.equals("intCtx")) {
4702 return (short) 17;
4703 } else if (key.equals("strContext")) {
4704 return "context value!";
4705 } else {
4706 return null;
4707 }
4708 }
4709}
4710----
4711--
4712+
4713This `retrieveContextInfo()` method is the only member of the
4714`IContextInfoRetriever` interface. Its role is to return the current
4715value of a state by name to create a context field. The names of the
4716context fields and which state variables they return depends on your
4717specific scenario.
4718+
4719All primitive types and objects are supported as context fields.
4720When `retrieveContextInfo()` returns an object, the context field
4721serializer calls its `toString()` method to add a string field to
4722event records. The method can also return `null`, which means that
4723no context field is available for the required name.
4724
4725. Register an instance of your context information retriever class to
4726 the context information manager singleton:
4727+
4728--
4729[source,java]
4730----
4731IContextInfoRetriever cir = new MyContextInfoRetriever();
4732ContextInfoManager cim = ContextInfoManager.getInstance();
4733cim.registerContextInfoRetriever("retrieverName", cir);
4734----
4735--
4736
4737. Before exiting the application, remove your context information
4738 retriever from the context information manager singleton:
4739+
4740--
4741[source,java]
4742----
4743ContextInfoManager cim = ContextInfoManager.getInstance();
4744cim.unregisterContextInfoRetriever("retrieverName");
4745----
4746--
4747+
4748This isn't strictly necessary, but it is recommended for a clean
05bbb80b 4749disposal of some resources of the manager.
96172a59
PP
4750
4751. Build your Java application with LTTng-UST Java agent support as
4752 usual, following the procedure for either the <<jul,JUL>> or
4753 <<log4j,Apache log4j>> framework.
4754
4755
4756.Provide application-specific context fields in a Java application.
4757====
4758[source,java]
4759.path:{Test.java}
4760----
4761import java.util.logging.Handler;
4762import java.util.logging.Logger;
4763import org.lttng.ust.agent.jul.LttngLogHandler;
4764import org.lttng.ust.agent.context.ContextInfoManager;
4765import org.lttng.ust.agent.context.IContextInfoRetriever;
4766
4767public class Test
4768{
4769 // Our context information retriever class
4770 private static class MyContextInfoRetriever
4771 implements IContextInfoRetriever
4772 {
4773 @Override
4774 public Object retrieveContextInfo(String key) {
4775 if (key.equals("intCtx")) {
4776 return (short) 17;
4777 } else if (key.equals("strContext")) {
4778 return "context value!";
4779 } else {
4780 return null;
4781 }
4782 }
4783 }
4784
4785 private static final int answer = 42;
4786
4787 public static void main(String args[]) throws Exception
4788 {
4789 // Get the context information manager instance
4790 ContextInfoManager cim = ContextInfoManager.getInstance();
4791
4792 // Create and register our context information retriever
4793 IContextInfoRetriever cir = new MyContextInfoRetriever();
4794 cim.registerContextInfoRetriever("myRetriever", cir);
4795
4796 // Create a logger
4797 Logger logger = Logger.getLogger("jello");
4798
4799 // Create an LTTng-UST log handler
4800 Handler lttngUstLogHandler = new LttngLogHandler();
4801
4802 // Add the LTTng-UST log handler to our logger
4803 logger.addHandler(lttngUstLogHandler);
4804
4805 // Log at will!
4806 logger.info("some info");
4807 logger.warning("some warning");
4808 Thread.sleep(500);
4809 logger.finer("finer information; the answer is " + answer);
4810 Thread.sleep(123);
4811 logger.severe("error!");
4812
4813 // Not mandatory, but cleaner
4814 logger.removeHandler(lttngUstLogHandler);
4815 lttngUstLogHandler.close();
4816 cim.unregisterContextInfoRetriever("myRetriever");
4817 }
4818}
4819----
4820
4821Build this example:
4822
4823[role="term"]
4824----
4825$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4826----
4827
4828<<creating-destroying-tracing-sessions,Create a tracing session>>
4829and <<enabling-disabling-events,create an event rule>> matching the
4830`jello` JUL logger:
4831
4832[role="term"]
4833----
4834$ lttng create
4835$ lttng enable-event --jul jello
4836----
4837
4838<<adding-context,Add the application-specific context fields>> to the
4839JUL channel:
4840
4841[role="term"]
4842----
4843$ lttng add-context --jul --type='$app.myRetriever:intCtx'
4844$ lttng add-context --jul --type='$app.myRetriever:strContext'
4845----
4846
4847<<basic-tracing-session-control,Start tracing>>:
4848
4849[role="term"]
4850----
4851$ lttng start
4852----
4853
4854Run the compiled class:
4855
4856[role="term"]
4857----
4858$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4859----
4860
4861<<basic-tracing-session-control,Stop tracing>> and inspect the
4862recorded events:
4863
4864[role="term"]
4865----
4866$ lttng stop
4867$ lttng view
4868----
4869====
4870
4871
4872[role="since-2.7"]
4873[[python-application]]
4874=== User space Python agent
4875
4876You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
4877uses the standard
4878https://docs.python.org/3/library/logging.html[`logging`] package.
4879
4880Each log statement emits an LTTng event once the
4881application module imports the
4882<<lttng-ust-agents,LTTng-UST Python agent>> package.
4883
4884[role="img-100"]
4885.A Python application importing the LTTng-UST Python agent.
4886image::python-app.png[]
4887
4888To use the LTTng-UST Python agent:
4889
05bbb80b
PP
4890. In the source code of the Python application, import the LTTng-UST
4891 Python agent:
96172a59
PP
4892+
4893--
4894[source,python]
4895----
4896import lttngust
4897----
4898--
4899+
4900The LTTng-UST Python agent automatically adds its logging handler to the
4901root logger at import time.
4902+
4903Any log statement that the application executes before this import does
4904not emit an LTTng event.
4905+
4906IMPORTANT: The LTTng-UST Python agent must be
4907<<installing-lttng,installed>>.
4908
4909. Use log statements and logging configuration as usual.
4910 Since the LTTng-UST Python agent adds a handler to the _root_
4911 logger, you can trace any log statement from any logger.
4912
4913.Use the LTTng-UST Python agent.
4914====
4915[source,python]
4916.path:{test.py}
4917----
4918import lttngust
4919import logging
4920import time
4921
4922
4923def example():
4924 logging.basicConfig()
4925 logger = logging.getLogger('my-logger')
4926
4927 while True:
4928 logger.debug('debug message')
4929 logger.info('info message')
4930 logger.warn('warn message')
4931 logger.error('error message')
4932 logger.critical('critical message')
4933 time.sleep(1)
4934
4935
4936if __name__ == '__main__':
4937 example()
4938----
4939
4940NOTE: `logging.basicConfig()`, which adds to the root logger a basic
4941logging handler which prints to the standard error stream, isn't
4942strictly required for LTTng-UST tracing to work, but in versions of
4943Python preceding{nbsp}3.2, you could see a warning message which indicates
4944that no handler exists for the logger `my-logger`.
4945
4946<<creating-destroying-tracing-sessions,Create a tracing session>>,
4947<<enabling-disabling-events,create an event rule>> matching the
4948`my-logger` Python logger, and <<basic-tracing-session-control,start
4949tracing>>:
4950
4951[role="term"]
4952----
4953$ lttng create
4954$ lttng enable-event --python my-logger
4955$ lttng start
4956----
4957
4958Run the Python script:
4959
4960[role="term"]
4961----
4962$ python test.py
4963----
4964
4965<<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4966events:
4967
4968[role="term"]
4969----
4970$ lttng stop
4971$ lttng view
4972----
4973====
4974
4975In the resulting trace, an <<event,event record>> generated by a Python
4976application is named `lttng_python:event` and has the following fields:
4977
4978`asctime`::
4979 Logging time (string).
4980
4981`msg`::
05bbb80b 4982 Log record message.
96172a59
PP
4983
4984`logger_name`::
4985 Logger name.
4986
4987`funcName`::
4988 Name of the function in which the log statement was executed.
4989
4990`lineno`::
4991 Line number at which the log statement was executed.
4992
4993`int_loglevel`::
4994 Log level integer value.
4995
4996`thread`::
4997 ID of the Python thread in which the log statement was executed.
4998
4999`threadName`::
5000 Name of the Python thread in which the log statement was executed.
5001
5002Use the opt:lttng-enable-event(1):--loglevel or
5003opt:lttng-enable-event(1):--loglevel-only option of the
5004man:lttng-enable-event(1) command to target a range of Python log levels
5005or a specific Python log level.
5006
5007When an application imports the LTTng-UST Python agent, the agent tries
5008to register to a <<lttng-sessiond,session daemon>>. Note that you must
5009<<start-sessiond,start the session daemon>> _before_ you run the Python
5010application. If a session daemon is found, the agent tries to register
5011to it during five seconds, after which the application continues
5012without LTTng tracing support. Override this timeout value with
5013the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5014(milliseconds).
5015
5016If the session daemon stops while a Python application with an imported
5017LTTng-UST Python agent runs, the agent retries to connect and to
5018register to a session daemon every three seconds. Override this
5019delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5020variable.
5021
5022
5023[role="since-2.5"]
5024[[proc-lttng-logger-abi]]
5025=== LTTng logger
5026
5027The `lttng-tracer` Linux kernel module, part of
5028<<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5029path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5030LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5031to any of those files to emit an LTTng event.
5032
5033[role="img-100"]
5034.An application writes to the LTTng logger file to emit an LTTng event.
5035image::lttng-logger.png[]
5036
5037The LTTng logger is the quickest method--not the most efficient,
5038however--to add instrumentation to an application. It is designed
5039mostly to instrument shell scripts:
5040
5041[role="term"]
5042----
5043$ echo "Some message, some $variable" > /dev/lttng-logger
5044----
5045
5046Any event that the LTTng logger emits is named `lttng_logger` and
5047belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5048other instrumentation points in the kernel tracing domain, **any Unix
5049user** can <<enabling-disabling-events,create an event rule>> which
5050matches its event name, not only the root user or users in the
5051<<tracing-group,tracing group>>.
5052
5053To use the LTTng logger:
5054
5055* From any application, write text data to the path:{/dev/lttng-logger}
5056 file.
5057
5058The `msg` field of `lttng_logger` event records contains the
5059recorded message.
5060
5061NOTE: The maximum message length of an LTTng logger event is
50621024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5063than one event to contain the remaining data.
5064
5065You shouldn't use the LTTng logger to trace a user application which
5066can be instrumented in a more efficient way, namely:
5067
5068* <<c-application,C and $$C++$$ applications>>.
5069* <<java-application,Java applications>>.
5070* <<python-application,Python applications>>.
5071
5072.Use the LTTng logger.
5073====
5074[source,bash]
5075.path:{test.bash}
5076----
5077echo 'Hello, World!' > /dev/lttng-logger
5078sleep 2
5079df --human-readable --print-type / > /dev/lttng-logger
5080----
5081
5082<<creating-destroying-tracing-sessions,Create a tracing session>>,
5083<<enabling-disabling-events,create an event rule>> matching the
5084`lttng_logger` Linux kernel tracepoint, and
5085<<basic-tracing-session-control,start tracing>>:
5086
5087[role="term"]
5088----
5089$ lttng create
5090$ lttng enable-event --kernel lttng_logger
5091$ lttng start
5092----
5093
5094Run the Bash script:
5095
5096[role="term"]
5097----
5098$ bash test.bash
5099----
5100
5101<<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5102events:
5103
5104[role="term"]
5105----
5106$ lttng stop
5107$ lttng view
5108----
5109====
5110
5111
5112[[instrumenting-linux-kernel]]
5113=== LTTng kernel tracepoints
5114
5115NOTE: This section shows how to _add_ instrumentation points to the
05bbb80b 5116Linux kernel. The subsystems of the kernel are already thoroughly
96172a59
PP
5117instrumented at strategic places for LTTng when you
5118<<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5119package.
5120
5121////
5122There are two methods to instrument the Linux kernel:
5123
5124. <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
5125 tracepoint which uses the `TRACE_EVENT()` API.
5126+
5127Choose this if you want to instrumentation a Linux kernel tree with an
5128instrumentation point compatible with ftrace, perf, and SystemTap.
5129
5130. Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
5131 instrument an out-of-tree kernel module.
5132+
5133Choose this if you don't need ftrace, perf, or SystemTap support.
5134////
5135
5136
5137[[linux-add-lttng-layer]]
5138==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5139
5140This section shows how to add an LTTng layer to existing ftrace
5141instrumentation using the `TRACE_EVENT()` API.
5142
5143This section doesn't document the `TRACE_EVENT()` macro. Read the
5144following articles to learn more about this API:
5145
5146* http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5147* http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5148* http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5149
5150The following procedure assumes that your ftrace tracepoints are
5151correctly defined in their own header and that they are created in
5152one source file using the `CREATE_TRACE_POINTS` definition.
5153
5154To add an LTTng layer over an existing ftrace tracepoint:
5155
5156. Make sure the following kernel configuration options are
5157 enabled:
5158+
5159--
5160* `CONFIG_MODULES`
5161* `CONFIG_KALLSYMS`
5162* `CONFIG_HIGH_RES_TIMERS`
5163* `CONFIG_TRACEPOINTS`
5164--
5165
5166. Build the Linux source tree with your custom ftrace tracepoints.
5167. Boot the resulting Linux image on your target system.
5168+
5169Confirm that the tracepoints exist by looking for their names in the
5170dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
05bbb80b 5171is your subsystem name.
96172a59
PP
5172
5173. Get a copy of the latest LTTng-modules{nbsp}{revision}:
5174+
5175--
5176[role="term"]
5177----
5178$ cd $(mktemp -d) &&
5179wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
5180tar -xf lttng-modules-latest-2.12.tar.bz2 &&
5181cd lttng-modules-2.12.*
5182----
5183--
5184
5185. In dir:{instrumentation/events/lttng-module}, relative to the root
5186 of the LTTng-modules source tree, create a header file named
5187 +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5188 LTTng-modules tracepoint definitions using the LTTng-modules
5189 macros in it.
5190+
5191Start with this template:
5192+
5193--
5194[source,c]
5195.path:{instrumentation/events/lttng-module/my_subsys.h}
5196----
5197#undef TRACE_SYSTEM
5198#define TRACE_SYSTEM my_subsys
5199
5200#if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5201#define _LTTNG_MY_SUBSYS_H
5202
5203#include "../../../probes/lttng-tracepoint-event.h"
5204#include <linux/tracepoint.h>
5205
5206LTTNG_TRACEPOINT_EVENT(
5207 /*
05bbb80b 5208 * Format is identical to the TRACE_EVENT() version for the three
96172a59
PP
5209 * following macro parameters:
5210 */
5211 my_subsys_my_event,
5212 TP_PROTO(int my_int, const char *my_string),
5213 TP_ARGS(my_int, my_string),
5214
5215 /* LTTng-modules specific macros */
5216 TP_FIELDS(
5217 ctf_integer(int, my_int_field, my_int)
5218 ctf_string(my_bar_field, my_bar)
5219 )
5220)
5221
5222#endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5223
5224#include "../../../probes/define_trace.h"
5225----
5226--
5227+
5228The entries in the `TP_FIELDS()` section are the list of fields for the
5229LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
05bbb80b 5230the `TRACE_EVENT()` ftrace macro.
96172a59
PP
5231+
5232See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5233complete description of the available `ctf_*()` macros.
5234
05bbb80b
PP
5235. Create the kernel module C{nbsp}source file of the LTTng-modules
5236 probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
96172a59
PP
5237 subsystem name:
5238+
5239--
5240[source,c]
5241.path:{probes/lttng-probe-my-subsys.c}
5242----
5243#include <linux/module.h>
5244#include "../lttng-tracer.h"
5245
5246/*
5247 * Build-time verification of mismatch between mainline
5248 * TRACE_EVENT() arguments and the LTTng-modules adaptation
5249 * layer LTTNG_TRACEPOINT_EVENT() arguments.
5250 */
5251#include <trace/events/my_subsys.h>
5252
5253/* Create LTTng tracepoint probes */
5254#define LTTNG_PACKAGE_BUILD
5255#define CREATE_TRACE_POINTS
5256#define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5257
5258#include "../instrumentation/events/lttng-module/my_subsys.h"
5259
5260MODULE_LICENSE("GPL and additional rights");
5261MODULE_AUTHOR("Your name <your-email>");
5262MODULE_DESCRIPTION("LTTng my_subsys probes");
5263MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5264 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5265 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5266 LTTNG_MODULES_EXTRAVERSION);
5267----
5268--
5269
5270. Edit path:{probes/KBuild} and add your new kernel module object
5271 next to the existing ones:
5272+
5273--
5274[source,make]
5275.path:{probes/KBuild}
5276----
5277# ...
5278
5279obj-m += lttng-probe-module.o
5280obj-m += lttng-probe-power.o
5281
5282obj-m += lttng-probe-my-subsys.o
5283
5284# ...
5285----
5286--
5287
5288. Build and install the LTTng kernel modules:
5289+
5290--
5291[role="term"]
5292----
5293$ make KERNELDIR=/path/to/linux
5294# make modules_install && depmod -a
5295----
5296--
5297+
5298Replace `/path/to/linux` with the path to the Linux source tree where
05bbb80b 5299you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
96172a59
PP
5300
5301Note that you can also use the
5302<<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5303instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5304C code that need to be executed before the event fields are recorded.
5305
5306The best way to learn how to use the previous LTTng-modules macros is to
5307inspect the existing LTTng-modules tracepoint definitions in the
5308dir:{instrumentation/events/lttng-module} header files. Compare them
5309with the Linux kernel mainline versions in the
5310dir:{include/trace/events} directory of the Linux source tree.
5311
5312
5313[role="since-2.7"]
5314[[lttng-tracepoint-event-code]]
5315===== Use custom C code to access the data for tracepoint fields
5316
5317Although we recommended to always use the
5318<<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5319the arguments and fields of an LTTng-modules tracepoint when possible,
5320sometimes you need a more complex process to access the data that the
5321tracer records as event record fields. In other words, you need local
5322variables and multiple C{nbsp}statements instead of simple
5323argument-based expressions that you pass to the
5324<<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5325
5326Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5327`LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5328a block of C{nbsp}code to be executed before LTTng records the fields.
5329The structure of this macro is:
5330
5331[source,c]
5332.`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5333----
5334LTTNG_TRACEPOINT_EVENT_CODE(
5335 /*
5336 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5337 * version for the following three macro parameters:
5338 */
5339 my_subsys_my_event,
5340 TP_PROTO(int my_int, const char *my_string),
5341 TP_ARGS(my_int, my_string),
5342
5343 /* Declarations of custom local variables */
5344 TP_locvar(
5345 int a = 0;
5346 unsigned long b = 0;
5347 const char *name = "(undefined)";
5348 struct my_struct *my_struct;
5349 ),
5350
5351 /*
5352 * Custom code which uses both tracepoint arguments
5353 * (in TP_ARGS()) and local variables (in TP_locvar()).
5354 *
5355 * Local variables are actually members of a structure pointed
5356 * to by the special variable tp_locvar.
5357 */
5358 TP_code(
5359 if (my_int) {
5360 tp_locvar->a = my_int + 17;
5361 tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5362 tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5363 tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5364 put_my_struct(tp_locvar->my_struct);
5365
5366 if (tp_locvar->b) {
5367 tp_locvar->a = 1;
5368 }
5369 }
5370 ),
5371
5372 /*
5373 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5374 * version for this, except that tp_locvar members can be
5375 * used in the argument expression parameters of
5376 * the ctf_*() macros.
5377 */
5378 TP_FIELDS(
5379 ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5380 ctf_integer(int, my_struct_a, tp_locvar->a)
5381 ctf_string(my_string_field, my_string)
5382 ctf_string(my_struct_name, tp_locvar->name)
5383 )
5384)
5385----
5386
5387IMPORTANT: The C code defined in `TP_code()` must not have any side
5388effects when executed. In particular, the code must not allocate
5389memory or get resources without deallocating this memory or putting
5390those resources afterwards.
5391
5392
5393[[instrumenting-linux-kernel-tracing]]
5394==== Load and unload a custom probe kernel module
5395
5396You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5397kernel module>> in the kernel before it can emit LTTng events.
5398
5399To load the default probe kernel modules and a custom probe kernel
5400module:
5401
5402* Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5403 probe modules to load when starting a root <<lttng-sessiond,session
5404 daemon>>:
5405+
5406--
5407.Load the `my_subsys`, `usb`, and the default probe modules.
5408====
5409[role="term"]
5410----
5411# lttng-sessiond --extra-kmod-probes=my_subsys,usb
5412----
5413====
5414--
5415+
5416You only need to pass the subsystem name, not the whole kernel module
5417name.
5418
5419To load _only_ a given custom probe kernel module:
5420
5421* Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5422 modules to load when starting a root session daemon:
5423+
5424--
5425.Load only the `my_subsys` and `usb` probe modules.
5426====
5427[role="term"]
5428----
5429# lttng-sessiond --kmod-probes=my_subsys,usb
5430----
5431====
5432--
5433
5434To confirm that a probe module is loaded:
5435
5436* Use man:lsmod(8):
5437+
5438--
5439[role="term"]
5440----
5441$ lsmod | grep lttng_probe_usb
5442----
5443--
5444
5445To unload the loaded probe modules:
5446
5447* Kill the session daemon with `SIGTERM`:
5448+
5449--
5450[role="term"]
5451----
5452# pkill lttng-sessiond
5453----
5454--
5455+
05bbb80b 5456You can also use the man:modprobe(8) `--remove` option if the session
96172a59
PP
5457daemon terminates abnormally.
5458
5459
5460[[controlling-tracing]]
5461== Tracing control
5462
5463Once an application or a Linux kernel is
5464<<instrumenting,instrumented>> for LTTng tracing,
5465you can _trace_ it.
5466
5467This section is divided in topics on how to use the various
5468<<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5469command-line tool>>, to _control_ the LTTng daemons and tracers.
5470
5471NOTE: In the following subsections, we refer to an man:lttng(1) command
5472using its man page name. For example, instead of _Run the `create`
5473command to..._, we use _Run the man:lttng-create(1) command to..._.
5474
5475
5476[[start-sessiond]]
5477=== Start a session daemon
5478
5479In some situations, you need to run a <<lttng-sessiond,session daemon>>
5480(man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5481command-line tool.
5482
5483You will see the following error when you run a command while no session
5484daemon is running:
5485
5486----
5487Error: No session daemon is available
5488----
5489
5490The only command that automatically runs a session daemon is
5491man:lttng-create(1), which you use to
5492<<creating-destroying-tracing-sessions,create a tracing session>>. While
5493this is most of the time the first operation that you do, sometimes it's
5494not. Some examples are:
5495
5496* <<list-instrumentation-points,List the available instrumentation points>>.
5497* <<saving-loading-tracing-session,Load a tracing session configuration>>.
5498
5499[[tracing-group]] Each Unix user must have its own running session
5500daemon to trace user applications. The session daemon that the root user
5501starts is the only one allowed to control the LTTng kernel tracer. Users
5502that are part of the _tracing group_ can control the root session
5503daemon. The default tracing group name is `tracing`; set it to something
5504else with the opt:lttng-sessiond(8):--group option when you start the
5505root session daemon.
5506
5507To start a user session daemon:
5508
5509* Run man:lttng-sessiond(8):
5510+
5511--
5512[role="term"]
5513----
5514$ lttng-sessiond --daemonize
5515----
5516--
5517
5518To start the root session daemon:
5519
5520* Run man:lttng-sessiond(8) as the root user:
5521+
5522--
5523[role="term"]
5524----
5525# lttng-sessiond --daemonize
5526----
5527--
5528
5529In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5530start the session daemon in foreground.
5531
5532To stop a session daemon, use man:kill(1) on its process ID (standard
5533`TERM` signal).
5534
5535Note that some Linux distributions could manage the LTTng session daemon
5536as a service. In this case, you should use the service manager to
5537start, restart, and stop session daemons.
5538
5539
5540[[creating-destroying-tracing-sessions]]
5541=== Create and destroy a tracing session
5542
5543Almost all the LTTng control operations happen in the scope of
5544a <<tracing-session,tracing session>>, which is the dialogue between the
5545<<lttng-sessiond,session daemon>> and you.
5546
5547To create a tracing session with a generated name:
5548
5549* Use the man:lttng-create(1) command:
5550+
5551--
5552[role="term"]
5553----
5554$ lttng create
5555----
5556--
5557
05bbb80b 5558The name of the created tracing session is `auto` followed by the
96172a59
PP
5559creation date.
5560
5561To create a tracing session with a specific name:
5562
5563* Use the optional argument of the man:lttng-create(1) command:
5564+
5565--
5566[role="term"]
5567----
5568$ lttng create my-session
5569----
5570--
5571+
5572Replace `my-session` with the specific tracing session name.
5573
05bbb80b
PP
5574LTTng appends the creation date to the name of the created tracing
5575session.
96172a59
PP
5576
5577LTTng writes the traces of a tracing session in
5578+$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5579name of the tracing session. Note that the env:LTTNG_HOME environment
5580variable defaults to `$HOME` if not set.
5581
5582To output LTTng traces to a non-default location:
5583
5584* Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5585+
5586--
5587[role="term"]
5588----
5589$ lttng create my-session --output=/tmp/some-directory
5590----
5591--
5592
5593You may create as many tracing sessions as you wish.
5594
5595To list all the existing tracing sessions for your Unix user:
5596
5597* Use the man:lttng-list(1) command:
5598+
5599--
5600[role="term"]
5601----
5602$ lttng list
5603----
5604--
5605
5606[[cur-tracing-session]]When you create a tracing session, it is set as
5607the _current tracing session_. The following man:lttng(1) commands
5608operate on the current tracing session when you don't specify one:
5609
5610[role="list-3-cols"]
5611* man:lttng-add-context(1)
5612* man:lttng-clear(1)
5613* man:lttng-destroy(1)
5614* man:lttng-disable-channel(1)
5615* man:lttng-disable-event(1)
5616* man:lttng-disable-rotation(1)
5617* man:lttng-enable-channel(1)
5618* man:lttng-enable-event(1)
5619* man:lttng-enable-rotation(1)
5620* man:lttng-load(1)
5621* man:lttng-regenerate(1)
5622* man:lttng-rotate(1)
5623* man:lttng-save(1)
5624* man:lttng-snapshot(1)
5625* man:lttng-start(1)
5626* man:lttng-status(1)
5627* man:lttng-stop(1)
5628* man:lttng-track(1)
5629* man:lttng-untrack(1)
5630* man:lttng-view(1)
5631
5632To change the current tracing session:
5633
5634* Use the man:lttng-set-session(1) command:
5635+
5636--
5637[role="term"]
5638----
5639$ lttng set-session new-session
5640----
5641--
5642+
5643Replace `new-session` by the name of the new current tracing session.
5644
5645When you're done tracing in a given tracing session, destroy it. This
5646operation frees the resources taken by the tracing session to destroy;
5647it doesn't destroy the trace data that LTTng wrote for this tracing
5648session (see <<clear,Clear a tracing session>> for one way to do this).
5649
5650To destroy the current tracing session:
5651
5652* Use the man:lttng-destroy(1) command:
5653+
5654--
5655[role="term"]
5656----
5657$ lttng destroy
5658----
5659--
5660
5661The man:lttng-destroy(1) command also runs the man:lttng-stop(1)
5662command implicitly (see <<basic-tracing-session-control,Start and stop a
5663tracing session>>). You need to stop tracing to make LTTng flush the
5664remaining trace data and make the trace readable.
5665
5666
5667[[list-instrumentation-points]]
5668=== List the available instrumentation points
5669
5670The <<lttng-sessiond,session daemon>> can query the running instrumented
5671user applications and the Linux kernel to get a list of available
5672instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5673they are tracepoints and system calls. For the user space tracing
5674domain, they are tracepoints. For the other tracing domains, they are
5675logger names.
5676
5677To list the available instrumentation points:
5678
05bbb80b
PP
5679* Use the man:lttng-list(1) command with the option of the requested
5680 tracing domain amongst:
96172a59
PP
5681+
5682--
5683opt:lttng-list(1):--kernel::
5684 Linux kernel tracepoints (your Unix user must be a root user, or it
5685 must be a member of the <<tracing-group,tracing group>>).
5686
5687opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
5688 Linux kernel system calls (your Unix user must be a root user, or it
5689 must be a member of the tracing group).
5690
5691opt:lttng-list(1):--userspace::
5692 User space tracepoints.
5693
5694opt:lttng-list(1):--jul::
5695 `java.util.logging` loggers.
5696
5697opt:lttng-list(1):--log4j::
5698 Apache log4j loggers.
5699
5700opt:lttng-list(1):--python::
5701 Python loggers.
5702--
5703
5704.List the available user space tracepoints.
5705====
5706[role="term"]
5707----
5708$ lttng list --userspace
5709----
5710====
5711
5712.List the available Linux kernel system call tracepoints.
5713====
5714[role="term"]
5715----
5716$ lttng list --kernel --syscall
5717----
5718====
5719
5720
5721[[enabling-disabling-events]]
5722=== Create and enable an event rule
5723
5724Once you <<creating-destroying-tracing-sessions,create a tracing
5725session>>, you can create <<event,event rules>> with the
5726man:lttng-enable-event(1) command.
5727
5728You specify each condition with a command-line option. The available
5729condition arguments are shown in the following table.
5730
5731[role="growable",cols="asciidoc,asciidoc,default"]
5732.Condition command-line arguments for the man:lttng-enable-event(1) command.
5733|====
5734|Argument |Description |Applicable tracing domains
5735
5736|
5737One of:
5738
5739. `--syscall`
5740. +--probe=__ADDR__+
5741. +--function=__ADDR__+
5742. +--userspace-probe=__PATH__:__SYMBOL__+
5743. +--userspace-probe=sdt:__PATH__:__PROVIDER__:__NAME__+
5744
5745|
5746Instead of using the default _tracepoint_ instrumentation type, use:
5747
5748. A Linux system call (entry and exit).
5749. A Linux https://lwn.net/Articles/132196/[kprobe] (symbol or address).
5750. The entry and return points of a Linux function (symbol or address).
5751. The entry point of a user application or library function (path to
5752 application/library and symbol).
5753. A https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[SystemTap
5754 Statically Defined Tracing] (USDT) probe (path to application/library,
5755 provider and probe names).
5756
5757|Linux kernel.
5758
5759|First positional argument.
5760
5761|
5762Tracepoint or system call name.
5763
5764With the opt:lttng-enable-event(1):--probe,
5765opt:lttng-enable-event(1):--function, and
5766opt:lttng-enable-event(1):--userspace-probe options, this is a custom
5767name given to the event rule. With the JUL, log4j, and Python domains,
5768this is a logger name.
5769
5770With a tracepoint, logger, or system call name, use the special
5771`*` globbing character to match anything (for example, `sched_*`,
5772`my_comp*:*msg_*`).
5773
5774|All.
5775
5776|
5777One of:
5778
5779. +--loglevel=__LEVEL__+
5780. +--loglevel-only=__LEVEL__+
5781
5782|
5783. Match only tracepoints or log statements with a logging level at
5784 least as severe as +__LEVEL__+.
5785. Match only tracepoints or log statements with a logging level
5786 equal to +__LEVEL__+.
5787
5788See man:lttng-enable-event(1) for the list of available logging level
5789names.
5790
5791|User space, JUL, log4j, and Python.
5792
5793|+--exclude=__EXCLUSIONS__+
5794
5795|
5796When you use a `*` character at the end of the tracepoint or logger
5797name (first positional argument), exclude the specific names in the
5798comma-delimited list +__EXCLUSIONS__+.
5799
5800|
5801User space, JUL, log4j, and Python.
5802
5803|+--filter=__EXPR__+
5804
5805|
5806Match only events which satisfy the expression +__EXPR__+.
5807
5808See man:lttng-enable-event(1) to learn more about the syntax of a
5809filter expression.
5810
5811|All.
5812
5813|====
5814
5815You attach an event rule to a <<channel,channel>> on creation. If you do
5816not specify the channel with the opt:lttng-enable-event(1):--channel
5817option, and if the event rule to create is the first in its
5818<<domain,tracing domain>> for a given tracing session, then LTTng
5819creates a _default channel_ for you. This default channel is reused in
5820subsequent invocations of the man:lttng-enable-event(1) command for the
5821same tracing domain.
5822
5823An event rule is always enabled at creation time.
5824
5825The following examples show how to combine the previous
5826command-line options to create simple to more complex event rules.
5827
5828.Create an event rule targetting a Linux kernel tracepoint (default channel).
5829====
5830[role="term"]
5831----
5832$ lttng enable-event --kernel sched_switch
5833----
5834====
5835
5836.Create an event rule matching four Linux kernel system calls (default channel).
5837====
5838[role="term"]
5839----
5840$ lttng enable-event --kernel --syscall open,write,read,close
5841----
5842====
5843
5844.Create event rules matching tracepoints with filter expressions (default channel).
5845====
5846[role="term"]
5847----
5848$ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5849----
5850
5851[role="term"]
5852----
5853$ lttng enable-event --kernel --all \
5854 --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5855----
5856
5857[role="term"]
5858----
5859$ lttng enable-event --jul my_logger \
5860 --filter='$app.retriever:cur_msg_id > 3'
5861----
5862
5863IMPORTANT: Make sure to always quote the filter string when you
5864use man:lttng(1) from a shell.
5865
5866See also <<pid-tracking,Track process attributes>> which offers another,
5867more efficient filtering mechanism for process ID, user ID, and group
5868ID attributes.
5869====
5870
5871.Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5872====
5873[role="term"]
5874----
5875$ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5876----
5877
5878IMPORTANT: Make sure to always quote the wildcard character when you
5879use man:lttng(1) from a shell.
5880====
5881
5882.Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5883====
5884[role="term"]
5885----
5886$ lttng enable-event --python my-app.'*' \
5887 --exclude='my-app.module,my-app.hello'
5888----
5889====
5890
5891.Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5892====
5893[role="term"]
5894----
5895$ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5896----
5897====
5898
5899.Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5900====
5901[role="term"]
5902----
5903$ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5904----
5905====
5906
5907.Create an event rule matching the `malloc` function entry in path:{/usr/lib/libc.so.6}:
5908====
5909[role="term"]
5910----
5911$ lttng enable-event --kernel --userspace-probe=/usr/lib/libc.so.6:malloc \
5912 libc_malloc
5913----
5914====
5915
5916.Create an event rule matching the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
5917====
5918[role="term"]
5919----
5920$ lttng enable-event --kernel --userspace-probe=sdt:serv:server:accept_request \
5921 server_accept_request
5922----
5923====
5924
5925The event rules of a given channel form a whitelist: as soon as an
5926emitted event passes one of them, LTTng can record the event. For
5927example, an event named `my_app:my_tracepoint` emitted from a user space
5928tracepoint with a `TRACE_ERROR` log level passes both of the following
5929rules:
5930
5931[role="term"]
5932----
5933$ lttng enable-event --userspace my_app:my_tracepoint
5934$ lttng enable-event --userspace my_app:my_tracepoint \
5935 --loglevel=TRACE_INFO
5936----
5937
5938The second event rule is redundant: the first one includes
5939the second one.
5940
5941
5942[[disable-event-rule]]
5943=== Disable an event rule
5944
5945To disable an event rule that you <<enabling-disabling-events,created>>
5946previously, use the man:lttng-disable-event(1) command. This command
5947disables _all_ the event rules (of a given tracing domain and channel)
5948which match an instrumentation point. The other conditions aren't
5949supported as of LTTng{nbsp}{revision}.
5950
5951The LTTng tracer doesn't record an emitted event which passes
5952a _disabled_ event rule.
5953
5954.Disable an event rule matching a Python logger (default channel).
5955====
5956[role="term"]
5957----
5958$ lttng disable-event --python my-logger
5959----
5960====
5961
5962.Disable an event rule matching all `java.util.logging` loggers (default channel).
5963====
5964[role="term"]
5965----
5966$ lttng disable-event --jul '*'
5967----
5968====
5969
5970.Disable _all_ the event rules of the default channel.
5971====
5972The opt:lttng-disable-event(1):--all-events option isn't, like the
5973opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
5974equivalent of the event name `*` (wildcard): it disables _all_ the event
5975rules of a given channel.
5976
5977[role="term"]
5978----
5979$ lttng disable-event --jul --all-events
5980----
5981====
5982
5983NOTE: You can't delete an event rule once you create it.
5984
5985
5986[[status]]
5987=== Get the status of a tracing session
5988
5989To get the status of the <<cur-tracing-session,current tracing
5990session>>, that is, its parameters, its channels, event rules, and their
5991attributes:
5992
5993* Use the man:lttng-status(1) command:
5994+
5995--
5996[role="term"]
5997----
5998$ lttng status
5999----
6000--
6001
6002To get the status of any tracing session:
6003
05bbb80b
PP
6004* Use the man:lttng-list(1) command with the name of the tracing
6005 session:
96172a59
PP
6006+
6007--
6008[role="term"]
6009----
6010$ lttng list my-session
6011----
6012--
6013+
05bbb80b 6014Replace `my-session` with the desired tracing session name.
96172a59
PP
6015
6016
6017[[basic-tracing-session-control]]
6018=== Start and stop a tracing session
6019
6020Once you <<creating-destroying-tracing-sessions,create a tracing
6021session>> and
6022<<enabling-disabling-events,create one or more event rules>>,
6023you can start and stop the tracers for this tracing session.
6024
6025To start tracing in the <<cur-tracing-session,current tracing session>>:
6026
6027* Use the man:lttng-start(1) command:
6028+
6029--
6030[role="term"]
6031----
6032$ lttng start
6033----
6034--
6035
6036LTTng is very flexible: you can launch user applications before
6037or after the you start the tracers. The tracers only record the events
6038if they pass enabled event rules and if they occur while the tracers are
6039started.
6040
6041To stop tracing in the current tracing session:
6042
6043* Use the man:lttng-stop(1) command:
6044+
6045--
6046[role="term"]
6047----
6048$ lttng stop
6049----
6050--
6051+
6052If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6053records>> or lost sub-buffers since the last time you ran
6054man:lttng-start(1), warnings are printed when you run the
6055man:lttng-stop(1) command.
6056
6057IMPORTANT: You need to stop tracing to make LTTng flush the remaining
6058trace data and make the trace readable. Note that the
6059man:lttng-destroy(1) command (see
6060<<creating-destroying-tracing-sessions,Create and destroy a tracing
6061session>>) also runs the man:lttng-stop(1) command implicitly.
6062
6063[role="since-2.12"]
6064[[clear]]
6065=== Clear a tracing session
6066
6067You might need to remove all the current tracing data of one or more
6068<<tracing-session,tracing sessions>> between multiple attempts to
6069reproduce a problem without interrupting the LTTng tracing activity.
6070
6071To clear the tracing data of the
6072<<cur-tracing-session,current tracing session>>:
6073
6074* Use the man:lttng-clear(1) command:
6075+
6076--
6077[role="term"]
6078----
6079$ lttng clear
6080----
6081--
6082
6083To clear the tracing data of all the tracing sessions:
6084
6085* Use the `lttng clear` command with the opt:lttng-clear(1):--all
6086 option:
6087+
6088--
6089[role="term"]
6090----
6091$ lttng clear --all
6092----
6093--
6094
6095
6096[[enabling-disabling-channels]]
6097=== Create a channel
6098
6099Once you create a tracing session, you can create a <<channel,channel>>
6100with the man:lttng-enable-channel(1) command.
6101
6102Note that LTTng automatically creates a default channel when, for a
6103given <<domain,tracing domain>>, no channels exist and you
6104<<enabling-disabling-events,create>> the first event rule. This default
6105channel is named `channel0` and its attributes are set to reasonable
6106values. Therefore, you only need to create a channel when you need
6107non-default attributes.
6108
6109You specify each non-default channel attribute with a command-line
6110option when you use the man:lttng-enable-channel(1) command. The
6111available command-line options are:
6112
6113[role="growable",cols="asciidoc,asciidoc"]
6114.Command-line options for the man:lttng-enable-channel(1) command.
6115|====
6116|Option |Description
6117
6118|`--overwrite`
6119
6120|
6121Use the _overwrite_
6122<<channel-overwrite-mode-vs-discard-mode,event record loss mode>> instead
6123of the default _discard_ mode.
6124
6125|`--buffers-pid` (user space tracing domain only)
6126
6127|
6128Use the per-process <<channel-buffering-schemes,buffering scheme>>
6129instead of the default per-user buffering scheme.
6130
6131|+--subbuf-size=__SIZE__+
6132
6133|
6134Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
6135either for each Unix user (default), or for each instrumented process.
6136
6137See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6138
6139|+--num-subbuf=__COUNT__+
6140
6141|
6142Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
6143for each Unix user (default), or for each instrumented process.
6144
6145See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6146
6147|+--tracefile-size=__SIZE__+
6148
6149|
6150Set the maximum size of each trace file that this channel writes within
6151a stream to +__SIZE__+ bytes instead of no maximum.
6152
6153See <<tracefile-rotation,Trace file count and size>>.
6154
6155|+--tracefile-count=__COUNT__+
6156
6157|
6158Limit the number of trace files that this channel creates to
6159+__COUNT__+ channels instead of no limit.
6160
6161See <<tracefile-rotation,Trace file count and size>>.
6162
6163|+--switch-timer=__PERIODUS__+
6164
6165|
6166Set the <<channel-switch-timer,switch timer period>>
6167to +__PERIODUS__+{nbsp}µs.
6168
6169|+--read-timer=__PERIODUS__+
6170
6171|
6172Set the <<channel-read-timer,read timer period>>
6173to +__PERIODUS__+{nbsp}µs.
6174
6175|[[opt-blocking-timeout]]+--blocking-timeout=__TIMEOUTUS__+
6176
6177|
6178Set the timeout of user space applications which load LTTng-UST
6179in blocking mode to +__TIMEOUTUS__+:
6180
61810 (default)::
6182 Never block (non-blocking mode).
6183
6184`inf`::
6185 Block forever until space is available in a sub-buffer to record
6186 the event.
6187
6188__n__, a positive value::
6189 Wait for at most __n__ µs when trying to write into a sub-buffer.
6190
6191Note that, for this option to have any effect on an instrumented
6192user space application, you need to run the application with a set
6193env:LTTNG_UST_ALLOW_BLOCKING environment variable.
6194
6195|+--output=__TYPE__+ (Linux kernel tracing domain only)
6196
6197|
05bbb80b
PP
6198Set the output type of the channel to +__TYPE__+, either `mmap` or
6199`splice`.
96172a59
PP
6200
6201|====
6202
6203You can only create a channel in the Linux kernel and user space
6204<<domain,tracing domains>>: other tracing domains have their own channel
6205created on the fly when <<enabling-disabling-events,creating event
6206rules>>.
6207
6208[IMPORTANT]
6209====
6210Because of a current LTTng limitation, you must create all channels
6211_before_ you <<basic-tracing-session-control,start tracing>> in a given
6212tracing session, that is, before the first time you run
6213man:lttng-start(1).
6214
6215Since LTTng automatically creates a default channel when you use the
6216man:lttng-enable-event(1) command with a specific tracing domain, you
6217can't, for example, create a Linux kernel event rule, start tracing,
6218and then create a user space event rule, because no user space channel
6219exists yet and it's too late to create one.
6220
6221For this reason, make sure to configure your channels properly
6222before starting the tracers for the first time!
6223====
6224
6225The following examples show how to combine the previous
6226command-line options to create simple to more complex channels.
6227
6228.Create a Linux kernel channel with default attributes.
6229====
6230[role="term"]
6231----
6232$ lttng enable-channel --kernel my-channel
6233----
6234====
6235
6236.Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6237====
6238[role="term"]
6239----
6240$ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6241 --buffers-pid my-channel
6242----
6243====
6244
6245.[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6246====
6247<<creating-destroying-tracing-sessions,Create a tracing-session>>,
6248create the channel, <<enabling-disabling-events,create an event rule>>,
6249and <<basic-tracing-session-control,start tracing>>:
6250
6251[role="term"]
6252----
6253$ lttng create
6254$ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
6255$ lttng enable-event --userspace --channel=blocking-channel --all
6256$ lttng start
6257----
6258
6259Run an application instrumented with LTTng-UST and allow it to block:
6260
6261[role="term"]
6262----
6263$ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6264----
6265====
6266
6267.Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream
6268====
6269[role="term"]
6270----
6271$ lttng enable-channel --kernel --tracefile-count=8 \
6272 --tracefile-size=4194304 my-channel
6273----
6274====
6275
6276.Create a user space channel in overwrite (or _flight recorder_) mode.
6277====
6278[role="term"]
6279----
6280$ lttng enable-channel --userspace --overwrite my-channel
6281----
6282====
6283
6284<<enabling-disabling-events,Create>> the same event rule in
6285two different channels:
6286
6287[role="term"]
6288----
6289$ lttng enable-event --userspace --channel=my-channel app:tp
6290$ lttng enable-event --userspace --channel=other-channel app:tp
6291----
6292
6293If both channels are enabled, when a tracepoint named `app:tp` is
6294reached, LTTng records two events, one for each channel.
6295
6296
6297[[disable-channel]]
6298=== Disable a channel
6299
6300To disable a specific channel that you <<enabling-disabling-channels,created>>
6301previously, use the man:lttng-disable-channel(1) command.
6302
6303.Disable a specific Linux kernel channel.
6304====
6305[role="term"]
6306----
6307$ lttng disable-channel --kernel my-channel
6308----
6309====
6310
6311The state of a channel precedes the individual states of event rules
6312attached to it: event rules which belong to a disabled channel, even if
6313they are enabled, are also considered disabled.
6314
6315
6316[[adding-context]]
6317=== Add context fields to a channel
6318
6319Event record fields in trace files provide important information about
6320events that occured previously, but sometimes some external context may
6321help you solve a problem faster.
6322
6323Examples of context fields are:
6324
6325* The **process ID**, **thread ID**, **process name**, and
6326 **process priority** of the thread in which the event occurs.
6327* The **hostname** of the system on which the event occurs.
6328* The Linux kernel and user call stacks (since
b52915bd 6329 LTTng{nbsp}2.11).
96172a59
PP
6330* The current values of many possible **performance counters** using
6331 perf, for example:
6332** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6333** Cache misses.
6334** Branch instructions, misses, and loads.
6335** CPU faults.
6336* Any context defined at the application level (supported for the
6337 JUL and log4j <<domain,tracing domains>>).
6338
6339To get the full list of available context fields, see
6340`lttng add-context --list`. Some context fields are reserved for a
6341specific <<domain,tracing domain>> (Linux kernel or user space).
6342
6343You add context fields to <<channel,channels>>. All the events
6344that a channel with added context fields records contain those fields.
6345
6346To add context fields to one or all the channels of a given tracing
6347session:
6348
6349* Use the man:lttng-add-context(1) command.
6350
6351.Add context fields to all the channels of the current tracing session.
6352====
6353The following command line adds the virtual process identifier and
6354the per-thread CPU cycles count fields to all the user space channels
6355of the
6356<<cur-tracing-session,current tracing session>>.
6357
6358[role="term"]
6359----
6360$ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6361----
6362====
6363
6364.Add performance counter context fields by raw ID
6365====
6366See man:lttng-add-context(1) for the exact format of the context field
6367type, which is partly compatible with the format used in
6368man:perf-record(1).
6369
6370[role="term"]
6371----
6372$ lttng add-context --userspace --type=perf:thread:raw:r0110:test
6373$ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6374----
6375====
6376
6377.Add context fields to a specific channel.
6378====
6379The following command line adds the thread identifier and user call
6380stack context fields to the Linux kernel channel named `my-channel` in
6381the current tracing session.
6382
6383[role="term"]
6384----
6385$ lttng add-context --kernel --channel=my-channel \
6386 --type=tid --type=callstack-user
6387----
6388====
6389
6390.Add an application-specific context field to a specific channel.
6391====
6392The following command line adds the `cur_msg_id` context field of the
6393`retriever` context retriever for all the instrumented
6394<<java-application,Java applications>> recording <<event,event records>>
6395in the channel named `my-channel`:
6396
6397[role="term"]
6398----
6399$ lttng add-context --kernel --channel=my-channel \
6400 --type='$app:retriever:cur_msg_id'
6401----
6402
6403IMPORTANT: Make sure to always quote the `$` character when you
6404use man:lttng-add-context(1) from a shell.
6405====
6406
6407NOTE: You can't remove context fields from a channel once you add it.
6408
6409
6410[role="since-2.7"]
6411[[pid-tracking]]
6412=== Track process attributes
6413
6414It's often useful to only allow processes with specific attributes to
6415emit events. For example, you may wish to record all the system calls
6416which a given process makes (à la
6417http://linux.die.net/man/1/strace[strace]).
6418
6419The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6420purpose. Both commands operate on _inclusion sets_ of process attribute
6421values. The available process attribute types are:
6422
6423Linux kernel <<domain,tracing domain>> only::
6424+
6425* Process ID (PID).
6426
6427* Virtual process ID (VPID).
6428+
6429This is the PID as seen by the application.
6430
6431* Unix user ID (UID) (since LTTng{nbsp}2.12).
6432
6433* Virtual Unix user ID (VUID) (since LTTng{nbsp}2.12).
6434+
6435This is the UID as seen by the application.
6436
6437* Unix group ID (GID) (since LTTng{nbsp}2.12).
6438
6439* Virtual Unix group ID (VGID) (since LTTng{nbsp}2.12).
6440+
6441This is the GID as seen by the application.
6442
6443
6444User space tracing domain::
6445+
6446* VPID.
6447* VUID (since LTTng{nbsp}2.12).
6448* VGID (since LTTng{nbsp}2.12).
6449
6450Each tracing domain has one inclusion set per process attribute type:
6451the Linux kernel tracing domain has six while the user space tracing
6452domain has three.
6453
6454For a given event which passes an enabled <<event,event rule>> to be
6455recorded, _all_ the attributes of its executing process must be part of
05bbb80b 6456the inclusion sets of the tracing domain of the event rule.
96172a59
PP
6457
6458Add entries to an inclusion set with the man:lttng-track(1) command and
6459remove entries with the man:lttng-untrack(1) command. A process
6460attribute is _tracked_ when it's part of an inclusion set and
6461_untracked_ otherwise.
6462
6463[NOTE]
6464====
6465The process attribute values are _numeric_.
6466
6467Should a process with a given tracked process ID, for example, exit, and
6468then a new process be given this ID, then the latter would also be
6469allowed to emit events.
6470
6471With the `lttng track` command, you can add Unix user and group _names_
6472to the user and group inclusion sets: the <<lttng-sessiond,session
6473daemon>> finds the corresponding UID, VUID, GID, or VGID once on
6474_addition_ to the inclusion set. This means that if you rename the user
6475or group after you run `lttng track`, its user/group ID remains tracked.
6476====
6477
6478.Track and untrack virtual process IDs.
6479====
6480For the sake of the following example, assume the target system has
648116{nbsp}possible VPIDs.
6482
6483When you
6484<<creating-destroying-tracing-sessions,create a tracing session>>,
6485the user space VPID inclusion set contains _all_ the possible VPIDs:
6486
6487[role="img-100"]
6488.All VPIDs are tracked.
6489image::track-all.png[]
6490
6491When the inclusion set is full and you use the man:lttng-track(1)
6492command to specify some VPIDs to track, LTTng first clears the inclusion
6493set, and then it adds the specific VPIDs to track. After:
6494
6495[role="term"]
6496----
6497$ lttng track --userspace --vpid=3,4,7,10,13
6498----
6499
6500the VPID inclusion set is:
6501
6502[role="img-100"]
6503.VPIDs 3, 4, 7, 10, and 13 are tracked.
6504image::track-3-4-7-10-13.png[]
6505
6506Add more VPIDs to the inclusion set afterwards:
6507
6508[role="term"]
6509----
6510$ lttng track --userspace --vpid=1,15,16
6511----
6512
6513The result is:
6514
6515[role="img-100"]
6516.VPIDs 1, 15, and 16 are added to the inclusion set.
6517image::track-1-3-4-7-10-13-15-16.png[]
6518
6519The man:lttng-untrack(1) command removes entries from process attribute
6520inclusion sets. Given the previous example, the following command:
6521
6522[role="term"]
6523----
6524$ lttng untrack --userspace --vpid=3,7,10,13
6525----
6526
6527leads to this VPID inclusion set:
6528
6529[role="img-100"]
6530.VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6531image::track-1-4-15-16.png[]
6532
6533LTTng can track all the possible VPIDs again using the
6534opt:lttng-track(1):--all option:
6535
6536[role="term"]
6537----
6538$ lttng track --userspace --vpid --all
6539----
6540
6541The result is, again:
6542
6543[role="img-100"]
6544.All VPIDs are tracked.
6545image::track-all.png[]
6546====
6547
6548.Track only specific process attributes.
6549====
6550A typical use case with process attribute tracking is to start with an
6551empty inclusion set, then <<basic-tracing-session-control,start the
6552tracers>>, and then add entries manually while the tracers are active.
6553
6554Use the opt:lttng-untrack(1):--all option of the
6555man:lttng-untrack(1) command to clear the inclusion set after you
6556<<creating-destroying-tracing-sessions,create a tracing session>>, for
6557example (with UIDs):
6558
6559[role="term"]
6560----
6561$ lttng untrack --kernel --uid --all
6562----
6563
6564gives:
6565
6566[role="img-100"]
6567.No UIDs are tracked.
6568image::untrack-all.png[]
6569
6570If you trace with this inclusion set configuration, the LTTng kernel
6571tracer records no events within the <<cur-tracing-session,current
6572tracing session>> because it doesn't track any UID. Use the
6573man:lttng-track(1) command as usual to track specific UIDs when you need
6574to, for example:
6575
6576[role="term"]
6577----
6578$ lttng track --kernel --uid=http,11
6579----
6580
6581Result:
6582
6583[role="img-100"]
6584.UIDs 6 (`http`) and 11 are tracked.
6585image::track-6-11.png[]
6586====
6587
6588
6589[role="since-2.5"]
6590[[saving-loading-tracing-session]]
6591=== Save and load tracing session configurations
6592
6593Configuring a <<tracing-session,tracing session>> can be long. Some of
6594the tasks involved are:
6595
6596* <<enabling-disabling-channels,Create channels>> with
6597 specific attributes.
6598* <<adding-context,Add context fields>> to specific channels.
6599* <<enabling-disabling-events,Create event rules>> with specific log
6600 level and filter conditions.
6601
6602If you use LTTng to solve real world problems, chances are you have to
6603record events using the same tracing session setup over and over,
6604modifying a few variables each time in your instrumented program
6605or environment. To avoid constant tracing session reconfiguration,
6606the man:lttng(1) command-line tool can save and load tracing session
6607configurations to/from XML files.
6608
6609To save a given tracing session configuration:
6610
6611* Use the man:lttng-save(1) command:
6612+
6613--
6614[role="term"]
6615----
6616$ lttng save my-session
6617----
6618--
6619+
6620Replace `my-session` with the name of the tracing session to save.
6621
6622LTTng saves tracing session configurations to
6623dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6624env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6625the opt:lttng-save(1):--output-path option to change this destination
6626directory.
6627
6628LTTng saves all configuration parameters, for example:
6629
6630* The tracing session name.
6631* The trace data output path.
6632* The channels with their state and all their attributes.
6633* The context fields you added to channels.
6634* The event rules with their state, log level and filter conditions.
6635
6636To load a tracing session:
6637
6638* Use the man:lttng-load(1) command:
6639+
6640--
6641[role="term"]
6642----
6643$ lttng load my-session
6644----
6645--
6646+
6647Replace `my-session` with the name of the tracing session to load.
6648
6649When LTTng loads a configuration, it restores your saved tracing session
6650as if you just configured it manually.
6651
6652See man:lttng-load(1) for the complete list of command-line options. You
6653can also save and load many sessions at a time, and decide in which
6654directory to output the XML files.
6655
6656
6657[[sending-trace-data-over-the-network]]
6658=== Send trace data over the network
6659
6660LTTng can send the recorded trace data to a remote system over the
6661network instead of writing it to the local file system.
6662
6663To send the trace data over the network:
6664
6665. On the _remote_ system (which can also be the target system),
6666 start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6667+
6668--
6669[role="term"]
6670----
6671$ lttng-relayd
6672----
6673--
6674
6675. On the _target_ system, create a tracing session configured to
6676 send trace data over the network:
6677+
6678--
6679[role="term"]
6680----
6681$ lttng create my-session --set-url=net://remote-system
6682----
6683--
6684+
6685Replace `remote-system` by the host name or IP address of the
6686remote system. See man:lttng-create(1) for the exact URL format.
6687
6688. On the target system, use the man:lttng(1) command-line tool as usual.
05bbb80b
PP
6689 When tracing is active, the consumer daemon of the target sends
6690 sub-buffers to the relay daemon running on the remote system instead
6691 of flushing them to the local file system. The relay daemon writes the
6692 received packets to the local file system.
96172a59
PP
6693
6694The relay daemon writes trace files to
6695+$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6696+__hostname__+ is the host name of the target system and +__session__+
6697is the tracing session name. Note that the env:LTTNG_HOME environment
6698variable defaults to `$HOME` if not set. Use the
6699opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6700trace files to another base directory.
6701
6702
6703[role="since-2.4"]
6704[[lttng-live]]
6705=== View events as LTTng emits them (noch:{LTTng} live)
6706
6707LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6708daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6709display events as LTTng emits them on the target system while tracing is
6710active.
6711
6712The relay daemon creates a _tee_: it forwards the trace data to both
6713the local file system and to connected live viewers:
6714
6715[role="img-90"]
6716.The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6717image::live.png[]
6718
6719To use LTTng live:
6720
6721. On the _target system_, create a <<tracing-session,tracing session>>
6722 in _live mode_:
6723+
6724--
6725[role="term"]
6726----
6727$ lttng create my-session --live
6728----
6729--
6730+
6731This spawns a local relay daemon.
6732
6733. Start the live viewer and configure it to connect to the relay
6734 daemon. For example, with
6735 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]:
6736+
6737--
6738[role="term"]
6739----
6740$ babeltrace2 net://localhost/host/hostname/my-session
6741----
6742--
6743+
6744Replace:
6745+
6746--
6747* `hostname` with the host name of the target system.
6748* `my-session` with the name of the tracing session to view.
6749--
6750
6751. Configure the tracing session as usual with the man:lttng(1)
6752 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6753
6754List the available live tracing sessions with Babeltrace{nbsp}2:
6755
6756[role="term"]
6757----
6758$ babeltrace2 net://localhost
6759----
6760
6761You can start the relay daemon on another system. In this case, you need
05bbb80b
PP
6762to specify the URL of the relay daemon when you create the tracing
6763session with the opt:lttng-create(1):--set-url option. You also need to
6764replace `localhost` in the procedure above with the host name of the
6765system on which the relay daemon is running.
96172a59
PP
6766
6767See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6768command-line options.
6769
6770
6771[role="since-2.3"]
6772[[taking-a-snapshot]]
6773=== Take a snapshot of the current sub-buffers of a tracing session
6774
6775The normal behavior of LTTng is to append full sub-buffers to growing
6776trace data files. This is ideal to keep a full history of the events
6777that occurred on the target system, but it can
6778represent too much data in some situations. For example, you may wish
6779to trace your application continuously until some critical situation
6780happens, in which case you only need the latest few recorded
6781events to perform the desired analysis, not multi-gigabyte trace files.
6782
6783With the man:lttng-snapshot(1) command, you can take a snapshot of the
6784current sub-buffers of a given <<tracing-session,tracing session>>.
6785LTTng can write the snapshot to the local file system or send it over
6786the network.
6787
6788[role="img-100"]
6789.A snapshot is a copy of the current sub-buffers, which aren't cleared after the operation.
6790image::snapshot.png[]
6791
6792If you wish to create unmanaged, self-contained, non-overlapping
6793trace chunk archives instead of a simple copy of the current
6794sub-buffers, see the <<session-rotation,tracing session rotation>>
6795feature (available since LTTng{nbsp}2.11).
6796
6797To take a snapshot:
6798
6799. Create a tracing session in _snapshot mode_:
6800+
6801--
6802[role="term"]
6803----
6804$ lttng create my-session --snapshot
6805----
6806--
6807+
6808The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
6809<<channel,channels>> created in this mode is automatically set to
6810_overwrite_ (flight recorder mode).
6811
6812. Configure the tracing session as usual with the man:lttng(1)
6813 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6814
6815. **Optional**: When you need to take a snapshot,
6816 <<basic-tracing-session-control,stop tracing>>.
6817+
6818You can take a snapshot when the tracers are active, but if you stop
6819them first, you're sure that the data in the sub-buffers doesn't
6820change before you actually take the snapshot.
6821
6822. Take a snapshot:
6823+
6824--
6825[role="term"]
6826----
6827$ lttng snapshot record --name=my-first-snapshot
6828----
6829--
6830+
05bbb80b
PP
6831LTTng writes the current sub-buffers of all the channels of the
6832<<cur-tracing-session,current tracing session>> to
96172a59
PP
6833trace files on the local file system. Those trace files have
6834`my-first-snapshot` in their name.
6835
6836There is no difference between the format of a normal trace file and the
6837format of a snapshot: viewers of LTTng traces also support LTTng
6838snapshots.
6839
6840By default, LTTng writes snapshot files to the path shown by
6841`lttng snapshot list-output`. You can change this path or decide to send
6842snapshots over the network using either:
6843
6844. An output path or URL that you specify when you
6845 <<creating-destroying-tracing-sessions,create the tracing session>>.
6846. A snapshot output path or URL that you add using
6847 `lttng snapshot add-output`.
6848. An output path or URL that you provide directly to the
6849 `lttng snapshot record` command.
6850
6851Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
6852you specify a URL, a relay daemon must listen on a remote system (see
6853<<sending-trace-data-over-the-network,Send trace data over the
6854network>>).
6855
6856
6857[role="since-2.11"]
6858[[session-rotation]]
6859=== Archive the current trace chunk (rotate a tracing session)
6860
05bbb80b
PP
6861The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
6862current sub-buffers of a tracing session to the file system or send them
6863over the network. When you take a snapshot, LTTng doesn't clear the ring
6864buffers of the tracing session: if you take another snapshot immediately
96172a59
PP
6865after, both snapshots could contain overlapping trace data.
6866
6867Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
6868_tracing session rotation_ is a feature which appends the content of the
6869ring buffers to what's already on the file system or sent over the
05bbb80b 6870network since the creation of the tracing session or since the last
96172a59
PP
6871rotation, and then clears those ring buffers to avoid trace data
6872overlaps.
6873
6874What LTTng is about to write when performing a tracing session rotation
6875is called the _current trace chunk_. When this current trace chunk is
6876written to the file system or sent over the network, it becomes a _trace
6877chunk archive_. Therefore, a tracing session rotation _archives_ the
6878current trace chunk.
6879
6880[role="img-100"]
6881.A tracing session rotation operation _archives_ the current trace chunk.
6882image::rotation.png[]
6883
6884A trace chunk archive is a self-contained LTTng trace which LTTng
6885doesn't manage anymore: you can read it, modify it, move it, or remove
6886it.
6887
6888There are two methods to perform a tracing session rotation: immediately
6889or with a rotation schedule.
6890
6891To perform an immediate tracing session rotation:
6892
6893. <<creating-destroying-tracing-sessions,Create a tracing session>>
6894 in _normal mode_ or _network streaming mode_
6895 (only those two creation modes support tracing session rotation):
6896+
6897--
6898[role="term"]
6899----
6900$ lttng create my-session
6901----
6902--
6903
6904. <<enabling-disabling-events,Create one or more event rules>>
6905 and <<basic-tracing-session-control,start tracing>>:
6906+
6907--
6908[role="term"]
6909----
6910$ lttng enable-event --kernel sched_'*'
6911$ lttng start
6912----
6913--
6914
6915. When needed, immediately rotate the
6916 <<cur-tracing-session,current tracing session>>:
6917+
6918--
6919[role="term"]
6920----
6921$ lttng rotate
6922----
6923--
6924+
6925The cmd:lttng-rotate command prints the path to the created trace
6926chunk archive. See man:lttng-rotate(1) to learn about the format
6927of trace chunk archive directory names.
6928+
6929Perform other immediate rotations while the tracing session is
6930active. It is guaranteed that all the trace chunk archives don't
6931contain overlapping trace data. You can also perform an immediate
6932rotation once you have <<basic-tracing-session-control,stopped>> the
6933tracing session.
6934
6935. When you're done tracing,
6936 <<creating-destroying-tracing-sessions,destroy the current tracing
6937 session>>:
6938+
6939--
6940[role="term"]
6941----
6942$ lttng destroy
6943----
6944--
6945+
6946The tracing session destruction operation creates one last trace
6947chunk archive from the current trace chunk.
6948
6949A tracing session rotation schedule is a planned rotation which LTTng
6950performs automatically based on one of the following conditions:
6951
6952* A timer with a configured period times out.
6953
6954* The total size of the flushed part of the current trace chunk
6955 becomes greater than or equal to a configured value.
6956
6957To schedule a tracing session rotation, set a _rotation schedule_:
6958
6959. <<creating-destroying-tracing-sessions,Create a tracing session>>
6960 in _normal mode_ or _network streaming mode_
6961 (only those two creation modes support tracing session rotation):
6962+
6963--
6964[role="term"]
6965----
6966$ lttng create my-session
6967----
6968--
6969
6970. <<enabling-disabling-events,Create one or more event rules>>:
6971+
6972--
6973[role="term"]
6974----
6975$ lttng enable-event --kernel sched_'*'
6976----
6977--
6978
6979. Set a tracing session rotation schedule:
6980+
6981--
6982[role="term"]
6983----
6984$ lttng enable-rotation --timer=10s
6985----
6986--
6987+
6988In this example, we set a rotation schedule so that LTTng performs a
6989tracing session rotation every ten seconds.
6990+
6991See man:lttng-enable-rotation(1) to learn more about other ways to set a
6992rotation schedule.
6993
6994. <<basic-tracing-session-control,Start tracing>>:
6995+
6996--
6997[role="term"]
6998----
6999$ lttng start
7000----
7001--
7002+
7003LTTng performs tracing session rotations automatically while the tracing
7004session is active thanks to the rotation schedule.
7005
7006. When you're done tracing,
7007 <<creating-destroying-tracing-sessions,destroy the current tracing
7008 session>>:
7009+
7010--
7011[role="term"]
7012----
7013$ lttng destroy
7014----
7015--
7016+
7017The tracing session destruction operation creates one last trace chunk
7018archive from the current trace chunk.
7019
7020Use man:lttng-disable-rotation(1) to unset a tracing session
7021rotation schedule.
7022
7023NOTE: man:lttng-rotate(1) and man:lttng-enable-rotation(1) list
7024limitations regarding those two commands.
7025
7026
7027[role="since-2.6"]
7028[[mi]]
7029=== Use the machine interface
7030
7031With any command of the man:lttng(1) command-line tool, set the
7032opt:lttng(1):--mi option to `xml` (before the command name) to get an
7033XML machine interface output, for example:
7034
7035[role="term"]
7036----
7037$ lttng --mi=xml enable-event --kernel --syscall open
7038----
7039
7040A schema definition (XSD) is
d267807c 7041https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/src/common/mi-lttng-4.0.xsd[available]
96172a59
PP
7042to ease the integration with external tools as much as possible.
7043
7044
7045[role="since-2.8"]
7046[[metadata-regenerate]]
7047=== Regenerate the metadata of an LTTng trace
7048
7049An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
7050data stream files and a metadata file. This metadata file contains,
7051amongst other things, information about the offset of the clock sources
7052used to timestamp <<event,event records>> when tracing.
7053
7054If, once a <<tracing-session,tracing session>> is
7055<<basic-tracing-session-control,started>>, a major
7056https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
05bbb80b 7057happens, the clock offset of the trace also needs to be updated. Use
96172a59
PP
7058the `metadata` item of the man:lttng-regenerate(1) command to do so.
7059
7060The main use case of this command is to allow a system to boot with
7061an incorrect wall time and trace it with LTTng before its wall time
7062is corrected. Once the system is known to be in a state where its
7063wall time is correct, it can run `lttng regenerate metadata`.
7064
7065To regenerate the metadata of an LTTng trace:
7066
7067* Use the `metadata` item of the man:lttng-regenerate(1) command:
7068+
7069--
7070[role="term"]
7071----
7072$ lttng regenerate metadata
7073----
7074--
7075
7076[IMPORTANT]
7077====
7078`lttng regenerate metadata` has the following limitations:
7079
7080* Tracing session <<creating-destroying-tracing-sessions,created>>
7081 in non-live mode.
7082* User space <<channel,channels>>, if any, are using
7083 <<channel-buffering-schemes,per-user buffering>>.
7084====
7085
7086
7087[role="since-2.9"]
7088[[regenerate-statedump]]
7089=== Regenerate the state dump of a tracing session
7090
7091The LTTng kernel and user space tracers generate state dump
7092<<event,event records>> when the application starts or when you
7093<<basic-tracing-session-control,start a tracing session>>. An analysis
7094can use the state dump event records to set an initial state before it
7095builds the rest of the state from the following event records.
7096http://tracecompass.org/[Trace Compass] is a notable example of an
7097application which uses the state dump of an LTTng trace.
7098
7099When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7100state dump event records aren't included in the snapshot because they
7101were recorded to a sub-buffer that has been consumed or overwritten
7102already.
7103
7104Use the `lttng regenerate statedump` command to emit the state
7105dump event records again.
7106
7107To regenerate the state dump of the current tracing session, provided
7108create it in snapshot mode, before you take a snapshot:
7109
7110. Use the `statedump` item of the man:lttng-regenerate(1) command:
7111+
7112--
7113[role="term"]
7114----
7115$ lttng regenerate statedump
7116----
7117--
7118
7119. <<basic-tracing-session-control,Stop the tracing session>>:
7120+
7121--
7122[role="term"]
7123----
7124$ lttng stop
7125----
7126--
7127
7128. <<taking-a-snapshot,Take a snapshot>>:
7129+
7130--
7131[role="term"]
7132----
7133$ lttng snapshot record --name=my-snapshot
7134----
7135--
7136
7137Depending on the event throughput, you should run steps 1 and 2
7138as closely as possible.
7139
7140NOTE: To record the state dump events, you need to
7141<<enabling-disabling-events,create event rules>> which enable them.
7142LTTng-UST state dump tracepoints start with `lttng_ust_statedump:`.
7143LTTng-modules state dump tracepoints start with `lttng_statedump_`.
7144
7145
7146[role="since-2.7"]
7147[[persistent-memory-file-systems]]
7148=== Record trace data on persistent memory file systems
7149
7150https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
7151(NVRAM) is random-access memory that retains its information when power
7152is turned off (non-volatile). Systems with such memory can store data
7153structures in RAM and retrieve them after a reboot, without flushing
7154to typical _storage_.
7155
7156Linux supports NVRAM file systems thanks to either
7157http://pramfs.sourceforge.net/[PRAMFS] or
7158https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
7159(requires Linux{nbsp}4.1+).
7160
7161This section doesn't describe how to operate such file systems;
7162we assume that you have a working persistent memory file system.
7163
7164When you create a <<tracing-session,tracing session>>, you can specify
7165the path of the shared memory holding the sub-buffers. If you specify a
7166location on an NVRAM file system, then you can retrieve the latest
7167recorded trace data when the system reboots after a crash.
7168
7169To record trace data on a persistent memory file system and retrieve the
7170trace data after a system crash:
7171
7172. Create a tracing session with a sub-buffer shared memory path located
7173 on an NVRAM file system:
7174+
7175--
7176[role="term"]
7177----
7178$ lttng create my-session --shm-path=/path/to/shm
7179----
7180--
7181
7182. Configure the tracing session as usual with the man:lttng(1)
7183 command-line tool, and <<basic-tracing-session-control,start tracing>>.
7184
7185. After a system crash, use the man:lttng-crash(1) command-line tool to
7186 view the trace data recorded on the NVRAM file system:
7187+
7188--
7189[role="term"]
7190----
7191$ lttng-crash /path/to/shm
7192----
7193--
7194
7195The binary layout of the ring buffer files isn't exactly the same as
7196the trace files layout. This is why you need to use man:lttng-crash(1)
7197instead of your preferred trace viewer directly.
7198
7199To convert the ring buffer files to LTTng trace files:
7200
7201* Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
7202+
7203--
7204[role="term"]
7205----
7206$ lttng-crash --extract=/path/to/trace /path/to/shm
7207----
7208--
7209
7210
7211[role="since-2.10"]
7212[[notif-trigger-api]]
05bbb80b 7213=== Get notified when the buffer usage of a channel is too high or too low
96172a59 7214
05bbb80b 7215With the $$C/C++$$ notification and trigger API of LTTng, your user
96172a59
PP
7216application can get notified when the buffer usage of one or more
7217<<channel,channels>> becomes too low or too high. Use this API
7218and enable or disable <<event,event rules>> during tracing to avoid
7219<<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
7220
05bbb80b 7221.Have a user application get notified when the buffer usage of an LTTng channel is too high.
96172a59
PP
7222====
7223In this example, we create and build an application which gets notified
7224when the buffer usage of a specific LTTng channel is higher than
722575{nbsp}%. We only print that it is the case in the example, but we
7226could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
7227disable event rules when this happens.
7228
05bbb80b 7229. Create the C{nbsp}source file of application:
96172a59
PP
7230+
7231--
7232[source,c]
7233.path:{notif-app.c}
7234----
7235#include <stdio.h>
7236#include <assert.h>
7237#include <lttng/domain.h>
7238#include <lttng/action/action.h>
7239#include <lttng/action/notify.h>
7240#include <lttng/condition/condition.h>
7241#include <lttng/condition/buffer-usage.h>
7242#include <lttng/condition/evaluation.h>
7243#include <lttng/notification/channel.h>
7244#include <lttng/notification/notification.h>
7245#include <lttng/trigger/trigger.h>
7246#include <lttng/endpoint.h>
7247
7248int main(int argc, char *argv[])
7249{
7250 int exit_status = 0;
7251 struct lttng_notification_channel *notification_channel;
7252 struct lttng_condition *condition;
7253 struct lttng_action *action;
7254 struct lttng_trigger *trigger;
7255 const char *tracing_session_name;
7256 const char *channel_name;
7257
7258 assert(argc >= 3);
7259 tracing_session_name = argv[1];
7260 channel_name = argv[2];
7261
7262 /*
7263 * Create a notification channel. A notification channel
7264 * connects the user application to the LTTng session daemon.
7265 * This notification channel can be used to listen to various
7266 * types of notifications.
7267 */
7268 notification_channel = lttng_notification_channel_create(
7269 lttng_session_daemon_notification_endpoint);
7270
7271 /*
7272 * Create a "high buffer usage" condition. In this case, the
7273 * condition is reached when the buffer usage is greater than or
7274 * equal to 75 %. We create the condition for a specific tracing
7275 * session name, channel name, and for the user space tracing
7276 * domain.
7277 *
7278 * The "low buffer usage" condition type also exists.
7279 */
7280 condition = lttng_condition_buffer_usage_high_create();
7281 lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
7282 lttng_condition_buffer_usage_set_session_name(
7283 condition, tracing_session_name);
7284 lttng_condition_buffer_usage_set_channel_name(condition,
7285 channel_name);
7286 lttng_condition_buffer_usage_set_domain_type(condition,
7287 LTTNG_DOMAIN_UST);
7288
7289 /*
7290 * Create an action (get a notification) to take when the
7291 * condition created above is reached.
7292 */
7293 action = lttng_action_notify_create();
7294
7295 /*
7296 * Create a trigger. A trigger associates a condition to an
7297 * action: the action is executed when the condition is reached.
7298 */
7299 trigger = lttng_trigger_create(condition, action);
7300
7301 /* Register the trigger to LTTng. */
7302 lttng_register_trigger(trigger);
7303
7304 /*
7305 * Now that we have registered a trigger, a notification will be
7306 * emitted everytime its condition is met. To receive this
7307 * notification, we must subscribe to notifications that match
7308 * the same condition.
7309 */
7310 lttng_notification_channel_subscribe(notification_channel,
7311 condition);
7312
7313 /*
7314 * Notification loop. Put this in a dedicated thread to avoid
7315 * blocking the main thread.
7316 */
7317 for (;;) {
7318 struct lttng_notification *notification;
7319 enum lttng_notification_channel_status status;
7320 const struct lttng_evaluation *notification_evaluation;
7321 const struct lttng_condition *notification_condition;
7322 double buffer_usage;
7323
7324 /* Receive the next notification. */
7325 status = lttng_notification_channel_get_next_notification(
7326 notification_channel, &notification);
7327
7328 switch (status) {
7329 case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7330 break;
7331 case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7332 /*
7333 * The session daemon can drop notifications if a monitoring
7334 * application isn't consuming the notifications fast
7335 * enough.
7336 */
7337 continue;
7338 case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7339 /*
7340 * The notification channel has been closed by the
7341 * session daemon. This is typically caused by a session
7342 * daemon shutting down.
7343 */
7344 goto end;
7345 default:
7346 /* Unhandled conditions or errors. */
7347 exit_status = 1;
7348 goto end;
7349 }
7350
7351 /*
7352 * A notification provides, amongst other things:
7353 *
7354 * * The condition that caused this notification to be
7355 * emitted.
7356 * * The condition evaluation, which provides more
7357 * specific information on the evaluation of the
7358 * condition.
7359 *
7360 * The condition evaluation provides the buffer usage
7361 * value at the moment the condition was reached.
7362 */
7363 notification_condition = lttng_notification_get_condition(
7364 notification);
7365 notification_evaluation = lttng_notification_get_evaluation(
7366 notification);
7367
7368 /* We're subscribed to only one condition. */
7369 assert(lttng_condition_get_type(notification_condition) ==
7370 LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
7371
7372 /*
7373 * Get the exact sampled buffer usage from the
7374 * condition evaluation.
7375 */
7376 lttng_evaluation_buffer_usage_get_usage_ratio(
7377 notification_evaluation, &buffer_usage);
7378
7379 /*
7380 * At this point, instead of printing a message, we
05bbb80b
PP
7381 * could do something to reduce the buffer usage of the channel,
7382 * like disable specific events.
96172a59
PP
7383 */
7384 printf("Buffer usage is %f %% in tracing session \"%s\", "
7385 "user space channel \"%s\".\n", buffer_usage * 100,
7386 tracing_session_name, channel_name);
7387 lttng_notification_destroy(notification);
7388 }
7389
7390end:
7391 lttng_action_destroy(action);
7392 lttng_condition_destroy(condition);
7393 lttng_trigger_destroy(trigger);
7394 lttng_notification_channel_destroy(notification_channel);
7395 return exit_status;
7396}
7397----
7398--
7399
7400. Build the `notif-app` application, linking it to `liblttng-ctl`:
7401+
7402--
7403[role="term"]
7404----
7405$ gcc -o notif-app notif-app.c -llttng-ctl
7406----
7407--
7408
7409. <<creating-destroying-tracing-sessions,Create a tracing session>>,
7410 <<enabling-disabling-events,create an event rule>> matching all the
7411 user space tracepoints, and
7412 <<basic-tracing-session-control,start tracing>>:
7413+
7414--
7415[role="term"]
7416----
7417$ lttng create my-session
7418$ lttng enable-event --userspace --all
7419$ lttng start
7420----
7421--
7422+
7423If you create the channel manually with the man:lttng-enable-channel(1)
05bbb80b
PP
7424command, control how frequently LTTng samples the current values of the
7425channel properties to evaluate user conditions with the
96172a59
PP
7426opt:lttng-enable-channel(1):--monitor-timer option.
7427
7428. Run the `notif-app` application. This program accepts the
7429 <<tracing-session,tracing session>> name and the user space channel
7430 name as its two first arguments. The channel which LTTng automatically
7431 creates with the man:lttng-enable-event(1) command above is named
7432 `channel0`:
7433+
7434--
7435[role="term"]
7436----
7437$ ./notif-app my-session channel0
7438----
7439--
7440
7441. In another terminal, run an application with a very high event
7442 throughput so that the 75{nbsp}% buffer usage condition is reached.
7443+
7444In the first terminal, the application should print lines like this:
7445+
7446----
7447Buffer usage is 81.45197 % in tracing session "my-session", user space
7448channel "channel0".
7449----
7450+
7451If you don't see anything, try modifying the condition in
7452path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
7453(step{nbsp}2) and running it again (step{nbsp}4).
7454====
7455
7456
7457[[reference]]
7458== Reference
7459
7460[[lttng-modules-ref]]
7461=== noch:{LTTng-modules}
7462
7463
7464[role="since-2.9"]
7465[[lttng-tracepoint-enum]]
7466==== `LTTNG_TRACEPOINT_ENUM()` usage
7467
7468Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
7469
7470[source,c]
7471----
7472LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
7473----
7474
7475Replace:
7476
7477* `name` with the name of the enumeration (C identifier, unique
7478 amongst all the defined enumerations).
7479* `entries` with a list of enumeration entries.
7480
7481The available enumeration entry macros are:
7482
7483+ctf_enum_value(__name__, __value__)+::
7484 Entry named +__name__+ mapped to the integral value +__value__+.
7485
7486+ctf_enum_range(__name__, __begin__, __end__)+::
7487 Entry named +__name__+ mapped to the range of integral values between
7488 +__begin__+ (included) and +__end__+ (included).
7489
7490+ctf_enum_auto(__name__)+::
7491 Entry named +__name__+ mapped to the integral value following the
05bbb80b 7492 last mapping value.
96172a59
PP
7493+
7494The last value of a `ctf_enum_value()` entry is its +__value__+
7495parameter.
7496+
7497The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
7498+
7499If `ctf_enum_auto()` is the first entry in the list, its integral
7500value is 0.
7501
7502Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
7503to use a defined enumeration as a tracepoint field.
7504
7505.Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
7506====
7507[source,c]
7508----
7509LTTNG_TRACEPOINT_ENUM(
7510 my_enum,
7511 TP_ENUM_VALUES(
7512 ctf_enum_auto("AUTO: EXPECT 0")
7513 ctf_enum_value("VALUE: 23", 23)
7514 ctf_enum_value("VALUE: 27", 27)
7515 ctf_enum_auto("AUTO: EXPECT 28")
7516 ctf_enum_range("RANGE: 101 TO 303", 101, 303)
7517 ctf_enum_auto("AUTO: EXPECT 304")
7518 )
7519)
7520----
7521====
7522
7523
7524[role="since-2.7"]
7525[[lttng-modules-tp-fields]]
7526==== Tracepoint fields macros (for `TP_FIELDS()`)
7527
7528[[tp-fast-assign]][[tp-struct-entry]]The available macros to define
7529tracepoint fields, which must be listed within `TP_FIELDS()` in
7530`LTTNG_TRACEPOINT_EVENT()`, are:
7531
7532[role="func-desc growable",cols="asciidoc,asciidoc"]
7533.Available macros to define LTTng-modules tracepoint fields
7534|====
7535|Macro |Description and parameters
7536
7537|
7538+ctf_integer(__t__, __n__, __e__)+
7539
7540+ctf_integer_nowrite(__t__, __n__, __e__)+
7541
7542+ctf_user_integer(__t__, __n__, __e__)+
7543
7544+ctf_user_integer_nowrite(__t__, __n__, __e__)+
7545|
7546Standard integer, displayed in base{nbsp}10.
7547
7548+__t__+::
7549 Integer C type (`int`, `long`, `size_t`, ...).
7550
7551+__n__+::
7552 Field name.
7553
7554+__e__+::
7555 Argument expression.
7556
7557|
7558+ctf_integer_hex(__t__, __n__, __e__)+
7559
7560+ctf_user_integer_hex(__t__, __n__, __e__)+
7561|
7562Standard integer, displayed in base{nbsp}16.
7563
7564+__t__+::
7565 Integer C type.
7566
7567+__n__+::
7568 Field name.
7569
7570+__e__+::
7571 Argument expression.
7572
7573|+ctf_integer_oct(__t__, __n__, __e__)+
7574|
7575Standard integer, displayed in base{nbsp}8.
7576
7577+__t__+::
7578 Integer C type.
7579
7580+__n__+::
7581 Field name.
7582
7583+__e__+::
7584 Argument expression.
7585
7586|
7587+ctf_integer_network(__t__, __n__, __e__)+
7588
7589+ctf_user_integer_network(__t__, __n__, __e__)+
7590|
7591Integer in network byte order (big-endian), displayed in base{nbsp}10.
7592
7593+__t__+::
7594 Integer C type.
7595
7596+__n__+::
7597 Field name.
7598
7599+__e__+::
7600 Argument expression.
7601
7602|
7603+ctf_integer_network_hex(__t__, __n__, __e__)+
7604
7605+ctf_user_integer_network_hex(__t__, __n__, __e__)+
7606|
7607Integer in network byte order, displayed in base{nbsp}16.
7608
7609+__t__+::
7610 Integer C type.
7611
7612+__n__+::
7613 Field name.
7614
7615+__e__+::
7616 Argument expression.
7617
7618|
7619+ctf_enum(__N__, __t__, __n__, __e__)+
7620
7621+ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
7622
7623+ctf_user_enum(__N__, __t__, __n__, __e__)+
7624
7625+ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
7626|
7627Enumeration.
7628
7629+__N__+::
7630 Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
7631
7632+__t__+::
7633 Integer C type (`int`, `long`, `size_t`, ...).
7634
7635+__n__+::
7636 Field name.
7637
7638+__e__+::
7639 Argument expression.
7640
7641|
7642+ctf_string(__n__, __e__)+
7643
7644+ctf_string_nowrite(__n__, __e__)+
7645
7646+ctf_user_string(__n__, __e__)+
7647
7648+ctf_user_string_nowrite(__n__, __e__)+
7649|
7650Null-terminated string; undefined behavior if +__e__+ is `NULL`.
7651
7652+__n__+::
7653 Field name.
7654
7655+__e__+::
7656 Argument expression.
7657
7658|
7659+ctf_array(__t__, __n__, __e__, __s__)+
7660
7661+ctf_array_nowrite(__t__, __n__, __e__, __s__)+
7662
7663+ctf_user_array(__t__, __n__, __e__, __s__)+
7664
7665+ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
7666|
7667Statically-sized array of integers.
7668
7669+__t__+::
7670 Array element C type.
7671
7672+__n__+::
7673 Field name.
7674
7675+__e__+::
7676 Argument expression.
7677
7678+__s__+::
7679 Number of elements.
7680
7681|
7682+ctf_array_bitfield(__t__, __n__, __e__, __s__)+
7683
7684+ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7685
7686+ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
7687
7688+ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7689|
7690Statically-sized array of bits.
7691
7692The type of +__e__+ must be an integer type. +__s__+ is the number
7693of elements of such type in +__e__+, not the number of bits.
7694
7695+__t__+::
7696 Array element C type.
7697
7698+__n__+::
7699 Field name.
7700
7701+__e__+::
7702 Argument expression.
7703
7704+__s__+::
7705 Number of elements.
7706
7707|
7708+ctf_array_text(__t__, __n__, __e__, __s__)+
7709
7710+ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
7711
7712+ctf_user_array_text(__t__, __n__, __e__, __s__)+
7713
7714+ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
7715|
7716Statically-sized array, printed as text.
7717
7718The string doesn't need to be null-terminated.
7719
7720+__t__+::
7721 Array element C type (always `char`).
7722
7723+__n__+::
7724 Field name.
7725
7726+__e__+::
7727 Argument expression.
7728
7729+__s__+::
7730 Number of elements.
7731
7732|
7733+ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
7734
7735+ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7736
7737+ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
7738
7739+ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7740|
7741Dynamically-sized array of integers.
7742
7743The type of +__E__+ must be unsigned.
7744
7745+__t__+::
7746 Array element C type.
7747
7748+__n__+::
7749 Field name.
7750
7751+__e__+::
7752 Argument expression.
7753
7754+__T__+::
7755 Length expression C type.
7756
7757+__E__+::
7758 Length expression.
7759
7760|
7761+ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7762
7763+ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7764|
7765Dynamically-sized array of integers, displayed in base{nbsp}16.
7766
7767The type of +__E__+ must be unsigned.
7768
7769+__t__+::
7770 Array element C type.
7771
7772+__n__+::
7773 Field name.
7774
7775+__e__+::
7776 Argument expression.
7777
7778+__T__+::
7779 Length expression C type.
7780
7781+__E__+::
7782 Length expression.
7783
7784|+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
7785|
7786Dynamically-sized array of integers in network byte order (big-endian),
7787displayed in base{nbsp}10.
7788
7789The type of +__E__+ must be unsigned.
7790
7791+__t__+::
7792 Array element C type.
7793
7794+__n__+::
7795 Field name.
7796
7797+__e__+::
7798 Argument expression.
7799
7800+__T__+::
7801 Length expression C type.
7802
7803+__E__+::
7804 Length expression.
7805
7806|
7807+ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7808
7809+ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7810
7811+ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7812
7813+ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7814|
7815Dynamically-sized array of bits.
7816
7817The type of +__e__+ must be an integer type. +__s__+ is the number
7818of elements of such type in +__e__+, not the number of bits.
7819
7820The type of +__E__+ must be unsigned.
7821
7822+__t__+::
7823 Array element C type.
7824
7825+__n__+::
7826 Field name.
7827
7828+__e__+::
7829 Argument expression.
7830
7831+__T__+::
7832 Length expression C type.
7833
7834+__E__+::
7835 Length expression.
7836
7837|
7838+ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7839
7840+ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7841
7842+ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7843
7844+ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7845|
7846Dynamically-sized array, displayed as text.
7847
7848The string doesn't need to be null-terminated.
7849
7850The type of +__E__+ must be unsigned.
7851
7852The behaviour is undefined if +__e__+ is `NULL`.
7853
7854+__t__+::
7855 Sequence element C type (always `char`).
7856
7857+__n__+::
7858 Field name.
7859
7860+__e__+::
7861 Argument expression.
7862
7863+__T__+::
7864 Length expression C type.
7865
7866+__E__+::
7867 Length expression.
7868|====
7869
7870Use the `_user` versions when the argument expression, `e`, is
7871a user space address. In the cases of `ctf_user_integer*()` and
7872`ctf_user_float*()`, `&e` must be a user space address, thus `e` must
7873be addressable.
7874
7875The `_nowrite` versions omit themselves from the session trace, but are
7876otherwise identical. This means the `_nowrite` fields won't be written
7877in the recorded trace. Their primary purpose is to make some
7878of the event context available to the
7879<<enabling-disabling-events,event filters>> without having to
7880commit the data to sub-buffers.
7881
7882
7883[[glossary]]
7884== Glossary
7885
7886Terms related to LTTng and to tracing in general:
7887
7888Babeltrace::
7889 The http://diamon.org/babeltrace[Babeltrace] project, which includes:
7890+
7891* The
7892 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
7893 command-line interface.
7894* The libbabeltrace2 library which offers a
7895 https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
7896* https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
7897* Plugins.
7898
7899[[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
7900 A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
7901
7902[[def-channel]]<<channel,channel>>::
7903 An entity which is responsible for a set of
7904 <<def-ring-buffer,ring buffers>>.
7905+
7906<<def-event-rule,Event rules>> are always attached to a specific
7907channel.
7908
7909clock::
7910 A source of time for a <<def-tracer,tracer>>.
7911
7912[[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
7913 A process which is responsible for consuming the full
7914 <<def-sub-buffer,sub-buffers>> and write them to a file system or
7915 send them over the network.
7916
7917[[def-current-trace-chunk]]current trace chunk::
7918 A <<def-trace-chunk,trace chunk>> which includes the current content
05bbb80b
PP
7919 of all the <<def-sub-buffer,sub-buffers>> of the
7920 <<def-tracing-session-rotation,tracing session>> and the stream files
7921 produced since the latest event amongst:
96172a59
PP
7922+
7923* The creation of the <<def-tracing-session,tracing session>>.
7924* The last tracing session rotation, if any.
7925
7926<<channel-overwrite-mode-vs-discard-mode,discard mode>>::
7927 The <<def-event-record-loss-mode,event record loss mode>> in which
7928 the <<def-tracer,tracer>> _discards_ new event records when there's no
7929 <<def-sub-buffer,sub-buffer>> space left to store them.
7930
7931[[def-event]]event::
7932 The consequence of the execution of an
7933 <<def-instrumentation-point,instrumentation point>>, like a
7934 <<def-tracepoint,tracepoint>> that you manually place in some source
7935 code, or a Linux kernel kprobe.
7936+
7937An event is said to _occur_ at a specific time. <<def-lttng,LTTng>> can
05bbb80b
PP
7938take various actions upon the occurrence of an event, like record its
7939payload to a <<def-sub-buffer,sub-buffer>>.
96172a59
PP
7940
7941[[def-event-name]]event name::
7942 The name of an <<def-event,event>>, which is also the name of the
7943 <<def-event-record,event record>>.
7944+
7945This is also called the _instrumentation point name_.
7946
7947[[def-event-record]]event record::
7948 A record, in a <<def-trace,trace>>, of the payload of an
7949 <<def-event,event>> which occured.
7950
7951[[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
7952 The mechanism by which event records of a given
7953 <<def-channel,channel>> are lost (not recorded) when there is no
7954 <<def-sub-buffer,sub-buffer>> space left to store them.
7955
7956[[def-event-rule]]<<event,event rule>>::
7957 Set of conditions which must be satisfied for one or more occuring
7958 <<def-event,events>> to be recorded.
7959
7960[[def-incl-set]]inclusion set::
7961 In the <<pid-tracking,process attribute tracking>> context: a
7962 set of <<def-proc-attr,process attributes>> of a given type.
7963
7964<<instrumenting,instrumentation>>::
7965 The use of <<def-lttng,LTTng>> probes to make a piece of software
7966 traceable.
7967
7968[[def-instrumentation-point]]instrumentation point::
7969 A point in the execution path of a piece of software that, when
7970 reached by this execution, can emit an <<def-event,event>>.
7971
7972instrumentation point name::
7973 See _<<def-event-name,event name>>_.
7974
7975`java.util.logging`::
05bbb80b
PP
7976 The
7977 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
7978 of the Java platform.
96172a59
PP
7979
7980log4j::
7981 A http://logging.apache.org/log4j/1.2/[logging library] for Java
7982 developed by the Apache Software Foundation.
7983
7984log level::
7985 Level of severity of a log statement or user space
7986 <<def-instrumentation-point,instrumentation point>>.
7987
7988[[def-lttng]]LTTng::
7989 The _Linux Trace Toolkit: next generation_ project.
7990
7991<<lttng-cli,cmd:lttng>>::
7992 A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
7993 project which you can use to send and receive control messages to and
7994 from a <<def-session-daemon,session daemon>>.
7995
7996LTTng analyses::
7997 The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
7998 which is a set of analyzing programs that you can use to obtain a
7999 higher level view of an <<def-lttng,LTTng>> <<def-trace,trace>>.
8000
8001cmd:lttng-consumerd::
8002 The name of the <<def-consumer-daemon,consumer daemon>> program.
8003
8004cmd:lttng-crash::
8005 A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8006 which can convert <<def-ring-buffer,ring buffer>> files (usually
8007 <<persistent-memory-file-systems,saved on a persistent memory file
8008 system>>) to <<def-trace,trace>> files.
8009+
8010See man:lttng-crash(1).
8011
8012LTTng Documentation::
8013 This document.
8014
8015<<lttng-live,LTTng live>>::
8016 A communication protocol between the <<lttng-relayd,relay daemon>> and
8017 live viewers which makes it possible to see <<def-event-record,event
8018 records>> ``live'', as they are received by the
8019 <<def-relay-daemon,relay daemon>>.
8020
8021<<lttng-modules,LTTng-modules>>::
8022 The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8023 which contains the Linux kernel modules to make the Linux kernel
8024 <<def-instrumentation-point,instrumentation points>> available for
8025 <<def-lttng,LTTng>> tracing.
8026
8027cmd:lttng-relayd::
8028 The name of the <<def-relay-daemon,relay daemon>> program.
8029
8030cmd:lttng-sessiond::
8031 The name of the <<def-session-daemon,session daemon>> program.
8032
8033[[def-lttng-tools]]LTTng-tools::
8034 The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8035 contains the various programs and libraries used to
8036 <<controlling-tracing,control tracing>>.
8037
8038[[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8039 The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8040 contains libraries to instrument
8041 <<def-user-application,user applications>>.
8042
8043<<lttng-ust-agents,LTTng-UST Java agent>>::
8044 A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8045 allow the LTTng instrumentation of `java.util.logging` and Apache
8046 log4j{nbsp}1.2 logging statements.
8047
8048<<lttng-ust-agents,LTTng-UST Python agent>>::
8049 A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8050 to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8051 statements.
8052
8053<<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8054 The <<def-event-record-loss-mode,event record loss mode>> in which new
8055 <<def-event-record,event records>> _overwrite_ older event records
8056 when there's no <<def-sub-buffer,sub-buffer>> space left to store
8057 them.
8058
8059<<channel-buffering-schemes,per-process buffering>>::
8060 A <<def-buffering-scheme,buffering scheme>> in which each instrumented
8061 process has its own <<def-sub-buffer,sub-buffers>> for a given user
8062 space <<def-channel,channel>>.
8063
8064<<channel-buffering-schemes,per-user buffering>>::
8065 A <<def-buffering-scheme,buffering scheme>> in which all the processes
8066 of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
8067 given user space <<def-channel,channel>>.
8068
8069[[def-proc-attr]]process attribute::
8070 In the <<pid-tracking,process attribute tracking>> context:
8071+
8072* A process ID.
8073* A virtual process ID.
8074* A Unix user ID.
8075* A virtual Unix user ID.
8076* A Unix group ID.
8077* A virtual Unix group ID.
8078
8079[[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
8080 A process which is responsible for receiving the <<def-trace,trace>>
8081 data which a distant <<def-consumer-daemon,consumer daemon>> sends.
8082
8083[[def-ring-buffer]]ring buffer::
8084 A set of <<def-sub-buffer,sub-buffers>>.
8085
8086rotation::
8087 See _<<def-tracing-session-rotation,tracing session rotation>>_.
8088
8089[[def-session-daemon]]<<lttng-sessiond,session daemon>>::
8090 A process which receives control commands from you and orchestrates
8091 the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
8092
8093<<taking-a-snapshot,snapshot>>::
8094 A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
8095 of a given <<def-tracing-session,tracing session>>, saved as
8096 <<def-trace,trace>> files.
8097
8098[[def-sub-buffer]]sub-buffer::
8099 One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
8100 which contains <<def-event-record,event records>>.
8101
8102timestamp::
8103 The time information attached to an <<def-event,event>> when it is
8104 emitted.
8105
8106[[def-trace]]trace (_noun_)::
8107 A set of:
8108+
8109* One http://diamon.org/ctf/[CTF] metadata stream file.
8110* One or more CTF data stream files which are the concatenations of one
8111 or more flushed <<def-sub-buffer,sub-buffers>>.
8112
8113[[def-trace-verb]]trace (_verb_)::
8114 The action of recording the <<def-event,events>> emitted by an
8115 application or by a system, or to initiate such recording by
8116 controlling a <<def-tracer,tracer>>.
8117
8118[[def-trace-chunk]]trace chunk::
8119 A self-contained <<def-trace,trace>> which is part of a
8120 <<def-tracing-session,tracing session>>. Each
8121 <<def-tracing-session-rotation, tracing session rotation>> produces a
8122 <<def-trace-chunk-archive,trace chunk archive>>.
8123
8124[[def-trace-chunk-archive]]trace chunk archive::
8125 The result of a <<def-tracing-session-rotation, tracing session rotation>>.
8126+
8127<<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
8128containing <<def-tracing-session,tracing session>> is still active: you
8129are free to read it, modify it, move it, or remove it.
8130
8131Trace Compass::
8132 The http://tracecompass.org[Trace Compass] project and application.
8133
8134[[def-tracepoint]]tracepoint::
8135 An instrumentation point using the tracepoint mechanism of the Linux
8136 kernel or of <<def-lttng-ust,LTTng-UST>>.
8137
8138tracepoint definition::
8139 The definition of a single <<def-tracepoint,tracepoint>>.
8140
8141tracepoint name::
8142 The name of a <<def-tracepoint,tracepoint>>.
8143
8144[[def-tracepoint-provider]]tracepoint provider::
8145 A set of functions providing <<def-tracepoint,tracepoints>> to an
8146 instrumented <<def-user-application,user application>>.
8147+
8148Not to be confused with a <<def-tracepoint-provider-package,tracepoint
8149provider package>>: many tracepoint providers can exist within a
8150tracepoint provider package.
8151
8152[[def-tracepoint-provider-package]]tracepoint provider package::
8153 One or more <<def-tracepoint-provider,tracepoint providers>> compiled
8154 as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
8155 link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
8156 library].
8157
8158[[def-tracer]]tracer::
8159 A software which records emitted <<def-event,events>>.
8160
8161<<domain,tracing domain>>::
8162 A namespace for <<def-event,event>> sources.
8163
8164<<tracing-group,tracing group>>::
8165 The Unix group in which a Unix user can be to be allowed to
8166 <<def-trace-verb,trace>> the Linux kernel.
8167
8168[[def-tracing-session]]<<tracing-session,tracing session>>::
8169 A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
8170
8171[[def-tracing-session-rotation]]<<session-rotation,tracing session rotation>>::
8172 The action of archiving the
8173 <<def-current-trace-chunk,current trace chunk>> of a
8174 <<def-tracing-session,tracing session>>.
8175
8176tracked <<def-proc-attr,process attribute>>::
8177 A process attribute which is part of an <<def-incl-set,inclusion
8178 set>>.
8179
8180untracked process attribute::
8181 A process attribute which isn't part of an <<def-incl-set,inclusion
8182 set>>.
8183
8184[[def-user-application]]user application::
8185 An application running in user space, as opposed to a Linux kernel
8186 module, for example.
This page took 0.317323 seconds and 4 git commands to generate.