The LTTng Documentation
=======================
Philippe Proulx <pproulx@efficios.com>
-v2.11, 22 October 2019
+v2.11, 25 February 2021
include::../common/copyright.txt[]
+
[role="term"]
----
-$ lttng enable-event --kernel --syscall read
+$ lttng enable-event --kernel --syscall open
$ lttng add-context --kernel --type=callstack-kernel --type=callstack-user
----
+
+
This change makes the tracer more efficient on NUMA systems.
+* The <<proc-lttng-logger-abi,LTTng logger>> kernel module now also
+ creates a ``misc'' device named `lttng-logger`, which udev will
+ make accessible as the path:{/dev/lttng-logger} special file.
++
+The `lttng-logger` device shares the `/proc/lttng-logger` file's ABI,
+but it works from within containers when the path is made accessible to
+them.
+
[[nuts-and-bolts]]
== Nuts and bolts
is the de facto function tracer of the Linux kernel. Its user
interface is a set of special files in sysfs.
* https://perf.wiki.kernel.org/[perf] is
- a performance analyzing tool for Linux which supports hardware
+ a performance analysis tool for Linux which supports hardware
performance counters, tracepoints, as well as other counters and
types of probes. perf's controlling utility is the cmd:perf command
line/curses tool.
* You only need to install LTTng-UST if you intend to trace user
applications.
-[IMPORTANT]
-====
-As of 22 October 2019, LTTng{nbsp}{revision} is not available
-as distribution packages, except for <<arch-linux,Arch Linux>>.
+[role="growable"]
+.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 5{nbsp}August{nbsp}2020.
+|====
+|Distribution |Available in releases
-You can <<building-from-source,build LTTng{nbsp}{revision} from source>>
-to install and use it.
-====
+|https://www.ubuntu.com/[Ubuntu]
+|<<ubuntu,Ubuntu{nbsp}20.04 _Focal Fossa_>>.
+Ubuntu{nbsp}16.04 _Xenial Xerus_ and Ubuntu{nbsp}18.04 _Bionic Beaver_:
+<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
-[[arch-linux]]
-=== Arch Linux
+|https://getfedora.org/[Fedora]
+|<<fedora,Fedora{nbsp}32>>.
-LTTng-UST{nbsp}{revision} is available in Arch Linux's _community_
-repository, while LTTng-tools{nbsp}{revision} and
-LTTng-modules{nbsp}{revision} are available in the
-https://aur.archlinux.org/[AUR].
+|https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
+|See http://packages.efficios.com/[EfficiOS Enterprise Packages].
-To install LTTng{nbsp}{revision} on Arch Linux, using
-https://github.com/actionless/pikaur[pikaur] for the AUR packages:
+|https://buildroot.org/[Buildroot]
+|xref:buildroot[Buildroot{nbsp}2019.11, Buildroot{nbsp}2020.02, and
+Buildroot{nbsp}2020.05].
+
+|https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
+https://www.yoctoproject.org/[Yocto]
+|<<oe-yocto,Yocto Project{nbsp}3.1 _Dunfell_>>
+(`openembedded-core` layer).
+|====
+
+
+[[ubuntu]]
+=== [[ubuntu-official-repositories]]Ubuntu
+
+LTTng{nbsp}{revision} is available on Ubuntu{nbsp}20.04 _Focal Fossa_.
+For previous supported releases of Ubuntu,
+<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
+
+To install LTTng{nbsp}{revision} on Ubuntu{nbsp}20.04 _Focal Fossa_:
. Install the main LTTng{nbsp}{revision} packages:
+
--
[role="term"]
----
-# pacman -Sy lttng-ust
-$ pikaur -Sy lttng-tools
-$ pikaur -Sy lttng-modules
+# apt-get install lttng-tools
+# apt-get install lttng-modules-dkms
+# apt-get install liblttng-ust-dev
----
--
-. **If you need to instrument and trace <<python-application,Python
- applications>>**, install the LTTng-UST Python agent:
+. **If you need to instrument and trace
+ <<java-application,Java applications>>**, install the LTTng-UST
+ Java agent:
+
--
[role="term"]
----
-# pacman -Sy python-lttngust
-# pacman -Sy python2-lttngust
+# apt-get install liblttng-ust-agent-java
----
--
+. **If you need to instrument and trace
+ <<python-application,Python{nbsp}3 applications>>**, install the
+ LTTng-UST Python agent:
++
+--
+[role="term"]
+----
+# apt-get install python3-lttngust
+----
+--
+
+[[ubuntu-ppa]]
+=== Ubuntu: noch:{LTTng} Stable {revision} PPA
+
+The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
+Stable{nbsp}{revision} PPA] offers the latest stable
+LTTng{nbsp}{revision} packages for Ubuntu{nbsp}16.04 _Xenial Xerus_ and
+Ubuntu{nbsp}18.04 _Bionic Beaver_.
+
+To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
+PPA:
+
+. Add the LTTng Stable{nbsp}{revision} PPA repository and update the
+ list of packages:
++
+--
+[role="term"]
+----
+# apt-add-repository ppa:lttng/stable-2.11
+# apt-get update
+----
+--
+
+. Install the main LTTng{nbsp}{revision} packages:
++
+--
+[role="term"]
+----
+# apt-get install lttng-tools
+# apt-get install lttng-modules-dkms
+# apt-get install liblttng-ust-dev
+----
+--
+
+. **If you need to instrument and trace
+ <<java-application,Java applications>>**, install the LTTng-UST
+ Java agent:
++
+--
+[role="term"]
+----
+# apt-get install liblttng-ust-agent-java
+----
+--
+
+. **If you need to instrument and trace
+ <<python-application,Python{nbsp}3 applications>>**, install the
+ LTTng-UST Python agent:
++
+--
+[role="term"]
+----
+# apt-get install python3-lttngust
+----
+--
+
+
+[[fedora]]
+=== Fedora
+
+To install LTTng{nbsp}{revision} on Fedora{nbsp}32:
+
+. Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
+ packages:
++
+--
+[role="term"]
+----
+# yum install lttng-tools
+# yum install lttng-ust
+----
+--
+
+. Download, build, and install the latest LTTng-modules{nbsp}{revision}:
++
+--
+[role="term"]
+----
+$ cd $(mktemp -d) &&
+wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.11.tar.bz2 &&
+tar -xf lttng-modules-latest-2.11.tar.bz2 &&
+cd lttng-modules-2.11.* &&
+make &&
+sudo make modules_install &&
+sudo depmod -a
+----
+--
+
+[IMPORTANT]
+.Java and Python application instrumentation and tracing
+====
+If you need to instrument and trace <<java-application,Java
+applications>> on Fedora, you need to build and install
+LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
+the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
+`--enable-java-agent-all` options to the `configure` script, depending
+on which Java logging framework you use.
+
+If you need to instrument and trace <<python-application,Python
+applications>> on Fedora, you need to build and install
+LTTng-UST{nbsp}{revision} from source and pass the
+`--enable-python-agent` option to the `configure` script.
+====
+
+
+[[buildroot]]
+=== Buildroot
+
+To install LTTng{nbsp}{revision} on Buildroot{nbsp}2019.11,
+Buildroot{nbsp}2020.02, or Buildroot{nbsp}2020.05:
+
+. Launch the Buildroot configuration tool:
++
+--
+[role="term"]
+----
+$ make menuconfig
+----
+--
+
+. In **Kernel**, check **Linux kernel**.
+. In **Toolchain**, check **Enable WCHAR support**.
+. In **Target packages**{nbsp}→ **Debugging, profiling and benchmark**,
+ check **lttng-modules** and **lttng-tools**.
+. In **Target packages**{nbsp}→ **Libraries**{nbsp}→
+ **Other**, check **lttng-libust**.
+
+
+[[oe-yocto]]
+=== OpenEmbedded and Yocto
+
+LTTng{nbsp}{revision} recipes are available in the
+https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
+layer for Yocto Project{nbsp}3.1 _Dunfell_ under the following names:
+
+* `lttng-tools`
+* `lttng-modules`
+* `lttng-ust`
+
+With BitBake, the simplest way to include LTTng recipes in your target
+image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
+
+----
+IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
+----
+
+If you use Hob:
+
+. Select a machine and an image recipe.
+. Click **Edit image recipe**.
+. Under the **All recipes** tab, search for **lttng**.
+. Check the desired LTTng recipes.
+
[[building-from-source]]
=== Build from source
_tracing session mode_ dictates where to send it. The following modes
are available in LTTng{nbsp}{revision}:
-Local mode::
+[[local-mode]]Local mode::
LTTng writes the traces to the file system of the machine it traces
(target system).
-Network streaming mode::
+[[net-streaming-mode]]Network streaming mode::
LTTng sends the traces over the network to a
<<lttng-relayd,relay daemon>> running on a remote system.
target's file system or to send it over the network to a
<<lttng-relayd,relay daemon>> running on a remote system.
-Live mode::
+[[live-mode]]Live mode::
This mode is similar to the network streaming mode, but a live
trace viewer can connect to the distant relay daemon to
<<lttng-live,view event records as LTTng generates them>>.
The available event record loss modes are:
Discard mode::
- Drop the newest event records until a the tracer releases a
- sub-buffer.
+ Drop the newest event records until the tracer releases a sub-buffer.
+
This is the only available mode when you specify a
<<opt-blocking-timeout,blocking timeout>>.
fit.
In discard mode, LTTng increments a count of lost event records when an
-event record is lost and saves this count to the trace. In overwrite
-mode, since LTTng{nbsp}2.8, LTTng increments a count of lost sub-buffers
-when a sub-buffer is lost and saves this count to the trace. In this
-mode, LTTng does not write to the trace the exact number of lost event
-records in those lost sub-buffers. Trace analyses can use the trace's
-saved discarded event record and sub-buffer counts to decide whether or
-not to perform the analyses even if trace data is known to be missing.
+event record is lost and saves this count to the trace. Since
+LTTng{nbsp}2.8, in overwrite mode, LTTng writes to a given sub-buffer
+its sequence number within its data stream. With a <<local-mode,local>>,
+<<net-streaming-mode,network streaming>>, or <<live-mode,live>>
+<<tracing-session,tracing session>>, a trace reader can use such
+sequence numbers to report lost packets. In overwrite mode, LTTng
+doesn't write to the trace the exact number of lost event records in
+those lost sub-buffers.
+
+Trace analyses can use saved discarded event record and sub-buffer
+(packet) counts of the trace to decide whether or not to perform the
+analyses even if trace data is known to be missing.
There are a few ways to decrease your probability of losing event
records.
You set the conditions when you <<enabling-disabling-events,create
an event rule>>.
-You always attach an event rule to <<channel,channel>> when you create
+You always attach an event rule to a <<channel,channel>> when you create
it.
When an event passes the conditions of an event rule, LTTng records it
sessions, but most of them require to create a handle first using
`lttng_create_handle()`.
-The best available developer documentation for `liblttng-ctl` is, as of
-LTTng{nbsp}{revision}, its installed header files. Every function and
-structure is thoroughly documented.
+As of LTTng{nbsp}{revision}, the best available developer documentation for
+`liblttng-ctl` is its installed header files. Every function and structure is
+thoroughly documented.
[[lttng-ust]]
* The _LTTng logger_ module.
+
The LTTng logger module implements the special path:{/proc/lttng-logger}
-file so that any executable can generate LTTng events by opening and
-writing to this file.
+(and path:{/dev/lttng-logger} since LTTng{nbsp}2.11) files so that any
+executable can generate LTTng events by opening and writing to those
+files.
+
See <<proc-lttng-logger-abi,LTTng logger>>.
[[prebuilt-ust-helpers]]
=== Prebuilt user space tracing helpers
-The LTTng-UST package provides a few helpers in the form or preloadable
+The LTTng-UST package provides a few helpers in the form of preloadable
shared objects which automatically instrument system functions and
calls.
.LTTng-UST Java agent imported by a Java application.
image::java-app.png[]
-Note that the methods described below are new in LTTng{nbsp}{revision}.
+Note that the methods described below are new in LTTng{nbsp}2.8.
Previous LTTng versions use another technique.
NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
=== LTTng logger
The `lttng-tracer` Linux kernel module, part of
-<<lttng-modules,LTTng-modules>>, creates the special LTTng logger file
-path:{/proc/lttng-logger} when it's loaded. Any application can write
-text data to this file to emit an LTTng event.
+<<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
+path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
+LTTng{nbsp}2.11) when it's loaded. Any application can write text data
+to any of those files to emit an LTTng event.
[role="img-100"]
.An application writes to the LTTng logger file to emit an LTTng event.
[role="term"]
----
-$ echo "Some message, some $variable" > /proc/lttng-logger
+$ echo "Some message, some $variable" > /dev/lttng-logger
----
Any event that the LTTng logger emits is named `lttng_logger` and
To use the LTTng logger:
-* From any application, write text data to the path:{/proc/lttng-logger}
+* From any application, write text data to the path:{/dev/lttng-logger}
file.
The `msg` field of `lttng_logger` event records contains the
[source,bash]
.path:{test.bash}
----
-echo 'Hello, World!' > /proc/lttng-logger
+echo 'Hello, World!' > /dev/lttng-logger
sleep 2
-df --human-readable --print-type / > /proc/lttng-logger
+df --human-readable --print-type / > /dev/lttng-logger
----
<<creating-destroying-tracing-sessions,Create a tracing session>>,
**process priority** of the thread in which the event occurs.
* The **hostname** of the system on which the event occurs.
* The Linux kernel and user call stacks (since
- LTTng{nbsp}{revision}).
+ LTTng{nbsp}2.11).
* The current values of many possible **performance counters** using
perf, for example:
** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
When LTTng loads a configuration, it restores your saved tracing session
as if you just configured it manually.
-See man:lttng(1) for the complete list of command-line options. You
-can also save and load all many sessions at a time, and decide in which
+See man:lttng-load(1) for the complete list of command-line options. You
+can also save and load many sessions at a time, and decide in which
directory to output the XML files.