The LTTng Documentation
=======================
Philippe Proulx <pproulx@efficios.com>
-v2.9, 29 November 2016
+v2.9, 25 February 2021
include::../common/copyright.txt[]
+include::../common/warning-not-maintained.txt[]
+
+
include::../common/welcome.txt[]
[[whats-new]]
== What's new in LTTng {revision}?
+LTTng{nbsp}{revision} bears the name _Joannès_. A Berliner Weisse style
+beer from the http://letreflenoir.com/[Trèfle Noir] microbrewery in
+https://en.wikipedia.org/wiki/Rouyn-Noranda[Rouyn-Noranda], the
+https://www.beeradvocate.com/beer/profile/20537/238967/[_**Joannès**_]
+is a tangy beer with a distinct pink dress and intense fruit flavor,
+thanks to the presence of fresh blackcurrant grown in Témiscamingue.
+
+New features and changes in LTTng{nbsp}{revision}:
+
* **Tracing control**:
** You can override the name or the URL of a tracing session
configuration when you use man:lttng-load(1) thanks to the new
--
[role="term"]
----
-lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
+$ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
----
--
+
[[installing-lttng]]
== Installation
+include::../common/warning-no-installation.txt[]
+
**LTTng** is a set of software <<plumbing,components>> which interact to
<<instrumenting,instrument>> the Linux kernel and user applications, and
to <<controlling-tracing,control tracing>> (start and stop
trace user applications.
Most distributions mark the LTTng-modules and LTTng-UST packages as
-optional when installing LTTng-tools (which is always required). In the
-following sections, we always provide the steps to install all three,
-but note that:
+optional when installing LTTng-tools (which is always required). Note
+that:
* You only need to install LTTng-modules if you intend to trace the
Linux kernel.
* You only need to install LTTng-UST if you intend to trace user
applications.
-[role="growable"]
-.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 29 November 2016.
-|====
-|Distribution |Available in releases |Alternatives
-
-|Ubuntu
-|Ubuntu{nbsp}14.04 _Trusty Tahr_ and Ubuntu{nbsp}16.04 _Xenial Xerus_:
-<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
-|link:/docs/v2.8#doc-ubuntu[LTTng{nbsp}2.8 for Ubuntu{nbsp}16.10 _Yakkety Yak_].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
-other Ubuntu releases.
-
-|Fedora
-|_Not available_
-|<<building-from-source,Build LTTng-modules{nbsp}{revision} from
-source>>.
-
-|Debian
-|_Not available_
-|<<building-from-source,Build LTTng-modules{nbsp}{revision} from
-source>>.
-
-|openSUSE
-|_Not available_
-|<<building-from-source,Build LTTng-modules{nbsp}{revision} from
-source>>.
-
-|Arch Linux
-|<<arch-linux,Latest AUR packages>>.
-|
-
-|Alpine Linux
-|_Not available_
-|link:/docs/v2.8#doc-alpine-linux[LTTng{nbsp}2.8 for Alpine Linux{nbsp}"edge"].
-
-LTTng{nbsp}2.8 for Alpine Linux{nbsp}3.5 (not released yet).
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
-other Alpine Linux releases.
-
-|RHEL and SLES
-|See http://packages.efficios.com/[EfficiOS Enterprise Packages].
-|
-
-|Buildroot
-|_Not available_
-|LTTng{nbsp}2.8 for Buildroot{nbsp}2016.11 (not released yet).
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
-other Buildroot releases.
-
-|OpenEmbedded and Yocto
-|_Not available_
-|link:/docs/v2.8#doc-oe-yocto[LTTng{nbsp}2.8 for Yocto Project{nbsp}2.2 _Morty_]
-(`openembedded-core` layer).
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
-other OpenEmbedded releases.
-|====
-
-
-[[ubuntu]]
-=== [[ubuntu-official-repositories]]Ubuntu
-
-[[ubuntu-ppa]]
-==== noch:{LTTng} Stable {revision} PPA
-
-The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
-Stable{nbsp}{revision} PPA] offers the latest stable
-LTTng{nbsp}{revision} packages for:
-
-* Ubuntu{nbsp}14.04 _Trusty Tahr_
-* Ubuntu{nbsp}16.04 _Xenial Xerus_
-
-To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision} PPA:
-
-. Add the LTTng Stable{nbsp}{revision} PPA repository and update the
- list of packages:
-+
---
-[role="term"]
-----
-sudo apt-add-repository ppa:lttng/stable-2.9
-sudo apt-get update
-----
---
-
-. Install the main LTTng{nbsp}{revision} packages:
-+
---
-[role="term"]
-----
-sudo apt-get install lttng-tools
-sudo apt-get install lttng-modules-dkms
-sudo apt-get install liblttng-ust-dev
-----
---
-
-. **If you need to instrument and trace
- <<java-application,Java applications>>**, install the LTTng-UST
- Java agent:
-+
---
-[role="term"]
-----
-sudo apt-get install liblttng-ust-agent-java
-----
---
-
-. **If you need to instrument and trace
- <<python-application,Python{nbsp}3 applications>>**, install the
- LTTng-UST Python agent:
-+
---
-[role="term"]
-----
-sudo apt-get install python3-lttngust
-----
---
-
-
-[[arch-linux]]
-=== Arch Linux
-
-To install LTTng{nbsp}{revision} on Arch Linux using
-https://archlinux.fr/yaourt-en[Yaourt]:
-
-. Install the main LTTng{nbsp}{revision} packages:
-+
---
-[role="term"]
-----
-yaourt -S lttng-tools
-yaourt -S lttng-ust
-yaourt -S lttng-modules
-----
---
-
-. **If you need to instrument and trace <<python-application,Python
- applications>>**, install the LTTng-UST Python agent:
-+
---
-[role="term"]
-----
-yaourt -S python-lttngust
-yaourt -S python2-lttngust
-----
---
-
-
-[[enterprise-distributions]]
-=== RHEL, SUSE, and other enterprise distributions
-
-To install LTTng on enterprise Linux distributions, such as Red Hat
-Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SUSE), please
-see http://packages.efficios.com/[EfficiOS Enterprise Packages].
-
[[building-from-source]]
=== Build from source
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.9.tar.bz2 &&
tar -xf lttng-modules-latest-2.9.tar.bz2 &&
cd lttng-modules-2.9.* &&
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.9.tar.bz2 &&
tar -xf lttng-ust-latest-2.9.tar.bz2 &&
cd lttng-ust-2.9.* &&
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.9.tar.bz2 &&
tar -xf lttng-tools-latest-2.9.tar.bz2 &&
cd lttng-tools-2.9.* &&
[[tracing-the-linux-kernel]]
=== Trace the Linux kernel
-The following command lines start with cmd:sudo because you need root
-privileges to trace the Linux kernel. You can avoid using cmd:sudo if
-your Unix user is a member of the <<lttng-sessiond,tracing group>>.
+The following command lines start with the `#` prompt because you need
+root privileges to trace the Linux kernel. You can also trace the kernel
+as a regular user if your Unix user is a member of the
+<<tracing-group,tracing group>>.
-. Create a <<tracing-session,tracing session>>:
+. Create a <<tracing-session,tracing session>> which writes its traces
+ to dir:{/tmp/my-kernel-trace}:
+
--
[role="term"]
----
-sudo lttng create my-kernel-session
+# lttng create my-kernel-session --output=/tmp/my-kernel-trace
----
--
--
[role="term"]
----
-lttng list --kernel
+# lttng list --kernel
+# lttng list --kernel --syscall
----
--
-. Create an <<event,event rule>> which matches the desired event names,
- for example `sched_switch` and `sched_process_fork`:
+. Create <<event,event rules>> which match the desired instrumentation
+ point names, for example the `sched_switch` and `sched_process_fork`
+ tracepoints, and the man:open(2) and man:close(2) system calls:
+
--
[role="term"]
----
-sudo lttng enable-event --kernel sched_switch,sched_process_fork
+# lttng enable-event --kernel sched_switch,sched_process_fork
+# lttng enable-event --kernel --syscall open,close
----
--
+
-You can also create an event rule which _matches_ all the Linux kernel
+You can also create an event rule which matches _all_ the Linux kernel
tracepoints (this will generate a lot of data when tracing):
+
--
[role="term"]
----
-sudo lttng enable-event --kernel --all
+# lttng enable-event --kernel --all
----
--
-. Start tracing:
+. <<basic-tracing-session-control,Start tracing>>:
+
--
[role="term"]
----
-sudo lttng start
+# lttng start
----
--
. Do some operation on your system for a few seconds. For example,
load a website, or list the files of a directory.
-. Stop tracing and destroy the tracing session:
+. <<basic-tracing-session-control,Stop tracing>> and destroy the
+ tracing session:
+
--
[role="term"]
----
-sudo lttng stop
-sudo lttng destroy
+# lttng stop
+# lttng destroy
----
--
+
The man:lttng-destroy(1) command does not destroy the trace data; it
only destroys the state of the tracing session.
-By default, LTTng saves the traces in
-+$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
-where +__name__+ is the tracing session name. Note that the
-env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
+. For the sake of this example, make the recorded trace accessible to
+ the non-root users:
++
+--
+[role="term"]
+----
+# chown -R $(whoami) /tmp/my-kernel-trace
+----
+--
See <<viewing-and-analyzing-your-traces,View and analyze the
recorded events>> to view the recorded events.
--
[role="term"]
----
-gcc -c -I. hello-tp.c
+$ gcc -c -I. hello-tp.c
----
--
--
[role="term"]
----
-gcc -c hello.c
+$ gcc -c hello.c
----
--
--
[role="term"]
----
-gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
+$ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
----
--
--
[role="term"]
----
-./hello world and beyond
+$ ./hello world and beyond
----
--
+
--
[role="term"]
----
-lttng-sessiond --daemonize
+$ lttng-sessiond --daemonize
----
--
+
--
[role="term"]
----
-lttng list --userspace
+$ lttng list --userspace
----
--
+
--
[role="term"]
----
-lttng create my-user-space-session
+$ lttng create my-user-space-session
----
--
--
[role="term"]
----
-lttng enable-event --userspace hello_world:my_first_tracepoint
+$ lttng enable-event --userspace hello_world:my_first_tracepoint
----
--
-. Start tracing:
+. <<basic-tracing-session-control,Start tracing>>:
+
--
[role="term"]
----
-lttng start
+$ lttng start
----
--
. Go back to the running `hello` application and press Enter. The
program executes all `tracepoint()` instrumentation points and exits.
-. Stop tracing and destroy the tracing session:
+. <<basic-tracing-session-control,Stop tracing>> and destroy the
+ tracing session:
+
--
[role="term"]
----
-sudo lttng stop
-sudo lttng destroy
+$ lttng stop
+$ lttng destroy
----
--
+
By default, LTTng saves the traces in
+$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
-where +__name__+ is the tracing session name. Note that the
+where +__name__+ is the tracing session name. The
env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
See <<viewing-and-analyzing-your-traces,View and analyze the
NOTE: This section assumes that the traces recorded during the previous
tutorials were saved to their default location, in the
-dir:{$LTTNG_HOME/lttng-traces} directory. Note that the env:LTTNG_HOME
+dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
environment variable defaults to `$HOME` if not set.
[role="term"]
----
-babeltrace ~/lttng-traces/my-user-space-session*
+$ babeltrace ~/lttng-traces/my-user-space-session*
----
cmd:babeltrace finds all traces recursively within the given path and
[role="term"]
----
-babeltrace ~/lttng-traces/my-kernel-session* | grep sys_
+$ babeltrace /tmp/my-kernel-trace | grep _switch
----
You can pipe the output of cmd:babeltrace into a tool like man:wc(1) to
[role="term"]
----
-babeltrace ~/lttng-traces/my-kernel-session* | grep sys_read | wc --lines
+$ babeltrace /tmp/my-kernel-trace | grep _open | wc --lines
----
[role="term"]
----
-python3 top5proc.py ~/lttng-traces/my-kernel-session*/kernel
+$ python3 top5proc.py /tmp/my-kernel-trace/kernel
----
Output example:
_tracing session mode_ dictates where to send it. The following modes
are available in LTTng{nbsp}{revision}:
-Local mode::
+[[local-mode]]Local mode::
LTTng writes the traces to the file system of the machine being traced
(target system).
-Network streaming mode::
+[[net-streaming-mode]]Network streaming mode::
LTTng sends the traces over the network to a
<<lttng-relayd,relay daemon>> running on a remote system.
or to send it over the network to a <<lttng-relayd,relay daemon>>
running on a remote system.
-Live mode::
+[[live-mode]]Live mode::
This mode is similar to the network streaming mode, but a live
trace viewer can connect to the distant relay daemon to
<<lttng-live,view event records as LTTng generates them>> by
discard mode, the tracer only discards the event record that doesn't
fit.
-In discard mode, LTTng increments a count of lost event records when
-an event record is lost and saves this count to the trace. In
-overwrite mode, LTTng keeps no information when it overwrites a
-sub-buffer before consuming it.
+In discard mode, LTTng increments a count of lost event records when an
+event record is lost and saves this count to the trace. Since
+LTTng{nbsp}2.8, in overwrite mode, LTTng writes to a given sub-buffer
+its sequence number within its data stream. With a <<local-mode,local>>,
+<<net-streaming-mode,network streaming>>, or <<live-mode,live>>
+<<tracing-session,tracing session>>, a trace reader can use such
+sequence numbers to report lost packets. In overwrite mode, LTTng
+doesn't write to the trace the exact number of lost event records in
+those lost sub-buffers.
+
+Trace analyses can use saved discarded event record and sub-buffer
+(packet) counts of the trace to decide whether or not to perform the
+analyses even if trace data is known to be missing.
There are a few ways to decrease your probability of losing event
records.
* **High event throughput**: In general, prefer bigger sub-buffers to
lower the risk of losing event records.
+
-Having bigger sub-buffers also ensures a lower sub-buffer switching
-frequency.
+Having bigger sub-buffers also ensures a lower
+<<channel-switch-timer,sub-buffer switching frequency>>.
+
The number of sub-buffers is only meaningful if you create the channel
in overwrite mode: in this case, if a sub-buffer overwrite happens, the
An **event** is the consequence of the execution of an _instrumentation
point_, like a tracepoint that you manually place in some source code,
or a Linux kernel KProbe. An event is said to _occur_ at a specific
-time. Different actions can be taken upon the occurance of an event,
+time. Different actions can be taken upon the occurrence of an event,
like record the event's payload to a buffer.
An **event record** is the representation of an event in a sub-buffer. A
* **LTTng-tools**: Libraries and command-line interface to
control tracing sessions.
** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
-** <<lttng-consumerd,Consumer daemon>> (man:lttng-consumerd(8)).
+** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
[role="term"]
----
-lttng <general options> <command> <command options>
+$ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
----
The <<controlling-tracing,Tracing control>> section explores the
.The consumer daemon.
image::plumbing-consumerd.png[]
-The _consumer daemon_, man:lttng-consumerd(8), is a daemon which shares
+The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
ring buffers with user applications or with the LTTng kernel modules to
collect trace data and send it to some location (on disk or to a
<<lttng-relayd,relay daemon>> over the network). The consumer daemon
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o tpp.o -llttng-ust -ldl
+$ gcc -o app app.o tpp.o -llttng-ust -ldl
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -c tpp.c
+$ gcc -I. -c tpp.c
----
--
--
[role="term"]
----
-ar rcs tpp.a tpp.o
+$ ar rcs tpp.a tpp.o
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o tpp.a -llttng-ust -ldl
+$ gcc -o app app.o tpp.a -llttng-ust -ldl
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -ldl -L. -ltpp
+$ gcc -o app app.o -ldl -L. -ltpp
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -ldl
+$ gcc -o app app.o -ldl
----
--
--
[role="term"]
----
-LD_PRELOAD=./libtpp.so ./app
+$ LD_PRELOAD=./libtpp.so ./app
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -ldl
+$ gcc -o app app.o -ldl
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
+$ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -L. -lemon
+$ gcc -o app app.o -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
+$ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -L. -lemon
+$ gcc -o app app.o -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl
+$ gcc -shared -o libemon.so emon.o -ldl
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -L. -lemon
+$ gcc -o app app.o -L. -lemon
----
--
--
[role="term"]
----
-LD_PRELOAD=./libtpp.so ./app
+$ LD_PRELOAD=./libtpp.so ./app
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl
+$ gcc -shared -o libemon.so emon.o -ldl
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -L. -lemon
+$ gcc -o app app.o -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
+$ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -ldl -L. -lemon
+$ gcc -o app app.o -ldl -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl
+$ gcc -shared -o libemon.so emon.o -ldl
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -ldl -L. -lemon
+$ gcc -o app app.o -ldl -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o -ldl
+$ gcc -shared -o libemon.so emon.o -ldl
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o -L. -lemon
+$ gcc -o app app.o -L. -lemon
----
--
--
[role="term"]
----
-LD_PRELOAD=./libtpp.so ./app
+$ LD_PRELOAD=./libtpp.so ./app
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o
+$ gcc -shared -o libemon.so emon.o
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
+$ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
--
[role="term"]
----
-gcc -c app.c
+$ gcc -c app.c
----
--
--
[role="term"]
----
-gcc -Wl,--export-dynamic -o app app.o tpp.o \
+$ gcc -Wl,--export-dynamic -o app app.o tpp.o \
-llttng-ust -ldl
----
--
--
[role="term"]
----
-gcc -I. -fpic -c emon.c
+$ gcc -I. -fpic -c emon.c
----
--
--
[role="term"]
----
-gcc -shared -o libemon.so emon.o
+$ gcc -shared -o libemon.so emon.o
----
--
--
[role="term"]
----
-./app
+$ ./app
----
--
|====
[role="term"]
----
-LD_PRELOAD=liblttng-ust-fork.so ./my-app
+$ LD_PRELOAD=liblttng-ust-fork.so ./my-app
----
If your tracepoint provider package is
[role="term"]
----
-LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
+$ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
----
[role="term"]
----
-LD_PRELOAD=liblttng-ust-fd.so ./my-app
+$ LD_PRELOAD=liblttng-ust-fd.so ./my-app
----
Typical use cases include closing all the file descriptors after
[role="term"]
----
-gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
+$ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
----
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
cd userspace-rcu-0.9.* &&
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.9.tar.bz2 &&
tar -xf lttng-ust-latest-2.9.tar.bz2 &&
cd lttng-ust-2.9.* &&
[role="term"]
----
-LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
+$ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
----
====
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.9.tar.bz2 &&
tar -xf lttng-tools-latest-2.9.tar.bz2 &&
cd lttng-tools-2.9.* &&
./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
- LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
+ LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
+ --disable-bin-lttng --disable-bin-lttng-crash \
+ --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
make &&
cd src/bin/lttng-consumerd &&
sudo make install &&
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.9.tar.bz2 &&
tar -xf lttng-tools-latest-2.9.tar.bz2 &&
cd lttng-tools-2.9.* &&
--
[role="term"]
----
-gcc -m32 -c -I. hello-tp.c
-gcc -m32 -c hello.c
-gcc -m32 -o hello hello.o hello-tp.o \
+$ gcc -m32 -c -I. hello-tp.c
+$ gcc -m32 -c hello.c
+$ gcc -m32 -o hello hello.o hello-tp.o \
-L/usr/lib32 -L/usr/local/lib32 \
-Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
-llttng-ust -ldl
--
[role="term"]
----
-gcc -o app app.c -llttng-ust
+$ gcc -o app app.c -llttng-ust
----
--
--
[role="term"]
----
-lttng enable-event --userspace 'lttng_ust_tracef:*'
+$ lttng enable-event --userspace 'lttng_ust_tracef:*'
----
--
--
[role="term"]
----
-gcc -o app app.c -llttng-ust
+$ gcc -o app app.c -llttng-ust
----
--
--
[role="term"]
----
-lttng enable-event --userspace 'lttng_ust_tracelog:*'
+$ lttng enable-event --userspace 'lttng_ust_tracelog:*'
--loglevel=TRACE_WARNING
----
--
--
[role="term"]
----
-lttng enable-event --userspace 'lttng_ust_tracelog:*'
+$ lttng enable-event --userspace 'lttng_ust_tracelog:*'
--loglevel-only=TRACE_INFO
----
--
[[prebuilt-ust-helpers]]
=== Prebuilt user space tracing helpers
-The LTTng-UST package provides a few helpers in the form or preloadable
+The LTTng-UST package provides a few helpers in the form of preloadable
shared objects which automatically instrument system functions and
calls.
--
[role="term"]
----
-LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
+$ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
----
--
+
--
[role="term"]
----
-LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
+$ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
----
--
to the entry and exit points of functions.
man:gcc(1) and man:clang(1) have an option named
-https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html[`-finstrument-functions`]
+https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
which generates instrumentation calls for entry and exit to functions.
The LTTng-UST function tracing helpers,
path:{liblttng-ust-cyg-profile.so} and
.LTTng-UST Java agent imported by a Java application.
image::java-app.png[]
-Note that the methods described below are new in LTTng{nbsp}{revision}.
+Note that the methods described below are new in LTTng{nbsp}2.8.
Previous LTTng versions use another technique.
NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
[role="term"]
----
-javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
+$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
----
<<creating-destroying-tracing-sessions,Create a tracing session>>,
[role="term"]
----
-lttng create
-lttng enable-event --jul jello
-lttng start
+$ lttng create
+$ lttng enable-event --jul jello
+$ lttng start
----
Run the compiled class:
[role="term"]
----
-java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
+$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
----
<<basic-tracing-session-control,Stop tracing>> and inspect the
[role="term"]
----
-lttng stop
-lttng view
+$ lttng stop
+$ lttng view
----
====
+In the resulting trace, an <<event,event record>> generated by a Java
+application using `java.util.logging` is named `lttng_jul:event` and
+has the following fields:
+
+`msg`::
+ Log record's message.
+
+`logger_name`::
+ Logger name.
+
+`class_name`::
+ Name of the class in which the log statement was executed.
+
+`method_name`::
+ Name of the method in which the log statement was executed.
+
+`long_millis`::
+ Logging time (timestamp in milliseconds).
+
+`int_loglevel`::
+ Log level integer value.
+
+`int_threadid`::
+ ID of the thread in which the log statement was executed.
+
You can use the opt:lttng-enable-event(1):--loglevel or
opt:lttng-enable-event(1):--loglevel-only option of the
man:lttng-enable-event(1) command to target a range of JUL log levels
[role="term"]
----
-javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
+$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
----
<<creating-destroying-tracing-sessions,Create a tracing session>>,
[role="term"]
----
-lttng create
-lttng enable-event --log4j jello
-lttng start
+$ lttng create
+$ lttng enable-event --log4j jello
+$ lttng start
----
Run the compiled class:
[role="term"]
----
-java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
+$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
----
<<basic-tracing-session-control,Stop tracing>> and inspect the
[role="term"]
----
-lttng stop
-lttng view
+$ lttng stop
+$ lttng view
----
====
+In the resulting trace, an <<event,event record>> generated by a Java
+application using log4j is named `lttng_log4j:event` and
+has the following fields:
+
+`msg`::
+ Log record's message.
+
+`logger_name`::
+ Logger name.
+
+`class_name`::
+ Name of the class in which the log statement was executed.
+
+`method_name`::
+ Name of the method in which the log statement was executed.
+
+`filename`::
+ Name of the file in which the executed log statement is located.
+
+`line_number`::
+ Line number at which the log statement was executed.
+
+`timestamp`::
+ Logging timestamp.
+
+`int_loglevel`::
+ Log level integer value.
+
+`thread_name`::
+ Name of the Java thread in which the log statement was executed.
+
You can use the opt:lttng-enable-event(1):--loglevel or
opt:lttng-enable-event(1):--loglevel-only option of the
man:lttng-enable-event(1) command to target a range of Apache log4j log levels
[role="term"]
----
-javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
+$ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
----
<<creating-destroying-tracing-sessions,Create a tracing session>>
[role="term"]
----
-lttng create
-lttng enable-event --jul jello
+$ lttng create
+$ lttng enable-event --jul jello
----
<<adding-context,Add the application-specific context fields>> to the
[role="term"]
----
-lttng add-context --jul --type='$app.myRetriever:intCtx'
-lttng add-context --jul --type='$app.myRetriever:strContext'
+$ lttng add-context --jul --type='$app.myRetriever:intCtx'
+$ lttng add-context --jul --type='$app.myRetriever:strContext'
----
<<basic-tracing-session-control,Start tracing>>:
[role="term"]
----
-lttng start
+$ lttng start
----
Run the compiled class:
[role="term"]
----
-java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
+$ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
----
<<basic-tracing-session-control,Stop tracing>> and inspect the
[role="term"]
----
-lttng stop
-lttng view
+$ lttng stop
+$ lttng view
----
====
[role="term"]
----
-lttng create
-lttng enable-event --python my-logger
-lttng start
+$ lttng create
+$ lttng enable-event --python my-logger
+$ lttng start
----
Run the Python script:
[role="term"]
----
-python test.py
+$ python test.py
----
<<basic-tracing-session-control,Stop tracing>> and inspect the recorded
[role="term"]
----
-lttng stop
-lttng view
+$ lttng stop
+$ lttng view
----
====
+In the resulting trace, an <<event,event record>> generated by a Python
+application is named `lttng_python:event` and has the following fields:
+
+`asctime`::
+ Logging time (string).
+
+`msg`::
+ Log record's message.
+
+`logger_name`::
+ Logger name.
+
+`funcName`::
+ Name of the function in which the log statement was executed.
+
+`lineno`::
+ Line number at which the log statement was executed.
+
+`int_loglevel`::
+ Log level integer value.
+
+`thread`::
+ ID of the Python thread in which the log statement was executed.
+
+`threadName`::
+ Name of the Python thread in which the log statement was executed.
+
You can use the opt:lttng-enable-event(1):--loglevel or
opt:lttng-enable-event(1):--loglevel-only option of the
man:lttng-enable-event(1) command to target a range of Python log levels
[role="term"]
----
-echo "Some message, some $variable" > /proc/lttng-logger
+$ echo "Some message, some $variable" > /proc/lttng-logger
----
Any event that the LTTng logger emits is named `lttng_logger` and
belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
other instrumentation points in the kernel tracing domain, **any Unix
user** can <<enabling-disabling-events,create an event rule>> which
-matches its event name, not only the root user or users in the tracing
-group.
+matches its event name, not only the root user or users in the
+<<tracing-group,tracing group>>.
To use the LTTng logger:
[role="term"]
----
-lttng create
-lttng enable-event --kernel lttng_logger
-lttng start
+$ lttng create
+$ lttng enable-event --kernel lttng_logger
+$ lttng start
----
Run the Bash script:
[role="term"]
----
-bash test.bash
+$ bash test.bash
----
<<basic-tracing-session-control,Stop tracing>> and inspect the recorded
[role="term"]
----
-lttng stop
-lttng view
+$ lttng stop
+$ lttng view
----
====
--
[role="term"]
----
-cd $(mktemp -d) &&
+$ cd $(mktemp -d) &&
wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.9.tar.bz2 &&
tar -xf lttng-modules-latest-2.9.tar.bz2 &&
cd lttng-modules-2.9.*
----
--
-. Edit path:{probes/Makefile} and add your new kernel module object
+. Edit path:{probes/KBuild} and add your new kernel module object
next to the existing ones:
+
--
[source,make]
-.path:{probes/Makefile}
+.path:{probes/KBuild}
----
# ...
--
[role="term"]
----
-make KERNELDIR=/path/to/linux
-sudo make modules_install
+$ make KERNELDIR=/path/to/linux
+# make modules_install && depmod -a
----
--
+
====
[role="term"]
----
-sudo lttng-sessiond --extra-kmod-probes=my_subsys,usb
+# lttng-sessiond --extra-kmod-probes=my_subsys,usb
----
====
--
====
[role="term"]
----
-sudo lttng-sessiond --kmod-probes=my_subsys,usb
+# lttng-sessiond --kmod-probes=my_subsys,usb
----
====
--
--
[role="term"]
----
-lsmod | grep lttng_probe_usb
+$ lsmod | grep lttng_probe_usb
----
--
--
[role="term"]
----
-sudo pkill lttng-sessiond
+# pkill lttng-sessiond
----
--
+
--
[role="term"]
----
-lttng-sessiond --daemonize
+$ lttng-sessiond --daemonize
----
--
--
[role="term"]
----
-sudo lttng-sessiond --daemonize
+# lttng-sessiond --daemonize
----
--
--
[role="term"]
----
-lttng create
+$ lttng create
----
--
--
[role="term"]
----
-lttng create my-session
+$ lttng create my-session
----
--
+
--
[role="term"]
----
-lttng create --output=/tmp/some-directory my-session
+$ lttng create my-session --output=/tmp/some-directory
----
--
--
[role="term"]
----
-lttng list
+$ lttng list
----
--
--
[role="term"]
----
-lttng set-session new-session
+$ lttng set-session new-session
----
--
+
--
[role="term"]
----
-lttng destroy
+$ lttng destroy
----
--
====
[role="term"]
----
-lttng list --userspace
+$ lttng list --userspace
----
====
====
[role="term"]
----
-lttng list --kernel --syscall
+$ lttng list --kernel --syscall
----
====
====
[role="term"]
----
-lttng enable-event --kernel sched_switch
+$ lttng enable-event --kernel sched_switch
----
====
====
[role="term"]
----
-lttng enable-event --kernel --syscall open,write,read,close
+$ lttng enable-event --kernel --syscall open,write,read,close
----
====
====
[role="term"]
----
-lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
+$ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
----
[role="term"]
----
-lttng enable-event --kernel --all \
+$ lttng enable-event --kernel --all \
--filter='$ctx.tid == 1988 || $ctx.tid == 1534'
----
[role="term"]
----
-lttng enable-event --jul my_logger \
+$ lttng enable-event --jul my_logger \
--filter='$app.retriever:cur_msg_id > 3'
----
====
[role="term"]
----
-lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
+$ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
----
IMPORTANT: Make sure to always quote the wildcard character when you
====
[role="term"]
----
-lttng enable-event --python my-app.'*' \
+$ lttng enable-event --python my-app.'*' \
--exclude='my-app.module,my-app.hello'
----
====
====
[role="term"]
----
-lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
+$ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
----
====
====
[role="term"]
----
-lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
+$ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
----
====
[role="term"]
----
-lttng enable-event --userspace my_app:my_tracepoint
-lttng enable-event --userspace my_app:my_tracepoint \
+$ lttng enable-event --userspace my_app:my_tracepoint
+$ lttng enable-event --userspace my_app:my_tracepoint \
--loglevel=TRACE_INFO
----
====
[role="term"]
----
-lttng disable-event --python my-logger
+$ lttng disable-event --python my-logger
----
====
====
[role="term"]
----
-lttng disable-event --jul '*'
+$ lttng disable-event --jul '*'
----
====
[role="term"]
----
-lttng disable-event --jul --all-events
+$ lttng disable-event --jul --all-events
----
====
--
[role="term"]
----
-lttng status
+$ lttng status
----
--
+
--
[role="term"]
----
-lttng list my-session
+$ lttng list my-session
----
--
+
--
[role="term"]
----
-lttng start
+$ lttng start
----
--
--
[role="term"]
----
-lttng stop
+$ lttng stop
----
--
+
====
[role="term"]
----
-lttng enable-channel --kernel my-channel
+$ lttng enable-channel --kernel my-channel
----
====
====
[role="term"]
----
-lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
+$ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
--buffers-pid my-channel
----
====
====
[role="term"]
----
-lttng enable-channel --kernel --tracefile-count=8 \
+$ lttng enable-channel --kernel --tracefile-count=8 \
--tracefile-size=4194304 my-channel
----
====
====
[role="term"]
----
-lttng enable-channel --userspace --overwrite my-channel
+$ lttng enable-channel --userspace --overwrite my-channel
----
====
[role="term"]
----
-lttng enable-event --userspace --channel=my-channel app:tp
-lttng enable-event --userspace --channel=other-channel app:tp
+$ lttng enable-event --userspace --channel=my-channel app:tp
+$ lttng enable-event --userspace --channel=other-channel app:tp
----
If both channels are enabled, when a tracepoint named `app:tp` is
====
[role="term"]
----
-lttng disable-channel --kernel my-channel
+$ lttng disable-channel --kernel my-channel
----
====
[role="term"]
----
-lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
+$ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
----
====
[role="term"]
----
-lttng add-context --userspace --type=perf:thread:raw:r0110:test
-lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
+$ lttng add-context --userspace --type=perf:thread:raw:r0110:test
+$ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
----
====
[role="term"]
----
-lttng add-context --kernel --channel=my-channel --type=tid
+$ lttng add-context --kernel --channel=my-channel --type=tid
----
====
[role="term"]
----
-lttng add-context --kernel --channel=my-channel \
+$ lttng add-context --kernel --channel=my-channel \
--type='$app:retriever:cur_msg_id'
----
[role="term"]
----
-lttng track --pid=3,4,7,10,13
+$ lttng track --pid=3,4,7,10,13
----
the whitelist is:
[role="term"]
----
-lttng track --pid=1,15,16
+$ lttng track --pid=1,15,16
----
The result is:
[role="term"]
----
-lttng untrack --pid=3,7,10,13
+$ lttng untrack --pid=3,7,10,13
----
leads to this whitelist:
.PIDs 3, 7, 10, and 13 are removed from the whitelist.
image::track-1-4-15-16.png[]
-LTTng can track all possible PIDs again using the opt:track(1):--all
-option:
+LTTng can track all possible PIDs again using the
+opt:lttng-track(1):--all option:
[role="term"]
----
-lttng track --pid --all
+$ lttng track --pid --all
----
The result is, again:
[role="term"]
----
-lttng untrack --pid --all
+$ lttng untrack --pid --all
----
gives:
[role="term"]
----
-lttng track --pid=6,11
+$ lttng track --pid=6,11
----
Result:
--
[role="term"]
----
-lttng save my-session
+$ lttng save my-session
----
--
+
--
[role="term"]
----
-lttng load my-session
+$ lttng load my-session
----
--
+
--
[role="term"]
----
-lttng-relayd
+$ lttng-relayd
----
--
--
[role="term"]
----
-lttng create my-session --set-url=net://remote-system
+$ lttng create my-session --set-url=net://remote-system
----
--
+
. On the target system, use the man:lttng(1) command-line tool as usual.
When tracing is active, the target's consumer daemon sends sub-buffers
- to the relay daemon running on the remote system intead of flushing
+ to the relay daemon running on the remote system instead of flushing
them to the local file system. The relay daemon writes the received
packets to the local file system.
--
[role="term"]
----
-lttng create --live my-session
+$ lttng create my-session --live
----
--
+
--
[role="term"]
----
-babeltrace --input-format=lttng-live net://localhost/host/hostname/my-session
+$ babeltrace --input-format=lttng-live \
+ net://localhost/host/hostname/my-session
----
--
+
[role="term"]
----
-babeltrace --input-format=lttng-live net://localhost
+$ babeltrace --input-format=lttng-live net://localhost
----
You can start the relay daemon on another system. In this case, you need
--
[role="term"]
----
-lttng create --snapshot my-session
+$ lttng create my-session --snapshot
----
--
+
--
[role="term"]
----
-lttng snapshot record --name=my-first-snapshot
+$ lttng snapshot record --name=my-first-snapshot
----
--
+
[role="term"]
----
-lttng --mi=xml enable-event --kernel --syscall open
+$ lttng --mi=xml enable-event --kernel --syscall open
----
A schema definition (XSD) is
--
[role="term"]
----
-lttng regenerate metadata
+$ lttng regenerate metadata
----
--
--
[role="term"]
----
-lttng regenerate statedump
+$ lttng regenerate statedump
----
--
--
[role="term"]
----
-lttng stop
+$ lttng stop
----
--
--
[role="term"]
----
-lttng snapshot record --name=my-snapshot
+$ lttng snapshot record --name=my-snapshot
----
--
--
[role="term"]
----
-lttng create --shm-path=/path/to/shm
+$ lttng create my-session --shm-path=/path/to/shm
----
--
--
[role="term"]
----
-lttng-crash /path/to/shm
+$ lttng-crash /path/to/shm
----
--
--
[role="term"]
----
-lttng-crash --extract=/path/to/trace /path/to/shm
+$ lttng-crash --extract=/path/to/trace /path/to/shm
----
--
or a Linux kernel KProbe.
+
An event is said to _occur_ at a specific time. Different actions can
-be taken upon the occurance of an event, like record the event's payload
+be taken upon the occurrence of an event, like record the event's payload
to a sub-buffer.
<<channel-overwrite-mode-vs-discard-mode,event loss mode>>::
<<domain,tracing domain>>::
A namespace for event sources.
-tracing group::
+<<tracing-group,tracing group>>::
The Unix group in which a Unix user can be to be allowed to trace the
Linux kernel.
projects / lttng-docs.git / blobdiff