The LTTng Documentation
=======================
Philippe Proulx <pproulx@efficios.com>
-v2.10, 25 July 2017
+v2.10, 22 January 2018
include::../common/copyright.txt[]
only at the end, when you <<enabling-disabling-events,create an event
rule>>, in both the instrumentation point name and the literal
strings of
- link:http://lttng.org/man/1/lttng-enable-event/v{revision}/#doc-filter-syntax[filter expressions]:
+ link:/man/1/lttng-enable-event/v{revision}/#doc-filter-syntax[filter expressions]:
+
--
[role="term"]
<<channel,channel>> buffer usage conditions are available.
Documentation is available in the
https://github.com/lttng/lttng-tools/tree/stable-{revision}/include/lttng[`liblttng-ctl`
- header files].
+ header files] and in
+ <<notif-trigger-api,Get notified when a channel's buffer usage is too
+ high or too low>>.
** You can now embed the whole textual LTTng-tools man pages into the
executables at build time with the `--enable-embedded-help`
[role="term"]
----
$ lttng create
-$ lttng enable-channel --userspace --blocking-timeout=-1 blocking-channel
+$ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
$ lttng enable-event --userspace --channel=blocking-channel --all
$ lttng start
$ LTTNG_UST_ALLOW_BLOCKING=1 my-app
applications.
[role="growable"]
-.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 25 July 2017.
+.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 22 January 2018.
|====
|Distribution |Available in releases |Alternatives
|https://www.ubuntu.com/[Ubuntu]
|Ubuntu{nbsp}14.04 _Trusty Tahr_ and Ubuntu{nbsp}16.04 _Xenial Xerus_:
<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
-|link:/docs/v2.9#doc-ubuntu[LTTng{nbsp}2.9 for Ubuntu{nbsp}17.04 _Zesty Zapus_].
+|link:/docs/v2.9#doc-ubuntu[LTTng{nbsp}2.9 for Ubuntu{nbsp}17.04 _Zesty Zapus_ and Ubuntu{nbsp}17.10 _Artful Aardvark_].
<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
other Ubuntu releases.
|https://getfedora.org/[Fedora]
-|_Not available_
-|link:/docs/v2.9#doc-fedora[LTTng{nbsp}2.9 for Fedora 26].
+|<<fedora,Fedora{nbsp}27>>.
+|link:/docs/v2.9#doc-fedora[LTTng{nbsp}2.9 for Fedora{nbsp}26].
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
+<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
+other Fedora releases.
|https://www.debian.org/[Debian]
-|_Not available_
-|link:/docs/v2.9#doc-debian[LTTng{nbsp}2.9 for Debian "stretch"
-(stable), Debian "buster" (testing), and Debian "sid" (unstable)].
+|<<debian,Debian "buster" (testing) and Debian "sid" (unstable)>>.
+|link:/docs/v2.9#doc-debian[LTTng{nbsp}2.9 for Debian "stretch" (stable)].
<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
|https://www.archlinux.org/[Arch Linux]
-|_Not available_
-|link:/docs/v2.9#doc-arch-linux[LTTng{nbsp}2.9 in the latest AUR packages].
+|<<arch-linux,Current Arch Linux build>>.
+|<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
|https://alpinelinux.org/[Alpine Linux]
-|_Not available_
-|link:/docs/v2.9#doc-alpine-linux[LTTng{nbsp}2.9 for Alpine Linux "edge"].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
+|<<alpine-linux,Alpine Linux{nbsp}3.7 and Alpine Linux{nbsp}"edge">>.
+|<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
|https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
|See http://packages.efficios.com/[EfficiOS Enterprise Packages].
|https://buildroot.org/[Buildroot]
|_Not available_
-|link:/docs/v2.9#doc-buildroot[LTTng{nbsp}2.9 for Buildroot{nbsp}2017.02 and
-Buildroot{nbsp}2017.05].
+|link:/docs/v2.9#doc-buildroot[LTTng{nbsp}2.9 for Buildroot{nbsp}2017.02,
+Buildroot{nbsp}2017.05, Buildroot{nbsp}2017.08, and Buildroot{nbsp}2017.11].
<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
|http://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
https://www.yoctoproject.org/[Yocto]
|_Not available_
-|link:/docs/v2.9#doc-oe-yocto[LTTng{nbsp}2.9 for Yocto Project{nbsp}2.3 _Pyro_]
+|link:/docs/v2.9#doc-oe-yocto[LTTng{nbsp}2.9 for Yocto Project{nbsp}2.3 _Pyro_
+and Yocto Project{nbsp}2.4 _Rocko_]
(`openembedded-core` layer).
<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
--
+[[fedora]]
+=== Fedora
+
+To install LTTng{nbsp}{revision} on Fedora{nbsp}27:
+
+. Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
+ packages:
++
+--
+[role="term"]
+----
+# yum install lttng-tools
+# yum install lttng-ust
+----
+--
+
+. Download, build, and install the latest LTTng-modules{nbsp}{revision}:
++
+--
+[role="term"]
+----
+$ cd $(mktemp -d) &&
+wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.10.tar.bz2 &&
+tar -xf lttng-modules-latest-2.10.tar.bz2 &&
+cd lttng-modules-2.10.* &&
+make &&
+sudo make modules_install &&
+sudo depmod -a
+----
+--
+
+[IMPORTANT]
+.Java and Python application instrumentation and tracing
+====
+If you need to instrument and trace <<java-application,Java
+applications>> on Fedora, you need to build and install
+LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
+the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
+`--enable-java-agent-all` options to the `configure` script, depending
+on which Java logging framework you use.
+
+If you need to instrument and trace <<python-application,Python
+applications>> on Fedora, you need to build and install
+LTTng-UST{nbsp}{revision} from source and pass the
+`--enable-python-agent` option to the `configure` script.
+====
+
+
+[[debian]]
+=== Debian
+
+To install LTTng{nbsp}{revision} on Debian "buster" (testing)
+or Debian "sid" (unstable):
+
+. Install the main LTTng{nbsp}{revision} packages:
++
+--
+[role="term"]
+----
+# apt-get install lttng-modules-dkms
+# apt-get install liblttng-ust-dev
+# apt-get install lttng-tools
+----
+--
+
+. **If you need to instrument and trace <<java-application,Java
+ applications>>**, install the LTTng-UST Java agent:
++
+--
+[role="term"]
+----
+# apt-get install liblttng-ust-agent-java
+----
+--
+
+. **If you need to instrument and trace <<python-application,Python
+ applications>>**, install the LTTng-UST Python agent:
++
+--
+[role="term"]
+----
+# apt-get install python3-lttngust
+----
+--
+
+
+[[arch-linux]]
+=== Arch Linux
+
+LTTng-UST{nbsp}{revision} is available in Arch Linux's _Community_
+repository, while LTTng-tools{nbsp}{revision} and
+LTTng-modules{nbsp}{revision} are available in the
+https://aur.archlinux.org/[AUR].
+
+To install LTTng{nbsp}{revision} on Arch Linux, using
+https://github.com/rmarquis/pacaur[pacaur] for the AUR packages:
+
+. Install the main LTTng{nbsp}{revision} packages:
++
+--
+[role="term"]
+----
+# pacman -Sy lttng-ust
+$ pacaur -Sy lttng-tools
+$ pacaur -Sy lttng-modules
+----
+--
+
+. **If you need to instrument and trace <<python-application,Python
+ applications>>**, install the LTTng-UST Python agent:
++
+--
+[role="term"]
+----
+# pacman -Sy python-lttngust
+# pacman -Sy python2-lttngust
+----
+--
+
+
+[[alpine-linux]]
+=== Alpine Linux
+
+To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
+Alpine Linux{nbsp}3.7 or Alpine Linux{nbsp}"edge":
+
+. **If you're installing for Alpine Linux{nbsp}"edge"**, make sure your
+ system is https://wiki.alpinelinux.org/wiki/Edge[configured for
+ "edge"].
+. **If you're installing for Alpine Linux{nbsp}"edge"**, enable the _testing_
+ repository by uncommenting the corresponding line in
+ path:{/etc/apk/repositories}.
+. Add the LTTng packages:
++
+--
+[role="term"]
+----
+# apk add lttng-tools
+# apk add lttng-ust-dev
+----
+--
+
+To install LTTng-modules{nbsp}{revision} (Linux kernel tracing support)
+on Alpine Linux{nbsp}3.7 or Alpine Linux{nbsp}"edge":
+
+. Add the vanilla Linux kernel:
++
+--
+[role="term"]
+----
+# apk add linux-vanilla linux-vanilla-dev
+----
+--
+
+. Reboot with the vanilla Linux kernel.
+. Download, build, and install the latest LTTng-modules{nbsp}{revision}:
++
+--
+[role="term"]
+----
+$ cd $(mktemp -d) &&
+wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.10.tar.bz2 &&
+tar -xf lttng-modules-latest-2.10.tar.bz2 &&
+cd lttng-modules-2.10.* &&
+make &&
+sudo make modules_install &&
+sudo depmod -a
+----
+--
+
+
[[enterprise-distributions]]
=== RHEL, SUSE, and other enterprise distributions
available event loss modes are:
Discard mode::
- Drop the newest event records until a the tracer
- releases a sub-buffer.
+ Drop the newest event records until a the tracer releases a
+ sub-buffer.
++
+This is the only available mode when you specify a
+<<opt-blocking-timeout,blocking timeout>>.
Overwrite mode::
Clear the sub-buffer containing the oldest event records and start
Which mechanism you should choose depends on your context: prioritize
the newest or the oldest event records in the ring buffer?
-Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
+Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
as soon as a there's no space left for a new event record, whereas in
discard mode, the tracer only discards the event record that doesn't
fit.
-In discard mode, LTTng increments a count of lost event records when
-an event record is lost and saves this count to the trace. In
-overwrite mode, LTTng keeps no information when it overwrites a
-sub-buffer before consuming it.
+In discard mode, LTTng increments a count of lost event records when an
+event record is lost and saves this count to the trace. In overwrite
+mode, since LTTng 2.8, LTTng increments a count of lost sub-buffers when
+a sub-buffer is lost and saves this count to the trace. In this mode,
+the exact number of lost event records in those lost sub-buffers is not
+saved to the trace. Trace analyses can use the trace's saved discarded
+event record and sub-buffer counts to decide whether or not to perform
+the analyses even if trace data is known to be missing.
There are a few ways to decrease your probability of losing event
records.
* **LTTng-tools**: Libraries and command-line interface to
control tracing sessions.
** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
-** <<lttng-consumerd,Consumer daemon>> (man:lttng-consumerd(8)).
+** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
.The consumer daemon.
image::plumbing-consumerd.png[]
-The _consumer daemon_, man:lttng-consumerd(8), is a daemon which shares
+The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
ring buffers with user applications or with the LTTng kernel modules to
collect trace data and send it to some location (on disk or to a
<<lttng-relayd,relay daemon>> over the network). The consumer daemon
0 (default)::
Never block (non-blocking mode).
--1::
+`inf`::
Block forever until space is available in a sub-buffer to record
the event.
----
====
-.[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout:
+.[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
====
<<creating-destroying-tracing-sessions,Create a tracing-session>>,
create the channel, <<enabling-disabling-events,create an event rule>>,
[role="term"]
----
$ lttng create
-$ lttng enable-channel --userspace --blocking-timeout=-1 blocking-channel
+$ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
$ lttng enable-event --userspace --channel=blocking-channel --all
$ lttng start
----
.PIDs 3, 7, 10, and 13 are removed from the whitelist.
image::track-1-4-15-16.png[]
-LTTng can track all possible PIDs again using the opt:track(1):--all
-option:
+LTTng can track all possible PIDs again using the
+opt:lttng-track(1):--all option:
[role="term"]
----
--
+[role="since-2.10"]
+[[notif-trigger-api]]
+=== Get notified when a channel's buffer usage is too high or too low
+
+With LTTng's $$C/C++$$ notification and trigger API, your user
+application can get notified when the buffer usage of one or more
+<<channel,channels>> becomes too low or too high. You can use this API
+and enable or disable <<event,event rules>> during tracing to avoid
+<<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
+
+.Have a user application get notified when an LTTng channel's buffer usage is too high.
+====
+In this example, we create and build an application which gets notified
+when the buffer usage of a specific LTTng channel is higher than
+75{nbsp}%. We only print that it is the case in the example, but we
+could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
+disable event rules when this happens.
+
+. Create the application's C source file:
++
+--
+[source,c]
+.path:{notif-app.c}
+----
+#include <stdio.h>
+#include <assert.h>
+#include <lttng/domain.h>
+#include <lttng/action/action.h>
+#include <lttng/action/notify.h>
+#include <lttng/condition/condition.h>
+#include <lttng/condition/buffer-usage.h>
+#include <lttng/condition/evaluation.h>
+#include <lttng/notification/channel.h>
+#include <lttng/notification/notification.h>
+#include <lttng/trigger/trigger.h>
+#include <lttng/endpoint.h>
+
+int main(int argc, char *argv[])
+{
+ int exit_status = 0;
+ struct lttng_notification_channel *notification_channel;
+ struct lttng_condition *condition;
+ struct lttng_action *action;
+ struct lttng_trigger *trigger;
+ const char *tracing_session_name;
+ const char *channel_name;
+
+ assert(argc >= 3);
+ tracing_session_name = argv[1];
+ channel_name = argv[2];
+
+ /*
+ * Create a notification channel. A notification channel
+ * connects the user application to the LTTng session daemon.
+ * This notification channel can be used to listen to various
+ * types of notifications.
+ */
+ notification_channel = lttng_notification_channel_create(
+ lttng_session_daemon_notification_endpoint);
+
+ /*
+ * Create a "high buffer usage" condition. In this case, the
+ * condition is reached when the buffer usage is greater than or
+ * equal to 75 %. We create the condition for a specific tracing
+ * session name, channel name, and for the user space tracing
+ * domain.
+ *
+ * The "low buffer usage" condition type also exists.
+ */
+ condition = lttng_condition_buffer_usage_high_create();
+ lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
+ lttng_condition_buffer_usage_set_session_name(
+ condition, tracing_session_name);
+ lttng_condition_buffer_usage_set_channel_name(condition,
+ channel_name);
+ lttng_condition_buffer_usage_set_domain_type(condition,
+ LTTNG_DOMAIN_UST);
+
+ /*
+ * Create an action (get a notification) to take when the
+ * condition created above is reached.
+ */
+ action = lttng_action_notify_create();
+
+ /*
+ * Create a trigger. A trigger associates a condition to an
+ * action: the action is executed when the condition is reached.
+ */
+ trigger = lttng_trigger_create(condition, action);
+
+ /* Register the trigger to LTTng. */
+ lttng_register_trigger(trigger);
+
+ /*
+ * Now that we have registered a trigger, a notification will be
+ * emitted everytime its condition is met. To receive this
+ * notification, we must subscribe to notifications that match
+ * the same condition.
+ */
+ lttng_notification_channel_subscribe(notification_channel,
+ condition);
+
+ /*
+ * Notification loop. You can put this in a dedicated thread to
+ * avoid blocking the main thread.
+ */
+ for (;;) {
+ struct lttng_notification *notification;
+ enum lttng_notification_channel_status status;
+ const struct lttng_evaluation *notification_evaluation;
+ const struct lttng_condition *notification_condition;
+ double buffer_usage;
+
+ /* Receive the next notification. */
+ status = lttng_notification_channel_get_next_notification(
+ notification_channel, ¬ification);
+
+ switch (status) {
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
+ break;
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
+ /*
+ * The session daemon can drop notifications if
+ * a monitoring application is not consuming the
+ * notifications fast enough.
+ */
+ continue;
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
+ /*
+ * The notification channel has been closed by the
+ * session daemon. This is typically caused by a session
+ * daemon shutting down.
+ */
+ goto end;
+ default:
+ /* Unhandled conditions or errors. */
+ exit_status = 1;
+ goto end;
+ }
+
+ /*
+ * A notification provides, amongst other things:
+ *
+ * * The condition that caused this notification to be
+ * emitted.
+ * * The condition evaluation, which provides more
+ * specific information on the evaluation of the
+ * condition.
+ *
+ * The condition evaluation provides the buffer usage
+ * value at the moment the condition was reached.
+ */
+ notification_condition = lttng_notification_get_condition(
+ notification);
+ notification_evaluation = lttng_notification_get_evaluation(
+ notification);
+
+ /* We're subscribed to only one condition. */
+ assert(lttng_condition_get_type(notification_condition) ==
+ LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
+
+ /*
+ * Get the exact sampled buffer usage from the
+ * condition evaluation.
+ */
+ lttng_evaluation_buffer_usage_get_usage_ratio(
+ notification_evaluation, &buffer_usage);
+
+ /*
+ * At this point, instead of printing a message, we
+ * could do something to reduce the channel's buffer
+ * usage, like disable specific events.
+ */
+ printf("Buffer usage is %f %% in tracing session \"%s\", "
+ "user space channel \"%s\".\n", buffer_usage * 100,
+ tracing_session_name, channel_name);
+ lttng_notification_destroy(notification);
+ }
+
+end:
+ lttng_action_destroy(action);
+ lttng_condition_destroy(condition);
+ lttng_trigger_destroy(trigger);
+ lttng_notification_channel_destroy(notification_channel);
+ return exit_status;
+}
+----
+--
+
+. Build the `notif-app` application, linking it to `liblttng-ctl`:
++
+--
+[role="term"]
+----
+$ gcc -o notif-app notif-app.c -llttng-ctl
+----
+--
+
+. <<creating-destroying-tracing-sessions,Create a tracing session>>,
+ <<enabling-disabling-events,create an event rule>> matching all the
+ user space tracepoints, and
+ <<basic-tracing-session-control,start tracing>>:
++
+--
+[role="term"]
+----
+$ lttng create my-session
+$ lttng enable-event --userspace --all
+$ lttng start
+----
+--
++
+If you create the channel manually with the man:lttng-enable-channel(1)
+command, you can control how frequently are the current values of the
+channel's properties sampled to evaluate user conditions with the
+opt:lttng-enable-channel(1):--monitor-timer option.
+
+. Run the `notif-app` application. This program accepts the
+ <<tracing-session,tracing session>> name and the user space channel
+ name as its two first arguments. The channel which LTTng automatically
+ creates with the man:lttng-enable-event(1) command above is named
+ `channel0`:
++
+--
+[role="term"]
+----
+$ ./notif-app my-session channel0
+----
+--
+
+. In another terminal, run an application with a very high event
+ throughput so that the 75{nbsp}% buffer usage condition is reached.
++
+In the first terminal, the application should print lines like this:
++
+----
+Buffer usage is 81.45197 % in tracing session "my-session", user space
+channel "channel0".
+----
++
+If you don't see anything, try modifying the condition in
+path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
+(step 2) and running it again (step 4).
+====
+
+
[[reference]]
== Reference