The LTTng Documentation
=======================
Philippe Proulx <pproulx@efficios.com>
-v2.10, 25 July 2017
+v2.10, 25 February 2021
include::../common/copyright.txt[]
+include::../common/warning-not-maintained.txt[]
+
+
include::../common/welcome.txt[]
only at the end, when you <<enabling-disabling-events,create an event
rule>>, in both the instrumentation point name and the literal
strings of
- link:http://lttng.org/man/1/lttng-enable-event/v{revision}/#doc-filter-syntax[filter expressions]:
+ link:/man/1/lttng-enable-event/v{revision}/#doc-filter-syntax[filter expressions]:
+
--
[role="term"]
<<channel,channel>> buffer usage conditions are available.
Documentation is available in the
https://github.com/lttng/lttng-tools/tree/stable-{revision}/include/lttng[`liblttng-ctl`
- header files].
+ header files] and in
+ <<notif-trigger-api,Get notified when a channel's buffer usage is too
+ high or too low>>.
** You can now embed the whole textual LTTng-tools man pages into the
executables at build time with the `--enable-embedded-help`
[role="term"]
----
$ lttng create
-$ lttng enable-channel --userspace --blocking-timeout=-1 blocking-channel
+$ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
$ lttng enable-event --userspace --channel=blocking-channel --all
$ lttng start
$ LTTNG_UST_ALLOW_BLOCKING=1 my-app
[[installing-lttng]]
== Installation
+include::../common/warning-no-installation.txt[]
+
**LTTng** is a set of software <<plumbing,components>> which interact to
<<instrumenting,instrument>> the Linux kernel and user applications, and
to <<controlling-tracing,control tracing>> (start and stop
trace user applications.
Most distributions mark the LTTng-modules and LTTng-UST packages as
-optional when installing LTTng-tools (which is always required). In the
-following sections, we always provide the steps to install all three,
-but note that:
+optional when installing LTTng-tools (which is always required). Note
+that:
* You only need to install LTTng-modules if you intend to trace the
Linux kernel.
* You only need to install LTTng-UST if you intend to trace user
applications.
-[role="growable"]
-.Availability of LTTng{nbsp}{revision} for major Linux distributions as of 25 July 2017.
-|====
-|Distribution |Available in releases |Alternatives
-
-|https://www.ubuntu.com/[Ubuntu]
-|Ubuntu{nbsp}14.04 _Trusty Tahr_ and Ubuntu{nbsp}16.04 _Xenial Xerus_:
-<<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
-|link:/docs/v2.9#doc-ubuntu[LTTng{nbsp}2.9 for Ubuntu{nbsp}17.04 _Zesty Zapus_].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>> for
-other Ubuntu releases.
-
-|https://getfedora.org/[Fedora]
-|_Not available_
-|link:/docs/v2.9#doc-fedora[LTTng{nbsp}2.9 for Fedora 26].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
-
-|https://www.debian.org/[Debian]
-|_Not available_
-|link:/docs/v2.9#doc-debian[LTTng{nbsp}2.9 for Debian "stretch"
-(stable), Debian "buster" (testing), and Debian "sid" (unstable)].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
-
-|https://www.archlinux.org/[Arch Linux]
-|_Not available_
-|link:/docs/v2.9#doc-arch-linux[LTTng{nbsp}2.9 in the latest AUR packages].
-
-|https://alpinelinux.org/[Alpine Linux]
-|_Not available_
-|link:/docs/v2.9#doc-alpine-linux[LTTng{nbsp}2.9 for Alpine Linux "edge"].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
-
-|https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
-|See http://packages.efficios.com/[EfficiOS Enterprise Packages].
-|
-
-|https://buildroot.org/[Buildroot]
-|_Not available_
-|link:/docs/v2.9#doc-buildroot[LTTng{nbsp}2.9 for Buildroot{nbsp}2017.02 and
-Buildroot{nbsp}2017.05].
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
-
-|http://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
-https://www.yoctoproject.org/[Yocto]
-|_Not available_
-|link:/docs/v2.9#doc-oe-yocto[LTTng{nbsp}2.9 for Yocto Project{nbsp}2.3 _Pyro_]
-(`openembedded-core` layer).
-
-<<building-from-source,Build LTTng{nbsp}{revision} from source>>.
-|====
-
-
-[[ubuntu]]
-=== [[ubuntu-official-repositories]]Ubuntu
-
-[[ubuntu-ppa]]
-==== noch:{LTTng} Stable {revision} PPA
-
-The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
-Stable{nbsp}{revision} PPA] offers the latest stable
-LTTng{nbsp}{revision} packages for:
-
-* Ubuntu{nbsp}14.04 _Trusty Tahr_
-* Ubuntu{nbsp}16.04 _Xenial Xerus_
-
-To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision} PPA:
-
-. Add the LTTng Stable{nbsp}{revision} PPA repository and update the
- list of packages:
-+
---
-[role="term"]
-----
-# apt-add-repository ppa:lttng/stable-2.10
-# apt-get update
-----
---
-
-. Install the main LTTng{nbsp}{revision} packages:
-+
---
-[role="term"]
-----
-# apt-get install lttng-tools
-# apt-get install lttng-modules-dkms
-# apt-get install liblttng-ust-dev
-----
---
-
-. **If you need to instrument and trace
- <<java-application,Java applications>>**, install the LTTng-UST
- Java agent:
-+
---
-[role="term"]
-----
-# apt-get install liblttng-ust-agent-java
-----
---
-
-. **If you need to instrument and trace
- <<python-application,Python{nbsp}3 applications>>**, install the
- LTTng-UST Python agent:
-+
---
-[role="term"]
-----
-# apt-get install python3-lttngust
-----
---
-
-
-[[enterprise-distributions]]
-=== RHEL, SUSE, and other enterprise distributions
-
-To install LTTng on enterprise Linux distributions, such as Red Hat
-Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SUSE), please
-see http://packages.efficios.com/[EfficiOS Enterprise Packages].
-
[[building-from-source]]
=== Build from source
. Do some operation on your system for a few seconds. For example,
load a website, or list the files of a directory.
-. <<basic-tracing-session-control,Stop tracing>> and destroy the
+. <<creating-destroying-tracing-sessions,Destroy>> the current
tracing session:
+
--
[role="term"]
----
-# lttng stop
# lttng destroy
----
--
+
The man:lttng-destroy(1) command does not destroy the trace data; it
only destroys the state of the tracing session.
++
+The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
+implicitly (see <<basic-tracing-session-control,Start and stop a tracing
+session>>). You need to stop tracing to make LTTng flush the remaining
+trace data and make the trace readable.
. For the sake of this example, make the recorded trace accessible to
the non-root users:
. Go back to the running `hello` application and press Enter. The
program executes all `tracepoint()` instrumentation points and exits.
-. <<basic-tracing-session-control,Stop tracing>> and destroy the
+. <<creating-destroying-tracing-sessions,Destroy>> the current
tracing session:
+
--
[role="term"]
----
-$ lttng stop
$ lttng destroy
----
--
+
The man:lttng-destroy(1) command does not destroy the trace data; it
only destroys the state of the tracing session.
++
+The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
+implicitly (see <<basic-tracing-session-control,Start and stop a tracing
+session>>). You need to stop tracing to make LTTng flush the remaining
+trace data and make the trace readable.
By default, LTTng saves the traces in
+$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
_tracing session mode_ dictates where to send it. The following modes
are available in LTTng{nbsp}{revision}:
-Local mode::
+[[local-mode]]Local mode::
LTTng writes the traces to the file system of the machine being traced
(target system).
-Network streaming mode::
+[[net-streaming-mode]]Network streaming mode::
LTTng sends the traces over the network to a
<<lttng-relayd,relay daemon>> running on a remote system.
or to send it over the network to a <<lttng-relayd,relay daemon>>
running on a remote system.
-Live mode::
+[[live-mode]]Live mode::
This mode is similar to the network streaming mode, but a live
trace viewer can connect to the distant relay daemon to
<<lttng-live,view event records as LTTng generates them>> by
available event loss modes are:
Discard mode::
- Drop the newest event records until a the tracer
- releases a sub-buffer.
+ Drop the newest event records until a the tracer releases a
+ sub-buffer.
++
+This is the only available mode when you specify a
+<<opt-blocking-timeout,blocking timeout>>.
Overwrite mode::
Clear the sub-buffer containing the oldest event records and start
Which mechanism you should choose depends on your context: prioritize
the newest or the oldest event records in the ring buffer?
-Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
+Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
as soon as a there's no space left for a new event record, whereas in
discard mode, the tracer only discards the event record that doesn't
fit.
-In discard mode, LTTng increments a count of lost event records when
-an event record is lost and saves this count to the trace. In
-overwrite mode, LTTng keeps no information when it overwrites a
-sub-buffer before consuming it.
+In discard mode, LTTng increments a count of lost event records when an
+event record is lost and saves this count to the trace. Since
+LTTng{nbsp}2.8, in overwrite mode, LTTng writes to a given sub-buffer
+its sequence number within its data stream. With a <<local-mode,local>>,
+<<net-streaming-mode,network streaming>>, or <<live-mode,live>>
+<<tracing-session,tracing session>>, a trace reader can use such
+sequence numbers to report lost packets. In overwrite mode, LTTng
+doesn't write to the trace the exact number of lost event records in
+those lost sub-buffers.
+
+Trace analyses can use saved discarded event record and sub-buffer
+(packet) counts of the trace to decide whether or not to perform the
+analyses even if trace data is known to be missing.
There are a few ways to decrease your probability of losing event
records.
<<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
-how you can fine-une the sub-buffer count and size of a channel to
+how you can fine-tune the sub-buffer count and size of a channel to
virtually stop losing event records, though at the cost of greater
memory usage.
* **LTTng-tools**: Libraries and command-line interface to
control tracing sessions.
** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
-** <<lttng-consumerd,Consumer daemon>> (man:lttng-consumerd(8)).
+** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
.The consumer daemon.
image::plumbing-consumerd.png[]
-The _consumer daemon_, man:lttng-consumerd(8), is a daemon which shares
+The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
ring buffers with user applications or with the LTTng kernel modules to
collect trace data and send it to some location (on disk or to a
<<lttng-relayd,relay daemon>> over the network). The consumer daemon
[[prebuilt-ust-helpers]]
=== Prebuilt user space tracing helpers
-The LTTng-UST package provides a few helpers in the form or preloadable
+The LTTng-UST package provides a few helpers in the form of preloadable
shared objects which automatically instrument system functions and
calls.
.LTTng-UST Java agent imported by a Java application.
image::java-app.png[]
-Note that the methods described below are new in LTTng{nbsp}{revision}.
+Note that the methods described below are new in LTTng{nbsp}2.8.
Previous LTTng versions use another technique.
NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
----
--
+The man:lttng-destroy(1) command also runs the man:lttng-stop(1)
+command implicitly (see <<basic-tracing-session-control,Start and stop a
+tracing session>>). You need to stop tracing to make LTTng flush the
+remaining trace data and make the trace readable.
+
[[list-instrumentation-points]]
=== List the available instrumentation points
function, this is a custom name given to the event rule. With the
JUL, log4j, and Python domains, this is a logger name.
-With a tracepoint, logger, or system call name, the last character
-can be `*` to match anything that remains.
+With a tracepoint, logger, or system call name, you can use the special
+`*` globbing character to match anything (for example, `sched_*`,
+`my_comp*:*msg_*`).
|All.
man:lttng-start(1), warnings are printed when you run the
man:lttng-stop(1) command.
+IMPORTANT: You need to stop tracing to make LTTng flush the remaining
+trace data and make the trace readable. Note that the
+man:lttng-destroy(1) command (see
+<<creating-destroying-tracing-sessions,Create and destroy a tracing
+session>>) also runs the man:lttng-stop(1) command implicitly.
+
[[enabling-disabling-channels]]
=== Create a channel
0 (default)::
Never block (non-blocking mode).
--1::
+`inf`::
Block forever until space is available in a sub-buffer to record
the event.
----
====
-.[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout:
+.[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
====
<<creating-destroying-tracing-sessions,Create a tracing-session>>,
create the channel, <<enabling-disabling-events,create an event rule>>,
[role="term"]
----
$ lttng create
-$ lttng enable-channel --userspace --blocking-timeout=-1 blocking-channel
+$ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
$ lttng enable-event --userspace --channel=blocking-channel --all
$ lttng start
----
.PIDs 3, 7, 10, and 13 are removed from the whitelist.
image::track-1-4-15-16.png[]
-LTTng can track all possible PIDs again using the opt:track(1):--all
-option:
+LTTng can track all possible PIDs again using the
+opt:lttng-track(1):--all option:
[role="term"]
----
--
+[role="since-2.10"]
+[[notif-trigger-api]]
+=== Get notified when a channel's buffer usage is too high or too low
+
+With LTTng's $$C/C++$$ notification and trigger API, your user
+application can get notified when the buffer usage of one or more
+<<channel,channels>> becomes too low or too high. You can use this API
+and enable or disable <<event,event rules>> during tracing to avoid
+<<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
+
+.Have a user application get notified when an LTTng channel's buffer usage is too high.
+====
+In this example, we create and build an application which gets notified
+when the buffer usage of a specific LTTng channel is higher than
+75{nbsp}%. We only print that it is the case in the example, but we
+could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
+disable event rules when this happens.
+
+. Create the application's C source file:
++
+--
+[source,c]
+.path:{notif-app.c}
+----
+#include <stdio.h>
+#include <assert.h>
+#include <lttng/domain.h>
+#include <lttng/action/action.h>
+#include <lttng/action/notify.h>
+#include <lttng/condition/condition.h>
+#include <lttng/condition/buffer-usage.h>
+#include <lttng/condition/evaluation.h>
+#include <lttng/notification/channel.h>
+#include <lttng/notification/notification.h>
+#include <lttng/trigger/trigger.h>
+#include <lttng/endpoint.h>
+
+int main(int argc, char *argv[])
+{
+ int exit_status = 0;
+ struct lttng_notification_channel *notification_channel;
+ struct lttng_condition *condition;
+ struct lttng_action *action;
+ struct lttng_trigger *trigger;
+ const char *tracing_session_name;
+ const char *channel_name;
+
+ assert(argc >= 3);
+ tracing_session_name = argv[1];
+ channel_name = argv[2];
+
+ /*
+ * Create a notification channel. A notification channel
+ * connects the user application to the LTTng session daemon.
+ * This notification channel can be used to listen to various
+ * types of notifications.
+ */
+ notification_channel = lttng_notification_channel_create(
+ lttng_session_daemon_notification_endpoint);
+
+ /*
+ * Create a "high buffer usage" condition. In this case, the
+ * condition is reached when the buffer usage is greater than or
+ * equal to 75 %. We create the condition for a specific tracing
+ * session name, channel name, and for the user space tracing
+ * domain.
+ *
+ * The "low buffer usage" condition type also exists.
+ */
+ condition = lttng_condition_buffer_usage_high_create();
+ lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
+ lttng_condition_buffer_usage_set_session_name(
+ condition, tracing_session_name);
+ lttng_condition_buffer_usage_set_channel_name(condition,
+ channel_name);
+ lttng_condition_buffer_usage_set_domain_type(condition,
+ LTTNG_DOMAIN_UST);
+
+ /*
+ * Create an action (get a notification) to take when the
+ * condition created above is reached.
+ */
+ action = lttng_action_notify_create();
+
+ /*
+ * Create a trigger. A trigger associates a condition to an
+ * action: the action is executed when the condition is reached.
+ */
+ trigger = lttng_trigger_create(condition, action);
+
+ /* Register the trigger to LTTng. */
+ lttng_register_trigger(trigger);
+
+ /*
+ * Now that we have registered a trigger, a notification will be
+ * emitted everytime its condition is met. To receive this
+ * notification, we must subscribe to notifications that match
+ * the same condition.
+ */
+ lttng_notification_channel_subscribe(notification_channel,
+ condition);
+
+ /*
+ * Notification loop. You can put this in a dedicated thread to
+ * avoid blocking the main thread.
+ */
+ for (;;) {
+ struct lttng_notification *notification;
+ enum lttng_notification_channel_status status;
+ const struct lttng_evaluation *notification_evaluation;
+ const struct lttng_condition *notification_condition;
+ double buffer_usage;
+
+ /* Receive the next notification. */
+ status = lttng_notification_channel_get_next_notification(
+ notification_channel, ¬ification);
+
+ switch (status) {
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
+ break;
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
+ /*
+ * The session daemon can drop notifications if
+ * a monitoring application is not consuming the
+ * notifications fast enough.
+ */
+ continue;
+ case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
+ /*
+ * The notification channel has been closed by the
+ * session daemon. This is typically caused by a session
+ * daemon shutting down.
+ */
+ goto end;
+ default:
+ /* Unhandled conditions or errors. */
+ exit_status = 1;
+ goto end;
+ }
+
+ /*
+ * A notification provides, amongst other things:
+ *
+ * * The condition that caused this notification to be
+ * emitted.
+ * * The condition evaluation, which provides more
+ * specific information on the evaluation of the
+ * condition.
+ *
+ * The condition evaluation provides the buffer usage
+ * value at the moment the condition was reached.
+ */
+ notification_condition = lttng_notification_get_condition(
+ notification);
+ notification_evaluation = lttng_notification_get_evaluation(
+ notification);
+
+ /* We're subscribed to only one condition. */
+ assert(lttng_condition_get_type(notification_condition) ==
+ LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
+
+ /*
+ * Get the exact sampled buffer usage from the
+ * condition evaluation.
+ */
+ lttng_evaluation_buffer_usage_get_usage_ratio(
+ notification_evaluation, &buffer_usage);
+
+ /*
+ * At this point, instead of printing a message, we
+ * could do something to reduce the channel's buffer
+ * usage, like disable specific events.
+ */
+ printf("Buffer usage is %f %% in tracing session \"%s\", "
+ "user space channel \"%s\".\n", buffer_usage * 100,
+ tracing_session_name, channel_name);
+ lttng_notification_destroy(notification);
+ }
+
+end:
+ lttng_action_destroy(action);
+ lttng_condition_destroy(condition);
+ lttng_trigger_destroy(trigger);
+ lttng_notification_channel_destroy(notification_channel);
+ return exit_status;
+}
+----
+--
+
+. Build the `notif-app` application, linking it to `liblttng-ctl`:
++
+--
+[role="term"]
+----
+$ gcc -o notif-app notif-app.c -llttng-ctl
+----
+--
+
+. <<creating-destroying-tracing-sessions,Create a tracing session>>,
+ <<enabling-disabling-events,create an event rule>> matching all the
+ user space tracepoints, and
+ <<basic-tracing-session-control,start tracing>>:
++
+--
+[role="term"]
+----
+$ lttng create my-session
+$ lttng enable-event --userspace --all
+$ lttng start
+----
+--
++
+If you create the channel manually with the man:lttng-enable-channel(1)
+command, you can control how frequently are the current values of the
+channel's properties sampled to evaluate user conditions with the
+opt:lttng-enable-channel(1):--monitor-timer option.
+
+. Run the `notif-app` application. This program accepts the
+ <<tracing-session,tracing session>> name and the user space channel
+ name as its two first arguments. The channel which LTTng automatically
+ creates with the man:lttng-enable-event(1) command above is named
+ `channel0`:
++
+--
+[role="term"]
+----
+$ ./notif-app my-session channel0
+----
+--
+
+. In another terminal, run an application with a very high event
+ throughput so that the 75{nbsp}% buffer usage condition is reached.
++
+In the first terminal, the application should print lines like this:
++
+----
+Buffer usage is 81.45197 % in tracing session "my-session", user space
+channel "channel0".
+----
++
+If you don't see anything, try modifying the condition in
+path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
+(step 2) and running it again (step 4).
+====
+
+
[[reference]]
== Reference
projects / lttng-docs.git / blobdiff