<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
- <title>Linux Trace Toolkit Quickstart</title>
+ <title>Linux Trace Toolkit Next Generation User Documentation</title>
</head>
<body>
-<h1>Linux Trace Toolkit Quickstart</h1>
+<h1>Linux Trace Toolkit Next Generation User Documentation</h1>
Author : Mathieu Desnoyers, September 2005<br>
Last update : January 21st, 2009<br>
+(originally known as the LTTng QUICKSTART guide)
<h2>Table of Contents</h2>
-<ul>
+<ul>
<li><a href="#intro" name="TOCintro">Introduction</a></li>
+<ul>
+<li><a href="#arch" name="TOCarch">Supported architectures</a></li>
+</ul>
+
<li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from
sources</a></li>
+<ul>
+<li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li>
+<li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li>
+<li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li>
+<li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li>
+<li><a href="#editconfig" name="TOCeditconfig">Editing the system wide
+configuration</a>
+<li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the
+ltt-control package</li>
+<li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li>
+<li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul>
+
<li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li>
+<ul>
+<li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control
+tracing and analyse traces</a></li>
+<li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to
+control tracing</a></li>
+<li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li>
+<li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li>
+<li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li>
+</ul>
+
<li><a href="#section3" name="TOCsection3">Adding kernel and user-space
-tracepoints</a></li>
+instrumentation</a>
+<ul>
+<li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li>
+<li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li>
+</ul>
+
<li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages
-from LTTV</a>
+from LTTV</a></li>
+<ul>
+<li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian
+<li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li>
+</ul>
+
+<li><a href="#section5" name="TOCsection5">Examples of LTTng use in the
+field</a></li>
</ul>
<h2><a href="#TOCintro" name="intro">Introduction</a></h2>
<p>
-This document is made of four parts : the first one explains how
+This document is made of five parts : the first one explains how
to install LTTng and LTTV from sources, the second one describes the steps
to follow to trace a system and view it. The third part explains
briefly how to add a new trace point to the kernel and to user space
-applications. The fourth and last part explains how to create Debian or RPM
-packages from the LTTng and LTTV sources.
+applications. The fourth part explains how to create Debian or RPM
+packages from the LTTng and LTTV sources. The fifth and last part describes use
+of LTTng in the field.
<p>
These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X
kernel. You will also find instructions for installation of LTTV 0.12.x : the
<br>
<br>
-Supported architectures :
+<h3><a href="#TOCarch" name="arch">Supported architectures</a></h3>
<br>
LTTng :<br>
+<br>
<li> x86 32/64 bits
<li> PowerPC 32 and 64 bits
<li> ARM (with limited timestamping precision, e.g. 1HZ. Need
architecture-specific support for better precision)
<li> MIPS
+<li> sh (partial architecture-specific instrumentation)
+<li> sparc64 (partial architecture-specific instrumentation)
+<li> s390 (partial architecture-specific instrumentation)
+<li> Other architectures supported without architecture-specific instrumentation
+and with low-resolution timestamps.<br>
<br>
<br>
LTTV :<br>
+<br>
<li> Intel 32/64 bits
<li> PowerPC 32 and 64 bits
<li> Possibly others. Takes care of endianness and type size difference between
<hr />
-<h2><a href="#TOCsection2" name="section2">Installation from sources</a></h2>
+<h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2>
<p>
-<li>Prerequisites</li>
+<h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3>
<ul>
<p>
Tools needed to follow the package download steps :
versions compatibility</a>.
-<li>Getting the LTTng packages</li>
+<h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3>
<PRE>
su -
</PRE>
-<li>Getting LTTng kernel sources</li>
+<h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3>
<PRE>
su -
</PRE>
-<li>Installing a LTTng kernel</li>
+<h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3>
<PRE>
su -
--
</PRE>
-<li>Editing the system wide configuration</li>
+<h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide
+configuration</a></h3>
<p>
You must activate debugfs and specify a mount point. This is typically done in
#echo lockdep-trace >> /etc/modules
</PRE>
-<li>Getting and installing the ltt-control package (on the traced machine)</li>
+
+<h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the
+ltt-control package (on the traced machine)</a></h3>
<p>
(note : the ltt-control package contains lttd and lttctl. Although it has the
same name as the ltt-control kernel module, they are *not* the same thing.)
make install
</PRE>
-<li>Userspace tracing</li>
+<h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3>
<PRE>
Make sure you selected the kernel menuconfig option :
event name : event
</PRE>
-<li>Getting and installing the LTTV package (on the visualisation machine, same
-or different from the visualisation machine)</li>
+<h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package
+(on the visualisation machine, same
+or different from the visualisation machine)</a></h3>
<PRE>
su -
make install
</PRE>
-<li>Getting and installing the markers-userspace package for user space
-tracing (experimental)</li>
-<p>
-See <a
-href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent.
-
<hr />
-<h2><a href="#TOCsection3" name="section3">Using LTTng and LTTV</a></h2>
+<h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2>
-<li>IMPORTANT : Arm Linux Kernel Markers after each boot</li>
+<li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li>
<PRE>
ltt-armall
</PRE>
-<li>Use graphical LTTV to control tracing and analyse traces</li>
+<h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control
+tracing and analyse traces</a></h3>
<PRE>
lttv-gui (or /usr/local/bin/lttv-gui)
- Spot the "Tracing Control" icon : click on it
* You should now see a trace
</PRE>
-<li>Use text mode LTTng to control tracing</li>
+<h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3>
<PRE>
The tracing can be controlled from a terminal by using the lttctl command (as
root).
lttctl -D trace1
see lttctl --help for details.
-<PRE>
+</PRE>
<p>
(note : to see if the buffers has been filled, look at the dmesg output after
lttctl -R or after stopping tracing from the GUI, it will show an event lost
how. lttv now also shows event lost messages in the console when loading a trace
with missing events or lost subbuffers.)
-<li>Use text mode LTTV</li>
+<h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3>
<p>
Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
graphical plugins available.
bottom right label "Current time". Support for this type of filtering should
be added to the filter module soon.
-<li>Hybrid mode</li>
+<h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3>
<p>
Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
hybrid. It can be especially useful when studying big workloads on a long period
<p>
Each "overwrite" channel is flight recorder channel.
+
+<h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3>
<li>Flight recorder mode</li>
<p>
The flight recorder mode writes data into overwritten buffers for all channels,
lttctl -D -w /tmp/trace3 trace3
</PRE>
-**************************************************************
-** Section 4 * Adding new instrumentations with the markers **
-**************************************************************
+<hr />
-See Documentation/markers.txt and Documentation/tracepoints.txt in your kernel
+
+<h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the
+markers</a></h2>
+<p>
+
+<h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel
+instrumentation</a></h3>
+
+<p>
+See <a
+href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a>
+and <a
+href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel
tree.
+<p>
+Also see <a
+href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a>
+for LTTng probe examples.
-* Add new events to userspace programs with userspace markers
-http://ltt.polymtl.ca/packages/
+<h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3>
+Add new events to userspace programs with
+<a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>.
Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It
allows inserting markers in executables and libraries, currently only on x86_32
and x86_64.
+See <a
+href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent.
-***********************************************************
-** Section 5 * Creating Debian or RPM packages **
-***********************************************************
+<p>
+Note that a new design document for a 3rd generation of tracepoint/marker-based
+userspace tracing is available at <a
+href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing
+Design</a>. This new infrastructure is not yet implemented.
-* Create custom LTTV Debian packages
+<p>
+The easy quick-and-dirty way to perform userspace tracing is currently to write
+an string to /mnt/debugfs/ltt/write_event. See <a
+href="#userspacetracing">Userspace tracing</a> in the
+installation for sources section of this document.
-Use : dpkg-buildpackage -rfakeroot
+<hr />
-You should then have your LTTV .deb files created for your architecture.
+<h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2>
+<p>
-* Create custom LTTng packages
+<h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3>
-For building LTTng Debian packages :
+<PRE>
+Use : dpkg-buildpackage -rfakeroot
+</PRE>
+<p>
+You should then have your LTTV .deb files created for your architecture.
-Get the build tree with patches applies as explained in section 2.
+<h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3>
+<p>
+For building LTTng Debian packages :
+get the build tree with patches applies as explained in section 2.
+<PRE>
make menuconfig (or xconfig or config) (customize your configuration)
make-kpkg kernel_image
-
+</PRE>
+<p>
You will then see your freshly created .deb in /usr/src. Install it with
+<PRE>
dpkg -i /usr/src/(image-name).deb
-
+</PRE>
+<p>
Then, follow the section "Editing the system wide configuration" in section 2.
+<hr />
+<h2><a href="#TOCsection5" name="section5">Examples of LTTng use in the field</a></h2>
+<p>
+A few examples of successful LTTng users :
+<ul>
+<li> Google are deploying LTTng on their servers. They want to use it to
+ monitor their production servers (with flight recorder mode tracing)
+ and to help them solve hard to reproduce problems. They have had
+ success with such tracing approach to fix "rare disk delay" issues and
+ VM-related issues presented in this article :
+<ul>
+ <li> <a href="http://ltt.polymtl.ca/papers/bligh-Reprint.pdf">Linux Kernel
+Debugging on Google-sized clusters at Ottawa Linux
+ Symposium 2007</a>
+</ul>
+<li> IBM Research have had problems with Commercial Scale-out applications,
+ which are being an increasing trend to split large server workloads.
+ They used LTTng successfully to solve a distributed filesystem-related
+ issue. It's presented in the same paper above.
+
+<li> Autodesk, in the development of their next-generation of Linux
+ audio/video edition applications, used LTTng extensively to solve
+ soft real-time issues they had. Also presented in the same paper.
+
+<li> Wind River included LTTng in their Linux distribution so their
+ clients, already familiar to Wind River own tracing solution in
+ VxWorks, car have the same kind of feature they have relied on for a
+ long time.
+
+<li> Montavista have integrated LTTng in their distribution for the same
+ reasons. It's used by Sony amongst others.
+
+<li> SuSE are currently integrating LTTng in their next SLES distribution,
+ because their clients asking for solutions which supports a kernel
+ closer to real-time need such tools to debug their problems.
+
+<li> A project between Ericsson, the Canadian Defense, NSERC and various
+ universities is just starting. It aims at monitoring and debugging
+ multi-core systems and provide automated and help user system behavior
+ analysis.
+
+<li> Siemens have been using LTTng internally for quite some time now.
+</ul>
</body>
</html>