-This document is made of four parts : the first one explains how +This document is made of five parts : the first one explains how to install LTTng and LTTV from sources, the second one describes the steps to follow to trace a system and view it. The third part explains briefly how to add a new trace point to the kernel and to user space -applications. The fourth and last part explains how to create Debian or RPM -packages from the LTTng and LTTV sources. - +applications. The fourth part explains how to create Debian or RPM +packages from the LTTng and LTTV sources. The fifth and last part describes use +of LTTng in the field.
These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X
kernel. You will also find instructions for installation of LTTV 0.12.x : the
@@ -43,21 +82,33 @@ To see the list of compatibilities between LTTng, ltt-control, LTTV, please
refer to :
LTTng+LTTV versions compatibility
-The lttng patch is necessary to have the tracing hooks in the kernel.
+
+The ongoing work had the Linux Kernel Markers integrated in the mainline Linux
+kernel since Linux 2.6.24 and the Tracepoints since 2.6.28. In its current
+state, the lttng patchset is necessary to have the trace clocksource, the
+instrumentation and the LTTng high-speed data extraction mechanism added to the
+kernel.
-Supported architectures :
+
-
Tools needed to follow the package download steps : @@ -121,26 +172,30 @@ libc6 development librairies
See the list of compatibilities between LTTng, ltt-control and LTTV at : LTTng+LTTV versions compatibility. -* Getting the LTTng packages +
su - mkdir /usr/src/lttng cd /usr/src/lttng (see http://ltt.polymtl.ca/lttng for package listing) wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - +-* Getting LTTng kernel sources +
su - cd /usr/src wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 @@ -151,10 +206,12 @@ cd linux-2.6.X or use quilt cd .. mv linux-2.6.X linux-2.6.X-lttng-0.x.xx +-* Installing a LTTng kernel +
su - cd /usr/src/linux-2.6.X-lttng-0.x.xx make menuconfig (or make xconfig or make config) @@ -207,29 +264,35 @@ select the right entry at the yaboot prompt (see choices : tab, select : type the kernel name followed by enter) Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. -- ++
You must activate debugfs and specify a mount point. This is typically done in -fstab such that it happens at boot time. - -If you have never used DebugFS before, these operation would do this for you : +fstab such that it happens at boot time. If you have never used DebugFS before, +these operation would do this for you : +
mkdir /mnt/debugfs cp /etc/fstab /etc/fstab.lttng.bkp echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab ++
then, rebooting or issuing the following command will activate debugfs : - +
mount /mnt/debugfs ++
You need to load the LTT modules to be able to control tracing from user space. This is done by issuing the following commands. Note however these commands load all LTT modules. Depending on what options you chose to compile statically, you may not need to issue all these commands. +
modprobe ltt-trace-control modprobe ltt-marker-control modprobe ltt-tracer @@ -246,15 +309,20 @@ modprobe syscall-trace modprobe trap-trace #if locking tracing is wanted, uncomment the following #modprobe lockdep-trace ++
If you want to have complete information about the kernel state (including all the process names), you need to load the ltt-statedump module. This is done by issuing the command : +
modprobe ltt-statedump - ++
You can automate at boot time loading the ltt-control module by : +
cp /etc/modules /etc/modules.bkp echo ltt-trace-control >> /etc/modules echo ltt-marker-control >> /etc/modules @@ -272,11 +340,16 @@ echo syscall-trace >> /etc/modules echo trap-trace >> /etc/modules #if locking tracing is wanted, uncomment the following #echo lockdep-trace >> /etc/modules +-* Getting and installing the ltt-control package (on the traced machine) +
(note : the ltt-control package contains lttd and lttctl. Although it has the same name as the ltt-control kernel module, they are *not* the same thing.) + +
su - cd /usr/src wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz @@ -287,10 +360,11 @@ system) ./configure make make install ++
Make sure you selected the kernel menuconfig option :
or <*> Support logging events from userspace
And that the ltt-userspace-event kernel module is loaded if selected as a
@@ -302,11 +376,13 @@ echo "some text to record" > /mnt/debugfs/ltt/write_event
It will appear in the trace under event :
channel : userspace
event name : event
+
+su - cd /usr/src wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz @@ -317,24 +393,21 @@ system) ./configure make make install ++
ltt-armall +-* Use graphical LTTV to control tracing and analyse traces - +
lttv-gui (or /usr/local/bin/lttv-gui)
- Spot the "Tracing Control" icon : click on it
(it's a traffic light icon)
@@ -343,9 +416,10 @@ lttv-gui (or /usr/local/bin/lttv-gui)
- click "stop"
- Yes
* You should now see a trace
+
-* Use text mode LTTng to control tracing
-
+The tracing can be controlled from a terminal by using the lttctl command (as root). @@ -358,103 +432,189 @@ Stop tracing and destroy trace channels : lttctl -D trace1 see lttctl --help for details. - ++
(note : to see if the buffers has been filled, look at the dmesg output after -lttctl -R or after stopping tracing from the GUI, it will show an event lost +lttctl -D or after stopping tracing from the GUI, it will show an event lost count. If it is the case, try using larger buffers. See lttctl --help to learn how. lttv now also shows event lost messages in the console when loading a trace with missing events or lost subbuffers.) -* Use text mode LTTV - +
Feel free to look in /usr/local/lib/lttv/plugins to see all the text and graphical plugins available. - +
For example, a simple trace dump in text format is available with : - +
lttv -m textDump -t /tmp/trace - -see lttv -m textDump --help for detailed command line options of textDump. - ++
+See lttv -m textDump --help for detailed command line options of textDump. +
It is, in the current state of the project, very useful to use "grep" on the text output to filter by specific event fields. You can later copy the timestamp of the events to the clipboard and paste them in the GUI by clicking on the bottom right label "Current time". Support for this type of filtering should be added to the filter module soon. -* Hybrid mode - +
Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : hybrid. It can be especially useful when studying big workloads on a long period of time. - +
When using this mode, the most important, low rate control information will be recorded during all the trace by lttd (i.e. process creation/exit). The high rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a flight recorder buffer (now named flight-channelname_X). - +
The following lttctl commands take an hybrid trace : - +
Create trace channel, start lttd on normal channels, start tracing: +
lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 - ++
Stop tracing, start lttd on flight recorder channels, destroy trace channels : +
lttctl -D -w /tmp/trace2 trace2 - ++
Each "overwrite" channel is flight recorder channel. -* Flight recorder mode +
The flight recorder mode writes data into overwritten buffers for all channels, including control channels, except for the facilities tracefiles. It consists of setting all channels to "overwrite". - +
The following lttctl commands take a flight recorder trace : - +
lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 ... lttctl -D -w /tmp/trace3 trace3 ++
-************************************************************** -** Section 4 * Adding new instrumentations with the markers ** -************************************************************** +
+See Documentation/markers.txt +and Documentation/tracepoints.txt in your kernel tree. +
+Also see ltt/probes/ +for LTTng probe examples. -* Add new events to userspace programs with userspace markers -http://ltt.polymtl.ca/packages/ +
+Note that a new design document for a 3rd generation of tracepoint/marker-based +userspace tracing is available at LTTng User-space Tracing +Design. This new infrastructure is not yet implemented. -* Create custom LTTV Debian packages +
+The easy quick-and-dirty way to perform userspace tracing is currently to write +an string to /mnt/debugfs/ltt/write_event. See Userspace tracing in the +installation for sources section of this document. -Use : dpkg-buildpackage -rfakeroot +
-* Create custom LTTng packages +
+Use : dpkg-buildpackage -rfakeroot ++
+You should then have your LTTV .deb files created for your architecture. -Get the build tree with patches applies as explained in section 2. +
+For building LTTng Debian packages : +get the build tree with patches applies as explained in section 2. +
make menuconfig (or xconfig or config) (customize your configuration) make-kpkg kernel_image - ++
You will then see your freshly created .deb in /usr/src. Install it with +
dpkg -i /usr/src/(image-name).deb - ++
Then, follow the section "Editing the system wide configuration" in section 2. +
+A few examples of successful LTTng users : + +