trunk/lttv/QUICKSTART.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
   2 <html>
   3 <head>
   4   <title>Linux Trace Toolkit Next Generation User Documentation</title>
   5 </head>
   6   <body>
   7
   8 <h1>Linux Trace Toolkit Next Generation User Documentation</h1>
   9
  10 Author : Mathieu Desnoyers, September 2005<br>
  11 Last update : January 21st, 2009<br>
  12 (originally known as the LTTng QUICKSTART guide)
  13
  14 <h2>Table of Contents</h2>
  15
  16 <ul>
  17 <li><a href="#intro" name="TOCintro">Introduction</a></li>
  18
  19 <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from
  20 sources</a></li>
  21 <ul>
  22 <li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li>
  23 <li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li>
  24 <li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li>
  25 <li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li>
  26 <li><a href="#editconfig" name="TOCeditconfig">Editing the system wide
  27 configuration</a>
  28 <li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the
  29 ltt-control package</li>
  30 <li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li>
  31 <li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul>
  32
  33 <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li>
  34 <ul>
  35 <li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control
  36 tracing and analyse traces</a></li>
  37 <li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to
  38 control tracing</a></li>
  39 <li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li>
  40 <li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li>
  41 <li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li>
  42 </ul>
  43
  44 <li><a href="#section3" name="TOCsection3">Adding kernel and user-space
  45 instrumentation</a>
  46 <ul>
  47 <li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li>
  48 <li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li>
  49 </ul>
  50
  51 <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages
  52 from LTTV</a></li>
  53 <ul>
  54 <li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian
  55 <li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li>
  56 </ul>
  57
  58 </ul>
  59
  60 <hr />
  61
  62 <h2><a href="#TOCintro" name="intro">Introduction</a></h2>
  63 <p>
  64 This document is made of four parts : the first one explains how
  65 to install LTTng and LTTV from sources, the second one describes the steps
  66 to follow to trace a system and view it. The third part explains
  67 briefly how to add a new trace point to the kernel and to user space
  68 applications. The fourth and last part explains how to create Debian or RPM
  69 packages from the LTTng and LTTV sources.
  70 <p>
  71 These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X
  72 kernel. You will also find instructions for installation of LTTV 0.12.x : the
  73 Linux Trace Toolkit Viewer.
  74 To see the list of compatibilities between LTTng, ltt-control, LTTV, please
  75 refer to :
  76 <a
  77 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a>
  78 The lttng patch is necessary to have the tracing hooks in the kernel.
  79
  80 <br>
  81 <br>
  82 Supported architectures :
  83 br>
  84 LTTng :<br>
  85 <li> x86 32/64 bits
  86 <li> PowerPC 32 and 64 bits
  87 <li> ARM (with limited timestamping precision, e.g. 1HZ. Need
  88 architecture-specific support for better precision)
  89 <li> MIPS
  90 <br>
  91 <br>
  92 LTTV :<br>
  93 <li> Intel 32/64 bits
  94 <li> PowerPC 32 and 64 bits
  95 <li> Possibly others. Takes care of endianness and type size difference between
  96 the LTTng traces and the LTTV analysis tool.
  97
  98 <hr />
  99
 100
 101 <h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2>
 102 <p>
 103
 104 <h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3>
 105 <ul>
 106 <p>
 107 Tools needed to follow the package download steps :
 108
 109 <li>wget
 110 <li>bzip2
 111 <li>gzip
 112 <li>tar
 113
 114 <p>
 115 You have to install the standard development libraries and programs necessary
 116 to compile a kernel :
 117
 118 <PRE>
 119 (from Documentation/Changes in the Linux kernel tree)
 120 Gnu C                2.95.3                # gcc --version
 121 Gnu make             3.79.1                # make --version
 122 binutils             2.12                  # ld -v
 123 util-linux           2.10o                 # fdformat --version
 124 module-init-tools    0.9.10                # depmod -V
 125 </PRE>
 126
 127 <p>
 128 You might also want to have libncurses5 to have the text mode kernel
 129 configuration menu, but there are alternatives.
 130
 131 <p>
 132 Prerequisites for LTTV 0.x.x installation are :
 133
 134 <PRE>
 135 gcc 3.2 or better
 136 gtk 2.4 or better development libraries
 137   (Debian : libgtk2.0, libgtk2.0-dev)
 138   (Fedora : gtk2, gtk2-devel)
 139   note : For Fedora users : this might require at least core 3 from Fedora,
 140   or you might have to compile your own GTK2 library.
 141 glib 2.4 or better development libraries
 142   (Debian : libglib2.0-0, libglib2.0-dev)
 143   (Fedora : glib2, glib2-devel)
 144 libpopt development libraries
 145   (Debian : libpopt0, libpopt-dev)
 146   (Fedora : popt)
 147 libpango development libraries
 148   (Debian : libpango1.0, libpango1.0-dev)
 149   (Fedora : pango, pango-devel)
 150 libc6 development librairies
 151   (Debian : libc6, libc6-dev)
 152   (Fedora : glibc, glibc)
 153 </PRE>
 154 </ul>
 155
 156 <li>Reminder</li>
 157
 158 <p>
 159 See the list of compatibilities between LTTng, ltt-control and LTTV at :
 160 <a
 161 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV
 162 versions compatibility</a>.
 163
 164
 165 <h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3>
 166
 167 <PRE>
 168 su -
 169 mkdir /usr/src/lttng
 170 cd /usr/src/lttng
 171 (see http://ltt.polymtl.ca/lttng for package listing)
 172 wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2
 173 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof -
 174 </PRE>
 175
 176
 177 <h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3>
 178
 179 <PRE>
 180 su -
 181 cd /usr/src
 182 wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2
 183 bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof -
 184 cd linux-2.6.X
 185 - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1
 186 - For LTTng 0.9.5+ apply the patches in the order specified in the series file,
 187   or use quilt
 188 cd ..
 189 mv linux-2.6.X linux-2.6.X-lttng-0.x.xx
 190 </PRE>
 191
 192
 193 <h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3>
 194
 195 <PRE>
 196 su -
 197 cd /usr/src/linux-2.6.X-lttng-0.x.xx
 198 make menuconfig (or make xconfig or make config)
 199     Select the < Help > button if you are not familiar with kernel
 200     configuration.
 201     Items preceded by [*] means they has to be built into the kernel.
 202     Items preceded by [M] means they has to be built as modules.
 203     Items preceded by [ ] means they should be removed.
 204   go to the "General setup" section
 205     Select the following options :
 206     [*] Prompt for development and/or incomplete code/drivers
 207     [*] Activate markers
 208     [*] Activate userspace markers ABI (experimental, optional)
 209     [*] Immediate value optimization (optional)
 210     [*] Linux Trace Toolkit Next Generation (LTTng) --->
 211     <M> or <*> Compile lttng tracing probes
 212     <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay
 213     <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay
 214     <M> or <*> Linux Trace Toolkit Serializer
 215     <M> or <*>   Linux Trace Toolkit Marker Control
 216     <M> or <*>     Linux Trace Toolkit Tracer
 217     [*] Align Linux Trace Toolkit Traces
 218     <M> or <*> Support logging events from userspace
 219     [*] Support trace extraction from crash dump
 220     <M> or <*> Linux Trace Toolkit Trace Controller
 221     <M> or <*> Linux Trace Toolkit State Dump
 222   Select <Exit>
 223   Select <Exit>
 224   Select <Yes>
 225 make
 226 make modules_install
 227 (if necessary, create a initrd with mkinitrd or your preferate alternative)
 228 (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx)
 229
 230 -- on X86, X86_64
 231 make install
 232 reboot
 233 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 234
 235 -- on PowerPC
 236 cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx
 237 cp System.map /boot/System.map-2.6.X-lttng-0.x.xx
 238 cp .config /boot/config-2.6.X-lttng-0.x.xx
 239 depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 240 mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 241 (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry
 242 that comes first is the default kernel)
 243 ybin
 244 select the right entry at the yaboot prompt (see choices : tab, select : type
 245 the kernel name followed by enter)
 246 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 247 --
 248 </PRE>
 249
 250 <h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide
 251 configuration</a></h3>
 252
 253 <p>
 254 You must activate debugfs and specify a mount point. This is typically done in
 255 fstab such that it happens at boot time.  If you have never used DebugFS before,
 256 these operation would do this for you :
 257
 258 <PRE>
 259 mkdir /mnt/debugfs
 260 cp /etc/fstab /etc/fstab.lttng.bkp
 261 echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
 262 </PRE>
 263
 264 <p>
 265 then, rebooting or issuing the following command will activate debugfs :
 266 <PRE>
 267 mount /mnt/debugfs
 268 </PRE>
 269
 270 <p>
 271 You need to load the LTT modules to be able to control tracing from user
 272 space. This is done by issuing the following commands. Note however
 273 these commands load all LTT modules. Depending on what options you chose to
 274 compile statically, you may not need to issue all these commands.
 275
 276 <PRE>
 277 modprobe ltt-trace-control
 278 modprobe ltt-marker-control
 279 modprobe ltt-tracer
 280 modprobe ltt-serialize
 281 modprobe ltt-relay
 282 modprobe ipc-trace
 283 modprobe kernel-trace
 284 modprobe mm-trace
 285 modprobe net-trace
 286 modprobe fs-trace
 287 modprobe jbd2-trace
 288 modprobe ext4-trace
 289 modprobe syscall-trace
 290 modprobe trap-trace
 291 #if locking tracing is wanted, uncomment the following
 292 #modprobe lockdep-trace
 293 </PRE>
 294
 295 <p>
 296 If you want to have complete information about the kernel state (including all
 297 the process names), you need to load the ltt-statedump module. This is done by
 298 issuing the command :
 299
 300 <PRE>
 301 modprobe ltt-statedump
 302 </PRE>
 303 <p>
 304 You can automate at boot time loading the ltt-control module by :
 305
 306 <PRE>
 307 cp /etc/modules /etc/modules.bkp
 308 echo ltt-trace-control >> /etc/modules
 309 echo ltt-marker-control >> /etc/modules
 310 echo ltt-tracer >> /etc/modules
 311 echo ltt-serialize >> /etc/modules
 312 echo ltt-relay >> /etc/modules
 313 echo ipc-trace >> /etc/modules
 314 echo kernel-trace >> /etc/modules
 315 echo mm-trace >> /etc/modules
 316 echo net-trace >> /etc/modules
 317 echo fs-trace >> /etc/modules
 318 echo jbd2-trace >> /etc/modules
 319 echo ext4-trace >> /etc/modules
 320 echo syscall-trace >> /etc/modules
 321 echo trap-trace >> /etc/modules
 322 #if locking tracing is wanted, uncomment the following
 323 #echo lockdep-trace >> /etc/modules
 324 </PRE>
 325
 326
 327 <h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the
 328 ltt-control package (on the traced machine)</a></h3>
 329 <p>
 330 (note : the ltt-control package contains lttd and lttctl. Although it has the
 331 same name as the ltt-control kernel module, they are *not* the same thing.)
 332
 333 <PRE>
 334 su -
 335 cd /usr/src
 336 wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz
 337 gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof -
 338 cd ltt-control-0.x-xxxx2006
 339 (refer to README to see the development libraries that must be installed on you
 340 system)
 341 ./configure
 342 make
 343 make install
 344 </PRE>
 345
 346 <h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3>
 347
 348 <PRE>
 349 Make sure you selected the kernel menuconfig option :
 350     <M> or <*> Support logging events from userspace
 351 And that the ltt-userspace-event kernel module is loaded if selected as a
 352 module.
 353
 354 Simple userspace tracing is available through
 355 echo "some text to record" > /mnt/debugfs/ltt/write_event
 356
 357 It will appear in the trace under event :
 358 channel : userspace
 359 event name : event
 360 </PRE>
 361
 362 <h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package
 363 (on the visualisation machine, same
 364 or different from the visualisation machine)</a></h3>
 365
 366 <PRE>
 367 su -
 368 cd /usr/src
 369 wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz
 370 gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof -
 371 cd lttv-0.x.xx-xxxx2008
 372 (refer to README to see the development libraries that must be installed on your
 373 system)
 374 ./configure
 375 make
 376 make install
 377 </PRE>
 378
 379 <hr />
 380
 381
 382 <h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2>
 383
 384 <li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li>
 385 <PRE>
 386 ltt-armall
 387 </PRE>
 388
 389 <h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control
 390 tracing and analyse traces</a></h3>
 391 <PRE>
 392 lttv-gui (or /usr/local/bin/lttv-gui)
 393   - Spot the "Tracing Control" icon : click on it
 394       (it's a traffic light icon)
 395     - enter the root password
 396     - click "start"
 397     - click "stop"
 398     - Yes
 399       * You should now see a trace
 400 </PRE>
 401
 402 <h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3>
 403 <PRE>
 404 The tracing can be controlled from a terminal by using the lttctl command (as
 405 root).
 406
 407 Start tracing :
 408
 409 lttctl -C -w /tmp/trace1 trace1
 410
 411 Stop tracing and destroy trace channels :
 412
 413 lttctl -D trace1
 414
 415 see lttctl --help for details.
 416 </PRE>
 417 <p>
 418 (note : to see if the buffers has been filled, look at the dmesg output after
 419 lttctl -R or after stopping tracing from the GUI, it will show an event lost
 420 count. If it is the case, try using larger buffers. See lttctl --help to learn
 421 how. lttv now also shows event lost messages in the console when loading a trace
 422 with missing events or lost subbuffers.)
 423
 424 <h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3>
 425 <p>
 426 Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
 427 graphical plugins available.
 428 <p>
 429 For example, a simple trace dump in text format is available with :
 430 <PRE>
 431 lttv -m textDump -t /tmp/trace
 432 </PRE>
 433 <p>
 434 See lttv -m textDump --help for detailed command line options of textDump.
 435 <p>
 436 It is, in the current state of the project, very useful to use "grep" on the
 437 text output to filter by specific event fields. You can later copy the timestamp
 438 of the events to the clipboard and paste them in the GUI by clicking on the
 439 bottom right label "Current time". Support for this type of filtering should
 440 be added to the filter module soon.
 441
 442 <h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3>
 443 <p>
 444 Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
 445 hybrid. It can be especially useful when studying big workloads on a long period
 446 of time.
 447 <p>
 448 When using this mode, the most important, low rate control information will be
 449 recorded during all the trace by lttd (i.e. process creation/exit). The high
 450 rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a
 451 flight recorder buffer (now named flight-channelname_X).
 452 <p>
 453 The following lttctl commands take an hybrid trace :
 454 <p>
 455 Create trace channel, start lttd on normal channels, start tracing:
 456 <PRE>
 457 lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2
 458 </PRE>
 459 <p>
 460 Stop tracing, start lttd on flight recorder channels, destroy trace channels :
 461 <PRE>
 462 lttctl -D -w /tmp/trace2 trace2
 463 </PRE>
 464 <p>
 465 Each "overwrite" channel is flight recorder channel.
 466
 467
 468 <h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3>
 469 <li>Flight recorder mode</li>
 470 <p>
 471 The flight recorder mode writes data into overwritten buffers for all channels,
 472 including control channels, except for the facilities tracefiles. It consists of
 473 setting all channels to "overwrite".
 474 <p>
 475 The following lttctl commands take a flight recorder trace :
 476 <PRE>
 477 lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3
 478 ...
 479 lttctl -D -w /tmp/trace3 trace3
 480 </PRE>
 481
 482 <hr />
 483
 484
 485 <h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the
 486 markers</a></h2>
 487 <p>
 488
 489 <h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel
 490 instrumentation</a></h3>
 491
 492 <p>
 493 See <a
 494 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a>
 495 and <a
 496 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel
 497 tree.
 498 <p>
 499 Also see <a
 500 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a>
 501 for LTTng probe examples.
 502
 503 <h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3>
 504
 505 Add new events to userspace programs with
 506 <a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>.
 507 Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It
 508 allows inserting markers in executables and libraries, currently only on x86_32
 509 and x86_64.
 510 See <a
 511 href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent.
 512
 513 <p>
 514 Note that a new design document for a 3rd generation of tracepoint/marker-based
 515 userspace tracing is available at <a
 516 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing
 517 Design</a>. This new infrastructure is not yet implemented.
 518
 519 <p>
 520 The easy quick-and-dirty way to perform userspace tracing is currently to write
 521 an string to /mnt/debugfs/ltt/write_event. See <a
 522 href="#userspacetracing">Userspace tracing</a> in the
 523 installation for sources section of this document.
 524
 525 <hr />
 526
 527 <h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2>
 528 <p>
 529
 530 <h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3>
 531
 532 <PRE>
 533 Use : dpkg-buildpackage -rfakeroot
 534 </PRE>
 535 <p>
 536 You should then have your LTTV .deb files created for your architecture.
 537
 538 <h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3>
 539 <p>
 540 For building LTTng Debian packages :
 541 get the build tree with patches applies as explained in section 2.
 542
 543 <PRE>
 544 make menuconfig (or xconfig or config) (customize your configuration)
 545 make-kpkg kernel_image
 546 </PRE>
 547 <p>
 548 You will then see your freshly created .deb in /usr/src. Install it with
 549 <PRE>
 550 dpkg -i /usr/src/(image-name).deb
 551 </PRE>
 552 <p>
 553 Then, follow the section "Editing the system wide configuration" in section 2.
 554
 555
 556
 557   </body>
 558 </html>