trunk/lttv/QUICKSTART.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
   2 <html>
   3 <head>
   4   <title>Linux Trace Toolkit Next Generation User Documentation</title>
   5 </head>
   6   <body>
   7
   8 <h1>Linux Trace Toolkit Next Generation User Documentation</h1>
   9
  10 Author : Mathieu Desnoyers, September 2005<br>
  11 Last update : January 21st, 2009<br>
  12 (originally known as the LTTng QUICKSTART guide)
  13
  14 <h2>Table of Contents</h2>
  15
  16 <ul>
  17 <li><a href="#intro" name="TOCintro">Introduction</a></li>
  18 <ul>
  19 <li><a href="#arch" name="TOCarch">Supported architectures</a></li>
  20 </ul>
  21
  22 <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from
  23 sources</a></li>
  24 <ul>
  25 <li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li>
  26 <li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li>
  27 <li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li>
  28 <li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li>
  29 <li><a href="#editconfig" name="TOCeditconfig">Editing the system wide
  30 configuration</a>
  31 <li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the
  32 ltt-control package</li>
  33 <li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li>
  34 <li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul>
  35
  36 <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li>
  37 <ul>
  38 <li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control
  39 tracing and analyse traces</a></li>
  40 <li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to
  41 control tracing</a></li>
  42 <li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li>
  43 <li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li>
  44 <li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li>
  45 </ul>
  46
  47 <li><a href="#section3" name="TOCsection3">Adding kernel and user-space
  48 instrumentation</a>
  49 <ul>
  50 <li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li>
  51 <li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li>
  52 </ul>
  53
  54 <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages
  55 from LTTV</a></li>
  56 <ul>
  57 <li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian
  58 <li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li>
  59 </ul>
  60
  61 <li><a href="#section5" name="TOCsection5">Examples of LTTng use in the
  62 field</a></li>
  63
  64 </ul>
  65
  66 <hr />
  67
  68 <h2><a href="#TOCintro" name="intro">Introduction</a></h2>
  69 <p>
  70 This document is made of five parts : the first one explains how
  71 to install LTTng and LTTV from sources, the second one describes the steps
  72 to follow to trace a system and view it. The third part explains
  73 briefly how to add a new trace point to the kernel and to user space
  74 applications. The fourth part explains how to create Debian or RPM
  75 packages from the LTTng and LTTV sources. The fifth and last part describes use
  76 of LTTng in the field.
  77 <p>
  78 These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X
  79 kernel. You will also find instructions for installation of LTTV 0.12.x : the
  80 Linux Trace Toolkit Viewer.
  81 To see the list of compatibilities between LTTng, ltt-control, LTTV, please
  82 refer to :
  83 <a
  84 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a>
  85 The lttng patch is necessary to have the tracing hooks in the kernel.
  86
  87 <br>
  88 <br>
  89 <h3><a href="#TOCarch" name="arch">Supported architectures</a></h3>
  90 <br>
  91 LTTng :<br>
  92 <br>
  93 <li> x86 32/64 bits
  94 <li> PowerPC 32 and 64 bits
  95 <li> ARM (with limited timestamping precision, e.g. 1HZ. Need
  96 architecture-specific support for better precision)
  97 <li> MIPS
  98 <br>
  99 <br>
 100 LTTV :<br>
 101 <br>
 102 <li> Intel 32/64 bits
 103 <li> PowerPC 32 and 64 bits
 104 <li> Possibly others. Takes care of endianness and type size difference between
 105 the LTTng traces and the LTTV analysis tool.
 106
 107 <hr />
 108
 109
 110 <h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2>
 111 <p>
 112
 113 <h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3>
 114 <ul>
 115 <p>
 116 Tools needed to follow the package download steps :
 117
 118 <li>wget
 119 <li>bzip2
 120 <li>gzip
 121 <li>tar
 122
 123 <p>
 124 You have to install the standard development libraries and programs necessary
 125 to compile a kernel :
 126
 127 <PRE>
 128 (from Documentation/Changes in the Linux kernel tree)
 129 Gnu C                2.95.3                # gcc --version
 130 Gnu make             3.79.1                # make --version
 131 binutils             2.12                  # ld -v
 132 util-linux           2.10o                 # fdformat --version
 133 module-init-tools    0.9.10                # depmod -V
 134 </PRE>
 135
 136 <p>
 137 You might also want to have libncurses5 to have the text mode kernel
 138 configuration menu, but there are alternatives.
 139
 140 <p>
 141 Prerequisites for LTTV 0.x.x installation are :
 142
 143 <PRE>
 144 gcc 3.2 or better
 145 gtk 2.4 or better development libraries
 146   (Debian : libgtk2.0, libgtk2.0-dev)
 147   (Fedora : gtk2, gtk2-devel)
 148   note : For Fedora users : this might require at least core 3 from Fedora,
 149   or you might have to compile your own GTK2 library.
 150 glib 2.4 or better development libraries
 151   (Debian : libglib2.0-0, libglib2.0-dev)
 152   (Fedora : glib2, glib2-devel)
 153 libpopt development libraries
 154   (Debian : libpopt0, libpopt-dev)
 155   (Fedora : popt)
 156 libpango development libraries
 157   (Debian : libpango1.0, libpango1.0-dev)
 158   (Fedora : pango, pango-devel)
 159 libc6 development librairies
 160   (Debian : libc6, libc6-dev)
 161   (Fedora : glibc, glibc)
 162 </PRE>
 163 </ul>
 164
 165 <li>Reminder</li>
 166
 167 <p>
 168 See the list of compatibilities between LTTng, ltt-control and LTTV at :
 169 <a
 170 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV
 171 versions compatibility</a>.
 172
 173
 174 <h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3>
 175
 176 <PRE>
 177 su -
 178 mkdir /usr/src/lttng
 179 cd /usr/src/lttng
 180 (see http://ltt.polymtl.ca/lttng for package listing)
 181 wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2
 182 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof -
 183 </PRE>
 184
 185
 186 <h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3>
 187
 188 <PRE>
 189 su -
 190 cd /usr/src
 191 wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2
 192 bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof -
 193 cd linux-2.6.X
 194 - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1
 195 - For LTTng 0.9.5+ apply the patches in the order specified in the series file,
 196   or use quilt
 197 cd ..
 198 mv linux-2.6.X linux-2.6.X-lttng-0.x.xx
 199 </PRE>
 200
 201
 202 <h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3>
 203
 204 <PRE>
 205 su -
 206 cd /usr/src/linux-2.6.X-lttng-0.x.xx
 207 make menuconfig (or make xconfig or make config)
 208     Select the < Help > button if you are not familiar with kernel
 209     configuration.
 210     Items preceded by [*] means they has to be built into the kernel.
 211     Items preceded by [M] means they has to be built as modules.
 212     Items preceded by [ ] means they should be removed.
 213   go to the "General setup" section
 214     Select the following options :
 215     [*] Prompt for development and/or incomplete code/drivers
 216     [*] Activate markers
 217     [*] Activate userspace markers ABI (experimental, optional)
 218     [*] Immediate value optimization (optional)
 219     [*] Linux Trace Toolkit Next Generation (LTTng) --->
 220     <M> or <*> Compile lttng tracing probes
 221     <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay
 222     <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay
 223     <M> or <*> Linux Trace Toolkit Serializer
 224     <M> or <*>   Linux Trace Toolkit Marker Control
 225     <M> or <*>     Linux Trace Toolkit Tracer
 226     [*] Align Linux Trace Toolkit Traces
 227     <M> or <*> Support logging events from userspace
 228     [*] Support trace extraction from crash dump
 229     <M> or <*> Linux Trace Toolkit Trace Controller
 230     <M> or <*> Linux Trace Toolkit State Dump
 231   Select <Exit>
 232   Select <Exit>
 233   Select <Yes>
 234 make
 235 make modules_install
 236 (if necessary, create a initrd with mkinitrd or your preferate alternative)
 237 (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx)
 238
 239 -- on X86, X86_64
 240 make install
 241 reboot
 242 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 243
 244 -- on PowerPC
 245 cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx
 246 cp System.map /boot/System.map-2.6.X-lttng-0.x.xx
 247 cp .config /boot/config-2.6.X-lttng-0.x.xx
 248 depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 249 mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 250 (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry
 251 that comes first is the default kernel)
 252 ybin
 253 select the right entry at the yaboot prompt (see choices : tab, select : type
 254 the kernel name followed by enter)
 255 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 256 --
 257 </PRE>
 258
 259 <h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide
 260 configuration</a></h3>
 261
 262 <p>
 263 You must activate debugfs and specify a mount point. This is typically done in
 264 fstab such that it happens at boot time.  If you have never used DebugFS before,
 265 these operation would do this for you :
 266
 267 <PRE>
 268 mkdir /mnt/debugfs
 269 cp /etc/fstab /etc/fstab.lttng.bkp
 270 echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
 271 </PRE>
 272
 273 <p>
 274 then, rebooting or issuing the following command will activate debugfs :
 275 <PRE>
 276 mount /mnt/debugfs
 277 </PRE>
 278
 279 <p>
 280 You need to load the LTT modules to be able to control tracing from user
 281 space. This is done by issuing the following commands. Note however
 282 these commands load all LTT modules. Depending on what options you chose to
 283 compile statically, you may not need to issue all these commands.
 284
 285 <PRE>
 286 modprobe ltt-trace-control
 287 modprobe ltt-marker-control
 288 modprobe ltt-tracer
 289 modprobe ltt-serialize
 290 modprobe ltt-relay
 291 modprobe ipc-trace
 292 modprobe kernel-trace
 293 modprobe mm-trace
 294 modprobe net-trace
 295 modprobe fs-trace
 296 modprobe jbd2-trace
 297 modprobe ext4-trace
 298 modprobe syscall-trace
 299 modprobe trap-trace
 300 #if locking tracing is wanted, uncomment the following
 301 #modprobe lockdep-trace
 302 </PRE>
 303
 304 <p>
 305 If you want to have complete information about the kernel state (including all
 306 the process names), you need to load the ltt-statedump module. This is done by
 307 issuing the command :
 308
 309 <PRE>
 310 modprobe ltt-statedump
 311 </PRE>
 312 <p>
 313 You can automate at boot time loading the ltt-control module by :
 314
 315 <PRE>
 316 cp /etc/modules /etc/modules.bkp
 317 echo ltt-trace-control >> /etc/modules
 318 echo ltt-marker-control >> /etc/modules
 319 echo ltt-tracer >> /etc/modules
 320 echo ltt-serialize >> /etc/modules
 321 echo ltt-relay >> /etc/modules
 322 echo ipc-trace >> /etc/modules
 323 echo kernel-trace >> /etc/modules
 324 echo mm-trace >> /etc/modules
 325 echo net-trace >> /etc/modules
 326 echo fs-trace >> /etc/modules
 327 echo jbd2-trace >> /etc/modules
 328 echo ext4-trace >> /etc/modules
 329 echo syscall-trace >> /etc/modules
 330 echo trap-trace >> /etc/modules
 331 #if locking tracing is wanted, uncomment the following
 332 #echo lockdep-trace >> /etc/modules
 333 </PRE>
 334
 335
 336 <h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the
 337 ltt-control package (on the traced machine)</a></h3>
 338 <p>
 339 (note : the ltt-control package contains lttd and lttctl. Although it has the
 340 same name as the ltt-control kernel module, they are *not* the same thing.)
 341
 342 <PRE>
 343 su -
 344 cd /usr/src
 345 wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz
 346 gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof -
 347 cd ltt-control-0.x-xxxx2006
 348 (refer to README to see the development libraries that must be installed on you
 349 system)
 350 ./configure
 351 make
 352 make install
 353 </PRE>
 354
 355 <h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3>
 356
 357 <PRE>
 358 Make sure you selected the kernel menuconfig option :
 359     <M> or <*> Support logging events from userspace
 360 And that the ltt-userspace-event kernel module is loaded if selected as a
 361 module.
 362
 363 Simple userspace tracing is available through
 364 echo "some text to record" > /mnt/debugfs/ltt/write_event
 365
 366 It will appear in the trace under event :
 367 channel : userspace
 368 event name : event
 369 </PRE>
 370
 371 <h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package
 372 (on the visualisation machine, same
 373 or different from the visualisation machine)</a></h3>
 374
 375 <PRE>
 376 su -
 377 cd /usr/src
 378 wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz
 379 gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof -
 380 cd lttv-0.x.xx-xxxx2008
 381 (refer to README to see the development libraries that must be installed on your
 382 system)
 383 ./configure
 384 make
 385 make install
 386 </PRE>
 387
 388 <hr />
 389
 390
 391 <h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2>
 392
 393 <li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li>
 394 <PRE>
 395 ltt-armall
 396 </PRE>
 397
 398 <h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control
 399 tracing and analyse traces</a></h3>
 400 <PRE>
 401 lttv-gui (or /usr/local/bin/lttv-gui)
 402   - Spot the "Tracing Control" icon : click on it
 403       (it's a traffic light icon)
 404     - enter the root password
 405     - click "start"
 406     - click "stop"
 407     - Yes
 408       * You should now see a trace
 409 </PRE>
 410
 411 <h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3>
 412 <PRE>
 413 The tracing can be controlled from a terminal by using the lttctl command (as
 414 root).
 415
 416 Start tracing :
 417
 418 lttctl -C -w /tmp/trace1 trace1
 419
 420 Stop tracing and destroy trace channels :
 421
 422 lttctl -D trace1
 423
 424 see lttctl --help for details.
 425 </PRE>
 426 <p>
 427 (note : to see if the buffers has been filled, look at the dmesg output after
 428 lttctl -R or after stopping tracing from the GUI, it will show an event lost
 429 count. If it is the case, try using larger buffers. See lttctl --help to learn
 430 how. lttv now also shows event lost messages in the console when loading a trace
 431 with missing events or lost subbuffers.)
 432
 433 <h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3>
 434 <p>
 435 Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
 436 graphical plugins available.
 437 <p>
 438 For example, a simple trace dump in text format is available with :
 439 <PRE>
 440 lttv -m textDump -t /tmp/trace
 441 </PRE>
 442 <p>
 443 See lttv -m textDump --help for detailed command line options of textDump.
 444 <p>
 445 It is, in the current state of the project, very useful to use "grep" on the
 446 text output to filter by specific event fields. You can later copy the timestamp
 447 of the events to the clipboard and paste them in the GUI by clicking on the
 448 bottom right label "Current time". Support for this type of filtering should
 449 be added to the filter module soon.
 450
 451 <h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3>
 452 <p>
 453 Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
 454 hybrid. It can be especially useful when studying big workloads on a long period
 455 of time.
 456 <p>
 457 When using this mode, the most important, low rate control information will be
 458 recorded during all the trace by lttd (i.e. process creation/exit). The high
 459 rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a
 460 flight recorder buffer (now named flight-channelname_X).
 461 <p>
 462 The following lttctl commands take an hybrid trace :
 463 <p>
 464 Create trace channel, start lttd on normal channels, start tracing:
 465 <PRE>
 466 lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2
 467 </PRE>
 468 <p>
 469 Stop tracing, start lttd on flight recorder channels, destroy trace channels :
 470 <PRE>
 471 lttctl -D -w /tmp/trace2 trace2
 472 </PRE>
 473 <p>
 474 Each "overwrite" channel is flight recorder channel.
 475
 476
 477 <h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3>
 478 <li>Flight recorder mode</li>
 479 <p>
 480 The flight recorder mode writes data into overwritten buffers for all channels,
 481 including control channels, except for the facilities tracefiles. It consists of
 482 setting all channels to "overwrite".
 483 <p>
 484 The following lttctl commands take a flight recorder trace :
 485 <PRE>
 486 lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3
 487 ...
 488 lttctl -D -w /tmp/trace3 trace3
 489 </PRE>
 490
 491 <hr />
 492
 493
 494 <h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the
 495 markers</a></h2>
 496 <p>
 497
 498 <h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel
 499 instrumentation</a></h3>
 500
 501 <p>
 502 See <a
 503 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a>
 504 and <a
 505 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel
 506 tree.
 507 <p>
 508 Also see <a
 509 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a>
 510 for LTTng probe examples.
 511
 512 <h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3>
 513
 514 Add new events to userspace programs with
 515 <a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>.
 516 Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It
 517 allows inserting markers in executables and libraries, currently only on x86_32
 518 and x86_64.
 519 See <a
 520 href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent.
 521
 522 <p>
 523 Note that a new design document for a 3rd generation of tracepoint/marker-based
 524 userspace tracing is available at <a
 525 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing
 526 Design</a>. This new infrastructure is not yet implemented.
 527
 528 <p>
 529 The easy quick-and-dirty way to perform userspace tracing is currently to write
 530 an string to /mnt/debugfs/ltt/write_event. See <a
 531 href="#userspacetracing">Userspace tracing</a> in the
 532 installation for sources section of this document.
 533
 534 <hr />
 535
 536 <h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2>
 537 <p>
 538
 539 <h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3>
 540
 541 <PRE>
 542 Use : dpkg-buildpackage -rfakeroot
 543 </PRE>
 544 <p>
 545 You should then have your LTTV .deb files created for your architecture.
 546
 547 <h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3>
 548 <p>
 549 For building LTTng Debian packages :
 550 get the build tree with patches applies as explained in section 2.
 551
 552 <PRE>
 553 make menuconfig (or xconfig or config) (customize your configuration)
 554 make-kpkg kernel_image
 555 </PRE>
 556 <p>
 557 You will then see your freshly created .deb in /usr/src. Install it with
 558 <PRE>
 559 dpkg -i /usr/src/(image-name).deb
 560 </PRE>
 561 <p>
 562 Then, follow the section "Editing the system wide configuration" in section 2.
 563
 564 <hr />
 565
 566 <h2><a href="#TOCsection5" name="section5">Examples of LTTng use in the field</a></h2>
 567 <p>
 568 A few examples of successful LTTng users :
 569
 570 <ul>
 571 <li> Google are deploying LTTng on their servers. They want to use it to
 572   monitor their production servers (with flight recorder mode tracing)
 573   and to help them solve hard to reproduce problems. They have had
 574   success with such tracing approach to fix "rare disk delay" issues and
 575   VM-related issues presented in this article :
 576 <ul>
 577   <li> <a href="http://ltt.polymtl.ca/papers/bligh-Reprint.pdf">Linux Kernel
 578 Debugging on Google-sized clusters at Ottawa Linux
 579     Symposium 2007</a>
 580 </ul>
 581 <li> IBM Research have had problems with Commercial Scale-out applications,
 582   which are being an increasing trend to split large server workloads.
 583   They used LTTng successfully to solve a distributed filesystem-related
 584   issue. It's presented in the same paper above.
 585
 586 <li> Autodesk, in the development of their next-generation of Linux
 587   audio/video edition applications, used LTTng extensively to solve
 588   soft real-time issues they had. Also presented in the same paper.
 589
 590 <li> Wind River included LTTng in their Linux distribution so their
 591   clients, already familiar to Wind River own tracing solution in
 592   VxWorks, car have the same kind of feature they have relied on for a
 593   long time.
 594
 595 <li> Montavista have integrated LTTng in their distribution for the same
 596   reasons. It's used by Sony amongst others.
 597
 598 <li> SuSE are currently integrating LTTng in their next SLES distribution,
 599   because their clients asking for solutions which supports a kernel
 600   closer to real-time need such tools to debug their problems.
 601
 602 <li> A project between Ericsson, the Canadian Defense, NSERC and various
 603   universities is just starting. It aims at monitoring and debugging
 604   multi-core systems and provide automated and help user system behavior
 605   analysis.
 606
 607 <li> Siemens have been using LTTng internally for quite some time now.
 608 </ul>
 609   </body>
 610 </html>