trunk/lttv/QUICKSTART.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
   2 <html>
   3 <head>
   4   <title>Linux Trace Toolkit Next Generation User Documentation</title>
   5 </head>
   6   <body>
   7
   8 <h1>Linux Trace Toolkit Next Generation User Documentation</h1>
   9
  10 Author : Mathieu Desnoyers, September 2005<br>
  11 Last update : January 21st, 2009<br>
  12 (originally known as the LTTng QUICKSTART guide)
  13
  14 <h2>Table of Contents</h2>
  15
  16 <ul>
  17 <li><a href="#intro" name="TOCintro">Introduction</a></li>
  18
  19 <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from
  20 sources</a></li>
  21 <ul>
  22 <li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li>
  23 <li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li>
  24 <li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li>
  25 <li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li>
  26 <li><a href="#editconfig" name="TOCeditconfig">Editing the system wide
  27 configuration</a>
  28 <li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the
  29 ltt-control package</li>
  30 <li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li>
  31 <li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul>
  32
  33 <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li>
  34 <ul>
  35 <li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control
  36 tracing and analyse traces</a></li>
  37 <li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to
  38 control tracing</a></li>
  39 <li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li>
  40 <li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li>
  41 <li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li>
  42 </ul>
  43
  44 <li><a href="#section3" name="TOCsection3">Adding kernel and user-space
  45 instrumentation</a>
  46 <ul>
  47 <li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li>
  48 <li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li>
  49 </ul>
  50
  51 <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages
  52 from LTTV</a></li>
  53 <ul>
  54 <li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian
  55 <li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li>
  56 </ul>
  57
  58 <li><a href="#section5" name="TOCsection5">Examples of LTTng use in the
  59 field</a></li>
  60
  61 </ul>
  62
  63 <hr />
  64
  65 <h2><a href="#TOCintro" name="intro">Introduction</a></h2>
  66 <p>
  67 This document is made of five parts : the first one explains how
  68 to install LTTng and LTTV from sources, the second one describes the steps
  69 to follow to trace a system and view it. The third part explains
  70 briefly how to add a new trace point to the kernel and to user space
  71 applications. The fourth part explains how to create Debian or RPM
  72 packages from the LTTng and LTTV sources. The fifth and last part describes use
  73 of LTTng in the field.
  74 <p>
  75 These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X
  76 kernel. You will also find instructions for installation of LTTV 0.12.x : the
  77 Linux Trace Toolkit Viewer.
  78 To see the list of compatibilities between LTTng, ltt-control, LTTV, please
  79 refer to :
  80 <a
  81 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a>
  82 The lttng patch is necessary to have the tracing hooks in the kernel.
  83
  84 <br>
  85 <br>
  86 Supported architectures :
  87 <br>
  88 LTTng :<br>
  89 <br>
  90 <li> x86 32/64 bits
  91 <li> PowerPC 32 and 64 bits
  92 <li> ARM (with limited timestamping precision, e.g. 1HZ. Need
  93 architecture-specific support for better precision)
  94 <li> MIPS
  95 <br>
  96 <br>
  97 LTTV :<br>
  98 <br>
  99 <li> Intel 32/64 bits
 100 <li> PowerPC 32 and 64 bits
 101 <li> Possibly others. Takes care of endianness and type size difference between
 102 the LTTng traces and the LTTV analysis tool.
 103
 104 <hr />
 105
 106
 107 <h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2>
 108 <p>
 109
 110 <h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3>
 111 <ul>
 112 <p>
 113 Tools needed to follow the package download steps :
 114
 115 <li>wget
 116 <li>bzip2
 117 <li>gzip
 118 <li>tar
 119
 120 <p>
 121 You have to install the standard development libraries and programs necessary
 122 to compile a kernel :
 123
 124 <PRE>
 125 (from Documentation/Changes in the Linux kernel tree)
 126 Gnu C                2.95.3                # gcc --version
 127 Gnu make             3.79.1                # make --version
 128 binutils             2.12                  # ld -v
 129 util-linux           2.10o                 # fdformat --version
 130 module-init-tools    0.9.10                # depmod -V
 131 </PRE>
 132
 133 <p>
 134 You might also want to have libncurses5 to have the text mode kernel
 135 configuration menu, but there are alternatives.
 136
 137 <p>
 138 Prerequisites for LTTV 0.x.x installation are :
 139
 140 <PRE>
 141 gcc 3.2 or better
 142 gtk 2.4 or better development libraries
 143   (Debian : libgtk2.0, libgtk2.0-dev)
 144   (Fedora : gtk2, gtk2-devel)
 145   note : For Fedora users : this might require at least core 3 from Fedora,
 146   or you might have to compile your own GTK2 library.
 147 glib 2.4 or better development libraries
 148   (Debian : libglib2.0-0, libglib2.0-dev)
 149   (Fedora : glib2, glib2-devel)
 150 libpopt development libraries
 151   (Debian : libpopt0, libpopt-dev)
 152   (Fedora : popt)
 153 libpango development libraries
 154   (Debian : libpango1.0, libpango1.0-dev)
 155   (Fedora : pango, pango-devel)
 156 libc6 development librairies
 157   (Debian : libc6, libc6-dev)
 158   (Fedora : glibc, glibc)
 159 </PRE>
 160 </ul>
 161
 162 <li>Reminder</li>
 163
 164 <p>
 165 See the list of compatibilities between LTTng, ltt-control and LTTV at :
 166 <a
 167 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV
 168 versions compatibility</a>.
 169
 170
 171 <h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3>
 172
 173 <PRE>
 174 su -
 175 mkdir /usr/src/lttng
 176 cd /usr/src/lttng
 177 (see http://ltt.polymtl.ca/lttng for package listing)
 178 wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2
 179 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof -
 180 </PRE>
 181
 182
 183 <h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3>
 184
 185 <PRE>
 186 su -
 187 cd /usr/src
 188 wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2
 189 bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof -
 190 cd linux-2.6.X
 191 - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1
 192 - For LTTng 0.9.5+ apply the patches in the order specified in the series file,
 193   or use quilt
 194 cd ..
 195 mv linux-2.6.X linux-2.6.X-lttng-0.x.xx
 196 </PRE>
 197
 198
 199 <h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3>
 200
 201 <PRE>
 202 su -
 203 cd /usr/src/linux-2.6.X-lttng-0.x.xx
 204 make menuconfig (or make xconfig or make config)
 205     Select the < Help > button if you are not familiar with kernel
 206     configuration.
 207     Items preceded by [*] means they has to be built into the kernel.
 208     Items preceded by [M] means they has to be built as modules.
 209     Items preceded by [ ] means they should be removed.
 210   go to the "General setup" section
 211     Select the following options :
 212     [*] Prompt for development and/or incomplete code/drivers
 213     [*] Activate markers
 214     [*] Activate userspace markers ABI (experimental, optional)
 215     [*] Immediate value optimization (optional)
 216     [*] Linux Trace Toolkit Next Generation (LTTng) --->
 217     <M> or <*> Compile lttng tracing probes
 218     <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay
 219     <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay
 220     <M> or <*> Linux Trace Toolkit Serializer
 221     <M> or <*>   Linux Trace Toolkit Marker Control
 222     <M> or <*>     Linux Trace Toolkit Tracer
 223     [*] Align Linux Trace Toolkit Traces
 224     <M> or <*> Support logging events from userspace
 225     [*] Support trace extraction from crash dump
 226     <M> or <*> Linux Trace Toolkit Trace Controller
 227     <M> or <*> Linux Trace Toolkit State Dump
 228   Select <Exit>
 229   Select <Exit>
 230   Select <Yes>
 231 make
 232 make modules_install
 233 (if necessary, create a initrd with mkinitrd or your preferate alternative)
 234 (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx)
 235
 236 -- on X86, X86_64
 237 make install
 238 reboot
 239 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 240
 241 -- on PowerPC
 242 cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx
 243 cp System.map /boot/System.map-2.6.X-lttng-0.x.xx
 244 cp .config /boot/config-2.6.X-lttng-0.x.xx
 245 depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 246 mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 247 (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry
 248 that comes first is the default kernel)
 249 ybin
 250 select the right entry at the yaboot prompt (see choices : tab, select : type
 251 the kernel name followed by enter)
 252 Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader.
 253 --
 254 </PRE>
 255
 256 <h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide
 257 configuration</a></h3>
 258
 259 <p>
 260 You must activate debugfs and specify a mount point. This is typically done in
 261 fstab such that it happens at boot time.  If you have never used DebugFS before,
 262 these operation would do this for you :
 263
 264 <PRE>
 265 mkdir /mnt/debugfs
 266 cp /etc/fstab /etc/fstab.lttng.bkp
 267 echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
 268 </PRE>
 269
 270 <p>
 271 then, rebooting or issuing the following command will activate debugfs :
 272 <PRE>
 273 mount /mnt/debugfs
 274 </PRE>
 275
 276 <p>
 277 You need to load the LTT modules to be able to control tracing from user
 278 space. This is done by issuing the following commands. Note however
 279 these commands load all LTT modules. Depending on what options you chose to
 280 compile statically, you may not need to issue all these commands.
 281
 282 <PRE>
 283 modprobe ltt-trace-control
 284 modprobe ltt-marker-control
 285 modprobe ltt-tracer
 286 modprobe ltt-serialize
 287 modprobe ltt-relay
 288 modprobe ipc-trace
 289 modprobe kernel-trace
 290 modprobe mm-trace
 291 modprobe net-trace
 292 modprobe fs-trace
 293 modprobe jbd2-trace
 294 modprobe ext4-trace
 295 modprobe syscall-trace
 296 modprobe trap-trace
 297 #if locking tracing is wanted, uncomment the following
 298 #modprobe lockdep-trace
 299 </PRE>
 300
 301 <p>
 302 If you want to have complete information about the kernel state (including all
 303 the process names), you need to load the ltt-statedump module. This is done by
 304 issuing the command :
 305
 306 <PRE>
 307 modprobe ltt-statedump
 308 </PRE>
 309 <p>
 310 You can automate at boot time loading the ltt-control module by :
 311
 312 <PRE>
 313 cp /etc/modules /etc/modules.bkp
 314 echo ltt-trace-control >> /etc/modules
 315 echo ltt-marker-control >> /etc/modules
 316 echo ltt-tracer >> /etc/modules
 317 echo ltt-serialize >> /etc/modules
 318 echo ltt-relay >> /etc/modules
 319 echo ipc-trace >> /etc/modules
 320 echo kernel-trace >> /etc/modules
 321 echo mm-trace >> /etc/modules
 322 echo net-trace >> /etc/modules
 323 echo fs-trace >> /etc/modules
 324 echo jbd2-trace >> /etc/modules
 325 echo ext4-trace >> /etc/modules
 326 echo syscall-trace >> /etc/modules
 327 echo trap-trace >> /etc/modules
 328 #if locking tracing is wanted, uncomment the following
 329 #echo lockdep-trace >> /etc/modules
 330 </PRE>
 331
 332
 333 <h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the
 334 ltt-control package (on the traced machine)</a></h3>
 335 <p>
 336 (note : the ltt-control package contains lttd and lttctl. Although it has the
 337 same name as the ltt-control kernel module, they are *not* the same thing.)
 338
 339 <PRE>
 340 su -
 341 cd /usr/src
 342 wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz
 343 gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof -
 344 cd ltt-control-0.x-xxxx2006
 345 (refer to README to see the development libraries that must be installed on you
 346 system)
 347 ./configure
 348 make
 349 make install
 350 </PRE>
 351
 352 <h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3>
 353
 354 <PRE>
 355 Make sure you selected the kernel menuconfig option :
 356     <M> or <*> Support logging events from userspace
 357 And that the ltt-userspace-event kernel module is loaded if selected as a
 358 module.
 359
 360 Simple userspace tracing is available through
 361 echo "some text to record" > /mnt/debugfs/ltt/write_event
 362
 363 It will appear in the trace under event :
 364 channel : userspace
 365 event name : event
 366 </PRE>
 367
 368 <h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package
 369 (on the visualisation machine, same
 370 or different from the visualisation machine)</a></h3>
 371
 372 <PRE>
 373 su -
 374 cd /usr/src
 375 wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz
 376 gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof -
 377 cd lttv-0.x.xx-xxxx2008
 378 (refer to README to see the development libraries that must be installed on your
 379 system)
 380 ./configure
 381 make
 382 make install
 383 </PRE>
 384
 385 <hr />
 386
 387
 388 <h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2>
 389
 390 <li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li>
 391 <PRE>
 392 ltt-armall
 393 </PRE>
 394
 395 <h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control
 396 tracing and analyse traces</a></h3>
 397 <PRE>
 398 lttv-gui (or /usr/local/bin/lttv-gui)
 399   - Spot the "Tracing Control" icon : click on it
 400       (it's a traffic light icon)
 401     - enter the root password
 402     - click "start"
 403     - click "stop"
 404     - Yes
 405       * You should now see a trace
 406 </PRE>
 407
 408 <h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3>
 409 <PRE>
 410 The tracing can be controlled from a terminal by using the lttctl command (as
 411 root).
 412
 413 Start tracing :
 414
 415 lttctl -C -w /tmp/trace1 trace1
 416
 417 Stop tracing and destroy trace channels :
 418
 419 lttctl -D trace1
 420
 421 see lttctl --help for details.
 422 </PRE>
 423 <p>
 424 (note : to see if the buffers has been filled, look at the dmesg output after
 425 lttctl -R or after stopping tracing from the GUI, it will show an event lost
 426 count. If it is the case, try using larger buffers. See lttctl --help to learn
 427 how. lttv now also shows event lost messages in the console when loading a trace
 428 with missing events or lost subbuffers.)
 429
 430 <h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3>
 431 <p>
 432 Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
 433 graphical plugins available.
 434 <p>
 435 For example, a simple trace dump in text format is available with :
 436 <PRE>
 437 lttv -m textDump -t /tmp/trace
 438 </PRE>
 439 <p>
 440 See lttv -m textDump --help for detailed command line options of textDump.
 441 <p>
 442 It is, in the current state of the project, very useful to use "grep" on the
 443 text output to filter by specific event fields. You can later copy the timestamp
 444 of the events to the clipboard and paste them in the GUI by clicking on the
 445 bottom right label "Current time". Support for this type of filtering should
 446 be added to the filter module soon.
 447
 448 <h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3>
 449 <p>
 450 Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
 451 hybrid. It can be especially useful when studying big workloads on a long period
 452 of time.
 453 <p>
 454 When using this mode, the most important, low rate control information will be
 455 recorded during all the trace by lttd (i.e. process creation/exit). The high
 456 rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a
 457 flight recorder buffer (now named flight-channelname_X).
 458 <p>
 459 The following lttctl commands take an hybrid trace :
 460 <p>
 461 Create trace channel, start lttd on normal channels, start tracing:
 462 <PRE>
 463 lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2
 464 </PRE>
 465 <p>
 466 Stop tracing, start lttd on flight recorder channels, destroy trace channels :
 467 <PRE>
 468 lttctl -D -w /tmp/trace2 trace2
 469 </PRE>
 470 <p>
 471 Each "overwrite" channel is flight recorder channel.
 472
 473
 474 <h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3>
 475 <li>Flight recorder mode</li>
 476 <p>
 477 The flight recorder mode writes data into overwritten buffers for all channels,
 478 including control channels, except for the facilities tracefiles. It consists of
 479 setting all channels to "overwrite".
 480 <p>
 481 The following lttctl commands take a flight recorder trace :
 482 <PRE>
 483 lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3
 484 ...
 485 lttctl -D -w /tmp/trace3 trace3
 486 </PRE>
 487
 488 <hr />
 489
 490
 491 <h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the
 492 markers</a></h2>
 493 <p>
 494
 495 <h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel
 496 instrumentation</a></h3>
 497
 498 <p>
 499 See <a
 500 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a>
 501 and <a
 502 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel
 503 tree.
 504 <p>
 505 Also see <a
 506 href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a>
 507 for LTTng probe examples.
 508
 509 <h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3>
 510
 511 Add new events to userspace programs with
 512 <a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>.
 513 Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It
 514 allows inserting markers in executables and libraries, currently only on x86_32
 515 and x86_64.
 516 See <a
 517 href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent.
 518
 519 <p>
 520 Note that a new design document for a 3rd generation of tracepoint/marker-based
 521 userspace tracing is available at <a
 522 href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing
 523 Design</a>. This new infrastructure is not yet implemented.
 524
 525 <p>
 526 The easy quick-and-dirty way to perform userspace tracing is currently to write
 527 an string to /mnt/debugfs/ltt/write_event. See <a
 528 href="#userspacetracing">Userspace tracing</a> in the
 529 installation for sources section of this document.
 530
 531 <hr />
 532
 533 <h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2>
 534 <p>
 535
 536 <h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3>
 537
 538 <PRE>
 539 Use : dpkg-buildpackage -rfakeroot
 540 </PRE>
 541 <p>
 542 You should then have your LTTV .deb files created for your architecture.
 543
 544 <h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3>
 545 <p>
 546 For building LTTng Debian packages :
 547 get the build tree with patches applies as explained in section 2.
 548
 549 <PRE>
 550 make menuconfig (or xconfig or config) (customize your configuration)
 551 make-kpkg kernel_image
 552 </PRE>
 553 <p>
 554 You will then see your freshly created .deb in /usr/src. Install it with
 555 <PRE>
 556 dpkg -i /usr/src/(image-name).deb
 557 </PRE>
 558 <p>
 559 Then, follow the section "Editing the system wide configuration" in section 2.
 560
 561 <hr />
 562
 563 <h2><a href="#TOCsection5" name="section5">Examples of LTTng use in the field</a></h2>
 564 <p>
 565 A few examples of successful LTTng users :
 566
 567 <ul>
 568 <li> Google are deploying LTTng on their servers. They want to use it to
 569   monitor their production servers (with flight recorder mode tracing)
 570   and to help them solve hard to reproduce problems. They have had
 571   success with such tracing approach to fix "rare disk delay" issues and
 572   VM-related issues presented in this article :
 573
 574   * "Linux Kernel Debugging on Google-sized clusters at Ottawa Linux
 575     Symposium 2007"
 576     http://ltt.polymtl.ca/papers/bligh-Reprint.pdf
 577
 578 <li> IBM Research have had problems with Commercial Scale-out applications,
 579   which are being an increasing trend to split large server workloads.
 580   They used LTTng successfully to solve a distributed filesystem-related
 581   issue. It's presented in the same paper above.
 582
 583 <li> Autodesk, in the development of their next-generation of Linux
 584   audio/video edition applications, used LTTng extensively to solve
 585   soft real-time issues they had. Also presented in the same paper.
 586
 587 <li> Wind River included LTTng in their Linux distribution so their
 588   clients, already familiar to Wind River own tracing solution in
 589   VxWorks, car have the same kind of feature they have relied on for a
 590   long time.
 591
 592 <li> Montavista have integrated LTTng in their distribution for the same
 593   reasons. It's used by Sony amongst others.
 594
 595 <li> SuSE are currently integrating LTTng in their next SLES distribution,
 596   because their clients asking for solutions which supports a kernel
 597   closer to real-time need such tools to debug their problems.
 598
 599 <li> A project between Ericsson, the Canadian Defense, NSERC and various
 600   universities is just starting. It aims at monitoring and debugging
 601   multi-core systems and provide automated and help user system behavior
 602   analysis.
 603
 604 <li> Siemens have been using LTTng internally for quite some time now.
 605 </ul>
 606   </body>
 607 </html>