| 1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| 2 | <html> |
| 3 | <head> |
| 4 | <title>Linux Trace Toolkit Next Generation User Documentation</title> |
| 5 | </head> |
| 6 | <body> |
| 7 | |
| 8 | <h1>Linux Trace Toolkit Next Generation User Documentation</h1> |
| 9 | |
| 10 | Author : Mathieu Desnoyers, September 2005<br> |
| 11 | Last update : January 21st, 2009<br> |
| 12 | (originally known as the LTTng QUICKSTART guide) |
| 13 | |
| 14 | <h2>Table of Contents</h2> |
| 15 | |
| 16 | <ul> |
| 17 | <li><a href="#intro" name="TOCintro">Introduction</a></li> |
| 18 | <ul> |
| 19 | <li><a href="#arch" name="TOCarch">Supported architectures</a></li> |
| 20 | </ul> |
| 21 | |
| 22 | <li><a href="#section1" name="TOCsection1">Installing LTTng and LTTV from |
| 23 | sources</a></li> |
| 24 | <ul> |
| 25 | <li><a href="#prerequisites" name="TOCprerequisites">Prerequisistes</li> |
| 26 | <li><a href="#getlttng" name="TOCgetlttng">Getting the LTTng packages</li> |
| 27 | <li><a href="#getlttngsrc" name="TOCgetlttngsrc">Getting the LTTng kernel sources</li> |
| 28 | <li><a href="#installlttng" name="TOCinstalllttng">Installing a LTTng kernel</li> |
| 29 | <li><a href="#editconfig" name="TOCeditconfig">Editing the system wide |
| 30 | configuration</a> |
| 31 | <li><a href="#getlttctl" name="TOCgetlttctl">Getting and installing the |
| 32 | ltt-control package</li> |
| 33 | <li><a href="#userspacetracing" name="TOCuserspacetracing">Userspace Tracing</li> |
| 34 | <li><a href="#getlttv" name="TOCgetlttv">Getting and installing the LTTV package</ul> |
| 35 | |
| 36 | <li><a href="#section2" name="TOCsection2">Using LTTng and LTTV</a></li> |
| 37 | <ul> |
| 38 | <li><a href="#uselttvgui" name="TOCuselttvgui">Use graphical LTTV to control |
| 39 | tracing and analyse traces</a></li> |
| 40 | <li><a href="#uselttngtext" name="TOCuselttngtext">Use text mode LTTng to |
| 41 | control tracing</a></li> |
| 42 | <li><a href="#uselttvtext" name="TOCuselttvtext">Use text mode LTTV</a></li> |
| 43 | <li><a href="#hybrid" name="TOChybrid">Tracing in "Hybrid" mode</a></li> |
| 44 | <li><a href="#flight" name="TOCflight">Tracing in flight recorder mode</a></li> |
| 45 | </ul> |
| 46 | |
| 47 | <li><a href="#section3" name="TOCsection3">Adding kernel and user-space |
| 48 | instrumentation</a> |
| 49 | <ul> |
| 50 | <li><a href="#kerneltp" name="TOCkerneltp">Adding kernel instrumentation</a></li> |
| 51 | <li><a href="#usertp" name="TOCusertp">Adding userspace instrumentation</a></li> |
| 52 | </ul> |
| 53 | |
| 54 | <li><a href="#section4" name="TOCsection4">Creating Debian and RPM packages |
| 55 | from LTTV</a></li> |
| 56 | <ul> |
| 57 | <li><a href="#pkgdebian" name="TOCpkgdebian">Create custom LTTV Debian |
| 58 | <li><a href="#pkglttng" name="TOCpkglttng">Create custom LTTng packages</a></li> |
| 59 | </ul> |
| 60 | |
| 61 | <li><a href="#section5" name="TOCsection5">Examples of LTTng use in the |
| 62 | field</a></li> |
| 63 | |
| 64 | </ul> |
| 65 | |
| 66 | <hr /> |
| 67 | |
| 68 | <h2><a href="#TOCintro" name="intro">Introduction</a></h2> |
| 69 | <p> |
| 70 | This document is made of five parts : the first one explains how |
| 71 | to install LTTng and LTTV from sources, the second one describes the steps |
| 72 | to follow to trace a system and view it. The third part explains |
| 73 | briefly how to add a new trace point to the kernel and to user space |
| 74 | applications. The fourth part explains how to create Debian or RPM |
| 75 | packages from the LTTng and LTTV sources. The fifth and last part describes use |
| 76 | of LTTng in the field. |
| 77 | <p> |
| 78 | These operations are made for installing the LTTng 0.86 tracer on a linux 2.6.X |
| 79 | kernel. You will also find instructions for installation of LTTV 0.12.x : the |
| 80 | Linux Trace Toolkit Viewer. |
| 81 | To see the list of compatibilities between LTTng, ltt-control, LTTV, please |
| 82 | refer to : |
| 83 | <a |
| 84 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV versions compatibility</a> |
| 85 | The lttng patch is necessary to have the tracing hooks in the kernel. |
| 86 | |
| 87 | <br> |
| 88 | <br> |
| 89 | <h3><a href="#TOCarch" name="arch">Supported architectures</a></h3> |
| 90 | <br> |
| 91 | LTTng :<br> |
| 92 | <br> |
| 93 | <li> x86 32/64 bits |
| 94 | <li> PowerPC 32 and 64 bits |
| 95 | <li> ARM (with limited timestamping precision, e.g. 1HZ. Need |
| 96 | architecture-specific support for better precision) |
| 97 | <li> MIPS |
| 98 | <br> |
| 99 | <br> |
| 100 | LTTV :<br> |
| 101 | <br> |
| 102 | <li> Intel 32/64 bits |
| 103 | <li> PowerPC 32 and 64 bits |
| 104 | <li> Possibly others. Takes care of endianness and type size difference between |
| 105 | the LTTng traces and the LTTV analysis tool. |
| 106 | |
| 107 | <hr /> |
| 108 | |
| 109 | |
| 110 | <h2><a href="#TOCsection1" name="section1">Installation from sources</a></h2> |
| 111 | <p> |
| 112 | |
| 113 | <h3><a href="#TOCprerequisites" name="prerequisites">Prerequisites</a></h3> |
| 114 | <ul> |
| 115 | <p> |
| 116 | Tools needed to follow the package download steps : |
| 117 | |
| 118 | <li>wget |
| 119 | <li>bzip2 |
| 120 | <li>gzip |
| 121 | <li>tar |
| 122 | |
| 123 | <p> |
| 124 | You have to install the standard development libraries and programs necessary |
| 125 | to compile a kernel : |
| 126 | |
| 127 | <PRE> |
| 128 | (from Documentation/Changes in the Linux kernel tree) |
| 129 | Gnu C 2.95.3 # gcc --version |
| 130 | Gnu make 3.79.1 # make --version |
| 131 | binutils 2.12 # ld -v |
| 132 | util-linux 2.10o # fdformat --version |
| 133 | module-init-tools 0.9.10 # depmod -V |
| 134 | </PRE> |
| 135 | |
| 136 | <p> |
| 137 | You might also want to have libncurses5 to have the text mode kernel |
| 138 | configuration menu, but there are alternatives. |
| 139 | |
| 140 | <p> |
| 141 | Prerequisites for LTTV 0.x.x installation are : |
| 142 | |
| 143 | <PRE> |
| 144 | gcc 3.2 or better |
| 145 | gtk 2.4 or better development libraries |
| 146 | (Debian : libgtk2.0, libgtk2.0-dev) |
| 147 | (Fedora : gtk2, gtk2-devel) |
| 148 | note : For Fedora users : this might require at least core 3 from Fedora, |
| 149 | or you might have to compile your own GTK2 library. |
| 150 | glib 2.4 or better development libraries |
| 151 | (Debian : libglib2.0-0, libglib2.0-dev) |
| 152 | (Fedora : glib2, glib2-devel) |
| 153 | libpopt development libraries |
| 154 | (Debian : libpopt0, libpopt-dev) |
| 155 | (Fedora : popt) |
| 156 | libpango development libraries |
| 157 | (Debian : libpango1.0, libpango1.0-dev) |
| 158 | (Fedora : pango, pango-devel) |
| 159 | libc6 development librairies |
| 160 | (Debian : libc6, libc6-dev) |
| 161 | (Fedora : glibc, glibc) |
| 162 | </PRE> |
| 163 | </ul> |
| 164 | |
| 165 | <li>Reminder</li> |
| 166 | |
| 167 | <p> |
| 168 | See the list of compatibilities between LTTng, ltt-control and LTTV at : |
| 169 | <a |
| 170 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/lttng-lttv-compatibility.html">LTTng+LTTV |
| 171 | versions compatibility</a>. |
| 172 | |
| 173 | |
| 174 | <h3><a href="#TOCgetlttng" name="getlttng">Getting the LTTng packages</a></h3> |
| 175 | |
| 176 | <PRE> |
| 177 | su - |
| 178 | mkdir /usr/src/lttng |
| 179 | cd /usr/src/lttng |
| 180 | (see http://ltt.polymtl.ca/lttng for package listing) |
| 181 | wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2 |
| 182 | bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof - |
| 183 | </PRE> |
| 184 | |
| 185 | |
| 186 | <h3><a href="#TOCgetlttngsrc" name="getlttngsrc">Getting LTTng kernel sources</a></h3> |
| 187 | |
| 188 | <PRE> |
| 189 | su - |
| 190 | cd /usr/src |
| 191 | wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2 |
| 192 | bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof - |
| 193 | cd linux-2.6.X |
| 194 | - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1 |
| 195 | - For LTTng 0.9.5+ apply the patches in the order specified in the series file, |
| 196 | or use quilt |
| 197 | cd .. |
| 198 | mv linux-2.6.X linux-2.6.X-lttng-0.x.xx |
| 199 | </PRE> |
| 200 | |
| 201 | |
| 202 | <h3><a href="#TOCinstalllttng" name="installlttng">Installing a LTTng kernel</a></h3> |
| 203 | |
| 204 | <PRE> |
| 205 | su - |
| 206 | cd /usr/src/linux-2.6.X-lttng-0.x.xx |
| 207 | make menuconfig (or make xconfig or make config) |
| 208 | Select the < Help > button if you are not familiar with kernel |
| 209 | configuration. |
| 210 | Items preceded by [*] means they has to be built into the kernel. |
| 211 | Items preceded by [M] means they has to be built as modules. |
| 212 | Items preceded by [ ] means they should be removed. |
| 213 | go to the "General setup" section |
| 214 | Select the following options : |
| 215 | [*] Prompt for development and/or incomplete code/drivers |
| 216 | [*] Activate markers |
| 217 | [*] Activate userspace markers ABI (experimental, optional) |
| 218 | [*] Immediate value optimization (optional) |
| 219 | [*] Linux Trace Toolkit Next Generation (LTTng) ---> |
| 220 | <M> or <*> Compile lttng tracing probes |
| 221 | <M> or <*> Linux Trace Toolkit High-speed Lockless Data Relay |
| 222 | <M> or <*> Linux Trace Toolkit Lock-Protected Data Relay |
| 223 | <M> or <*> Linux Trace Toolkit Serializer |
| 224 | <M> or <*> Linux Trace Toolkit Marker Control |
| 225 | <M> or <*> Linux Trace Toolkit Tracer |
| 226 | [*] Align Linux Trace Toolkit Traces |
| 227 | <M> or <*> Support logging events from userspace |
| 228 | [*] Support trace extraction from crash dump |
| 229 | <M> or <*> Linux Trace Toolkit Trace Controller |
| 230 | <M> or <*> Linux Trace Toolkit State Dump |
| 231 | Select <Exit> |
| 232 | Select <Exit> |
| 233 | Select <Yes> |
| 234 | make |
| 235 | make modules_install |
| 236 | (if necessary, create a initrd with mkinitrd or your preferate alternative) |
| 237 | (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx) |
| 238 | |
| 239 | -- on X86, X86_64 |
| 240 | make install |
| 241 | reboot |
| 242 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
| 243 | |
| 244 | -- on PowerPC |
| 245 | cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx |
| 246 | cp System.map /boot/System.map-2.6.X-lttng-0.x.xx |
| 247 | cp .config /boot/config-2.6.X-lttng-0.x.xx |
| 248 | depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| 249 | mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx |
| 250 | (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry |
| 251 | that comes first is the default kernel) |
| 252 | ybin |
| 253 | select the right entry at the yaboot prompt (see choices : tab, select : type |
| 254 | the kernel name followed by enter) |
| 255 | Select the Linux 2.6.X-lttng-0.x.xx kernel in your boot loader. |
| 256 | -- |
| 257 | </PRE> |
| 258 | |
| 259 | <h3><a href="#TOCeditconfig" name="editconfig">Editing the system wide |
| 260 | configuration</a></h3> |
| 261 | |
| 262 | <p> |
| 263 | You must activate debugfs and specify a mount point. This is typically done in |
| 264 | fstab such that it happens at boot time. If you have never used DebugFS before, |
| 265 | these operation would do this for you : |
| 266 | |
| 267 | <PRE> |
| 268 | mkdir /mnt/debugfs |
| 269 | cp /etc/fstab /etc/fstab.lttng.bkp |
| 270 | echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab |
| 271 | </PRE> |
| 272 | |
| 273 | <p> |
| 274 | then, rebooting or issuing the following command will activate debugfs : |
| 275 | <PRE> |
| 276 | mount /mnt/debugfs |
| 277 | </PRE> |
| 278 | |
| 279 | <p> |
| 280 | You need to load the LTT modules to be able to control tracing from user |
| 281 | space. This is done by issuing the following commands. Note however |
| 282 | these commands load all LTT modules. Depending on what options you chose to |
| 283 | compile statically, you may not need to issue all these commands. |
| 284 | |
| 285 | <PRE> |
| 286 | modprobe ltt-trace-control |
| 287 | modprobe ltt-marker-control |
| 288 | modprobe ltt-tracer |
| 289 | modprobe ltt-serialize |
| 290 | modprobe ltt-relay |
| 291 | modprobe ipc-trace |
| 292 | modprobe kernel-trace |
| 293 | modprobe mm-trace |
| 294 | modprobe net-trace |
| 295 | modprobe fs-trace |
| 296 | modprobe jbd2-trace |
| 297 | modprobe ext4-trace |
| 298 | modprobe syscall-trace |
| 299 | modprobe trap-trace |
| 300 | #if locking tracing is wanted, uncomment the following |
| 301 | #modprobe lockdep-trace |
| 302 | </PRE> |
| 303 | |
| 304 | <p> |
| 305 | If you want to have complete information about the kernel state (including all |
| 306 | the process names), you need to load the ltt-statedump module. This is done by |
| 307 | issuing the command : |
| 308 | |
| 309 | <PRE> |
| 310 | modprobe ltt-statedump |
| 311 | </PRE> |
| 312 | <p> |
| 313 | You can automate at boot time loading the ltt-control module by : |
| 314 | |
| 315 | <PRE> |
| 316 | cp /etc/modules /etc/modules.bkp |
| 317 | echo ltt-trace-control >> /etc/modules |
| 318 | echo ltt-marker-control >> /etc/modules |
| 319 | echo ltt-tracer >> /etc/modules |
| 320 | echo ltt-serialize >> /etc/modules |
| 321 | echo ltt-relay >> /etc/modules |
| 322 | echo ipc-trace >> /etc/modules |
| 323 | echo kernel-trace >> /etc/modules |
| 324 | echo mm-trace >> /etc/modules |
| 325 | echo net-trace >> /etc/modules |
| 326 | echo fs-trace >> /etc/modules |
| 327 | echo jbd2-trace >> /etc/modules |
| 328 | echo ext4-trace >> /etc/modules |
| 329 | echo syscall-trace >> /etc/modules |
| 330 | echo trap-trace >> /etc/modules |
| 331 | #if locking tracing is wanted, uncomment the following |
| 332 | #echo lockdep-trace >> /etc/modules |
| 333 | </PRE> |
| 334 | |
| 335 | |
| 336 | <h3><a href="#TOCgetlttctl" name="getlttctl">Getting and installing the |
| 337 | ltt-control package (on the traced machine)</a></h3> |
| 338 | <p> |
| 339 | (note : the ltt-control package contains lttd and lttctl. Although it has the |
| 340 | same name as the ltt-control kernel module, they are *not* the same thing.) |
| 341 | |
| 342 | <PRE> |
| 343 | su - |
| 344 | cd /usr/src |
| 345 | wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz |
| 346 | gzip -cd ltt-control-0.x-xxxx2008.tar.gz | tar xvof - |
| 347 | cd ltt-control-0.x-xxxx2006 |
| 348 | (refer to README to see the development libraries that must be installed on you |
| 349 | system) |
| 350 | ./configure |
| 351 | make |
| 352 | make install |
| 353 | </PRE> |
| 354 | |
| 355 | <h3><a href="#TOCuserspacetracing" name="userspacetracing">Userspace tracing</a></h3> |
| 356 | |
| 357 | <PRE> |
| 358 | Make sure you selected the kernel menuconfig option : |
| 359 | <M> or <*> Support logging events from userspace |
| 360 | And that the ltt-userspace-event kernel module is loaded if selected as a |
| 361 | module. |
| 362 | |
| 363 | Simple userspace tracing is available through |
| 364 | echo "some text to record" > /mnt/debugfs/ltt/write_event |
| 365 | |
| 366 | It will appear in the trace under event : |
| 367 | channel : userspace |
| 368 | event name : event |
| 369 | </PRE> |
| 370 | |
| 371 | <h3><a href="#TOCgetlttv" name="getlttv">Getting and installing the LTTV package |
| 372 | (on the visualisation machine, same |
| 373 | or different from the visualisation machine)</a></h3> |
| 374 | |
| 375 | <PRE> |
| 376 | su - |
| 377 | cd /usr/src |
| 378 | wget http://ltt.polymtl.ca/packages/lttv-0.x.xx-xxxx2008.tar.gz |
| 379 | gzip -cd lttv-0.x.xx-xxxx2008.tar.gz | tar xvof - |
| 380 | cd lttv-0.x.xx-xxxx2008 |
| 381 | (refer to README to see the development libraries that must be installed on your |
| 382 | system) |
| 383 | ./configure |
| 384 | make |
| 385 | make install |
| 386 | </PRE> |
| 387 | |
| 388 | <hr /> |
| 389 | |
| 390 | |
| 391 | <h2><a href="#TOCsection2" name="section2">Using LTTng and LTTV</a></h2> |
| 392 | |
| 393 | <li><b>IMPORTANT : Arm Linux Kernel Markers after each boot before tracing</b></li> |
| 394 | <PRE> |
| 395 | ltt-armall |
| 396 | </PRE> |
| 397 | |
| 398 | <h3><a href="#TOCuselttvgui" name="uselttvgui">Use graphical LTTV to control |
| 399 | tracing and analyse traces</a></h3> |
| 400 | <PRE> |
| 401 | lttv-gui (or /usr/local/bin/lttv-gui) |
| 402 | - Spot the "Tracing Control" icon : click on it |
| 403 | (it's a traffic light icon) |
| 404 | - enter the root password |
| 405 | - click "start" |
| 406 | - click "stop" |
| 407 | - Yes |
| 408 | * You should now see a trace |
| 409 | </PRE> |
| 410 | |
| 411 | <h3><a href="#TOCuselttngtext" name="uselttngtext">Use text mode LTTng to control tracing</a></h3> |
| 412 | <PRE> |
| 413 | The tracing can be controlled from a terminal by using the lttctl command (as |
| 414 | root). |
| 415 | |
| 416 | Start tracing : |
| 417 | |
| 418 | lttctl -C -w /tmp/trace1 trace1 |
| 419 | |
| 420 | Stop tracing and destroy trace channels : |
| 421 | |
| 422 | lttctl -D trace1 |
| 423 | |
| 424 | see lttctl --help for details. |
| 425 | </PRE> |
| 426 | <p> |
| 427 | (note : to see if the buffers has been filled, look at the dmesg output after |
| 428 | lttctl -R or after stopping tracing from the GUI, it will show an event lost |
| 429 | count. If it is the case, try using larger buffers. See lttctl --help to learn |
| 430 | how. lttv now also shows event lost messages in the console when loading a trace |
| 431 | with missing events or lost subbuffers.) |
| 432 | |
| 433 | <h3><a href="#TOCuselttvtext" name="uselttvtext">Use text mode LTTV</a></h3> |
| 434 | <p> |
| 435 | Feel free to look in /usr/local/lib/lttv/plugins to see all the text and |
| 436 | graphical plugins available. |
| 437 | <p> |
| 438 | For example, a simple trace dump in text format is available with : |
| 439 | <PRE> |
| 440 | lttv -m textDump -t /tmp/trace |
| 441 | </PRE> |
| 442 | <p> |
| 443 | See lttv -m textDump --help for detailed command line options of textDump. |
| 444 | <p> |
| 445 | It is, in the current state of the project, very useful to use "grep" on the |
| 446 | text output to filter by specific event fields. You can later copy the timestamp |
| 447 | of the events to the clipboard and paste them in the GUI by clicking on the |
| 448 | bottom right label "Current time". Support for this type of filtering should |
| 449 | be added to the filter module soon. |
| 450 | |
| 451 | <h3><a href="#TOChybrid" name="hybrid">Tracing in "Hybrid" mode</a></h3> |
| 452 | <p> |
| 453 | Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used : |
| 454 | hybrid. It can be especially useful when studying big workloads on a long period |
| 455 | of time. |
| 456 | <p> |
| 457 | When using this mode, the most important, low rate control information will be |
| 458 | recorded during all the trace by lttd (i.e. process creation/exit). The high |
| 459 | rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a |
| 460 | flight recorder buffer (now named flight-channelname_X). |
| 461 | <p> |
| 462 | The following lttctl commands take an hybrid trace : |
| 463 | <p> |
| 464 | Create trace channel, start lttd on normal channels, start tracing: |
| 465 | <PRE> |
| 466 | lttctl -C -w /tmp/trace2 -o channel.kernel.overwrite=1 trace2 |
| 467 | </PRE> |
| 468 | <p> |
| 469 | Stop tracing, start lttd on flight recorder channels, destroy trace channels : |
| 470 | <PRE> |
| 471 | lttctl -D -w /tmp/trace2 trace2 |
| 472 | </PRE> |
| 473 | <p> |
| 474 | Each "overwrite" channel is flight recorder channel. |
| 475 | |
| 476 | |
| 477 | <h3><a href="#TOCflight" name="flight">Tracing in flight recorder mode</a></h3> |
| 478 | <li>Flight recorder mode</li> |
| 479 | <p> |
| 480 | The flight recorder mode writes data into overwritten buffers for all channels, |
| 481 | including control channels, except for the facilities tracefiles. It consists of |
| 482 | setting all channels to "overwrite". |
| 483 | <p> |
| 484 | The following lttctl commands take a flight recorder trace : |
| 485 | <PRE> |
| 486 | lttctl -C -w /tmp/trace3 -o channel.all.overwrite=1 trace3 |
| 487 | ... |
| 488 | lttctl -D -w /tmp/trace3 trace3 |
| 489 | </PRE> |
| 490 | |
| 491 | <hr /> |
| 492 | |
| 493 | |
| 494 | <h2><a href="#TOCsection3" name="section3">Adding new instrumentations with the |
| 495 | markers</a></h2> |
| 496 | <p> |
| 497 | |
| 498 | <h3><a href="#TOCkerneltp" name="kerneltp">Adding kernel |
| 499 | instrumentation</a></h3> |
| 500 | |
| 501 | <p> |
| 502 | See <a |
| 503 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/markers.txt">Documentation/markers.txt</a> |
| 504 | and <a |
| 505 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=blob;f=Documentation/tracepoints.txt">Documentation/tracepoints.txt</a> in your kernel |
| 506 | tree. |
| 507 | <p> |
| 508 | Also see <a |
| 509 | href="http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=tree;f=ltt/probes">ltt/probes/</a> |
| 510 | for LTTng probe examples. |
| 511 | |
| 512 | <h3><a href="#TOCusertp" name="usertp">Adding userspace instrumentation</a></h3> |
| 513 | |
| 514 | Add new events to userspace programs with |
| 515 | <a href="http://ltt.polymtl.ca/packages/">userspace markers packages</a>. |
| 516 | Get the latest markers-userspace-*.tar.bz2 and see the Makefile and examples. It |
| 517 | allows inserting markers in executables and libraries, currently only on x86_32 |
| 518 | and x86_64. |
| 519 | See <a |
| 520 | href="http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2">markers-userspace-0.5.tar.bz2</a> or more recent. |
| 521 | |
| 522 | <p> |
| 523 | Note that a new design document for a 3rd generation of tracepoint/marker-based |
| 524 | userspace tracing is available at <a |
| 525 | href="http://ltt.polymtl.ca/svn/trunk/lttv/doc/developer/ust.html">LTTng User-space Tracing |
| 526 | Design</a>. This new infrastructure is not yet implemented. |
| 527 | |
| 528 | <p> |
| 529 | The easy quick-and-dirty way to perform userspace tracing is currently to write |
| 530 | an string to /mnt/debugfs/ltt/write_event. See <a |
| 531 | href="#userspacetracing">Userspace tracing</a> in the |
| 532 | installation for sources section of this document. |
| 533 | |
| 534 | <hr /> |
| 535 | |
| 536 | <h2><a href="#TOCsection4" name="section4">Creating Debian or RPM packages</a></h2> |
| 537 | <p> |
| 538 | |
| 539 | <h3><a href="#TOCpkgdebian" name="pkgdebian">Create custom LTTV Debian packages</a></h3> |
| 540 | |
| 541 | <PRE> |
| 542 | Use : dpkg-buildpackage -rfakeroot |
| 543 | </PRE> |
| 544 | <p> |
| 545 | You should then have your LTTV .deb files created for your architecture. |
| 546 | |
| 547 | <h3><a href="#TOCpkglttng" name="pkglttng">Create custom LTTng packages</a></h3> |
| 548 | <p> |
| 549 | For building LTTng Debian packages : |
| 550 | get the build tree with patches applies as explained in section 2. |
| 551 | |
| 552 | <PRE> |
| 553 | make menuconfig (or xconfig or config) (customize your configuration) |
| 554 | make-kpkg kernel_image |
| 555 | </PRE> |
| 556 | <p> |
| 557 | You will then see your freshly created .deb in /usr/src. Install it with |
| 558 | <PRE> |
| 559 | dpkg -i /usr/src/(image-name).deb |
| 560 | </PRE> |
| 561 | <p> |
| 562 | Then, follow the section "Editing the system wide configuration" in section 2. |
| 563 | |
| 564 | <hr /> |
| 565 | |
| 566 | <h2><a href="#TOCsection5" name="section5">Examples of LTTng use in the field</a></h2> |
| 567 | <p> |
| 568 | A few examples of successful LTTng users : |
| 569 | |
| 570 | <ul> |
| 571 | <li> Google are deploying LTTng on their servers. They want to use it to |
| 572 | monitor their production servers (with flight recorder mode tracing) |
| 573 | and to help them solve hard to reproduce problems. They have had |
| 574 | success with such tracing approach to fix "rare disk delay" issues and |
| 575 | VM-related issues presented in this article : |
| 576 | <ul> |
| 577 | <li> <a href="http://ltt.polymtl.ca/papers/bligh-Reprint.pdf">Linux Kernel |
| 578 | Debugging on Google-sized clusters at Ottawa Linux |
| 579 | Symposium 2007</a> |
| 580 | </ul> |
| 581 | <li> IBM Research have had problems with Commercial Scale-out applications, |
| 582 | which are being an increasing trend to split large server workloads. |
| 583 | They used LTTng successfully to solve a distributed filesystem-related |
| 584 | issue. It's presented in the same paper above. |
| 585 | |
| 586 | <li> Autodesk, in the development of their next-generation of Linux |
| 587 | audio/video edition applications, used LTTng extensively to solve |
| 588 | soft real-time issues they had. Also presented in the same paper. |
| 589 | |
| 590 | <li> Wind River included LTTng in their Linux distribution so their |
| 591 | clients, already familiar to Wind River own tracing solution in |
| 592 | VxWorks, car have the same kind of feature they have relied on for a |
| 593 | long time. |
| 594 | |
| 595 | <li> Montavista have integrated LTTng in their distribution for the same |
| 596 | reasons. It's used by Sony amongst others. |
| 597 | |
| 598 | <li> SuSE are currently integrating LTTng in their next SLES distribution, |
| 599 | because their clients asking for solutions which supports a kernel |
| 600 | closer to real-time need such tools to debug their problems. |
| 601 | |
| 602 | <li> A project between Ericsson, the Canadian Defense, NSERC and various |
| 603 | universities is just starting. It aims at monitoring and debugging |
| 604 | multi-core systems and provide automated and help user system behavior |
| 605 | analysis. |
| 606 | |
| 607 | <li> Siemens have been using LTTng internally for quite some time now. |
| 608 | </ul> |
| 609 | </body> |
| 610 | </html> |