ltt/branches/poly/QUICKSTART

   1 Linux Trace Toolkit Quickstart
   2 ------------------------------
   3 Author : Mathieu Desnoyers, September 2005
   4 Last update : May 14, 2007
   5
   6
   7 This document is made of four parts : the first one explains how to install
   8 LTTng and LTTV from Debian and RPM binary packages, the second one explains how
   9 to install LTTng and LTTV from sources and the third one describes the steps
  10 to follow to trace a system and view it. The fourth and last part explains
  11 briefly how to add a new trace point to the kernel and to user space
  12 applications.
  13
  14 What you will typically want is to read sections 2 and 3 : install LTTng from
  15 sources and use it.
  16
  17 These operations are made for installing the LTTng 0.6.X tracer on a
  18 linux 2.6.X kernel. You will also find instructions for installation of
  19 LTTV 0.8.x : the Linux Trace Toolkit Viewer.
  20
  21 To see the list of compatibilities between LTTng, ltt-control, LTTV, genevent
  22 and ltt-usertrace, please refer to :
  23 http://ltt.polymtl.ca > LTTng+LTTV versions compatibility
  24
  25
  26
  27 The following lttng patch is necessary to have the tracing hooks in the kernel.
  28 The following ltt-control module controls the tracing.
  29
  30 Required programs and libraries are assumed to be automatically installed in an
  31 installation with Debian or RPM packages. In the case of an installation from
  32 sources, the dependencies are listed.
  33
  34
  35 ** Current development status **
  36
  37 LTTng :
  38 supported architectures :
  39 Intel Pentium (UP/SMP) with TSC
  40 PowerPC 32 and 64 bits
  41 ARM
  42 x86_64
  43 C2 Microsystems (variant of MIPS)
  44
  45 LTTV :
  46 supported architectures :
  47 Intel i386 and better
  48 Intel 64 bits
  49 PowerPC 32 and 64 bits
  50
  51
  52 ***********************************************************
  53 ** Section 1 * Installation from Debian or RPM packages  **
  54 ***********************************************************
  55
  56 ** NOTE : RPM and Debian packages are only made once a version has been
  57    thoroughly tested. If they do not exist at the moment, please install from
  58    sources (see section 2 below). To see the list of compatibilities between
  59    LTTng, ltt-control, LTTV, genevent and lttng-modules, please refer to
  60    http://ltt.polymtl.ca > LTTng+LTTV versions compatibility
  61
  62
  63 * Install from RPM packages on Fedora Core 4 :
  64
  65 Get LTTV RPM from :
  66
  67 http://ltt.polymtl.ca/packages/fedora/RPMS
  68
  69 LTTV RPM are ready.
  70
  71 LTTng kernel and lttng-modules RPM are available for some architectures (i586,
  72 i686). Feel free to help fix the spec files to have correct lttng-modules RPM
  73 package.
  74
  75
  76 * Install from .deb packages on Debian :
  77
  78 You can use the ltt.polymtl.ca apt source to get LTTV for Debian :
  79
  80 Add the following two sources to your /etc/apt/sources.list :
  81
  82 deb http://ltt.polymtl.ca/packages/debian experimental main
  83 deb-src http://ltt.polymtl.ca/packages/debian experimental main
  84
  85
  86 * Install from precompiled binary packages (LTTV compiled only for i386, and
  87   LTTng only for i686 smp), perform the following :
  88
  89 su -
  90 apt-get update
  91 apt-get install lttv lttv-doc
  92 apt-get install kernel-image-2.6.12-rc4-mm2-lttng-0.4.2
  93 apt-get install lttng-modules-modules-2.6.12-rc4-mm2-lttng-0.4.2
  94   * note : the packages are signed by myself. I am not considered a trusted
  95     Debian source yet, so warnings are normal.
  96
  97 Then, follow the section "Editing the system wide configuration" in section 2.
  98
  99 * Create custom LTTV Debian packages
 100
 101 Binary packages are only available for i386. If you want to create your own LTTV
 102 packages for other platforms, do :
 103
 104 su -
 105 cd /usr/src
 106 apt-get source lttv
 107 cd lttv-0.6.9
 108 dpkg-buildpackage -rfakeroot
 109
 110 You should then have your LTTV .deb files created for your architecture.
 111
 112 * Create custom LTTng packages
 113
 114 For building LTTng Debian packages :
 115
 116 su -
 117 apt-get install kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
 118 cd /usr/src
 119 bzip2 -cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2.tar.bz2 | tar xvof -
 120 cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
 121 make menuconfig (or xconfig or config) (customize your configuration)
 122 make-kpkg kernel_image
 123
 124 You will then see your freshly created .deb in /usr/src. Install it with
 125 dpkg -i /usr/src/(image-name).deb
 126
 127 You will also need to create a package for the lttng-modules :
 128
 129 su -
 130 cd /usr/src
 131 apt-get source lttng-modules
 132 cd kernel-source-2.6.12-rc4-mm2-lttng-0.4.2
 133 make-kpkg --added_modules /usr/src/lttng-modules-0.3 modules_image
 134
 135 You will then see your freshly created .deb in /usr/src. Install it with
 136 dpkg -i /usr/src/lttng-modules-modules-(your version).deb
 137
 138
 139 Then, follow the section "Editing the system wide configuration" in section 2.
 140
 141
 142 ***********************************************************
 143 ** Section 2 * Installation from sources                 **
 144 ***********************************************************
 145
 146 * Prerequisites
 147
 148 Tools needed to follow the package download steps :
 149
 150 o wget
 151 o bzip2
 152 o gzip
 153 o tar
 154
 155 You have to install the standard development libraries and programs necessary
 156 to compile a kernel :
 157
 158 (from Documentation/Changes in the Linux kernel tree)
 159 o  Gnu C                  2.95.3                  # gcc --version
 160 o  Gnu make               3.79.1                  # make --version
 161 o  binutils               2.12                    # ld -v
 162 o  util-linux             2.10o                   # fdformat --version
 163 o  module-init-tools      0.9.10                  # depmod -V
 164
 165 You might also want to have libncurses5 to have the text mode kernel
 166 configuration menu, but there are alternatives.
 167
 168 Prerequisites for LTTV 0.x.x installation are :
 169
 170 gcc 3.2 or better
 171 gtk 2.4 or better development libraries
 172   (Debian : libgtk2.0, libgtk2.0-dev)
 173   (Fedora : gtk2, gtk2-devel)
 174   note : For Fedora users : this might require at least core 3 from Fedora,
 175   or you might have to compile your own GTK2 library.
 176 glib 2.4 or better development libraries
 177   (Debian : libglib2.0-0, libglib2.0-dev)
 178   (Fedora : glib2, glib2-devel)
 179 libpopt development libraries
 180   (Debian : libpopt0, libpopt-dev)
 181   (Fedora : popt)
 182 libpango development libraries
 183   (Debian : libpango1.0, libpango1.0-dev)
 184   (Fedora : pango, pango-devel)
 185 libc6 development librairies
 186   (Debian : libc6, libc6-dev)
 187   (Fedora : glibc, glibc)
 188
 189
 190 * Getting the LTTng packages
 191
 192 su -
 193 mkdir /usr/src/lttng
 194 cd /usr/src/lttng
 195 (see http://ltt.polymtl.ca/lttng for package listing)
 196 wget http://ltt.polymtl.ca/lttng/patch-2.6.X-lttng-0.x.xx.tar.bz2
 197 bzip2 -cd patch-2.6.X-lttng-0.x.xx.tar.bz2 | tar xvof -
 198
 199
 200 * Getting LTTng kernel sources
 201
 202 su -
 203 cd /usr/src
 204 wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.X.tar.bz2
 205 bzip2 -cd linux-2.6.X.tar.bz2 | tar xvof -
 206 cd linux-2.6.X
 207 - For LTTng 0.9.4- cat /usr/src/lttng/patch*-2.6.X-lttng-0.x.xx* | patch -p1
 208 - For LTTng 0.9.5+ apply the patches in the order specified in the series file,
 209   or use quilt
 210 cd ..
 211 mv linux-2.6.X linux-2.6.X-lttng-0.x.xx
 212
 213
 214 * Installing a LTTng kernel
 215
 216 su -
 217 cd /usr/src/linux-2.6.X-lttng-0.x.xx
 218 make menuconfig (or make xconfig or make config)
 219     Select the < Help > button if you are not familiar with kernel
 220     configuration.
 221     Items preceded by [*] means they has to be built into the kernel.
 222     Items preceded by [M] means they has to be built as modules.
 223     Items preceded by [ ] means they should be removed.
 224   go to the "Instrumentation Support" section
 225     Select the following options :
 226     [*] Linux Trace Toolkit Instrumentation Support
 227     <M> or <*> Linux Trace Toolkit Tracer
 228       <M> or <*> Linux Trace Toolkit Relay+DebugFS Support
 229         It makes no difference for the rest of the procedure whether the Tracer
 230         is compiled built-in or as a module.
 231     activate :
 232        [*] Align Linux Trace Toolkit Traces
 233        [*] Allow tracing from userspace
 234        <M> Linux Trace Toolkit Netlink Controller
 235        <M> Linux Trace Toolkit State Dump
 236     your choice (see < Help >) :
 237        [ ] Activate Linux Trace Toolkit Heartbeat Timer
 238     You may or may not decide to compile probes. Afterward, you will have to
 239     load the probe modules to enable tracing of their events. The probes
 240     automatically select the appropriate facilities.
 241     Static instrumentation is a more invasive type of instrumentation that gives
 242     the address taking a lock or doing a printk.
 243   Select <Exit>
 244   Select <Exit>
 245   Select <Yes>
 246 make
 247 make modules_install
 248 (if necessary, create a initrd with mkinitrd or your preferate alternative)
 249 (mkinitrd -o /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx)
 250
 251 -- on X86, X86_64
 252 make install
 253 reboot
 254 Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader.
 255
 256 -- on PowerPC
 257 cp vmlinux.strip /boot/vmlinux-2.6.X-lttng-0.x.xx
 258 cp System.map /boot/System.map-2.6.X-lttng-0.x.xx
 259 cp .config /boot/config-2.6.X-lttng-0.x.xx
 260 depmod -ae -F /boot/System.map-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 261 mkinitrd /boot/initrd.img-2.6.X-lttng-0.x.xx 2.6.X-lttng-0.x.xx
 262 (edit /etc/yaboot.conf to add a new entry pointing to your kernel : the entry
 263 that comes first is the default kernel)
 264 ybin
 265 select the right entry at the yaboot prompt (see choices : tab, select : type
 266 the kernel name followed by enter)
 267 Select the Linux 2.6.17-lttng-0.x.xx kernel in your boot loader.
 268 --
 269
 270
 271
 272 * Editing the system wide configuration
 273
 274 You must activate debugfs and specify a mount point. This is typically done in
 275 fstab such that it happens at boot time.
 276
 277 If you have never used DebugFS before, these operation would do this for you :
 278
 279 mkdir /mnt/debugfs
 280 cp /etc/fstab /etc/fstab.lttng.bkp
 281 echo "debugfs         /mnt/debugfs    debugfs rw              0       0"  >> /etc/fstab
 282
 283 then, rebooting or issuing the following command will activate debugfs :
 284
 285 mount /mnt/debugfs
 286
 287 You need to load the LTT modules to be able to control tracing from user
 288 space. This is done by issuing the following commands. Note however
 289 these commands load all LTT modules. Depending on what options you chose to
 290 compile statically, you may not need to issue all these commands.
 291
 292 modprobe ltt-control
 293 modprobe ltt-core
 294 modprobe ltt-relay
 295 modprobe ltt-tracer
 296 modprobe ltt-probe-mm
 297 modprobe ltt-probe-kernel
 298 modprobe ltt-probe-i386 (or x86_64, powerpc, ppc, arm, mips)
 299 modprobe ltt-probe-net
 300 modprobe ltt-probe-list
 301 modprobe ltt-probe-ipc
 302 modprobe ltt-probe-fs
 303
 304 If you want to have complete information about the kernel state (including all
 305 the process names), you need to load the ltt-statedump module. This is done by
 306 issuing the command :
 307
 308 modprobe ltt-statedump
 309
 310 You can automate at boot time loading the ltt-control module by :
 311
 312 cp /etc/modules /etc/modules.bkp
 313 echo ltt-control >> /etc/modules
 314 echo ltt-core >> /etc/modules
 315 echo ltt-relay >> /etc/modules
 316 echo ltt-tracer >> /etc/modules
 317 echo ltt-probe-mm >> /etc/modules
 318 echo ltt-probe-kernel >> /etc/modules
 319 echo ltt-probe-i386 >> /etc/modules (or x86_64, powerpc, ppc, arm, mips)
 320 echo ltt-probe-net >> /etc/modules
 321 echo ltt-probe-list >> /etc/modules
 322 echo ltt-probe-ipc >> /etc/modules
 323 echo ltt-probe-fs >> /etc/modules
 324 echo ltt-statedump >> /etc/modules
 325
 326 (note : if you want to probe a marker which is within a module, make sure you
 327 load the probe _after_ the module, otherwise the probe will not be able to
 328 connect itself to the marker.)
 329
 330
 331 * Getting and installing the ltt-control package (on the traced machine)
 332 (note : the ltt-control package contains lttd and lttctl. Although it has the
 333 same name as the ltt-control kernel module, they are *not* the same thing.)
 334 su -
 335 cd /usr/src
 336 wget http://ltt.polymtl.ca/lttng/ltt-control-0.x-xxxx2006.tar.gz
 337 gzip -cd ltt-control-0.x-xxxx2006.tar.gz | tar xvof -
 338 cd ltt-control-0.x-xxxx2006
 339 (refer to README to see the development libraries that must be installed on you
 340 system)
 341 ./configure
 342 make
 343 make install
 344
 345 * Getting and installing the ltt-usertrace package for user space tracing
 346 See http://ltt.polymtl.ca/ > USERSPACE TRACING QUICKSTART
 347
 348
 349 * Getting and installing the LTTV package (on the visualisation machine, same or
 350   different from the visualisation machine)
 351
 352 su -
 353 cd /usr/src
 354 wget http://ltt.polymtl.ca/packages/LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz
 355 gzip -cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006.tar.gz | tar xvof -
 356 cd LinuxTraceToolkitViewer-0.x.xx-xxxx2006
 357 (refer to README to see the development libraries that must be installed on your
 358 system)
 359 ./configure
 360 make
 361 make install
 362
 363
 364
 365
 366 ***********************************************************
 367 ** Section 3 * Using LTTng and LTTV                      **
 368 ***********************************************************
 369
 370 * Arm Linux Kernel Markers after each boot
 371
 372 ltt-armall
 373
 374 * Use graphical LTTV to control tracing and analyse traces
 375
 376 lttv-gui (or /usr/local/bin/lttv-gui)
 377   - Spot the "Tracing Control" icon : click on it
 378       (it's a traffic light icon)
 379     - enter the root password
 380     - click "start"
 381     - click "stop"
 382     - Yes
 383       * You should now see a trace
 384
 385 * Use text mode LTTng to control tracing
 386
 387 The tracing can be controlled from a terminal by using the lttctl command (as
 388 root).
 389
 390 Start tracing :
 391
 392 lttctl -n trace -d -l /mnt/debugfs/ltt -t /tmp/trace
 393
 394 Stop tracing and destroy trace channels :
 395
 396 lttctl -n trace -R
 397
 398 see lttctl --help for details.
 399
 400 (note : to see if the buffers has been filled, look at the dmesg output after
 401 lttctl -R or after stopping tracing from the GUI, it will show an event lost
 402 count. If it is the case, try using larger buffers. See lttctl --help to learn
 403 how.)
 404
 405 * Use text mode LTTV
 406
 407 Feel free to look in /usr/local/lib/lttv/plugins to see all the text and
 408 graphical plugins available.
 409
 410 For example, a simple trace dump in text format is available with :
 411
 412 lttv -m textDump -t /tmp/trace
 413
 414 see lttv -m textDump --help for detailed command line options of textDump.
 415
 416 It is, in the current state of the project, very useful to use "grep" on the
 417 text output to filter by specific event fields. You can later copy the timestamp
 418 of the events to the clipboard and paste them in the GUI by clicking on the
 419 bottom right label "Current time". Support for this type of filtering should
 420 be added to the filter module soon.
 421
 422 * Hybrid mode
 423
 424 Starting from LTTng 0.5.105 and ltt-control 0.20, a new mode can be used :
 425 hybrid. It can be especially useful when studying big workloads on a long period
 426 of time.
 427
 428 When using this mode, the most important, low rate control information will be
 429 recorded during all the trace by lttd (i.e. process creation/exit). The high
 430 rate information (i.e. interrupt/traps/syscall entry/exit) will be kept in a
 431 flight recorder buffer (now named flight-channelname_X).
 432
 433 The following lttctl commands take an hybrid trace :
 434
 435 Create trace channel, start lttd on normal channels, start tracing:
 436 lttctl -n tracename -d -l /mnt/relayfs/ltt -t /tmp/trace1 -m hybrid
 437
 438 Stop tracing, start lttd on flight recorder channels, destroy trace channels :
 439 lttctl -n tracename -f -l /mnt/relayfs/ltt -t /tmp/trace1 -m hybrid
 440
 441
 442 We will need to tweak what we consider "important" medium rate events. For
 443 instance, thread branding events are actually considered a "high rate" event
 444 when it should be considered "medium rate". The same should apply for the
 445 state dump process enumeration.
 446
 447 * Flight recorder mode
 448
 449 The flight recorder mode writes data into overwritten buffers for all channels,
 450 including control channels, except for the facilities tracefiles.
 451
 452 The following lttctl commands take a flight recorder trace :
 453
 454 lttctl -n trace -c -m flight
 455 lttd -n -d -t /tmp/trace -c /sys/kernel/debug/ltt
 456 lttctl -n trace -s
 457 .. do stuff
 458 lttctl -n trace -q
 459 lttd -f -d -t /tmp/trace -c /sys/kernel/debug/ltt
 460 lttctl -m trace -r
 461
 462
 463 ***********************************************************
 464 ** Section 4 * Adding new instrumentations with genevent **
 465 ***********************************************************
 466
 467 * Getting and installing genevent
 468
 469 su -
 470 cd /usr/src
 471 wget http://ltt.polymtl.ca/packages/genevent-0.xx.tar.gz
 472 gzip -cd genevent-0.xx.tar.gz | tar xvof -
 473 cd genevent-0.xx
 474 make
 475 make install
 476
 477
 478 * Add new events to the kernel with genevent (deprecated in LTTng 0.9.x)
 479
 480 su -
 481 cd /usr/local/share/ltt-control/facilities
 482 cp process.xml yourfacility.xml
 483   * edit yourfacility.xml to fit your needs.
 484 cd /tmp
 485 /usr/local/bin/genevent /usr/local/share/ltt-control/facilities/yourfacility.xml
 486 cp ltt-facility-yourfacility.h ltt-facility-id-yourfacility.h \
 487          /usr/src/linux-2.6.17-lttng-0.x.xx8/include/ltt
 488 cp ltt-facility-loader-yourfacility.c ltt-facility-loader-yourfacility.h \
 489          /usr/src/linux-2.6.17-lttng-0.x.xx/ltt/facilities
 490   * edit the kernel file you want to instrument to add a marker to it. See
 491     include/linux/marker.h.
 492   * create a dynamically loadable probe. See ltt/probes for examples. The probe
 493     will be connected to your marker and will typically call the logging
 494     functions found in the header file you created with genevent.
 495
 496 * Add new kernel events
 497
 498 *Important* note : in its current state, LTTng and LTTV needs the programmer
 499 to keep the marker/probe format string and the XML description of the
 500 event data types in sync by hand. Failure to do so will result in errors in
 501 LTTV.
 502
 503 See the markers documentation to see how to describe the marker. You will need
 504 to clone probe modules found in ltt/probes to connect them to the markers so
 505 that the information can be recorded in the trace.
 506
 507 * Add new events to userspace programs with genevent
 508 See http://ltt.polymtl.ca/ > USERSPACE TRACING QUICKSTART
 509
 510 User-space tracing still uses genevent, which is subject to change in a near
 511 future.