| 1 | .TH "LTTNG" "1" "February 9, 2012" "" "" |
| 2 | |
| 3 | .SH "NAME" |
| 4 | lttng \(em LTTng 2.0 tracer control command line tool |
| 5 | |
| 6 | .SH "SYNOPSIS" |
| 7 | |
| 8 | .PP |
| 9 | .nf |
| 10 | lttng [OPTIONS] <COMMAND> |
| 11 | .fi |
| 12 | .SH "DESCRIPTION" |
| 13 | |
| 14 | .PP |
| 15 | The LTTng project aims at providing highly efficient tracing tools for Linux. |
| 16 | It's tracers help tracking down performance issues and debugging problems |
| 17 | involving multiple concurrent processes and threads. Tracing across multiple |
| 18 | systems is also possible. |
| 19 | |
| 20 | The \fBlttng\fP command line tool from lttng-tools package is used to control |
| 21 | both kernel and user-space tracing. Every interactions with the tracer should |
| 22 | be done by this tool or by the liblttng-ctl provided with the lttng-tools |
| 23 | package. |
| 24 | |
| 25 | LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry, |
| 26 | which permits you to interact with multiple tracers (kernel and user-space) |
| 27 | inside the same container, a tracing session. Traces can be gathered from the |
| 28 | kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading |
| 29 | those traces is done using the babeltrace(1) text viewer. |
| 30 | |
| 31 | In order to trace the kernel, the session daemon needs to be running as root. |
| 32 | LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is |
| 33 | in that group can interact with the root session daemon and thus trace the |
| 34 | kernel. Session daemons can co-exist meaning that you can have a session daemon |
| 35 | running as Alice that can be use to trace her applications along side with a |
| 36 | root daemon or even a Bob daemon. We highly recommand to start the session |
| 37 | daemon at boot time for stable and long term tracing. |
| 38 | |
| 39 | Every user-space applications instrumented with lttng-ust(3), will |
| 40 | automatically register to the session daemon. This feature gives you the |
| 41 | ability to list available traceable applications and tracepoints on a per user |
| 42 | basis. (See \fBlist\fP command). |
| 43 | .SH "OPTIONS" |
| 44 | |
| 45 | .PP |
| 46 | This program follow the usual GNU command line syntax with long options starting with |
| 47 | two dashes. Below is a summary of the available options. |
| 48 | .PP |
| 49 | |
| 50 | .TP |
| 51 | .BR "-h, --help" |
| 52 | Show summary of possible options and commands. |
| 53 | .TP |
| 54 | .BR "-v, --verbose" |
| 55 | Increase verbosity. |
| 56 | FIXME : details (-v : sessiond verbose, -vv : consumerd verbose, etc) ? |
| 57 | .TP |
| 58 | .BR "-q, --quiet" |
| 59 | Suppress all messages (even errors). |
| 60 | .TP |
| 61 | .BR "-g, --group NAME" |
| 62 | Set unix tracing group name. (default: tracing) |
| 63 | .TP |
| 64 | .BR "-n, --no-sessiond" |
| 65 | Don't automatically spawn a session daemon. |
| 66 | .TP |
| 67 | .BR "--sessiond-path" |
| 68 | Set session daemon full binary path. |
| 69 | .TP |
| 70 | .BR "--list-options" |
| 71 | Simple listing of lttng options. |
| 72 | .TP |
| 73 | .BR "--list-commands" |
| 74 | Simple listing of lttng commands. |
| 75 | .SH "COMMANDS" |
| 76 | |
| 77 | .TP |
| 78 | \fBadd-context\fP |
| 79 | .nf |
| 80 | Add context to event(s) and/or channel(s). |
| 81 | |
| 82 | A context is basically extra information appended to a channel or event. For |
| 83 | instance, you could ask the tracer to add the PID information within the |
| 84 | "sched_switch" kernel event. You can also add performance monitoring unit |
| 85 | counters (perf PMU) using the perf kernel API). |
| 86 | |
| 87 | For example, this command will add the context information 'prio' and two perf |
| 88 | counters (hardware branch misses and cache misses), to all events in the trace |
| 89 | data output: |
| 90 | |
| 91 | # lttng add-context -k -t prio -t perf:branch-misses -t perf:cache-misses |
| 92 | |
| 93 | Please take a look at the help (-h/--help) for a detailed list of available |
| 94 | contexts. |
| 95 | |
| 96 | If no channel and no event is given (-c/-e), the context is added to all |
| 97 | channels (which applies automatically to all events in that channel). Otherwise |
| 98 | the context will be added only to the channel (-c) and/or event (-e) indicated. |
| 99 | |
| 100 | If \fB-s, --session\fP is omitted, the session name is taken from the .lttngrc |
| 101 | file. |
| 102 | .fi |
| 103 | |
| 104 | .B OPTIONS: |
| 105 | |
| 106 | .nf |
| 107 | -h, --help |
| 108 | Show summary of possible options and commands. |
| 109 | -s, --session NAME |
| 110 | Apply on session name. |
| 111 | -c, --channel NAME |
| 112 | Apply on channel name. |
| 113 | -e, --event NAME |
| 114 | Apply on event name. |
| 115 | -k, --kernel |
| 116 | Apply for the kernel tracer |
| 117 | -u, --userspace |
| 118 | Apply for the user-space tracer |
| 119 | -t, --type TYPE |
| 120 | Context type. You can repeat this option on the command line. Please |
| 121 | use "lttng add-context -h" to list all available types. |
| 122 | .fi |
| 123 | |
| 124 | .IP |
| 125 | |
| 126 | .IP "\fBcalibrate\fP" |
| 127 | .nf |
| 128 | Quantify LTTng overhead |
| 129 | |
| 130 | The LTTng calibrate command can be used to find out the combined average |
| 131 | overhead of the LTTng tracer and the instrumentation mechanisms used. This |
| 132 | overhead can be calibrated in terms of time or using any of the PMU performance |
| 133 | counter available on the system. |
| 134 | |
| 135 | For now, the only calibration implemented is that of the kernel function |
| 136 | instrumentation (kretprobes). |
| 137 | |
| 138 | * Calibrate kernel function instrumentation |
| 139 | |
| 140 | Let's use an example to show this calibration. We use an i7 processor with 4 |
| 141 | general-purpose PMU registers. This information is available by issuing dmesg, |
| 142 | looking for "generic registers". |
| 143 | |
| 144 | This sequence of commands will gather a trace executing a kretprobe hooked on |
| 145 | an empty function, gathering PMU counters LLC (Last Level Cache) misses |
| 146 | information (see lttng add-context --help to see the list of available PMU |
| 147 | counters). |
| 148 | |
| 149 | # lttng create calibrate-function |
| 150 | # lttng enable-event calibrate --kernel --function lttng_calibrate_kretprobe |
| 151 | # lttng add-context --kernel -t perf:LLC-load-misses -t perf:LLC-store-misses \\ |
| 152 | -t perf:LLC-prefetch-misses |
| 153 | # lttng start |
| 154 | # for a in $(seq 1 10); do \\ |
| 155 | lttng calibrate --kernel --function; |
| 156 | done |
| 157 | # lttng destroy |
| 158 | # babeltrace $(ls -1drt ~/lttng-traces/calibrate-function-* | tail -n 1) |
| 159 | |
| 160 | The output from babeltrace can be saved to a text file and opened in a |
| 161 | spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between |
| 162 | consecutive "calibrate_entry" and "calibrate_return" events. Note that these |
| 163 | counters are per-CPU, so scheduling events would need to be present to account |
| 164 | for migration between CPU. Therefore, for calibration purposes, only events |
| 165 | staying on the same CPU must be considered. |
| 166 | |
| 167 | The average result, for the i7, on 10 samples: |
| 168 | |
| 169 | Average Std.Dev. |
| 170 | perf_LLC_load_misses: 5.0 0.577 |
| 171 | perf_LLC_store_misses: 1.6 0.516 |
| 172 | perf_LLC_prefetch_misses: 9.0 14.742 |
| 173 | |
| 174 | As we can notice, the load and store misses are relatively stable across runs |
| 175 | (their standard deviation is relatively low) compared to the prefetch misses. |
| 176 | We can conclude from this information that LLC load and store misses can be |
| 177 | accounted for quite precisely, but prefetches within a function seems to behave |
| 178 | too erratically (not much causality link between the code executed and the CPU |
| 179 | prefetch activity) to be accounted for. |
| 180 | .fi |
| 181 | |
| 182 | .B OPTIONS: |
| 183 | |
| 184 | .nf |
| 185 | -h, --help |
| 186 | Show summary of possible options and commands. |
| 187 | -k, --kernel |
| 188 | Apply for the kernel tracer |
| 189 | -u, --userspace |
| 190 | Apply for the user-space tracer |
| 191 | --function |
| 192 | Dynamic function entry/return probe (default) |
| 193 | .fi |
| 194 | |
| 195 | .IP |
| 196 | |
| 197 | .IP "\fBcreate\fP [OPTIONS] [NAME] |
| 198 | .nf |
| 199 | Create tracing session. |
| 200 | |
| 201 | A tracing session contains channel(s) which contains event(s). It is domain |
| 202 | agnostic meaning that you can enable channels and events for either the |
| 203 | user-space tracer and/or the kernel tracer. It acts like a container |
| 204 | aggregating multiple tracing sources. |
| 205 | |
| 206 | On creation, a \fB.lttngrc\fP file is created in your $HOME directory |
| 207 | containing the current session name. If NAME is omitted, a session name is |
| 208 | automatically created having this form: 'auto-yyyymmdd-hhmms'. |
| 209 | |
| 210 | If no \fB-o, --output\fP is specified, the traces will be written in |
| 211 | $HOME/lttng-traces. |
| 212 | .fi |
| 213 | |
| 214 | .B OPTIONS: |
| 215 | |
| 216 | .nf |
| 217 | -h, --help |
| 218 | Show summary of possible options and commands. |
| 219 | --list-options |
| 220 | Simple listing of options |
| 221 | -o, --output PATH |
| 222 | Specify output path for traces |
| 223 | .fi |
| 224 | |
| 225 | .IP |
| 226 | |
| 227 | .IP "\fBdestroy\fP [OPTIONS] [NAME]" |
| 228 | .nf |
| 229 | Teardown tracing session |
| 230 | |
| 231 | Free memory on the session daemon and tracer side. It's gone! |
| 232 | |
| 233 | If NAME is omitted, the session name is taken from the .lttngrc file. |
| 234 | .fi |
| 235 | |
| 236 | .B OPTIONS: |
| 237 | |
| 238 | .nf |
| 239 | -h, --help |
| 240 | Show summary of possible options and commands. |
| 241 | --list-options |
| 242 | Simple listing of options |
| 243 | .fi |
| 244 | |
| 245 | .IP |
| 246 | |
| 247 | .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]" |
| 248 | .nf |
| 249 | Enable tracing channel |
| 250 | |
| 251 | If \fB-s, --session\fP is omitted, the session name is taken from the .lttngrc |
| 252 | file. |
| 253 | .fi |
| 254 | |
| 255 | .B OPTIONS: |
| 256 | |
| 257 | .nf |
| 258 | -h, --help |
| 259 | Show this help |
| 260 | --list-options |
| 261 | Simple listing of options |
| 262 | -s, --session |
| 263 | Apply on session name |
| 264 | -k, --kernel |
| 265 | Apply to the kernel tracer |
| 266 | -u, --userspace |
| 267 | Apply to the user-space tracer |
| 268 | |
| 269 | --discard |
| 270 | Discard event when subbuffers are full (default) |
| 271 | --overwrite |
| 272 | Flight recorder mode : overwrites events when subbuffers are full |
| 273 | --subbuf-size |
| 274 | Subbuffer size in bytes (default: 4096, kernel default: 262144) |
| 275 | --num-subbuf |
| 276 | Number of subbufers (default: 8, kernel default: 4) |
| 277 | --switch-timer |
| 278 | Switch subbuffer timer interval in usec (default: 0) |
| 279 | --read-timer |
| 280 | Read timer interval in usec (default: 200) |
| 281 | .fi |
| 282 | |
| 283 | .IP |
| 284 | |
| 285 | .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]" |
| 286 | .nf |
| 287 | Enable tracing event |
| 288 | |
| 289 | A tracing event is always assigned to a channel. If \fB-c, --channel\fP is |
| 290 | omitted, a default channel named '\fBchannel0\fP' is created and the event is |
| 291 | added to it. For the user-space tracer, using \fB-a, --all\fP is the same as |
| 292 | using the wildcard "*". |
| 293 | |
| 294 | If \fB-s, --session\fP is omitted, the session name is taken from the .lttngrc |
| 295 | file. |
| 296 | .fi |
| 297 | |
| 298 | .B OPTIONS: |
| 299 | |
| 300 | .nf |
| 301 | -h, --help |
| 302 | Show summary of possible options and commands. |
| 303 | --list-options |
| 304 | Simple listing of options |
| 305 | -s, --session |
| 306 | Apply on session name |
| 307 | -c, --channel |
| 308 | Apply on channel name |
| 309 | -a, --all |
| 310 | Enable all tracepoints |
| 311 | -k, --kernel |
| 312 | Apply for the kernel tracer |
| 313 | -u, --userspace |
| 314 | Apply for the user-space tracer |
| 315 | |
| 316 | --tracepoint |
| 317 | Tracepoint event (default) |
| 318 | - userspace tracer supports wildcards at end of string. Don't forget to |
| 319 | quote to deal with bash expansion. |
| 320 | e.g.: |
| 321 | "*" |
| 322 | "app_component:na*" |
| 323 | --loglevel |
| 324 | Tracepoint loglevel |
| 325 | --probe [addr | symbol | symbol+offset] |
| 326 | Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...) |
| 327 | or hexadecimal (0xNNN...) |
| 328 | --function [addr | symbol | symbol+offset] |
| 329 | Dynamic function entry/return probe. Addr and offset can be octal |
| 330 | (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...) |
| 331 | --syscall |
| 332 | System call event |
| 333 | Enabling syscalls tracing (kernel tracer), you will not be able to disable them |
| 334 | with disable-event. This is a known limitation. You can disable the entire |
| 335 | channel to do the trick. |
| 336 | .fi |
| 337 | |
| 338 | .IP "\fBdisable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]" |
| 339 | .nf |
| 340 | Disable tracing channel |
| 341 | |
| 342 | Disabling a channel makes all event(s) in that channel to stop tracing. You can |
| 343 | enable it back by calling \fBlttng enable-channel NAME\fP again. |
| 344 | |
| 345 | If \fB-s, --session\fP is omitted, the session name is taken from the .lttngrc |
| 346 | file. |
| 347 | .fi |
| 348 | |
| 349 | .B OPTIONS: |
| 350 | |
| 351 | .nf |
| 352 | -h, --help |
| 353 | Show summary of possible options and commands. |
| 354 | --list-options |
| 355 | Simple listing of options |
| 356 | -s, --session |
| 357 | Apply on session name |
| 358 | -k, --kernel |
| 359 | Apply for the kernel tracer |
| 360 | -u, --userspace |
| 361 | Apply for the user-space tracer |
| 362 | .fi |
| 363 | |
| 364 | .IP "\fBdisable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]" |
| 365 | .nf |
| 366 | Disable tracing event |
| 367 | |
| 368 | The event, once disabled, can be re-enabled by calling \fBlttng enable-event |
| 369 | NAME\fP again. |
| 370 | |
| 371 | If \fB-s, --session\fP is omitted, the session name is taken from the .lttngrc |
| 372 | file. |
| 373 | .fi |
| 374 | |
| 375 | .B OPTIONS: |
| 376 | |
| 377 | .nf |
| 378 | -h, --help |
| 379 | Show summary of possible options and commands. |
| 380 | --list-options |
| 381 | Simple listing of options |
| 382 | -s, --session |
| 383 | Apply on session name |
| 384 | -k, --kernel |
| 385 | Apply for the kernel tracer |
| 386 | -u, --userspace |
| 387 | Apply for the user-space tracer |
| 388 | .fi |
| 389 | |
| 390 | .IP "\fBlist\fP [-k|-u] [SESSION [SESSION_OPTIONS]]" |
| 391 | .nf |
| 392 | List tracing session informations. |
| 393 | |
| 394 | With no arguments, it will list available tracing session(s). |
| 395 | |
| 396 | With -k alone, it will list all available kernel events (except the system |
| 397 | calls events). |
| 398 | With -u alone, it will list all available user-space events from registered |
| 399 | applications. Here is an example of 'lttng list -u': |
| 400 | |
| 401 | PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello |
| 402 | ust_tests_hello:tptest_sighandler (type: tracepoint) |
| 403 | ust_tests_hello:tptest (type: tracepoint) |
| 404 | |
| 405 | You can now enable any event listed by using the name : |
| 406 | \fBust_tests_hello:tptest\fP. |
| 407 | .fi |
| 408 | |
| 409 | .B OPTIONS: |
| 410 | |
| 411 | .nf |
| 412 | -h, --help |
| 413 | Show summary of possible options and commands. |
| 414 | --list-options |
| 415 | Simple listing of options |
| 416 | -k, --kernel |
| 417 | Select kernel domain (FIXME : apparition de la notion de "domain" ici) |
| 418 | -u, --userspace |
| 419 | Select user-space domain. |
| 420 | |
| 421 | Session options: |
| 422 | -c, --channel NAME |
| 423 | List details of a channel |
| 424 | -d, --domain |
| 425 | List available domain(s) |
| 426 | .fi |
| 427 | |
| 428 | .IP "\fBset-session\fP NAME" |
| 429 | .nf |
| 430 | Set current session name |
| 431 | |
| 432 | Will change the session name in the .lttngrc file. |
| 433 | .fi |
| 434 | |
| 435 | .B OPTIONS: |
| 436 | |
| 437 | .nf |
| 438 | -h, --help |
| 439 | Show summary of possible options and commands. |
| 440 | --list-options |
| 441 | Simple listing of options |
| 442 | .fi |
| 443 | |
| 444 | .IP |
| 445 | |
| 446 | .IP "\fBstart\fP [OPTIONS] [NAME]" |
| 447 | .nf |
| 448 | Start tracing |
| 449 | |
| 450 | It will start tracing for all tracers for a specific tracing session. |
| 451 | |
| 452 | If NAME is omitted, the session name is taken from the .lttngrc file. |
| 453 | .fi |
| 454 | |
| 455 | .B OPTIONS: |
| 456 | |
| 457 | .nf |
| 458 | -h, --help |
| 459 | Show summary of possible options and commands. |
| 460 | --list-options |
| 461 | Simple listing of options |
| 462 | .fi |
| 463 | |
| 464 | .IP |
| 465 | |
| 466 | .IP "\fBstop\fP [OPTIONS] [NAME]" |
| 467 | .nf |
| 468 | Stop tracing |
| 469 | |
| 470 | It will stop tracing for all tracers for a specific tracing session. |
| 471 | |
| 472 | If NAME is omitted, the session name is taken from the .lttngrc file. |
| 473 | .fi |
| 474 | |
| 475 | .B OPTIONS: |
| 476 | |
| 477 | .nf |
| 478 | -h, --help |
| 479 | Show summary of possible options and commands. |
| 480 | --list-options |
| 481 | Simple listing of options |
| 482 | .fi |
| 483 | |
| 484 | .IP |
| 485 | |
| 486 | .IP "\fBversion\fP" |
| 487 | .nf |
| 488 | Show version information |
| 489 | .fi |
| 490 | |
| 491 | .B OPTIONS: |
| 492 | |
| 493 | .nf |
| 494 | -h, --help |
| 495 | Show summary of possible options and commands. |
| 496 | --list-options |
| 497 | Simple listing of options |
| 498 | .fi |
| 499 | |
| 500 | .IP |
| 501 | |
| 502 | .IP "\fBview\fP [SESSION_NAME] [OPTIONS]" |
| 503 | .nf |
| 504 | View traces of a tracing session |
| 505 | |
| 506 | By default, the babeltrace viewer will be used for text viewing. |
| 507 | |
| 508 | The SESSION_NAME is an optional session name. If not specified, lttng will get |
| 509 | it from the configuration file (.lttngrc). |
| 510 | .fi |
| 511 | |
| 512 | .B OPTIONS: |
| 513 | |
| 514 | .nf |
| 515 | -h, --help |
| 516 | Show this help |
| 517 | --list-options |
| 518 | Simple listing of options |
| 519 | -t, --trace-path PATH |
| 520 | Trace directory path for the viewer |
| 521 | -e, --viewer CMD |
| 522 | Specify viewer and/or options to use |
| 523 | This will completely override the default viewers so |
| 524 | please make sure to specify the full command. The trace |
| 525 | directory path of the session will be appended at the end |
| 526 | to the arguments |
| 527 | .fi |
| 528 | |
| 529 | .SH "ENVIRONMENT VARIABLES" |
| 530 | |
| 531 | .PP |
| 532 | Note that all command line options override environment variables. |
| 533 | .PP |
| 534 | |
| 535 | .PP |
| 536 | .IP "LTTNG_SESSIOND_PATH_ENV" |
| 537 | Allows to specify the full session daemon binary path to lttng command line |
| 538 | tool. You can also use --sessiond-path option having the same effect. |
| 539 | .SH "SEE ALSO" |
| 540 | |
| 541 | .PP |
| 542 | babeltrace(1), lttng-ust(3), lttng-sessiond(8) |
| 543 | .PP |
| 544 | .SH "BUGS" |
| 545 | |
| 546 | .PP |
| 547 | No show stopper bugs known yet at this stable version. |
| 548 | |
| 549 | If you encounter any issues or usability problem, please report it on our |
| 550 | mailing list <lttng-dev@lists.lttng.org> to help improve this project. |
| 551 | .SH "CREDITS" |
| 552 | |
| 553 | .PP |
| 554 | lttng is distributed under the GNU public license version 2. See the file |
| 555 | COPYING for details. |
| 556 | .PP |
| 557 | A Web site is available at http://lttng.org for more information on the LTTng |
| 558 | project. |
| 559 | .PP |
| 560 | You can also find our git tree at http://git.lttng.org. |
| 561 | .PP |
| 562 | Mailing lists for support and development: <lttng-dev@lists.lttng.org>. |
| 563 | .PP |
| 564 | You can find us on IRC server irc.oftc.net (OFTC) in #lttng. |
| 565 | .PP |
| 566 | .SH "THANKS" |
| 567 | |
| 568 | .PP |
| 569 | Thanks to Yannick Brosseau without whom this project would never have been so |
| 570 | lean and mean! Also thanks to the Ericsson teams working on tracing which |
| 571 | helped us greatly with detailled bug reports and unusual test cases. |
| 572 | |
| 573 | Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA |
| 574 | maintainer) and Jon Bernard for our Debian packages. |
| 575 | |
| 576 | Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de |
| 577 | Montreal for the LTTng journey. |
| 578 | .pp |
| 579 | .SH "AUTHORS" |
| 580 | |
| 581 | .PP |
| 582 | lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and |
| 583 | David Goulet. More people have since contributed to it. It is currently |
| 584 | maintained by David Goulet <dgoulet@efficios.com>. |
| 585 | .PP |