2.13/lttng-docs-2.13.txt

   1 The LTTng Documentation
   2 =======================
   3 Philippe Proulx <pproulx@efficios.com>
   4 v2.13, 28 November 2023
   5
   6
   7 include::../common/copyright.txt[]
   8
   9
  10 include::../common/welcome.txt[]
  11
  12
  13 include::../common/audience.txt[]
  14
  15
  16 [[chapters]]
  17 === What's in this documentation?
  18
  19 The LTTng Documentation is divided into the following sections:
  20
  21 * ``**<<nuts-and-bolts,Nuts and bolts>>**'' explains the
  22   rudiments of software tracing and the rationale behind the
  23   LTTng project.
  24 +
  25 Skip this section if you’re familiar with software tracing and with the
  26 LTTng project.
  27
  28 * ``**<<installing-lttng,Installation>>**'' describes the steps to
  29   install the LTTng packages on common Linux distributions and from
  30   their sources.
  31 +
  32 Skip this section if you already properly installed LTTng on your target
  33 system.
  34
  35 * ``**<<getting-started,Quick start>>**'' is a concise guide to
  36   get started quickly with LTTng kernel and user space tracing.
  37 +
  38 We recommend this section if you're new to LTTng or to software tracing
  39 in general.
  40 +
  41 Skip this section if you're not new to LTTng.
  42
  43 * ``**<<core-concepts,Core concepts>>**'' explains the concepts at
  44   the heart of LTTng.
  45 +
  46 It's a good idea to become familiar with the core concepts
  47 before attempting to use the toolkit.
  48
  49 * ``**<<plumbing,Components of LTTng>>**'' describes the various
  50   components of the LTTng machinery, like the daemons, the libraries,
  51   and the command-line interface.
  52
  53 * ``**<<instrumenting,Instrumentation>>**'' shows different ways to
  54   instrument user applications and the Linux kernel for LTTng tracing.
  55 +
  56 Instrumenting source code is essential to provide a meaningful
  57 source of events.
  58 +
  59 Skip this section if you don't have a programming background.
  60
  61 * ``**<<controlling-tracing,Tracing control>>**'' is divided into topics
  62   which demonstrate how to use the vast array of features that
  63   LTTng{nbsp}{revision} offers.
  64
  65 * ``**<<reference,Reference>>**'' contains API reference tables.
  66
  67 * ``**<<glossary,Glossary>>**'' is a specialized dictionary of terms
  68   related to LTTng or to the field of software tracing.
  69
  70
  71 include::../common/convention.txt[]
  72
  73
  74 include::../common/acknowledgements.txt[]
  75
  76
  77 [[whats-new]]
  78 == What's new in LTTng{nbsp}{revision}?
  79
  80 LTTng{nbsp}{revision} bears the name _Nordicité_, the product of a
  81 collaboration between https://champlibre.co/[Champ Libre] and
  82 https://www.boreale.com/[Boréale]. This farmhouse IPA is brewed with
  83 https://en.wikipedia.org/wiki/Kveik[Kveik] yeast and Québec-grown
  84 barley, oats, and juniper branches. The result is a remarkable, fruity,
  85 hazy golden IPA that offers a balanced touch of resinous and woodsy
  86 bitterness.
  87
  88 New features and changes in LTTng{nbsp}{revision}:
  89
  90 General::
  91 +
  92 * The LTTng trigger API of <<liblttng-ctl-lttng,`liblttng-ctl`>> now
  93   offers the ``__event rule matches__'' condition (an <<event-rule,event
  94   rule>> matches an event) as well as the following new actions:
  95 +
  96 --
  97 * <<basic-tracing-session-control,Start or stop>> a recording session.
  98 * <<session-rotation,Archive the current trace chunk>> of a
  99   recording session (rotate).
 100 * <<taking-a-snapshot,Take a snapshot>> of a recording session.
 101 --
 102 +
 103 As a reminder, a <<trigger,trigger>> is a condition-actions pair. When
 104 the condition of a trigger is satisfied, LTTng attempts to execute its
 105 actions.
 106 +
 107 This feature is also available with the new man:lttng-add-trigger(1),
 108 man:lttng-remove-trigger(1), and man:lttng-list-triggers(1)
 109 <<lttng-cli,cmd:lttng>> commands.
 110 +
 111 Starting from LTTng{nbsp}{revision}, a trigger may have more than one
 112 action.
 113 +
 114 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
 115 trigger to a session daemon>>” to learn more.
 116
 117 * The LTTng <<lttng-ust,user space>> and <<lttng-modules,kernel>>
 118   tracers offer the new namespace context field `time_ns`, which is the
 119   inode number, in the proc file system, of the current clock namespace.
 120 +
 121 See man:lttng-add-context(1), man:lttng-ust(3), and
 122 man:time_namespaces(7).
 123
 124 * The link:/man[manual pages] of LTTng-tools now have a terminology and
 125   style which match the LTTng Documentation, many fixes, more internal
 126   and manual page links, clearer lists and procedures, superior
 127   consistency, and usage examples.
 128 +
 129 The new man:lttng-event-rule(7) manual page explains the new, common
 130 way to specify an event rule on the command line.
 131 +
 132 The new man:lttng-concepts(7) manual page explains the core concepts of
 133 LTTng. Its contents is essentially the ``<<core-concepts,Core
 134 concepts>>'' section of this documentation, but more adapted to the
 135 manual page style.
 136
 137 User space tracing::
 138 +
 139 [IMPORTANT]
 140 ====
 141 The major version part of the `liblttng-ust`
 142 https://en.wikipedia.org/wiki/Soname[soname] is bumped, which means you
 143 **must recompile** your instrumented applications/libraries and
 144 <<tracepoint-provider,tracepoint provider packages>> to use
 145 LTTng-UST{nbsp}{revision}.
 146
 147 This change became a necessity to clean up the library and for
 148 `liblttng-ust` to stop exporting private symbols.
 149
 150 Also, LTTng{nbsp}{revision} prepends the `lttng_ust_` and `LTTNG_UST_`
 151 prefix to all public macro/definition/function names to offer a
 152 consistent API namespace. The LTTng{nbsp}2.12 API is still available;
 153 see the ``Compatibility with previous APIs'' section of
 154 man:lttng-ust(3).
 155 ====
 156 +
 157 Other notable changes:
 158 +
 159 * The `liblttng-ust` C{nbsp}API offers the new man:lttng_ust_vtracef(3)
 160   and man:lttng_ust_vtracelog(3) macros which are to
 161   man:lttng_ust_tracef(3) and man:lttng_ust_tracelog(3) what
 162   man:vprintf(3) is to man:printf(3).
 163
 164 * LTTng-UST now only depends on https://liburcu.org/[`liburcu`] at build
 165   time, not at run time.
 166
 167 Kernel tracing::
 168 +
 169 * The preferred display base of event record integer fields which
 170   contain memory addresses is now hexadecimal instead of decimal.
 171
 172 * The `pid` field is removed from `lttng_statedump_file_descriptor`
 173   event records and the `file_table_address` field is added.
 174 +
 175 This new field is the address of the `files_struct` structure which
 176 contains the file descriptor.
 177 +
 178 See the
 179 ``https://github.com/lttng/lttng-modules/commit/e7a0ca7205fd4be7c829d171baa8823fe4784c90[statedump: introduce `file_table_address`]''
 180 patch to learn more.
 181
 182 * The `flags` field of `syscall_entry_clone` event records is now a
 183   structure containing two enumerations (exit signal and options).
 184 +
 185 This change makes the flag values more readable and meaningful.
 186 +
 187 See the
 188 ``https://github.com/lttng/lttng-modules/commit/d775625e2ba4825b73b5897e7701ad6e2bdba115[syscalls: Make `clone()`'s `flags` field a 2 enum struct]''
 189 patch to learn more.
 190
 191 * The memory footprint of the kernel tracer is improved: the latter only
 192   generates metadata for the specific system call recording event rules
 193   that you <<enabling-disabling-events,create>>.
 194
 195
 196 [[nuts-and-bolts]]
 197 == Nuts and bolts
 198
 199 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
 200 generation_ is a modern toolkit for tracing Linux systems and
 201 applications. So your first question might be:
 202 **what is tracing?**
 203
 204
 205 [[what-is-tracing]]
 206 === What is tracing?
 207
 208 As the history of software engineering progressed and led to what
 209 we now take for granted--complex, numerous and
 210 interdependent software applications running in parallel on
 211 sophisticated operating systems like Linux--the authors of such
 212 components, software developers, began feeling a natural
 213 urge to have tools that would ensure the robustness and good performance
 214 of their masterpieces.
 215
 216 One major achievement in this field is, inarguably, the
 217 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
 218 an essential tool for developers to find and fix bugs. But even the best
 219 debugger won't help make your software run faster, and nowadays, faster
 220 software means either more work done by the same hardware, or cheaper
 221 hardware for the same work.
 222
 223 A _profiler_ is often the tool of choice to identify performance
 224 bottlenecks. Profiling is suitable to identify _where_ performance is
 225 lost in a given piece of software. The profiler outputs a profile, a
 226 statistical summary of observed events, which you may use to discover
 227 which functions took the most time to execute. However, a profiler won't
 228 report _why_ some identified functions are the bottleneck. Bottlenecks
 229 might only occur when specific conditions are met, conditions that are
 230 sometimes impossible to capture by a statistical profiler, or impossible
 231 to reproduce with an application altered by the overhead of an
 232 event-based profiler. For a thorough investigation of software
 233 performance issues, a history of execution is essential, with the
 234 recorded values of variables and context fields you choose, and with as
 235 little influence as possible on the instrumented application. This is
 236 where tracing comes in handy.
 237
 238 _Tracing_ is a technique used to understand what goes on in a running
 239 software system. The piece of software used for tracing is called a
 240 _tracer_, which is conceptually similar to a tape recorder. When
 241 recording, specific instrumentation points placed in the software source
 242 code generate events that are saved on a giant tape: a _trace_ file. You
 243 can record user application and operating system events at the same
 244 time, opening the possibility of resolving a wide range of problems that
 245 would otherwise be extremely challenging.
 246
 247 Tracing is often compared to _logging_. However, tracers and loggers are
 248 two different tools, serving two different purposes. Tracers are
 249 designed to record much lower-level events that occur much more
 250 frequently than log messages, often in the range of thousands per
 251 second, with very little execution overhead. Logging is more appropriate
 252 for a very high-level analysis of less frequent events: user accesses,
 253 exceptional conditions (errors and warnings, for example), database
 254 transactions, instant messaging communications, and such. Simply put,
 255 logging is one of the many use cases that can be satisfied with tracing.
 256
 257 The list of recorded events inside a trace file can be read manually
 258 like a log file for the maximum level of detail, but it's generally
 259 much more interesting to perform application-specific analyses to
 260 produce reduced statistics and graphs that are useful to resolve a
 261 given problem. Trace viewers and analyzers are specialized tools
 262 designed to do this.
 263
 264 In the end, this is what LTTng is: a powerful, open source set of
 265 tools to trace the Linux kernel and user applications at the same time.
 266 LTTng is composed of several components actively maintained and
 267 developed by its link:/community/#where[community].
 268
 269
 270 [[lttng-alternatives]]
 271 === Alternatives to noch:{LTTng}
 272
 273 Excluding proprietary solutions, a few competing software tracers
 274 exist for Linux:
 275
 276 https://github.com/dtrace4linux/linux[dtrace4linux]::
 277     A port of Sun Microsystems' DTrace to Linux.
 278 +
 279 The cmd:dtrace tool interprets user scripts and is responsible for
 280 loading code into the Linux kernel for further execution and collecting
 281 the outputted data.
 282
 283 https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
 284     A subsystem in the Linux kernel in which a virtual machine can
 285     execute programs passed from the user space to the kernel.
 286 +
 287 You can attach such programs to tracepoints and kprobes thanks to a
 288 system call, and they can output data to the user space when executed
 289 thanks to different mechanisms (pipe, VM register values, and eBPF maps,
 290 to name a few).
 291
 292 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
 293     The de facto function tracer of the Linux kernel.
 294 +
 295 Its user interface is a set of special files in sysfs.
 296
 297 https://perf.wiki.kernel.org/[perf]::
 298     A performance analysis tool for Linux which supports hardware
 299     performance counters, tracepoints, as well as other counters and
 300     types of probes.
 301 +
 302 The controlling utility of perf is the cmd:perf command line/text UI
 303 tool.
 304
 305 https://linux.die.net/man/1/strace[strace]::
 306     A command-line utility which records system calls made by a
 307     user process, as well as signal deliveries and changes of process
 308     state.
 309 +
 310 strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
 311 fulfill its function.
 312
 313 https://www.sysdig.org/[sysdig]::
 314     Like SystemTap, uses scripts to analyze Linux kernel events.
 315 +
 316 You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
 317 sysdig executes them while it traces the system or afterwards. The
 318 interface of sysdig is the cmd:sysdig command-line tool as well as the
 319 text UI-based cmd:csysdig tool.
 320
 321 https://sourceware.org/systemtap/[SystemTap]::
 322     A Linux kernel and user space tracer which uses custom user scripts
 323     to produce plain text traces.
 324 +
 325 SystemTap converts the scripts to the C language, and then compiles them
 326 as Linux kernel modules which are loaded to produce trace data. The
 327 primary user interface of SystemTap is the cmd:stap command-line tool.
 328
 329 The main distinctive features of LTTng is that it produces correlated
 330 kernel and user space traces, as well as doing so with the lowest
 331 overhead amongst other solutions. It produces trace files in the
 332 https://diamon.org/ctf[CTF] format, a file format optimized
 333 for the production and analyses of multi-gigabyte data.
 334
 335 LTTng is the result of more than 10{nbsp}years of active open source
 336 development by a community of passionate developers. LTTng is currently
 337 available on major desktop and server Linux distributions.
 338
 339 The main interface for tracing control is a single command-line tool
 340 named cmd:lttng. The latter can create several recording sessions, enable
 341 and disable recording event rules on the fly, filter events efficiently
 342 with custom user expressions, start and stop tracing, and much more.
 343 LTTng can write the traces on the file system or send them over the
 344 network, and keep them totally or partially. You can make LTTng execute
 345 user-defined actions when LTTng emits an event. You can view the traces
 346 once tracing becomes inactive or as LTTng records events.
 347
 348 <<installing-lttng,Install LTTng now>> and
 349 <<getting-started,start tracing>>!
 350
 351
 352 [[installing-lttng]]
 353 == Installation
 354
 355 **LTTng** is a set of software <<plumbing,components>> which interact to
 356 <<instrumenting,instrument>> the Linux kernel and user applications, and
 357 to <<controlling-tracing,control tracing>> (start and stop
 358 recording, create recording event rules, and the rest). Those
 359 components are bundled into the following packages:
 360
 361 LTTng-tools::
 362     Libraries and command-line interface to control tracing.
 363
 364 LTTng-modules::
 365     Linux kernel modules to instrument and trace the kernel.
 366
 367 LTTng-UST::
 368     Libraries and Java/Python packages to instrument and trace user
 369     applications.
 370
 371 Most distributions mark the LTTng-modules and LTTng-UST packages as
 372 optional when installing LTTng-tools (which is always required). In the
 373 following sections, we always provide the steps to install all three,
 374 but note that:
 375
 376 * You only need to install LTTng-modules if you intend to use
 377   the Linux kernel LTTng tracer.
 378
 379 * You only need to install LTTng-UST if you intend to use the user
 380   space LTTng tracer.
 381
 382 [role="growable"]
 383 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 17{nbsp}October{nbsp}2023.
 384
 385 |===
 386 |Distribution |Available in releases
 387
 388 |https://www.ubuntu.com/[Ubuntu]
 389 |xref:ubuntu[Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_].
 390
 391 Ubuntu{nbsp}18.04 LTS _Bionic Beaver_ and Ubuntu{nbsp}20.04 LTS _Focal Fossa_:
 392 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 393
 394 |https://www.debian.org/[Debian]
 395 |<<debian,Debian{nbsp}12 _bookworm_>>.
 396
 397 |https://getfedora.org/[Fedora]
 398 |xref:fedora[Fedora{nbsp}37, Fedora{nbsp}38, and Fedora{nbsp}39].
 399
 400 |https://www.archlinux.org/[Arch Linux]
 401 |<<arch-linux,_extra_ repository and AUR>>.
 402
 403 |https://alpinelinux.org/[Alpine Linux]
 404 |xref:alpine-linux[Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, and Alpine Linux{nbsp}3.18].
 405
 406 |https://buildroot.org/[Buildroot]
 407 |xref:buildroot[Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 408 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 409 Buildroot{nbsp}2023.05, and Buildroot{nbsp}2023.08].
 410
 411 |https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
 412 https://www.yoctoproject.org/[Yocto]
 413 |xref:oe-yocto[Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 414 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 415 Yocto Project{nbsp}4.3 _Nanbield_].
 416
 417 |====
 418
 419 [NOTE]
 420 ====
 421 For https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
 422 packages, see https://packages.efficios.com/[EfficiOS Enterprise
 423 Packages].
 424
 425 For other distributions, <<building-from-source,build LTTng from
 426 source>>.
 427 ====
 428
 429 [[ubuntu]]
 430 === [[ubuntu-official-repository]]Ubuntu
 431
 432 LTTng{nbsp}{revision} is available on Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_. For previous supported releases of Ubuntu, <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 433
 434 To install LTTng{nbsp}{revision} on Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_:
 435
 436 . Install the main LTTng{nbsp}{revision} packages:
 437 +
 438 --
 439 [role="term"]
 440 ----
 441 # apt-get install lttng-tools
 442 # apt-get install lttng-modules-dkms
 443 # apt-get install liblttng-ust-dev
 444 ----
 445 --
 446
 447 . **If you need to instrument and trace <<java-application,Java applications>>**,
 448 install the LTTng-UST Java agent:
 449 +
 450 --
 451 [role="term"]
 452 ----
 453 # apt-get install liblttng-ust-agent-java
 454 ----
 455 --
 456
 457 . **If you need to instrument and trace <<python-application,Python{nbsp}3
 458 applications>>**, install the LTTng-UST Python agent:
 459 +
 460 --
 461 [role="term"]
 462 ----
 463 # apt-get install python3-lttngust
 464 ----
 465 --
 466
 467 [[ubuntu-ppa]]
 468 === Ubuntu: noch:{LTTng} Stable {revision} PPA
 469
 470 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
 471 Stable{nbsp}{revision} PPA] offers the latest stable LTTng{nbsp}{revision}
 472 packages for Ubuntu{nbsp}18.04 LTS _Bionic Beaver_, Ubuntu{nbsp}20.04 LTS _Focal Fossa_,
 473 and Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_.
 474
 475 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
 476 PPA:
 477
 478 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
 479   list of packages:
 480 +
 481 --
 482 [role="term",subs="attributes"]
 483 ----
 484 # apt-add-repository ppa:lttng/stable-{revision}
 485 # apt-get update
 486 ----
 487 --
 488
 489 . Install the main LTTng{nbsp}{revision} packages:
 490 +
 491 --
 492 [role="term"]
 493 ----
 494 # apt-get install lttng-tools
 495 # apt-get install lttng-modules-dkms
 496 # apt-get install liblttng-ust-dev
 497 ----
 498 --
 499
 500 . **If you need to instrument and trace
 501   <<java-application,Java applications>>**, install the LTTng-UST
 502   Java agent:
 503 +
 504 --
 505 [role="term"]
 506 ----
 507 # apt-get install liblttng-ust-agent-java
 508 ----
 509 --
 510
 511 . **If you need to instrument and trace
 512   <<python-application,Python{nbsp}3 applications>>**, install the
 513   LTTng-UST Python agent:
 514 +
 515 --
 516 [role="term"]
 517 ----
 518 # apt-get install python3-lttngust
 519 ----
 520 --
 521
 522 [[debian]]
 523 === Debian
 524
 525 To install LTTng{nbsp}{revision} on Debian{nbsp}12 _bookworm_:
 526
 527 . Install the main LTTng{nbsp}{revision} packages:
 528 +
 529 --
 530 [role="term"]
 531 ----
 532 # apt install lttng-modules-dkms
 533 # apt install liblttng-ust-dev
 534 # apt install lttng-tools
 535 ----
 536 --
 537
 538 . **If you need to instrument and trace <<java-application,Java
 539   applications>>**, install the LTTng-UST Java agent:
 540 +
 541 --
 542 [role="term"]
 543 ----
 544 # apt install liblttng-ust-agent-java
 545 ----
 546 --
 547
 548 . **If you need to instrument and trace <<python-application,Python
 549   applications>>**, install the LTTng-UST Python agent:
 550 +
 551 --
 552 [role="term"]
 553 ----
 554 # apt install python3-lttngust
 555 ----
 556 --
 557
 558 [[fedora]]
 559 === Fedora
 560
 561 To install LTTng{nbsp}{revision} on Fedora{nbsp}37, Fedora{nbsp}38, or
 562 Fedora{nbsp}39:
 563
 564 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
 565   packages:
 566 +
 567 --
 568 [role="term"]
 569 ----
 570 # yum install lttng-tools
 571 # yum install lttng-ust
 572 ----
 573 --
 574
 575 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 576 +
 577 --
 578 [role="term",subs="attributes,specialcharacters"]
 579 ----
 580 $ cd $(mktemp -d) &&
 581   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 582   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 583   cd lttng-modules-{revision}.* &&
 584   make &&
 585   sudo make modules_install &&
 586   sudo depmod -a
 587 ----
 588 --
 589
 590 [IMPORTANT]
 591 .Java and Python application instrumentation and tracing
 592 ====
 593 If you need to instrument and trace <<java-application,Java
 594 applications>> on Fedora, you need to build and install
 595 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
 596 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
 597 `--enable-java-agent-all` options to the `configure` script, depending
 598 on which Java logging framework you use.
 599
 600 If you need to instrument and trace <<python-application,Python
 601 applications>> on Fedora, you need to build and install
 602 LTTng-UST{nbsp}{revision} from source and pass the
 603 `--enable-python-agent` option to the `configure` script.
 604 ====
 605
 606
 607 [[arch-linux]]
 608 === Arch Linux
 609
 610 LTTng-UST{nbsp}{revision} is available in the _extra_
 611 repository of Arch Linux, while LTTng-tools{nbsp}{revision} and
 612 LTTng-modules{nbsp}{revision} are available in the
 613 https://aur.archlinux.org/[AUR].
 614
 615 To install LTTng{nbsp}{revision} on Arch Linux, using
 616 https://github.com/Jguer/yay[yay] for the AUR packages:
 617
 618 . Install the main LTTng{nbsp}{revision} packages:
 619 +
 620 --
 621 [role="term"]
 622 ----
 623 # pacman -Sy lttng-ust
 624 $ yay -Sy lttng-tools
 625 $ yay -Sy lttng-modules
 626 ----
 627 --
 628
 629 . **If you need to instrument and trace <<python-application,Python
 630   applications>>**, install the LTTng-UST Python agent:
 631 +
 632 --
 633 [role="term"]
 634 ----
 635 # pacman -Sy python-lttngust
 636 ----
 637 --
 638
 639
 640 [[alpine-linux]]
 641 === Alpine Linux
 642
 643 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
 644 Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, or Alpine Linux{nbsp}3.18:
 645
 646 . Add the LTTng packages:
 647 +
 648 --
 649 [role="term"]
 650 ----
 651 # apk add lttng-tools
 652 # apk add lttng-ust-dev
 653 ----
 654 --
 655
 656 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 657 +
 658 --
 659 [role="term",subs="attributes,specialcharacters"]
 660 ----
 661 $ cd $(mktemp -d) &&
 662   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 663   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 664   cd lttng-modules-{revision}.* &&
 665   make &&
 666   sudo make modules_install &&
 667   sudo depmod -a
 668 ----
 669 --
 670
 671
 672 [[buildroot]]
 673 === Buildroot
 674
 675 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 676 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 677 Buildroot{nbsp}2023.05, or Buildroot{nbsp}2023.08:
 678
 679 . Launch the Buildroot configuration tool:
 680 +
 681 --
 682 [role="term"]
 683 ----
 684 $ make menuconfig
 685 ----
 686 --
 687
 688 . In **Kernel**, check **Linux kernel**.
 689 . In **Toolchain**, check **Enable WCHAR support**.
 690 . In **Target packages**{nbsp}&#8594; **Debugging, profiling and benchmark**,
 691   check **lttng-modules** and **lttng-tools**.
 692 . In **Target packages**{nbsp}&#8594; **Libraries**{nbsp}&#8594;
 693   **Other**, check **lttng-libust**.
 694
 695
 696 [[oe-yocto]]
 697 === OpenEmbedded and Yocto
 698
 699 LTTng{nbsp}{revision} recipes are available in the
 700 https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
 701 layer for Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 702 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 703 Yocto Project{nbsp}4.3 _Nanbield_ under the following names:
 704
 705 * `lttng-tools`
 706 * `lttng-modules`
 707 * `lttng-ust`
 708
 709 With BitBake, the simplest way to include LTTng recipes in your target
 710 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
 711
 712 ----
 713 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
 714 ----
 715
 716 If you use Hob:
 717
 718 . Select a machine and an image recipe.
 719 . Click **Edit image recipe**.
 720 . Under the **All recipes** tab, search for **lttng**.
 721 . Check the desired LTTng recipes.
 722
 723
 724 [[building-from-source]]
 725 === Build from source
 726
 727 To build and install LTTng{nbsp}{revision} from source:
 728
 729 . Using the package manager of your distribution, or from source,
 730   install the following dependencies of LTTng-tools and LTTng-UST:
 731 +
 732 --
 733 * https://sourceforge.net/projects/libuuid/[libuuid]
 734 * https://directory.fsf.org/wiki/Popt[popt]
 735 * https://liburcu.org/[Userspace RCU]
 736 * http://www.xmlsoft.org/[libxml2]
 737 * **Optional**: https://github.com/numactl/numactl[numactl]
 738 --
 739
 740 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 741 +
 742 --
 743 [role="term"]
 744 ----
 745 $ cd $(mktemp -d) &&
 746   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
 747   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
 748   cd lttng-modules-2.13.* &&
 749   make &&
 750   sudo make modules_install &&
 751   sudo depmod -a
 752 ----
 753 --
 754
 755 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
 756 +
 757 --
 758 [role="term"]
 759 ----
 760 $ cd $(mktemp -d) &&
 761   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
 762   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
 763   cd lttng-ust-2.13.* &&
 764   ./configure &&
 765   make &&
 766   sudo make install &&
 767   sudo ldconfig
 768 ----
 769 --
 770 +
 771 Add `--disable-numa` to `./configure` if you don't have
 772 https://github.com/numactl/numactl[numactl].
 773 +
 774 --
 775 [IMPORTANT]
 776 .Java and Python application tracing
 777 ====
 778 If you need to instrument and have LTTng trace <<java-application,Java
 779 applications>>, pass the `--enable-java-agent-jul`,
 780 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
 781 `configure` script, depending on which Java logging framework you use.
 782
 783 If you need to instrument and have LTTng trace
 784 <<python-application,Python applications>>, pass the
 785 `--enable-python-agent` option to the `configure` script. You can set
 786 the env:PYTHON environment variable to the path to the Python interpreter
 787 for which to install the LTTng-UST Python agent package.
 788 ====
 789 --
 790 +
 791 --
 792 [NOTE]
 793 ====
 794 By default, LTTng-UST libraries are installed to
 795 dir:{/usr/local/lib}, which is the de facto directory in which to
 796 keep self-compiled and third-party libraries.
 797
 798 When <<building-tracepoint-providers-and-user-application,linking an
 799 instrumented user application with `liblttng-ust`>>:
 800
 801 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
 802   variable.
 803
 804 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
 805   man:gcc(1), man:g++(1), or man:clang(1).
 806 ====
 807 --
 808
 809 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
 810 +
 811 --
 812 [role="term"]
 813 ----
 814 $ cd $(mktemp -d) &&
 815   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
 816   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
 817   cd lttng-tools-2.13.* &&
 818   ./configure &&
 819   make &&
 820   sudo make install &&
 821   sudo ldconfig
 822 ----
 823 --
 824
 825 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
 826 previous steps automatically for a given version of LTTng and confine
 827 the installed files to a specific directory. This can be useful to try
 828 LTTng without installing it on your system.
 829
 830 [[linux-kernel-sig]]
 831 === Linux kernel module signature
 832
 833 Linux kernel modules require trusted signatures in order to be loaded
 834 when any of the following is true:
 835
 836 * The system boots with
 837   https://uefi.org/specs/UEFI/2.10/32_Secure_Boot_and_Driver_Signing.html#secure-boot-and-driver-signing[Secure Boot]
 838   enabled.
 839
 840 * The Linux kernel which boots is configured with
 841   `CONFIG_MODULE_SIG_FORCE`.
 842
 843 * The Linux kernel boots with a command line containing
 844   `module.sig_enforce=1`.
 845
 846 .`root` user running <<lttng-sessiond,`lttng-sessiond`>> which fails to load a required <<lttng-modules,kernel module>> due to the signature enforcement policies.
 847 ====
 848 [role="term"]
 849 ----
 850 # lttng-sessiond
 851 Warning: No tracing group detected
 852 modprobe: ERROR: could not insert 'lttng_ring_buffer_client_discard': Key was rejected by service
 853 Error: Unable to load required module lttng-ring-buffer-client-discard
 854 Warning: No kernel tracer available
 855 ----
 856 ====
 857
 858 There are several methods to enroll trusted keys for signing modules
 859 that are built from source. The precise details vary from one Linux
 860 version to another, and distributions may have their own mechanisms. For
 861 example, https://github.com/dell/dkms[DKMS] may autogenerate a key and
 862 sign modules, but the key isn't automatically enrolled.
 863
 864 See
 865 https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html[Kernel
 866 module signing facility] and the documentation of your distribution
 867 to learn more about signing Linux kernel modules.
 868
 869 [[getting-started]]
 870 == Quick start
 871
 872 This is a short guide to get started quickly with LTTng kernel and user
 873 space tracing.
 874
 875 Before you follow this guide, make sure to <<installing-lttng,install>>
 876 LTTng.
 877
 878 This tutorial walks you through the steps to:
 879
 880 . <<tracing-the-linux-kernel,Record Linux kernel events>>.
 881
 882 . <<tracing-your-own-user-application,Record the events of a user
 883   application>> written in C.
 884
 885 . <<viewing-and-analyzing-your-traces,View and analyze the
 886   recorded events>>.
 887
 888
 889 [[tracing-the-linux-kernel]]
 890 === Record Linux kernel events
 891
 892 NOTE: The following command lines start with the `#` prompt because you
 893 need root privileges to control the Linux kernel LTTng tracer. You can
 894 also control the kernel tracer as a regular user if your Unix user is a
 895 member of the <<tracing-group,tracing group>>.
 896
 897 . Create a <<tracing-session,recording session>> to write LTTng traces
 898   to dir:{/tmp/my-kernel-trace}:
 899 +
 900 --
 901 [role="term"]
 902 ----
 903 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
 904 ----
 905 --
 906
 907 . List the available kernel tracepoints and system calls:
 908 +
 909 --
 910 [role="term"]
 911 ----
 912 # lttng list --kernel
 913 # lttng list --kernel --syscall
 914 ----
 915 --
 916
 917 . Create <<event,recording event rules>> which match events having
 918   the desired names, for example the `sched_switch` and
 919   `sched_process_fork` tracepoints, and the man:open(2) and man:close(2)
 920   system calls:
 921 +
 922 --
 923 [role="term"]
 924 ----
 925 # lttng enable-event --kernel sched_switch,sched_process_fork
 926 # lttng enable-event --kernel --syscall open,close
 927 ----
 928 --
 929 +
 930 Create a recording event rule which matches _all_ the Linux kernel
 931 tracepoint events with the opt:lttng-enable-event(1):--all option
 932 (recording with such a recording event rule generates a lot of data):
 933 +
 934 --
 935 [role="term"]
 936 ----
 937 # lttng enable-event --kernel --all
 938 ----
 939 --
 940
 941 . <<basic-tracing-session-control,Start recording>>:
 942 +
 943 --
 944 [role="term"]
 945 ----
 946 # lttng start
 947 ----
 948 --
 949
 950 . Do some operation on your system for a few seconds. For example,
 951   load a website, or list the files of a directory.
 952
 953 . <<creating-destroying-tracing-sessions,Destroy>> the current
 954   recording session:
 955 +
 956 --
 957 [role="term"]
 958 ----
 959 # lttng destroy
 960 ----
 961 --
 962 +
 963 The man:lttng-destroy(1) command doesn't destroy the trace data; it
 964 only destroys the state of the recording session.
 965 +
 966 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
 967 implicitly (see ``<<basic-tracing-session-control,Start and stop a
 968 recording session>>''). You need to stop recording to make LTTng flush
 969 the remaining trace data and make the trace readable.
 970
 971 . For the sake of this example, make the recorded trace accessible to
 972   the non-root users:
 973 +
 974 --
 975 [role="term"]
 976 ----
 977 # chown -R $(whoami) /tmp/my-kernel-trace
 978 ----
 979 --
 980
 981 See ``<<viewing-and-analyzing-your-traces,View and analyze the
 982 recorded events>>'' to view the recorded events.
 983
 984
 985 [[tracing-your-own-user-application]]
 986 === Record user application events
 987
 988 This section walks you through a simple example to record the events of
 989 a _Hello world_ program written in{nbsp}C.
 990
 991 To create the traceable user application:
 992
 993 . Create the tracepoint provider header file, which defines the
 994   tracepoints and the events they can generate:
 995 +
 996 --
 997 [source,c]
 998 .path:{hello-tp.h}
 999 ----
1000 #undef LTTNG_UST_TRACEPOINT_PROVIDER
1001 #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
1002
1003 #undef LTTNG_UST_TRACEPOINT_INCLUDE
1004 #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
1005
1006 #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
1007 #define _HELLO_TP_H
1008
1009 #include <lttng/tracepoint.h>
1010
1011 LTTNG_UST_TRACEPOINT_EVENT(
1012     hello_world,
1013     my_first_tracepoint,
1014     LTTNG_UST_TP_ARGS(
1015         int, my_integer_arg,
1016         char *, my_string_arg
1017     ),
1018     LTTNG_UST_TP_FIELDS(
1019         lttng_ust_field_string(my_string_field, my_string_arg)
1020         lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
1021     )
1022 )
1023
1024 #endif /* _HELLO_TP_H */
1025
1026 #include <lttng/tracepoint-event.h>
1027 ----
1028 --
1029
1030 . Create the tracepoint provider package source file:
1031 +
1032 --
1033 [source,c]
1034 .path:{hello-tp.c}
1035 ----
1036 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
1037 #define LTTNG_UST_TRACEPOINT_DEFINE
1038
1039 #include "hello-tp.h"
1040 ----
1041 --
1042
1043 . Build the tracepoint provider package:
1044 +
1045 --
1046 [role="term"]
1047 ----
1048 $ gcc -c -I. hello-tp.c
1049 ----
1050 --
1051
1052 . Create the _Hello World_ application source file:
1053 +
1054 --
1055 [source,c]
1056 .path:{hello.c}
1057 ----
1058 #include <stdio.h>
1059 #include "hello-tp.h"
1060
1061 int main(int argc, char *argv[])
1062 {
1063     unsigned int i;
1064
1065     puts("Hello, World!\nPress Enter to continue...");
1066
1067     /*
1068      * The following getchar() call only exists for the purpose of this
1069      * demonstration, to pause the application in order for you to have
1070      * time to list its tracepoints. You don't need it otherwise.
1071      */
1072     getchar();
1073
1074     /*
1075      * An lttng_ust_tracepoint() call.
1076      *
1077      * Arguments, as defined in `hello-tp.h`:
1078      *
1079      * 1. Tracepoint provider name (required)
1080      * 2. Tracepoint name (required)
1081      * 3. `my_integer_arg` (first user-defined argument)
1082      * 4. `my_string_arg` (second user-defined argument)
1083      *
1084      * Notice the tracepoint provider and tracepoint names are
1085      * C identifiers, NOT strings: they're in fact parts of variables
1086      * that the macros in `hello-tp.h` create.
1087      */
1088     lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,
1089                          "hi there!");
1090
1091     for (i = 0; i < argc; i++) {
1092         lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1093                              i, argv[i]);
1094     }
1095
1096     puts("Quitting now!");
1097     lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1098                          i * i, "i^2");
1099     return 0;
1100 }
1101 ----
1102 --
1103
1104 . Build the application:
1105 +
1106 --
1107 [role="term"]
1108 ----
1109 $ gcc -c hello.c
1110 ----
1111 --
1112
1113 . Link the application with the tracepoint provider package,
1114   `liblttng-ust` and `libdl`:
1115 +
1116 --
1117 [role="term"]
1118 ----
1119 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1120 ----
1121 --
1122
1123 Here's the whole build process:
1124
1125 [role="img-100"]
1126 .Build steps of the user space tracing tutorial.
1127 image::ust-flow.png[]
1128
1129 To record the events of the user application:
1130
1131 . Run the application with a few arguments:
1132 +
1133 --
1134 [role="term"]
1135 ----
1136 $ ./hello world and beyond
1137 ----
1138 --
1139 +
1140 You see:
1141 +
1142 --
1143 ----
1144 Hello, World!
1145 Press Enter to continue...
1146 ----
1147 --
1148
1149 . Start an LTTng <<lttng-sessiond,session daemon>>:
1150 +
1151 --
1152 [role="term"]
1153 ----
1154 $ lttng-sessiond --daemonize
1155 ----
1156 --
1157 +
1158 NOTE: A session daemon might already be running, for example as a
1159 service that the service manager of your distribution started.
1160
1161 . List the available user space tracepoints:
1162 +
1163 --
1164 [role="term"]
1165 ----
1166 $ lttng list --userspace
1167 ----
1168 --
1169 +
1170 You see the `hello_world:my_first_tracepoint` tracepoint listed
1171 under the `./hello` process.
1172
1173 . Create a <<tracing-session,recording session>>:
1174 +
1175 --
1176 [role="term"]
1177 ----
1178 $ lttng create my-user-space-session
1179 ----
1180 --
1181
1182 . Create a <<event,recording event rule>> which matches user space
1183   tracepoint events named `hello_world:my_first_tracepoint`:
1184 +
1185 --
1186 [role="term"]
1187 ----
1188 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1189 ----
1190 --
1191
1192 . <<basic-tracing-session-control,Start recording>>:
1193 +
1194 --
1195 [role="term"]
1196 ----
1197 $ lttng start
1198 ----
1199 --
1200
1201 . Go back to the running `hello` application and press **Enter**.
1202 +
1203 The program executes all `lttng_ust_tracepoint()` instrumentation
1204 points, emitting events as the event rule you created in step{nbsp}5
1205 matches them, and
1206 exits.
1207
1208 . <<creating-destroying-tracing-sessions,Destroy>> the current
1209   recording session:
1210 +
1211 --
1212 [role="term"]
1213 ----
1214 $ lttng destroy
1215 ----
1216 --
1217 +
1218 The man:lttng-destroy(1) command doesn't destroy the trace data; it
1219 only destroys the state of the recording session.
1220 +
1221 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1222 implicitly (see ``<<basic-tracing-session-control,Start and stop a
1223 recording session>>''). You need to stop recording to make LTTng flush
1224 the remaining trace data and make the trace readable.
1225
1226 By default, LTTng saves the traces to the
1227 +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+ directory, where
1228 +__NAME__+ is the recording session name. The env:LTTNG_HOME environment
1229 variable defaults to `$HOME` if not set.
1230
1231
1232 [[viewing-and-analyzing-your-traces]]
1233 === View and analyze the recorded events
1234
1235 Once you have completed the <<tracing-the-linux-kernel,Record Linux
1236 kernel events>> and <<tracing-your-own-user-application,Record user
1237 application events>> tutorials, you can inspect the recorded events.
1238
1239 There are tools you can use to read LTTng traces:
1240
1241 https://babeltrace.org/[Babeltrace{nbsp}2]::
1242     A rich, flexible trace manipulation toolkit which includes
1243     a versatile command-line interface
1244     (man:babeltrace2(1)),
1245     a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C{nbsp}library],
1246     and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1247     so that you can easily process or convert an LTTng trace with
1248     your own script.
1249 +
1250 The Babeltrace{nbsp}2 project ships with a plugin
1251 (man:babeltrace2-plugin-ctf(7)) which supports the format of the traces
1252 which LTTng produces, https://diamon.org/ctf/[CTF].
1253
1254 http://tracecompass.org/[Trace Compass]::
1255     A graphical user interface for viewing and analyzing any type of
1256     logs or traces, including those of LTTng.
1257
1258 NOTE: This section assumes that LTTng wrote the traces it recorded
1259 during the previous tutorials to their default location, in the
1260 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1261 environment variable defaults to `$HOME` if not set.
1262
1263
1264 [[viewing-and-analyzing-your-traces-bt]]
1265 ==== Use the cmd:babeltrace2 command-line tool
1266
1267 The simplest way to list all the recorded events of an LTTng trace is to
1268 pass its path to man:babeltrace2(1), without options:
1269
1270 [role="term"]
1271 ----
1272 $ babeltrace2 ~/lttng-traces/my-user-space-session*
1273 ----
1274
1275 The cmd:babeltrace2 command finds all traces recursively within the
1276 given path and prints all their events, sorting them chronologically.
1277
1278 Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1279 further filtering:
1280
1281 [role="term"]
1282 ----
1283 $ babeltrace2 /tmp/my-kernel-trace | grep _switch
1284 ----
1285
1286 Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1287 the recorded events:
1288
1289 [role="term"]
1290 ----
1291 $ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1292 ----
1293
1294
1295 [[viewing-and-analyzing-your-traces-bt-python]]
1296 ==== Use the Babeltrace{nbsp}2 Python bindings
1297
1298 The <<viewing-and-analyzing-your-traces-bt,text output of
1299 cmd:babeltrace2>> is useful to isolate event records by simple matching
1300 using man:grep(1) and similar utilities. However, more elaborate
1301 filters, such as keeping only event records with a field value falling
1302 within a specific range, are not trivial to write using a shell.
1303 Moreover, reductions and even the most basic computations involving
1304 multiple event records are virtually impossible to implement.
1305
1306 Fortunately, Babeltrace{nbsp}2 ships with
1307 https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1308 which make it easy to read the event records of an LTTng trace
1309 sequentially and compute the desired information.
1310
1311 The following script accepts an LTTng Linux kernel trace path as its
1312 first argument and prints the short names of the top five running
1313 processes on CPU{nbsp}0 during the whole trace:
1314
1315 [source,python]
1316 .path:{top5proc.py}
1317 ----
1318 import bt2
1319 import sys
1320 import collections
1321
1322
1323 def top5proc():
1324     # Get the trace path from the first command-line argument
1325     it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1326
1327     # This counter dictionary will hold execution times:
1328     #
1329     #     Task command name -> Total execution time (ns)
1330     exec_times = collections.Counter()
1331
1332     # This holds the last `sched_switch` timestamp
1333     last_ts = None
1334
1335     for msg in it:
1336         # We only care about event messages
1337         if type(msg) is not bt2._EventMessageConst:
1338             continue
1339
1340         # Event of the event message
1341         event = msg.event
1342
1343         # Keep only `sched_switch` events
1344         if event.cls.name != 'sched_switch':
1345             continue
1346
1347         # Keep only records of events which LTTng emitted from CPU 0
1348         if event.packet.context_field['cpu_id'] != 0:
1349             continue
1350
1351         # Event timestamp (ns)
1352         cur_ts = msg.default_clock_snapshot.ns_from_origin
1353
1354         if last_ts is None:
1355             # Start here
1356             last_ts = cur_ts
1357
1358         # (Short) name of the previous task command
1359         prev_comm = str(event.payload_field['prev_comm'])
1360
1361         # Initialize an entry in our dictionary if not done yet
1362         if prev_comm not in exec_times:
1363             exec_times[prev_comm] = 0
1364
1365         # Compute previous command execution time
1366         diff = cur_ts - last_ts
1367
1368         # Update execution time of this command
1369         exec_times[prev_comm] += diff
1370
1371         # Update last timestamp
1372         last_ts = cur_ts
1373
1374     # Print top 5
1375     for name, ns in exec_times.most_common(5):
1376         print('{:20}{} s'.format(name, ns / 1e9))
1377
1378
1379 if __name__ == '__main__':
1380     top5proc()
1381 ----
1382
1383 Run this script:
1384
1385 [role="term"]
1386 ----
1387 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1388 ----
1389
1390 Output example:
1391
1392 ----
1393 swapper/0           48.607245889 s
1394 chromium            7.192738188 s
1395 pavucontrol         0.709894415 s
1396 Compositor          0.660867933 s
1397 Xorg.bin            0.616753786 s
1398 ----
1399
1400 Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1401 since we weren't using the CPU that much when recording, its first
1402 position in the list makes sense.
1403
1404
1405 [[core-concepts]]
1406 == [[understanding-lttng]]Core concepts
1407
1408 From a user's perspective, the LTTng system is built on a few concepts,
1409 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1410 operates by sending commands to the <<lttng-sessiond,session daemon>>
1411 (through <<liblttng-ctl-lttng,`liblttng-ctl`>>).
1412
1413 Understanding how those objects relate to each other is key to master
1414 the toolkit.
1415
1416 The core concepts of LTTng are:
1417
1418 * <<"event-rule","Instrumentation point, event rule, and event">>
1419 * <<trigger,Trigger>>
1420 * <<tracing-session,Recording session>>
1421 * <<domain,Tracing domain>>
1422 * <<channel,Channel and ring buffer>>
1423 * <<event,Recording event rule and event record>>
1424
1425 NOTE: The man:lttng-concepts(7) manual page also documents the core
1426 concepts of LTTng, with more links to other LTTng-tools manual pages.
1427
1428
1429 [[event-rule]]
1430 === Instrumentation point, event rule, and event
1431
1432 An _instrumentation point_ is a point, within a piece of software,
1433 which, when executed, creates an LTTng _event_.
1434
1435 LTTng offers various <<instrumentation-point-types,types of
1436 instrumentation>>.
1437
1438 An _event rule_ is a set of conditions to match a set of events.
1439
1440 When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is
1441 said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies _all_ the
1442 conditions of{nbsp}__ER__. This concept is similar to a
1443 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
1444 which matches a set of strings.
1445
1446 When an event rule matches an event, LTTng _emits_ the event, therefore
1447 attempting to execute one or more actions.
1448
1449 [IMPORTANT]
1450 ====
1451 [[event-creation-emission-opti]]The event creation and emission
1452 processes are documentation concepts to help understand the journey from
1453 an instrumentation point to the execution of actions.
1454
1455 The actual creation of an event can be costly because LTTng needs to
1456 evaluate the arguments of the instrumentation point.
1457
1458 In practice, LTTng implements various optimizations for the Linux kernel
1459 and user space <<domain,tracing domains>> to avoid actually creating an
1460 event when the tracer knows, thanks to properties which are independent
1461 from the event payload and current context, that it would never emit
1462 such an event. Those properties are:
1463
1464 * The <<instrumentation-point-types,instrumentation point type>>.
1465
1466 * The instrumentation point name.
1467
1468 * The instrumentation point log level.
1469
1470 * For a <<event,recording event rule>>:
1471 ** The status of the rule itself.
1472 ** The status of the <<channel,channel>>.
1473 ** The activity of the <<tracing-session,recording session>>.
1474 ** Whether or not the process for which LTTng would create the event is
1475    <<pid-tracking,allowed to record events>>.
1476
1477 In other words: if, for a given instrumentation point{nbsp}__IP__, the
1478 LTTng tracer knows that it would never emit an event,
1479 executing{nbsp}__IP__ represents a simple boolean variable check and,
1480 for a Linux kernel recording event rule, a few process attribute checks.
1481 ====
1482
1483 As of LTTng{nbsp}{revision}, there are two places where you can find an
1484 event rule:
1485
1486 <<event,Recording event rule>>::
1487     A specific type of event rule of which the action is to record the
1488     matched event as an event record.
1489 +
1490 See ``<<enabling-disabling-events,Create and enable a recording event
1491 rule>>'' to learn more.
1492
1493 ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13)::
1494     When the event rule of the trigger condition matches an event, LTTng
1495     can execute user-defined actions such as sending an LTTng
1496     notification,
1497     <<basic-tracing-session-control,starting a recording session>>,
1498     and more.
1499 +
1500 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
1501 trigger to a session daemon>>” to learn more.
1502
1503 For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy _all_ the
1504 basic conditions of an event rule{nbsp}__ER__, that is:
1505
1506 * The instrumentation point from which LTTng
1507   creates{nbsp}__E__ has a specific
1508   <<instrumentation-point-types,type>>.
1509
1510 * A pattern matches the name of{nbsp}__E__ while another pattern
1511   doesn't.
1512
1513 * The log level of the instrumentation point from which LTTng
1514   creates{nbsp}__E__ is at least as severe as some value, or is exactly
1515   some value.
1516
1517 * The fields of the payload of{nbsp}__E__ and the current context fields
1518   satisfy a filter expression.
1519
1520 A <<event,recording event rule>> has additional, implicit conditions to
1521 satisfy.
1522
1523
1524 [[instrumentation-point-types]]
1525 ==== Instrumentation point types
1526
1527 As of LTTng{nbsp}{revision}, the available instrumentation point
1528 types are, depending on the <<domain,tracing domain>>:
1529
1530 Linux kernel::
1531     LTTng tracepoint:::
1532         A statically defined point in the source code of the kernel
1533         image or of a kernel module using the
1534         <<lttng-modules,LTTng-modules>> macros.
1535
1536     Linux kernel system call:::
1537         Entry, exit, or both of a Linux kernel system call.
1538
1539     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kprobe]:::
1540         A single probe dynamically placed in the compiled kernel code.
1541 +
1542 When you create such an instrumentation point, you set its memory
1543 address or symbol name.
1544
1545     Linux user space probe:::
1546         A single probe dynamically placed at the entry of a compiled
1547         user space application/library function through the kernel.
1548 +
1549 When you create such an instrumentation point, you set:
1550 +
1551 --
1552 With the ELF method::
1553     Its application/library path and its symbol name.
1554
1555 With the USDT method::
1556     Its application/library path, its provider name, and its probe name.
1557 +
1558 ``USDT'' stands for _SystemTap User-level Statically Defined Tracing_,
1559 a http://dtrace.org/blogs/about/[DTrace]-style marker.
1560 --
1561 +
1562 As of LTTng{nbsp}{revision}, LTTng only supports USDT probes which
1563 are _not_ reference-counted.
1564
1565     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kretprobe]:::
1566         Entry, exit, or both of a Linux kernel function.
1567 +
1568 When you create such an instrumentation point, you set the memory
1569 address or symbol name of its function.
1570
1571 User space::
1572     LTTng tracepoint:::
1573         A statically defined point in the source code of a C/$$C++$$
1574         application/library using the
1575         <<lttng-ust,LTTng-UST>> macros.
1576
1577 `java.util.logging`, Apache log4j, and Python::
1578     Java or Python logging statement:::
1579         A method call on a Java or Python logger attached to an
1580         LTTng-UST handler.
1581
1582 See ``<<list-instrumentation-points,List the available instrumentation
1583 points>>'' to learn how to list available Linux kernel, user space, and
1584 logging instrumentation points.
1585
1586
1587 [[trigger]]
1588 === Trigger
1589
1590 A _trigger_ associates a condition to one or more actions.
1591
1592 When the condition of a trigger is satisfied, LTTng attempts to execute
1593 its actions.
1594
1595 As of LTTng{nbsp}{revision}, the available trigger conditions and
1596 actions are:
1597
1598 Conditions::
1599 +
1600 * The consumed buffer size of a given <<tracing-session,recording
1601   session>> becomes greater than some value.
1602
1603 * The buffer usage of a given <<channel,channel>> becomes greater than
1604   some value.
1605
1606 * The buffer usage of a given channel becomes less than some value.
1607
1608 * There's an ongoing <<session-rotation,recording session rotation>>.
1609
1610 * A recording session rotation becomes completed.
1611
1612 * An <<add-event-rule-matches-trigger,event rule matches>> an event.
1613
1614 Actions::
1615 +
1616 * <<trigger-event-notif,Send a notification>> to a user application.
1617 * <<basic-tracing-session-control,Start>> a given recording session.
1618 * <<basic-tracing-session-control,Stop>> a given recording session.
1619 * <<session-rotation,Archive the current trace chunk>> of a given
1620   recording session (rotate).
1621 * <<taking-a-snapshot,Take a snapshot>> of a given recording session.
1622
1623 A trigger belongs to a <<lttng-sessiond,session daemon>>, not to a
1624 specific recording session. For a given session daemon, each Unix user has
1625 its own, private triggers. Note, however, that the `root` Unix user may,
1626 for the root session daemon:
1627
1628 * Add a trigger as another Unix user.
1629
1630 * List all the triggers, regardless of their owner.
1631
1632 * Remove a trigger which belongs to another Unix user.
1633
1634 For a given session daemon and Unix user, a trigger has a unique name.
1635
1636
1637 [[tracing-session]]
1638 === Recording session
1639
1640 A _recording session_ (named ``tracing session'' prior to
1641 LTTng{nbsp}2.13) is a stateful dialogue between you and a
1642 <<lttng-sessiond,session daemon>> for everything related to
1643 <<event,event recording>>.
1644
1645 Everything that you do when you control LTTng tracers to record events
1646 happens within a recording session. In particular, a recording session:
1647
1648 * Has its own name, unique for a given session daemon.
1649
1650 * Has its own set of trace files, if any.
1651
1652 * Has its own state of activity (started or stopped).
1653 +
1654 An active recording session is an implicit <<event,recording event rule>>
1655 condition.
1656
1657 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1658   snapshot, or live).
1659
1660 * Has its own <<channel,channels>> to which are attached their own
1661   recording event rules.
1662
1663 * Has its own <<pid-tracking,process attribute inclusion sets>>.
1664
1665 [role="img-100"]
1666 .A _recording session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,recording event rules>>.
1667 image::concepts.png[]
1668
1669 Those attributes and objects are completely isolated between different
1670 recording sessions.
1671
1672 A recording session is like an
1673 https://en.wikipedia.org/wiki/Automated_teller_machine[ATM] session: the
1674 operations you do on the banking system through the ATM don't alter the
1675 data of other users of the same system. In the case of the ATM, a
1676 session lasts as long as your bank card is inside. In the case of LTTng,
1677 a recording session lasts from the man:lttng-create(1) command to the
1678 man:lttng-destroy(1) command.
1679
1680 [role="img-100"]
1681 .Each Unix user has its own set of recording sessions.
1682 image::many-sessions.png[]
1683
1684 A recording session belongs to a <<lttng-sessiond,session daemon>>. For a
1685 given session daemon, each Unix user has its own, private recording
1686 sessions. Note, however, that the `root` Unix user may operate on or
1687 destroy another user's recording session.
1688
1689
1690 [[tracing-session-mode]]
1691 ==== Recording session mode
1692
1693 LTTng offers four recording session modes:
1694
1695 [[local-mode]]Local mode::
1696     Write the trace data to the local file system.
1697
1698 [[net-streaming-mode]]Network streaming mode::
1699     Send the trace data over the network to a listening
1700     <<lttng-relayd,relay daemon>>.
1701
1702 [[snapshot-mode]]Snapshot mode::
1703     Only write the trace data to the local file system or send it to a
1704     listening relay daemon when LTTng <<taking-a-snapshot,takes a
1705     snapshot>>.
1706 +
1707 LTTng forces all the <<channel,channels>>
1708 to be created to be configured to be snapshot-ready.
1709 +
1710 LTTng takes a snapshot of such a recording session when:
1711 +
1712 --
1713 * You run the man:lttng-snapshot(1) command.
1714
1715 * LTTng executes a `snapshot-session` <<trigger,trigger>> action.
1716 --
1717
1718 [[live-mode]]Live mode::
1719     Send the trace data over the network to a listening relay daemon
1720     for <<lttng-live,live reading>>.
1721 +
1722 An LTTng live reader (for example, man:babeltrace2(1)) can connect to
1723 the same relay daemon to receive trace data while the recording session is
1724 active.
1725
1726
1727 [[domain]]
1728 === Tracing domain
1729
1730 A _tracing domain_ identifies a type of LTTng tracer.
1731
1732 A tracing domain has its own properties and features.
1733
1734 There are currently five available tracing domains:
1735
1736 * Linux kernel
1737 * User space
1738 * `java.util.logging` (JUL)
1739 * log4j
1740 * Python
1741
1742 You must specify a tracing domain to target a type of LTTng tracer when
1743 using some <<lttng-cli,cmd:lttng>> commands to avoid ambiguity. For
1744 example, because the Linux kernel and user space tracing domains support
1745 named tracepoints as <<event-rule,instrumentation points>>, you need to
1746 specify a tracing domain when you <<enabling-disabling-events,create
1747 an event rule>> because both tracing domains could have tracepoints
1748 sharing the same name.
1749
1750 You can create <<channel,channels>> in the Linux kernel and user space
1751 tracing domains. The other tracing domains have a single, default
1752 channel.
1753
1754
1755 [[channel]]
1756 === Channel and ring buffer
1757
1758 A _channel_ is an object which is responsible for a set of
1759 _ring buffers_.
1760
1761 Each ring buffer is divided into multiple _sub-buffers_. When a
1762 <<event,recording event rule>>
1763 matches an event, LTTng can record it to one or more sub-buffers of one
1764 or more channels.
1765
1766 When you <<enabling-disabling-channels,create a channel>>, you set its
1767 final attributes, that is:
1768
1769 * Its <<channel-buffering-schemes,buffering scheme>>.
1770
1771 * What to do <<channel-overwrite-mode-vs-discard-mode,when there's no
1772   space left>> for a new event record because all sub-buffers are full.
1773
1774 * The <<channel-subbuf-size-vs-subbuf-count,size of each ring buffer and
1775   how many sub-buffers>> a ring buffer has.
1776
1777 * The <<tracefile-rotation,size of each trace file LTTng writes for this
1778   channel and the maximum count>> of trace files.
1779
1780 * The periods of its <<channel-read-timer,read>>,
1781   <<channel-switch-timer,switch>>, and <<channel-monitor-timer,monitor>>
1782   timers.
1783
1784 * For a Linux kernel channel: its output type.
1785 +
1786 See the opt:lttng-enable-channel(1):--output option of the
1787 man:lttng-enable-channel(1) command.
1788
1789 * For a user space channel: the value of its
1790   <<blocking-timeout-example,blocking timeout>>.
1791
1792 A channel is always associated to a <<domain,tracing domain>>. The
1793 `java.util.logging` (JUL), log4j, and Python tracing domains each have a
1794 default channel which you can't configure.
1795
1796 A channel owns <<event,recording event rules>>.
1797
1798
1799 [[channel-buffering-schemes]]
1800 ==== Buffering scheme
1801
1802 A channel has at least one ring buffer _per CPU_. LTTng always records
1803 an event to the ring buffer dedicated to the CPU which emits it.
1804
1805 The buffering scheme of a user space channel determines what has its own
1806 set of per-CPU ring buffers:
1807
1808 Per-user buffering::
1809   Allocate one set of ring buffers--one per CPU--shared by all the
1810   instrumented processes of:
1811     If your Unix user is `root`:::
1812       Each Unix user.
1813 +
1814 --
1815 [role="img-100"]
1816 .Per-user buffering scheme (recording session belongs to the `root` Unix user).
1817 image::per-user-buffering-root.png[]
1818 --
1819
1820     Otherwise:::
1821       Your Unix user.
1822 +
1823 --
1824 [role="img-100"]
1825 .Per-user buffering scheme (recording session belongs to the `Bob` Unix user).
1826 image::per-user-buffering.png[]
1827 --
1828
1829 Per-process buffering::
1830   Allocate one set of ring buffers--one per CPU--for each
1831   instrumented process of:
1832     If your Unix user is `root`:::
1833       All Unix users.
1834 +
1835 --
1836 [role="img-100"]
1837 .Per-process buffering scheme (recording session belongs to the `root` Unix user).
1838 image::per-process-buffering-root.png[]
1839 --
1840
1841     Otherwise:::
1842       Your Unix user.
1843 +
1844 --
1845 [role="img-100"]
1846 .Per-process buffering scheme (recording session belongs to the `Bob` Unix user).
1847 image::per-process-buffering.png[]
1848 --
1849
1850 The per-process buffering scheme tends to consume more memory than the
1851 per-user option because systems generally have more instrumented
1852 processes than Unix users running instrumented processes. However, the
1853 per-process buffering scheme ensures that one process having a high
1854 event throughput won't fill all the shared sub-buffers of the same Unix
1855 user, only its own.
1856
1857 The buffering scheme of a Linux kernel channel is always to allocate a
1858 single set of ring buffers for the whole system. This scheme is similar
1859 to the per-user option, but with a single, global user ``running'' the
1860 kernel.
1861
1862
1863 [[channel-overwrite-mode-vs-discard-mode]]
1864 ==== Event record loss mode
1865
1866 When LTTng emits an event, LTTng can record it to a specific, available
1867 sub-buffer within the ring buffers of specific channels. When there's no
1868 space left in a sub-buffer, the tracer marks it as consumable and
1869 another, available sub-buffer starts receiving the following event
1870 records. An LTTng <<lttng-consumerd,consumer daemon>> eventually
1871 consumes the marked sub-buffer, which returns to the available state.
1872
1873 [NOTE]
1874 [role="docsvg-channel-subbuf-anim"]
1875 ====
1876 {note-no-anim}
1877 ====
1878
1879 In an ideal world, sub-buffers are consumed faster than they're filled.
1880 In the real world, however, all sub-buffers can be full at some point,
1881 leaving no space to record the following events.
1882
1883 In an ideal world, sub-buffers are consumed faster than they're filled,
1884 as it's the case in the previous animation. In the real world,
1885 however, all sub-buffers can be full at some point, leaving no space to
1886 record the following events.
1887
1888 By default, <<lttng-modules,LTTng-modules>> and <<lttng-ust,LTTng-UST>>
1889 are _non-blocking_ tracers: when there's no available sub-buffer to
1890 record an event, it's acceptable to lose event records when the
1891 alternative would be to cause substantial delays in the execution of the
1892 instrumented application. LTTng privileges performance over integrity;
1893 it aims at perturbing the instrumented application as little as possible
1894 in order to make the detection of subtle race conditions and rare
1895 interrupt cascades possible.
1896
1897 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1898 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1899 example>> to learn how to use the blocking mode.
1900
1901 When it comes to losing event records because there's no available
1902 sub-buffer, or because the blocking timeout of
1903 the channel is reached, the _event record loss mode_ of the channel
1904 determines what to do. The available event record loss modes are:
1905
1906 [[discard-mode]]Discard mode::
1907     Drop the newest event records until a sub-buffer becomes available.
1908 +
1909 This is the only available mode when you specify a blocking timeout.
1910 +
1911 With this mode, LTTng increments a count of lost event records when an
1912 event record is lost and saves this count to the trace. A trace reader
1913 can use the saved discarded event record count of the trace to decide
1914 whether or not to perform some analysis even if trace data is known to
1915 be missing.
1916
1917 [[overwrite-mode]]Overwrite mode::
1918     Clear the sub-buffer containing the oldest event records and start
1919     writing the newest event records there.
1920 +
1921 This mode is sometimes called _flight recorder mode_ because it's
1922 similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight
1923 recorder]: always keep a fixed amount of the latest data. It's also
1924 similar to the roll mode of an oscilloscope.
1925 +
1926 Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer
1927 its sequence number within its data stream. With a <<local-mode,local>>,
1928 <<net-streaming-mode,network streaming>>, or <<live-mode,live>> recording
1929 session, a trace reader can use such sequence numbers to report lost
1930 packets. A trace reader can use the saved discarded sub-buffer (packet)
1931 count of the trace to decide whether or not to perform some analysis
1932 even if trace data is known to be missing.
1933 +
1934 With this mode, LTTng doesn't write to the trace the exact number of
1935 lost event records in the lost sub-buffers.
1936
1937 Which mechanism you should choose depends on your context: prioritize
1938 the newest or the oldest event records in the ring buffer?
1939
1940 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1941 as soon as a there's no space left for a new event record, whereas in
1942 discard mode, the tracer only discards the event record that doesn't
1943 fit.
1944
1945 There are a few ways to decrease your probability of losing event
1946 records. The ``<<channel-subbuf-size-vs-subbuf-count,Sub-buffer size and
1947 count>>'' section shows how to fine-tune the sub-buffer size and count
1948 of a channel to virtually stop losing event records, though at the cost
1949 of greater memory usage.
1950
1951
1952 [[channel-subbuf-size-vs-subbuf-count]]
1953 ==== Sub-buffer size and count
1954
1955 A channel has one or more ring buffer for each CPU of the target system.
1956
1957 See the ``<<channel-buffering-schemes,Buffering scheme>>'' section to
1958 learn how many ring buffers of a given channel are dedicated to each CPU
1959 depending on its buffering scheme.
1960
1961 Set the size of each sub-buffer the ring buffers of a channel contain
1962 and how many there are
1963 when you <<enabling-disabling-channels,create it>>.
1964
1965 Note that LTTng switching the current sub-buffer of a ring buffer
1966 (marking a full one as consumable and switching to an available one for
1967 LTTng to record the next events) introduces noticeable CPU overhead.
1968 Knowing this, the following list presents a few practical situations
1969 along with how to configure the sub-buffer size and count for them:
1970
1971 High event throughput::
1972     In general, prefer large sub-buffers to lower the risk of losing
1973     event records.
1974 +
1975 Having larger sub-buffers also ensures a lower sub-buffer switching
1976 frequency.
1977 +
1978 The sub-buffer count is only meaningful if you create the channel in
1979 <<overwrite-mode,overwrite mode>>: in this case, if LTTng overwrites a
1980 sub-buffer, then the other sub-buffers are left unaltered.
1981
1982 Low event throughput::
1983     In general, prefer smaller sub-buffers since the risk of losing
1984     event records is low.
1985 +
1986 Because LTTng emits events less frequently, the sub-buffer switching
1987 frequency should remain low and therefore the overhead of the tracer
1988 shouldn't be a problem.
1989
1990 Low memory system::
1991     If your target system has a low memory limit, prefer fewer first,
1992     then smaller sub-buffers.
1993 +
1994 Even if the system is limited in memory, you want to keep the
1995 sub-buffers as large as possible to avoid a high sub-buffer switching
1996 frequency.
1997
1998 Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format,
1999 which means event record data is very compact. For example, the average
2000 LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a
2001 sub-buffer size of 1{nbsp}MiB is considered large.
2002
2003 The previous scenarios highlight the major trade-off between a few large
2004 sub-buffers and more, smaller sub-buffers: sub-buffer switching
2005 frequency vs. how many event records are lost in overwrite mode.
2006 Assuming a constant event throughput and using the overwrite mode, the
2007 two following configurations have the same ring buffer total size:
2008
2009 [NOTE]
2010 [role="docsvg-channel-subbuf-size-vs-count-anim"]
2011 ====
2012 {note-no-anim}
2013 ====
2014
2015 Two sub-buffers of 4{nbsp}MiB each::
2016     Expect a very low sub-buffer switching frequency, but if LTTng
2017     ever needs to overwrite a sub-buffer, half of the event records so
2018     far (4{nbsp}MiB) are definitely lost.
2019
2020 Eight sub-buffers of 1{nbsp}MiB each::
2021     Expect four times the tracer overhead of the configuration above,
2022     but if LTTng needs to overwrite a sub-buffer, only the eighth of
2023     event records so far (1{nbsp}MiB) are definitely lost.
2024
2025 In <<discard-mode,discard mode>>, the sub-buffer count parameter is
2026 pointless: use two sub-buffers and set their size according to your
2027 requirements.
2028
2029
2030 [[tracefile-rotation]]
2031 ==== Maximum trace file size and count (trace file rotation)
2032
2033 By default, trace files can grow as large as needed.
2034
2035 Set the maximum size of each trace file that LTTng writes of a given
2036 channel when you <<enabling-disabling-channels,create it>>.
2037
2038 When the size of a trace file reaches the fixed maximum size of the
2039 channel, LTTng creates another file to contain the next event records.
2040 LTTng appends a file count to each trace file name in this case.
2041
2042 If you set the trace file size attribute when you create a channel, the
2043 maximum number of trace files that LTTng creates is _unlimited_ by
2044 default. To limit them, set a maximum number of trace files. When the
2045 number of trace files reaches the fixed maximum count of the channel,
2046 LTTng overwrites the oldest trace file. This mechanism is called _trace
2047 file rotation_.
2048
2049 [IMPORTANT]
2050 ====
2051 Even if you don't limit the trace file count, always assume that LTTng
2052 manages all the trace files of the recording session.
2053
2054 In other words, there's no safe way to know if LTTng still holds a given
2055 trace file open with the trace file rotation feature.
2056
2057 The only way to obtain an unmanaged, self-contained LTTng trace before
2058 you <<creating-destroying-tracing-sessions,destroy the recording session>>
2059 is with the <<session-rotation,recording session rotation>> feature, which
2060 is available since LTTng{nbsp}2.11.
2061 ====
2062
2063
2064 [[channel-timers]]
2065 ==== Timers
2066
2067 Each channel can have up to three optional timers:
2068
2069 [[channel-switch-timer]]Switch timer::
2070     When this timer expires, a sub-buffer switch happens: for each ring
2071     buffer of the channel, LTTng marks the current sub-buffer as
2072     consumable and _switches_ to an available one to record the next
2073     events.
2074 +
2075 [NOTE]
2076 [role="docsvg-channel-switch-timer"]
2077 ====
2078 {note-no-anim}
2079 ====
2080 +
2081 A switch timer is useful to ensure that LTTng consumes and commits trace
2082 data to trace files or to a distant <<lttng-relayd,relay daemon>>
2083 periodically in case of a low event throughput.
2084 +
2085 Such a timer is also convenient when you use large
2086 <<channel-subbuf-size-vs-subbuf-count,sub-buffers>> to cope with a
2087 sporadic high event throughput, even if the throughput is otherwise low.
2088 +
2089 Set the period of the switch timer of a channel when you
2090 <<enabling-disabling-channels,create it>> with
2091 the opt:lttng-enable-channel(1):--switch-timer option.
2092
2093 [[channel-read-timer]]Read timer::
2094     When this timer expires, LTTng checks for full, consumable
2095     sub-buffers.
2096 +
2097 By default, the LTTng tracers use an asynchronous message mechanism to
2098 signal a full sub-buffer so that a <<lttng-consumerd,consumer daemon>>
2099 can consume it.
2100 +
2101 When such messages must be avoided, for example in real-time
2102 applications, use this timer instead.
2103 +
2104 Set the period of the read timer of a channel when you
2105 <<enabling-disabling-channels,create it>> with the
2106 opt:lttng-enable-channel(1):--read-timer option.
2107
2108 [[channel-monitor-timer]]Monitor timer::
2109     When this timer expires, the consumer daemon samples some channel
2110     statistics to evaluate the following <<trigger,trigger>>
2111     conditions:
2112 +
2113 --
2114 . The consumed buffer size of a given <<tracing-session,recording
2115   session>> becomes greater than some value.
2116 . The buffer usage of a given channel becomes greater than some value.
2117 . The buffer usage of a given channel becomes less than some value.
2118 --
2119 +
2120 If you disable the monitor timer of a channel{nbsp}__C__:
2121 +
2122 --
2123 * The consumed buffer size value of the recording session of{nbsp}__C__
2124   could be wrong for trigger condition type{nbsp}1: the consumed buffer
2125   size of{nbsp}__C__ won't be part of the grand total.
2126
2127 * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3)
2128   for{nbsp}__C__ will never be satisfied.
2129 --
2130 +
2131 Set the period of the monitor timer of a channel when you
2132 <<enabling-disabling-channels,create it>> with the
2133 opt:lttng-enable-channel(1):--monitor-timer option.
2134
2135
2136 [[event]]
2137 === Recording event rule and event record
2138
2139 A _recording event rule_ is a specific type of <<event-rule,event rule>>
2140 of which the action is to serialize and record the matched event as an
2141 _event record_.
2142
2143 Set the explicit conditions of a recording event rule when you
2144 <<enabling-disabling-events,create it>>. A recording event rule also has
2145 the following implicit conditions:
2146
2147 * The recording event rule itself is enabled.
2148 +
2149 A recording event rule is enabled on creation.
2150
2151 * The <<channel,channel>> to which the recording event rule is attached
2152   is enabled.
2153 +
2154 A channel is enabled on creation.
2155
2156 * The <<tracing-session,recording session>> of the recording event rule is
2157   <<basic-tracing-session-control,active>> (started).
2158 +
2159 A recording session is inactive (stopped) on creation.
2160
2161 * The process for which LTTng creates an event to match is
2162   <<pid-tracking,allowed to record events>>.
2163 +
2164 All processes are allowed to record events on recording session
2165 creation.
2166
2167 You always attach a recording event rule to a channel, which belongs to
2168 a recording session, when you create it.
2169
2170 When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__,
2171 LTTng attempts to serialize and record{nbsp}__E__ to one of the
2172 available sub-buffers of the channel to which{nbsp}__E__ is attached.
2173
2174 When multiple matching recording event rules are attached to the same
2175 channel, LTTng attempts to serialize and record the matched event
2176 _once_. In the following example, the second recording event rule is
2177 redundant when both are enabled:
2178
2179 [role="term"]
2180 ----
2181 $ lttng enable-event --userspace hello:world
2182 $ lttng enable-event --userspace hello:world --loglevel=INFO
2183 ----
2184
2185 [role="img-100"]
2186 .Logical path from an instrumentation point to an event record.
2187 image::event-rule.png[]
2188
2189 As of LTTng{nbsp}{revision}, you cannot remove a recording event
2190 rule: it exists as long as its recording session exists.
2191
2192
2193 [[plumbing]]
2194 == Components of noch:{LTTng}
2195
2196 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
2197 to call LTTng a simple _tool_ since it's composed of multiple
2198 interacting components.
2199
2200 This section describes those components, explains their respective
2201 roles, and shows how they connect together to form the LTTng ecosystem.
2202
2203 The following diagram shows how the most important components of LTTng
2204 interact with user applications, the Linux kernel, and you:
2205
2206 [role="img-100"]
2207 .Control and trace data paths between LTTng components.
2208 image::plumbing.png[]
2209
2210 The LTTng project integrates:
2211
2212 LTTng-tools::
2213     Libraries and command-line interface to control recording sessions:
2214 +
2215 * <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
2216 * <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
2217 * <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
2218 * <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
2219 * <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
2220 * <<persistent-memory-file-systems,`lttng-crash` command-line tool>>
2221   (man:lttng-crash(1)).
2222
2223 LTTng-UST::
2224     Libraries and Java/Python packages to instrument and trace user
2225     applications:
2226 +
2227 * <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
2228   headers to instrument and trace any native user application.
2229 * <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
2230 ** `liblttng-ust-libc-wrapper`
2231 ** `liblttng-ust-pthread-wrapper`
2232 ** `liblttng-ust-cyg-profile`
2233 ** `liblttng-ust-cyg-profile-fast`
2234 ** `liblttng-ust-dl`
2235 * <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
2236   Java applications using `java.util.logging` or
2237   Apache log4j{nbsp}1.2 logging.
2238 * <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
2239   Python applications using the standard `logging` package.
2240
2241 LTTng-modules::
2242     <<lttng-modules,Linux kernel modules>> to instrument and trace the
2243     kernel:
2244 +
2245 * LTTng kernel tracer module.
2246 * Recording ring buffer kernel modules.
2247 * Probe kernel modules.
2248 * LTTng logger kernel module.
2249
2250
2251 [[lttng-cli]]
2252 === Tracing control command-line interface
2253
2254 The _man:lttng(1) command-line tool_ is the standard user interface to
2255 control LTTng <<tracing-session,recording sessions>>.
2256
2257 The cmd:lttng tool is part of LTTng-tools.
2258
2259 The cmd:lttng tool is linked with
2260 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
2261 one or more <<lttng-sessiond,session daemons>> behind the scenes.
2262
2263 The cmd:lttng tool has a Git-like interface:
2264
2265 [role="term"]
2266 ----
2267 $ lttng [GENERAL OPTIONS] <COMMAND> [COMMAND OPTIONS]
2268 ----
2269
2270 The ``<<controlling-tracing,Tracing control>>'' section explores the
2271 available features of LTTng through its cmd:lttng tool.
2272
2273
2274 [[liblttng-ctl-lttng]]
2275 === Tracing control library
2276
2277 [role="img-100"]
2278 .The tracing control library.
2279 image::plumbing-liblttng-ctl.png[]
2280
2281 The _LTTng control library_, `liblttng-ctl`, is used to communicate with
2282 a <<lttng-sessiond,session daemon>> using a C{nbsp}API that hides the
2283 underlying details of the protocol.
2284
2285 `liblttng-ctl` is part of LTTng-tools.
2286
2287 The <<lttng-cli,cmd:lttng command-line tool>> is linked with
2288 `liblttng-ctl`.
2289
2290 Use `liblttng-ctl` in C or $$C++$$ source code by including its
2291 ``master'' header:
2292
2293 [source,c]
2294 ----
2295 #include <lttng/lttng.h>
2296 ----
2297
2298 As of LTTng{nbsp}{revision}, the best available developer documentation
2299 for `liblttng-ctl` is its installed header files. Functions and
2300 structures are documented with header comments.
2301
2302
2303 [[lttng-ust]]
2304 === User space tracing library
2305
2306 [role="img-100"]
2307 .The user space tracing library.
2308 image::plumbing-liblttng-ust.png[]
2309
2310 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
2311 is the LTTng user space tracer.
2312
2313 `liblttng-ust` receives commands from a <<lttng-sessiond,session
2314 daemon>>, for example to allow specific instrumentation points to emit
2315 LTTng <<event-rule,events>>, and writes event records to <<channel,ring
2316 buffers>> shared with a <<lttng-consumerd,consumer daemon>>.
2317
2318 `liblttng-ust` is part of LTTng-UST.
2319
2320 `liblttng-ust` can also send asynchronous messages to the session daemon
2321 when it emits an event. This supports the ``event rule matches''
2322 <<trigger,trigger>> condition feature (see
2323 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2324 to a session daemon>>”).
2325
2326 Public C{nbsp}header files are installed beside `liblttng-ust` to
2327 instrument any <<c-application,C or $$C++$$ application>>.
2328
2329 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
2330 packages, use their own <<tracepoint-provider,tracepoint provider
2331 package>> which is linked with `liblttng-ust`.
2332
2333 An application or library doesn't have to initialize `liblttng-ust`
2334 manually: its constructor does the necessary tasks to register the
2335 application to a session daemon. The initialization phase also
2336 configures instrumentation points depending on the <<event-rule,event
2337 rules>> that you already created.
2338
2339
2340 [[lttng-ust-agents]]
2341 === User space tracing agents
2342
2343 [role="img-100"]
2344 .The user space tracing agents.
2345 image::plumbing-lttng-ust-agents.png[]
2346
2347 The _LTTng-UST Java and Python agents_ are regular Java and Python
2348 packages which add LTTng tracing capabilities to the
2349 native logging frameworks.
2350
2351 The LTTng-UST agents are part of LTTng-UST.
2352
2353 In the case of Java, the
2354 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
2355 core logging facilities] and
2356 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
2357 Note that Apache Log4j{nbsp}2 isn't supported.
2358
2359 In the case of Python, the standard
2360 https://docs.python.org/3/library/logging.html[`logging`] package
2361 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
2362 LTTng-UST Python agent package.
2363
2364 The applications using the LTTng-UST agents are in the
2365 `java.util.logging` (JUL), log4j, and Python <<domain,tracing domains>>.
2366
2367 Both agents use the same mechanism to convert log statements to LTTng
2368 events. When an agent initializes, it creates a log handler that
2369 attaches to the root logger. The agent also registers to a
2370 <<lttng-sessiond,session daemon>>. When the user application executes a
2371 log statement, the root logger passes it to the log handler of the
2372 agent. The custom log handler of the agent calls a native function in a
2373 tracepoint provider package shared library linked with
2374 <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
2375 other fields, like its logger name and its log level. This native
2376 function contains a user space instrumentation point, therefore tracing
2377 the log statement.
2378
2379 The log level condition of a <<event,recording event rule>> is
2380 considered when tracing a Java or a Python application, and it's
2381 compatible with the standard `java.util.logging`, log4j, and Python log
2382 levels.
2383
2384
2385 [[lttng-modules]]
2386 === LTTng kernel modules
2387
2388 [role="img-100"]
2389 .The LTTng kernel modules.
2390 image::plumbing-lttng-modules.png[]
2391
2392 The _LTTng kernel modules_ are a set of Linux kernel modules
2393 which implement the kernel tracer of the LTTng project.
2394
2395 The LTTng kernel modules are part of LTTng-modules.
2396
2397 The LTTng kernel modules include:
2398
2399 * A set of _probe_ modules.
2400 +
2401 Each module attaches to a specific subsystem
2402 of the Linux kernel using its tracepoint instrument points.
2403 +
2404 There are also modules to attach to the entry and return points of the
2405 Linux system call functions.
2406
2407 * _Ring buffer_ modules.
2408 +
2409 A ring buffer implementation is provided as kernel modules. The LTTng
2410 kernel tracer writes to ring buffers; a
2411 <<lttng-consumerd,consumer daemon>> reads from ring buffers.
2412
2413 * The _LTTng kernel tracer_ module.
2414 * The <<proc-lttng-logger-abi,_LTTng logger_>> module.
2415 +
2416 The LTTng logger module implements the special path:{/proc/lttng-logger}
2417 (and path:{/dev/lttng-logger}, since LTTng{nbsp}2.11) files so that any
2418 executable can generate LTTng events by opening those files and
2419 writing to them.
2420
2421 The LTTng kernel tracer can also send asynchronous messages to the
2422 <<lttng-sessiond,session daemon>> when it emits an event.
2423 This supports the ``event rule matches''
2424 <<trigger,trigger>> condition feature (see
2425 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2426 to a session daemon>>”).
2427
2428 Generally, you don't have to load the LTTng kernel modules manually
2429 (using man:modprobe(8), for example): a root session daemon loads the
2430 necessary modules when starting. If you have extra probe modules, you
2431 can specify to load them to the session daemon on the command line
2432 (see the opt:lttng-sessiond(8):--extra-kmod-probes option). See also
2433 <<linux-kernel-sig,Linux kernel module signature>>.
2434
2435 The LTTng kernel modules are installed in
2436 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
2437 the kernel release (output of `uname --kernel-release`).
2438
2439
2440 [[lttng-sessiond]]
2441 === Session daemon
2442
2443 [role="img-100"]
2444 .The session daemon.
2445 image::plumbing-sessiond.png[]
2446
2447 The _session daemon_, man:lttng-sessiond(8), is a
2448 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which:
2449
2450 * Manages <<tracing-session,recording sessions>>.
2451
2452 * Controls the various components (like tracers and
2453   <<lttng-consumerd,consumer daemons>>) of LTTng.
2454
2455 * Sends <<notif-trigger-api,asynchronous notifications>> to user
2456   applications.
2457
2458 The session daemon is part of LTTng-tools.
2459
2460 The session daemon sends control requests to and receives control
2461 responses from:
2462
2463 * The <<lttng-ust,user space tracing library>>.
2464 +
2465 Any instance of the user space tracing library first registers to
2466 a session daemon. Then, the session daemon can send requests to
2467 this instance, such as:
2468 +
2469 --
2470 ** Get the list of tracepoints.
2471 ** Share a <<event,recording event rule>> so that the user space tracing
2472    library can decide whether or not a given tracepoint can emit events.
2473    Amongst the possible conditions of a recording event rule is a filter
2474    expression which `liblttng-ust` evaluates before it emits an event.
2475 ** Share <<channel,channel>> attributes and ring buffer locations.
2476 --
2477 +
2478 The session daemon and the user space tracing library use a Unix
2479 domain socket to communicate.
2480
2481 * The <<lttng-ust-agents,user space tracing agents>>.
2482 +
2483 Any instance of a user space tracing agent first registers to
2484 a session daemon. Then, the session daemon can send requests to
2485 this instance, such as:
2486 +
2487 --
2488 ** Get the list of loggers.
2489 ** Enable or disable a specific logger.
2490 --
2491 +
2492 The session daemon and the user space tracing agent use a TCP connection
2493 to communicate.
2494
2495 * The <<lttng-modules,LTTng kernel tracer>>.
2496 * The <<lttng-consumerd,consumer daemon>>.
2497 +
2498 The session daemon sends requests to the consumer daemon to instruct
2499 it where to send the trace data streams, amongst other information.
2500
2501 * The <<lttng-relayd,relay daemon>>.
2502
2503 The session daemon receives commands from the
2504 <<liblttng-ctl-lttng,tracing control library>>.
2505
2506 The session daemon can receive asynchronous messages from the
2507 <<lttng-ust,user space>> and <<lttng-modules,kernel>> tracers
2508 when they emit events. This supports the ``event rule matches''
2509 <<trigger,trigger>> condition feature (see
2510 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2511 to a session daemon>>”).
2512
2513 The root session daemon loads the appropriate
2514 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2515 one or more <<lttng-consumerd,consumer daemons>> as soon as you create
2516 a <<event,recording event rule>>.
2517
2518 The session daemon doesn't send and receive trace data: this is the
2519 role of the <<lttng-consumerd,consumer daemon>> and
2520 <<lttng-relayd,relay daemon>>. It does, however, generate the
2521 https://diamon.org/ctf/[CTF] metadata stream.
2522
2523 Each Unix user can have its own session daemon instance. The
2524 recording sessions which different session daemons manage are completely
2525 independent.
2526
2527 The root user's session daemon is the only one which is
2528 allowed to control the LTTng kernel tracer, and its spawned consumer
2529 daemon is the only one which is allowed to consume trace data from the
2530 LTTng kernel tracer. Note, however, that any Unix user which is a member
2531 of the <<tracing-group,tracing group>> is allowed
2532 to create <<channel,channels>> in the
2533 Linux kernel <<domain,tracing domain>>, and therefore to use the Linux
2534 kernel LTTng tracer.
2535
2536 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2537 session daemon when using its `create` command if none is currently
2538 running. You can also start the session daemon manually.
2539
2540
2541 [[lttng-consumerd]]
2542 === Consumer daemon
2543
2544 [role="img-100"]
2545 .The consumer daemon.
2546 image::plumbing-consumerd.png[]
2547
2548 The _consumer daemon_, cmd:lttng-consumerd, is a
2549 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which shares
2550 ring buffers with user applications or with the LTTng kernel modules to
2551 collect trace data and send it to some location (file system or to a
2552 <<lttng-relayd,relay daemon>> over the network).
2553
2554 The consumer daemon is part of LTTng-tools.
2555
2556 You don't start a consumer daemon manually: a consumer daemon is always
2557 spawned by a <<lttng-sessiond,session daemon>> as soon as you create a
2558 <<event,recording event rule>>, that is, before you start recording. When
2559 you kill its owner session daemon, the consumer daemon also exits
2560 because it's the child process of the session daemon. Command-line
2561 options of man:lttng-sessiond(8) target the consumer daemon process.
2562
2563 There are up to two running consumer daemons per Unix user, whereas only
2564 one session daemon can run per user. This is because each process can be
2565 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2566 and 64-bit processes, it's more efficient to have separate
2567 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2568 exception: it can have up to _three_ running consumer daemons: 32-bit
2569 and 64-bit instances for its user applications, and one more
2570 reserved for collecting kernel trace data.
2571
2572
2573 [[lttng-relayd]]
2574 === Relay daemon
2575
2576 [role="img-100"]
2577 .The relay daemon.
2578 image::plumbing-relayd.png[]
2579
2580 The _relay daemon_, man:lttng-relayd(8), is a
2581 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] acting as a bridge
2582 between remote session and consumer daemons, local trace files, and a
2583 remote live trace reader.
2584
2585 The relay daemon is part of LTTng-tools.
2586
2587 The main purpose of the relay daemon is to implement a receiver of
2588 <<sending-trace-data-over-the-network,trace data over the network>>.
2589 This is useful when the target system doesn't have much file system
2590 space to write trace files locally.
2591
2592 The relay daemon is also a server to which a
2593 <<lttng-live,live trace reader>> can
2594 connect. The live trace reader sends requests to the relay daemon to
2595 receive trace data as the target system records events. The
2596 communication protocol is named _LTTng live_; it's used over TCP
2597 connections.
2598
2599 Note that you can start the relay daemon on the target system directly.
2600 This is the setup of choice when the use case is to view/analyze events
2601 as the target system records them without the need of a remote system.
2602
2603
2604 [[instrumenting]]
2605 == [[using-lttng]]Instrumentation
2606
2607 There are many examples of tracing and monitoring in our everyday life:
2608
2609 * You have access to real-time and historical weather reports and
2610   forecasts thanks to weather stations installed around the country.
2611 * You know your heart is safe thanks to an electrocardiogram.
2612 * You make sure not to drive your car too fast and to have enough fuel
2613   to reach your destination thanks to gauges visible on your dashboard.
2614
2615 All the previous examples have something in common: they rely on
2616 **instruments**. Without the electrodes attached to the surface of your
2617 body skin, cardiac monitoring is futile.
2618
2619 LTTng, as a tracer, is no different from those real life examples. If
2620 you're about to trace a software system or, in other words, record its
2621 history of execution, you better have **instrumentation points** in the
2622 subject you're tracing, that is, the actual software system.
2623
2624 <<instrumentation-point-types,Various ways>> were developed to
2625 instrument a piece of software for LTTng tracing. The most
2626 straightforward one is to manually place static instrumentation points,
2627 called _tracepoints_, in the source code of the application. The Linux
2628 kernel <<domain,tracing domain>> also makes it possible to dynamically
2629 add instrumentation points.
2630
2631 If you're only interested in tracing the Linux kernel, your
2632 instrumentation needs are probably already covered by the built-in
2633 <<lttng-modules,Linux kernel instrumentation points>> of LTTng. You may
2634 also wish to have LTTng trace a user application which is already
2635 instrumented for LTTng tracing. In such cases, skip this whole section
2636 and read the topics of the ``<<controlling-tracing,Tracing control>>''
2637 section.
2638
2639 Many methods are available to instrument a piece of software for LTTng
2640 tracing:
2641
2642 * <<c-application,Instrument a C/$$C++$$ user application>>.
2643 * <<prebuilt-ust-helpers,Load a prebuilt user space tracing helper>>.
2644 * <<java-application,Instrument a Java application>>.
2645 * <<python-application,Instrument a Python application>>.
2646 * <<proc-lttng-logger-abi,Use the LTTng logger>>.
2647 * <<instrumenting-linux-kernel,Instrument a Linux kernel image or module>>.
2648
2649
2650 [[c-application]]
2651 === [[cxx-application]]Instrument a C/$$C++$$ user application
2652
2653 The high level procedure to instrument a C or $$C++$$ user application
2654 with the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`,
2655 is:
2656
2657 . <<tracepoint-provider,Create the source files of a tracepoint provider
2658   package>>.
2659
2660 . <<probing-the-application-source-code,Add tracepoints to
2661   the source code of the application>>.
2662
2663 . <<building-tracepoint-providers-and-user-application,Build and link
2664   a tracepoint provider package and the user application>>.
2665
2666 If you need quick, man:printf(3)-like instrumentation, skip those steps
2667 and use <<tracef,`lttng_ust_tracef()`>> or
2668 <<tracelog,`lttng_ust_tracelog()`>> instead.
2669
2670 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2671 instrument a user application with `liblttng-ust`.
2672
2673
2674 [[tracepoint-provider]]
2675 ==== Create the source files of a tracepoint provider package
2676
2677 A _tracepoint provider_ is a set of compiled functions which provide
2678 **tracepoints** to an application, the type of instrumentation point
2679 which LTTng-UST provides.
2680
2681 Those functions can make LTTng emit events with user-defined fields and
2682 serialize those events as event records to one or more LTTng-UST
2683 <<channel,channel>> sub-buffers. The `lttng_ust_tracepoint()` macro,
2684 which you <<probing-the-application-source-code,insert in the source
2685 code of a user application>>, calls those functions.
2686
2687 A _tracepoint provider package_ is an object file (`.o`) or a shared
2688 library (`.so`) which contains one or more tracepoint providers. Its
2689 source files are:
2690
2691 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2692 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2693
2694 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2695 the LTTng user space tracer, at run time.
2696
2697 [role="img-100"]
2698 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2699 image::ust-app.png[]
2700
2701 NOTE: If you need quick, man:printf(3)-like instrumentation, skip
2702 creating and using a tracepoint provider and use
2703 <<tracef,`lttng_ust_tracef()`>> or <<tracelog,`lttng_ust_tracelog()`>>
2704 instead.
2705
2706
2707 [[tpp-header]]
2708 ===== Create a tracepoint provider header file template
2709
2710 A _tracepoint provider header file_ contains the tracepoint definitions
2711 of a tracepoint provider.
2712
2713 To create a tracepoint provider header file:
2714
2715 . Start from this template:
2716 +
2717 --
2718 [source,c]
2719 .Tracepoint provider header file template (`.h` file extension).
2720 ----
2721 #undef LTTNG_UST_TRACEPOINT_PROVIDER
2722 #define LTTNG_UST_TRACEPOINT_PROVIDER provider_name
2723
2724 #undef LTTNG_UST_TRACEPOINT_INCLUDE
2725 #define LTTNG_UST_TRACEPOINT_INCLUDE "./tp.h"
2726
2727 #if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
2728 #define _TP_H
2729
2730 #include <lttng/tracepoint.h>
2731
2732 /*
2733  * Use LTTNG_UST_TRACEPOINT_EVENT(), LTTNG_UST_TRACEPOINT_EVENT_CLASS(),
2734  * LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(), and
2735  * LTTNG_UST_TRACEPOINT_LOGLEVEL() here.
2736  */
2737
2738 #endif /* _TP_H */
2739
2740 #include <lttng/tracepoint-event.h>
2741 ----
2742 --
2743 +
2744 Replace:
2745 +
2746 * +__provider_name__+ with the name of your tracepoint provider.
2747 * `"tp.h"` with the name of your tracepoint provider header file.
2748
2749 . Below the `#include <lttng/tracepoint.h>` line, put your
2750   <<defining-tracepoints,tracepoint definitions>>.
2751
2752 Your tracepoint provider name must be unique amongst all the possible
2753 tracepoint provider names used on the same target system. We suggest to
2754 include the name of your project or company in the name, for example,
2755 `org_lttng_my_project_tpp`.
2756
2757
2758 [[defining-tracepoints]]
2759 ===== Create a tracepoint definition
2760
2761 A _tracepoint definition_ defines, for a given tracepoint:
2762
2763 * Its **input arguments**.
2764 +
2765 They're the macro parameters that the `lttng_ust_tracepoint()` macro
2766 accepts for this particular tracepoint in the source code of the user
2767 application.
2768
2769 * Its **output event fields**.
2770 +
2771 They're the sources of event fields that form the payload of any event
2772 that the execution of the `lttng_ust_tracepoint()` macro emits for this
2773 particular tracepoint.
2774
2775 Create a tracepoint definition with the
2776 `LTTNG_UST_TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2777 line in the
2778 <<tpp-header,tracepoint provider header file template>>.
2779
2780 The syntax of the `LTTNG_UST_TRACEPOINT_EVENT()` macro is:
2781
2782 [source,c]
2783 .`LTTNG_UST_TRACEPOINT_EVENT()` macro syntax.
2784 ----
2785 LTTNG_UST_TRACEPOINT_EVENT(
2786     /* Tracepoint provider name */
2787     provider_name,
2788
2789     /* Tracepoint name */
2790     tracepoint_name,
2791
2792     /* Input arguments */
2793     LTTNG_UST_TP_ARGS(
2794         arguments
2795     ),
2796
2797     /* Output event fields */
2798     LTTNG_UST_TP_FIELDS(
2799         fields
2800     )
2801 )
2802 ----
2803
2804 Replace:
2805
2806 * +__provider_name__+ with your tracepoint provider name.
2807 * +__tracepoint_name__+ with your tracepoint name.
2808 * +__arguments__+ with the <<tpp-def-input-args,input arguments>>.
2809 * +__fields__+ with the <<tpp-def-output-fields,output event field>>
2810   definitions.
2811
2812 The full name of this tracepoint is `provider_name:tracepoint_name`.
2813
2814 [IMPORTANT]
2815 .Event name length limitation
2816 ====
2817 The concatenation of the tracepoint provider name and the tracepoint
2818 name must not exceed **254{nbsp}characters**. If it does, the
2819 instrumented application compiles and runs, but LTTng throws multiple
2820 warnings and you could experience serious issues.
2821 ====
2822
2823 [[tpp-def-input-args]]The syntax of the `LTTNG_UST_TP_ARGS()` macro is:
2824
2825 [source,c]
2826 .`LTTNG_UST_TP_ARGS()` macro syntax.
2827 ----
2828 LTTNG_UST_TP_ARGS(
2829     type, arg_name
2830 )
2831 ----
2832
2833 Replace:
2834
2835 * +__type__+ with the C{nbsp}type of the argument.
2836 * +__arg_name__+ with the argument name.
2837
2838 You can repeat +__type__+ and +__arg_name__+ up to 10{nbsp}times to have
2839 more than one argument.
2840
2841 .`LTTNG_UST_TP_ARGS()` usage with three arguments.
2842 ====
2843 [source,c]
2844 ----
2845 LTTNG_UST_TP_ARGS(
2846     int, count,
2847     float, ratio,
2848     const char*, query
2849 )
2850 ----
2851 ====
2852
2853 The `LTTNG_UST_TP_ARGS()` and `LTTNG_UST_TP_ARGS(void)` forms are valid
2854 to create a tracepoint definition with no input arguments.
2855
2856 [[tpp-def-output-fields]]The `LTTNG_UST_TP_FIELDS()` macro contains a
2857 list of `lttng_ust_field_*()` macros. Each `lttng_ust_field_*()` macro
2858 defines one event field. See man:lttng-ust(3) for a complete description
2859 of the available `lttng_ust_field_*()` macros. A `lttng_ust_field_*()`
2860 macro specifies the type, size, and byte order of one event field.
2861
2862 Each `lttng_ust_field_*()` macro takes an _argument expression_
2863 parameter. This is a C{nbsp}expression that the tracer evaluates at the
2864 `lttng_ust_tracepoint()` macro site in the source code of the
2865 application. This expression provides the source of data of a field. The
2866 argument expression can include input argument names listed in the
2867 `LTTNG_UST_TP_ARGS()` macro.
2868
2869 Each `lttng_ust_field_*()` macro also takes a _field name_ parameter.
2870 Field names must be unique within a given tracepoint definition.
2871
2872 Here's a complete tracepoint definition example:
2873
2874 .Tracepoint definition.
2875 ====
2876 The following tracepoint definition defines a tracepoint which takes
2877 three input arguments and has four output event fields.
2878
2879 [source,c]
2880 ----
2881 #include "my-custom-structure.h"
2882
2883 LTTNG_UST_TRACEPOINT_EVENT(
2884     my_provider,
2885     my_tracepoint,
2886     LTTNG_UST_TP_ARGS(
2887         const struct my_custom_structure *, my_custom_structure,
2888         float, ratio,
2889         const char *, query
2890     ),
2891     LTTNG_UST_TP_FIELDS(
2892         lttng_ust_field_string(query_field, query)
2893         lttng_ust_field_float(double, ratio_field, ratio)
2894         lttng_ust_field_integer(int, recv_size,
2895                                 my_custom_structure->recv_size)
2896         lttng_ust_field_integer(int, send_size,
2897                                 my_custom_structure->send_size)
2898     )
2899 )
2900 ----
2901
2902 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
2903 macro in the source code of your application like this:
2904
2905 [source,c]
2906 ----
2907 lttng_ust_tracepoint(my_provider, my_tracepoint,
2908                      my_structure, some_ratio, the_query);
2909 ----
2910 ====
2911
2912 NOTE: The LTTng-UST tracer only evaluates the arguments of a tracepoint
2913 at run time when such a tracepoint _could_ emit an event. See
2914 <<event-creation-emission-opti,this note>> to learn more.
2915
2916
2917 [[using-tracepoint-classes]]
2918 ===== Use a tracepoint class
2919
2920 A _tracepoint class_ is a class of tracepoints which share the same
2921 output event field definitions. A _tracepoint instance_ is one
2922 instance of such a defined tracepoint class, with its own tracepoint
2923 name.
2924
2925 The <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()` macro>> is
2926 actually a shorthand which defines both a tracepoint class and a
2927 tracepoint instance at the same time.
2928
2929 When you build a tracepoint provider package, the C or $$C++$$ compiler
2930 creates one serialization function for each **tracepoint class**. A
2931 serialization function is responsible for serializing the event fields
2932 of a tracepoint to a sub-buffer when recording.
2933
2934 For various performance reasons, when your situation requires multiple
2935 tracepoint definitions with different names, but with the same event
2936 fields, we recommend that you manually create a tracepoint class and
2937 instantiate as many tracepoint instances as needed. One positive effect
2938 of such a design, amongst other advantages, is that all tracepoint
2939 instances of the same tracepoint class reuse the same serialization
2940 function, thus reducing
2941 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2942
2943 .Use a tracepoint class and tracepoint instances.
2944 ====
2945 Consider the following three tracepoint definitions:
2946
2947 [source,c]
2948 ----
2949 LTTNG_UST_TRACEPOINT_EVENT(
2950     my_app,
2951     get_account,
2952     LTTNG_UST_TP_ARGS(
2953         int, userid,
2954         size_t, len
2955     ),
2956     LTTNG_UST_TP_FIELDS(
2957         lttng_ust_field_integer(int, userid, userid)
2958         lttng_ust_field_integer(size_t, len, len)
2959     )
2960 )
2961
2962 LTTNG_UST_TRACEPOINT_EVENT(
2963     my_app,
2964     get_settings,
2965     LTTNG_UST_TP_ARGS(
2966         int, userid,
2967         size_t, len
2968     ),
2969     LTTNG_UST_TP_FIELDS(
2970         lttng_ust_field_integer(int, userid, userid)
2971         lttng_ust_field_integer(size_t, len, len)
2972     )
2973 )
2974
2975 LTTNG_UST_TRACEPOINT_EVENT(
2976     my_app,
2977     get_transaction,
2978     LTTNG_UST_TP_ARGS(
2979         int, userid,
2980         size_t, len
2981     ),
2982     LTTNG_UST_TP_FIELDS(
2983         lttng_ust_field_integer(int, userid, userid)
2984         lttng_ust_field_integer(size_t, len, len)
2985     )
2986 )
2987 ----
2988
2989 In this case, we create three tracepoint classes, with one implicit
2990 tracepoint instance for each of them: `get_account`, `get_settings`, and
2991 `get_transaction`. However, they all share the same event field names
2992 and types. Hence three identical, yet independent serialization
2993 functions are created when you build the tracepoint provider package.
2994
2995 A better design choice is to define a single tracepoint class and three
2996 tracepoint instances:
2997
2998 [source,c]
2999 ----
3000 /* The tracepoint class */
3001 LTTNG_UST_TRACEPOINT_EVENT_CLASS(
3002     /* Tracepoint class provider name */
3003     my_app,
3004
3005     /* Tracepoint class name */
3006     my_class,
3007
3008     /* Input arguments */
3009     LTTNG_UST_TP_ARGS(
3010         int, userid,
3011         size_t, len
3012     ),
3013
3014     /* Output event fields */
3015     LTTNG_UST_TP_FIELDS(
3016         lttng_ust_field_integer(int, userid, userid)
3017         lttng_ust_field_integer(size_t, len, len)
3018     )
3019 )
3020
3021 /* The tracepoint instances */
3022 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3023     /* Tracepoint class provider name */
3024     my_app,
3025
3026     /* Tracepoint class name */
3027     my_class,
3028
3029     /* Instance provider name */
3030     my_app,
3031
3032     /* Tracepoint name */
3033     get_account,
3034
3035     /* Input arguments */
3036     LTTNG_UST_TP_ARGS(
3037         int, userid,
3038         size_t, len
3039     )
3040 )
3041 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3042     my_app,
3043     my_class,
3044     get_settings,
3045     LTTNG_UST_TP_ARGS(
3046         int, userid,
3047         size_t, len
3048     )
3049 )
3050 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3051     my_app,
3052     my_class,
3053     get_transaction,
3054     LTTNG_UST_TP_ARGS(
3055         int, userid,
3056         size_t, len
3057     )
3058 )
3059 ----
3060 ====
3061
3062 The tracepoint class and instance provider names must be the same if the
3063 `LTTNG_UST_TRACEPOINT_EVENT_CLASS()` and
3064 `LTTNG_UST_TRACEPOINT_EVENT_INSTANCE()` expansions are part of the same
3065 translation unit. See man:lttng-ust(3) to learn more.
3066
3067
3068 [[assigning-log-levels]]
3069 ===== Assign a log level to a tracepoint definition
3070
3071 Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
3072 with the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro.
3073
3074 Assigning different levels of severity to tracepoint definitions can be
3075 useful: when you <<enabling-disabling-events,create a recording event
3076 rule>>, you can target tracepoints having a log level at least as severe
3077 as a specific value.
3078
3079 The concept of LTTng-UST log levels is similar to the levels found
3080 in typical logging frameworks:
3081
3082 * In a logging framework, the log level is given by the function
3083   or method name you use at the log statement site: `debug()`,
3084   `info()`, `warn()`, `error()`, and so on.
3085
3086 * In LTTng-UST, you statically assign the log level to a tracepoint
3087   definition; any `lttng_ust_tracepoint()` macro invocation which refers
3088   to this definition has this log level.
3089
3090 You must use `LTTNG_UST_TRACEPOINT_LOGLEVEL()` _after_ the
3091 <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()`>> or
3092 <<using-tracepoint-classes,`LTTNG_UST_TRACEPOINT_INSTANCE()`>> macro for
3093 a given tracepoint.
3094
3095 The syntax of the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro is:
3096
3097 [source,c]
3098 .`LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro syntax.
3099 ----
3100 LTTNG_UST_TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
3101 ----
3102
3103 Replace:
3104
3105 * +__provider_name__+ with the tracepoint provider name.
3106 * +__tracepoint_name__+ with the tracepoint name.
3107 * +__log_level__+ with the log level to assign to the tracepoint
3108   definition named +__tracepoint_name__+ in the +__provider_name__+
3109   tracepoint provider.
3110 +
3111 See man:lttng-ust(3) for a list of available log level names.
3112
3113 .Assign the `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT` log level to a tracepoint definition.
3114 ====
3115 [source,c]
3116 ----
3117 /* Tracepoint definition */
3118 LTTNG_UST_TRACEPOINT_EVENT(
3119     my_app,
3120     get_transaction,
3121     LTTNG_UST_TP_ARGS(
3122         int, userid,
3123         size_t, len
3124     ),
3125     LTTNG_UST_TP_FIELDS(
3126         lttng_ust_field_integer(int, userid, userid)
3127         lttng_ust_field_integer(size_t, len, len)
3128     )
3129 )
3130
3131 /* Log level assignment */
3132 LTTNG_UST_TRACEPOINT_LOGLEVEL(my_app, get_transaction,
3133                               LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT)
3134 ----
3135 ====
3136
3137
3138 [[tpp-source]]
3139 ===== Create a tracepoint provider package source file
3140
3141 A _tracepoint provider package source file_ is a C source file which
3142 includes a <<tpp-header,tracepoint provider header file>> to expand its
3143 macros into event serialization and other functions.
3144
3145 Use the following tracepoint provider package source file template:
3146
3147 [source,c]
3148 .Tracepoint provider package source file template.
3149 ----
3150 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
3151
3152 #include "tp.h"
3153 ----
3154
3155 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
3156 header file>> name. You may also include more than one tracepoint
3157 provider header file here to create a tracepoint provider package
3158 holding more than one tracepoint providers.
3159
3160
3161 [[probing-the-application-source-code]]
3162 ==== Add tracepoints to the source code of an application
3163
3164 Once you <<tpp-header,create a tracepoint provider header file>>, use
3165 the `lttng_ust_tracepoint()` macro in the source code of your
3166 application to insert the tracepoints that this header
3167 <<defining-tracepoints,defines>>.
3168
3169 The `lttng_ust_tracepoint()` macro takes at least two parameters: the
3170 tracepoint provider name and the tracepoint name. The corresponding
3171 tracepoint definition defines the other parameters.
3172
3173 .`lttng_ust_tracepoint()` usage.
3174 ====
3175 The following <<defining-tracepoints,tracepoint definition>> defines a
3176 tracepoint which takes two input arguments and has two output event
3177 fields.
3178
3179 [source,c]
3180 .Tracepoint provider header file.
3181 ----
3182 #include "my-custom-structure.h"
3183
3184 LTTNG_UST_TRACEPOINT_EVENT(
3185     my_provider,
3186     my_tracepoint,
3187     LTTNG_UST_TP_ARGS(
3188         int, argc,
3189         const char *, cmd_name
3190     ),
3191     LTTNG_UST_TP_FIELDS(
3192         lttng_ust_field_string(cmd_name, cmd_name)
3193         lttng_ust_field_integer(int, number_of_args, argc)
3194     )
3195 )
3196 ----
3197
3198 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3199 macro in the source code of your application like this:
3200
3201 [source,c]
3202 .Application source file.
3203 ----
3204 #include "tp.h"
3205
3206 int main(int argc, char* argv[])
3207 {
3208     lttng_ust_tracepoint(my_provider, my_tracepoint, argc, argv[0]);
3209     return 0;
3210 }
3211 ----
3212
3213 Note how the source code of the application includes
3214 the tracepoint provider header file containing the tracepoint
3215 definitions to use, path:{tp.h}.
3216 ====
3217
3218 .`lttng_ust_tracepoint()` usage with a complex tracepoint definition.
3219 ====
3220 Consider this complex tracepoint definition, where multiple event
3221 fields refer to the same input arguments in their argument expression
3222 parameter:
3223
3224 [source,c]
3225 .Tracepoint provider header file.
3226 ----
3227 /* For `struct stat` */
3228 #include <sys/types.h>
3229 #include <sys/stat.h>
3230 #include <unistd.h>
3231
3232 LTTNG_UST_TRACEPOINT_EVENT(
3233     my_provider,
3234     my_tracepoint,
3235     LTTNG_UST_TP_ARGS(
3236         int, my_int_arg,
3237         char *, my_str_arg,
3238         struct stat *, st
3239     ),
3240     LTTNG_UST_TP_FIELDS(
3241         lttng_ust_field_integer(int, my_constant_field, 23 + 17)
3242         lttng_ust_field_integer(int, my_int_arg_field, my_int_arg)
3243         lttng_ust_field_integer(int, my_int_arg_field2,
3244                                 my_int_arg * my_int_arg)
3245         lttng_ust_field_integer(int, sum4_field,
3246                                 my_str_arg[0] + my_str_arg[1] +
3247                                 my_str_arg[2] + my_str_arg[3])
3248         lttng_ust_field_string(my_str_arg_field, my_str_arg)
3249         lttng_ust_field_integer_hex(off_t, size_field, st->st_size)
3250         lttng_ust_field_float(double, size_dbl_field, (double) st->st_size)
3251         lttng_ust_field_sequence_text(char, half_my_str_arg_field,
3252                                       my_str_arg, size_t,
3253                                       strlen(my_str_arg) / 2)
3254     )
3255 )
3256 ----
3257
3258 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3259 macro in the source code of your application like this:
3260
3261 [source,c]
3262 .Application source file.
3263 ----
3264 #define LTTNG_UST_TRACEPOINT_DEFINE
3265 #include "tp.h"
3266
3267 int main(void)
3268 {
3269     struct stat s;
3270
3271     stat("/etc/fstab", &s);
3272     lttng_ust_tracepoint(my_provider, my_tracepoint, 23,
3273                          "Hello, World!", &s);
3274
3275     return 0;
3276 }
3277 ----
3278
3279 If you look at the event record that LTTng writes when recording this
3280 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
3281 it should look like this:
3282
3283 .Event record fields
3284 |====
3285 |Field name              |Field value
3286 |`my_constant_field`     |40
3287 |`my_int_arg_field`      |23
3288 |`my_int_arg_field2`     |529
3289 |`sum4_field`            |389
3290 |`my_str_arg_field`      |`Hello, World!`
3291 |`size_field`            |0x12d
3292 |`size_dbl_field`        |301.0
3293 |`half_my_str_arg_field` |`Hello,`
3294 |====
3295 ====
3296
3297 Sometimes, the arguments you pass to `lttng_ust_tracepoint()` are
3298 expensive to evaluate--they use the call stack, for example. To avoid
3299 this computation when LTTng wouldn't emit any event anyway, use the
3300 `lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros.
3301
3302 The syntax of the `lttng_ust_tracepoint_enabled()` and
3303 `lttng_ust_do_tracepoint()` macros is:
3304
3305 [source,c]
3306 .`lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros syntax.
3307 ----
3308 lttng_ust_tracepoint_enabled(provider_name, tracepoint_name)
3309
3310 lttng_ust_do_tracepoint(provider_name, tracepoint_name, ...)
3311 ----
3312
3313 Replace:
3314
3315 * +__provider_name__+ with the tracepoint provider name.
3316 * +__tracepoint_name__+ with the tracepoint name.
3317
3318 `lttng_ust_tracepoint_enabled()` returns a non-zero value if executing
3319 the tracepoint named `tracepoint_name` from the provider named
3320 `provider_name` _could_ make LTTng emit an event, depending on the
3321 payload of said event.
3322
3323 `lttng_ust_do_tracepoint()` is like `lttng_ust_tracepoint()`, except
3324 that it doesn't check what `lttng_ust_tracepoint_enabled()` checks.
3325 Using `lttng_ust_tracepoint()` with `lttng_ust_tracepoint_enabled()` is
3326 dangerous because `lttng_ust_tracepoint()` also contains the
3327 `lttng_ust_tracepoint_enabled()` check; therefore, a race condition is
3328 possible in this situation:
3329
3330 [source,c]
3331 .Possible race condition when using `lttng_ust_tracepoint_enabled()` with `lttng_ust_tracepoint()`.
3332 ----
3333 if (lttng_ust_tracepoint_enabled(my_provider, my_tracepoint)) {
3334     stuff = prepare_stuff();
3335 }
3336
3337 lttng_ust_tracepoint(my_provider, my_tracepoint, stuff);
3338 ----
3339
3340 If `lttng_ust_tracepoint_enabled()` is false, but would be true after
3341 the conditional block, then `stuff` isn't prepared: the emitted event
3342 will either contain wrong data, or the whole application could crash
3343 (with a segmentation fault, for example).
3344
3345 NOTE: Neither `lttng_ust_tracepoint_enabled()` nor
3346 `lttng_ust_do_tracepoint()` have an `STAP_PROBEV()` call. If you need
3347 it, you must emit this call yourself.
3348
3349
3350 [[building-tracepoint-providers-and-user-application]]
3351 ==== Build and link a tracepoint provider package and an application
3352
3353 Once you have one or more <<tpp-header,tracepoint provider header
3354 files>> and a <<tpp-source,tracepoint provider package source file>>,
3355 create the tracepoint provider package by compiling its source
3356 file. From here, multiple build and run scenarios are possible. The
3357 following table shows common application and library configurations
3358 along with the required command lines to achieve them.
3359
3360 In the following diagrams, we use the following file names:
3361
3362 `app`::
3363   Executable application.
3364
3365 `app.o`::
3366   Application object file.
3367
3368 `tpp.o`::
3369   Tracepoint provider package object file.
3370
3371 `tpp.a`::
3372   Tracepoint provider package archive file.
3373
3374 `libtpp.so`::
3375   Tracepoint provider package shared object file.
3376
3377 `emon.o`::
3378   User library object file.
3379
3380 `libemon.so`::
3381   User library shared object file.
3382
3383 We use the following symbols in the diagrams of table below:
3384
3385 [role="img-100"]
3386 .Symbols used in the build scenario diagrams.
3387 image::ust-sit-symbols.png[]
3388
3389 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
3390 variable in the following instructions.
3391
3392 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
3393 .Common tracepoint provider package scenarios.
3394 |====
3395 |Scenario |Instructions
3396
3397 |
3398 The instrumented application is statically linked with
3399 the tracepoint provider package object.
3400
3401 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
3402
3403 |
3404 include::../common/ust-sit-step-tp-o.txt[]
3405
3406 To build the instrumented application:
3407
3408 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3409 +
3410 --
3411 [source,c]
3412 ----
3413 #define LTTNG_UST_TRACEPOINT_DEFINE
3414 ----
3415 --
3416
3417 . Compile the application source file:
3418 +
3419 --
3420 [role="term"]
3421 ----
3422 $ gcc -c app.c
3423 ----
3424 --
3425
3426 . Build the application:
3427 +
3428 --
3429 [role="term"]
3430 ----
3431 $ gcc -o app app.o tpp.o -llttng-ust -ldl
3432 ----
3433 --
3434
3435 To run the instrumented application:
3436
3437 * Start the application:
3438 +
3439 --
3440 [role="term"]
3441 ----
3442 $ ./app
3443 ----
3444 --
3445
3446 |
3447 The instrumented application is statically linked with the
3448 tracepoint provider package archive file.
3449
3450 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
3451
3452 |
3453 To create the tracepoint provider package archive file:
3454
3455 . Compile the <<tpp-source,tracepoint provider package source file>>:
3456 +
3457 --
3458 [role="term"]
3459 ----
3460 $ gcc -I. -c tpp.c
3461 ----
3462 --
3463
3464 . Create the tracepoint provider package archive file:
3465 +
3466 --
3467 [role="term"]
3468 ----
3469 $ ar rcs tpp.a tpp.o
3470 ----
3471 --
3472
3473 To build the instrumented application:
3474
3475 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3476 +
3477 --
3478 [source,c]
3479 ----
3480 #define LTTNG_UST_TRACEPOINT_DEFINE
3481 ----
3482 --
3483
3484 . Compile the application source file:
3485 +
3486 --
3487 [role="term"]
3488 ----
3489 $ gcc -c app.c
3490 ----
3491 --
3492
3493 . Build the application:
3494 +
3495 --
3496 [role="term"]
3497 ----
3498 $ gcc -o app app.o tpp.a -llttng-ust -ldl
3499 ----
3500 --
3501
3502 To run the instrumented application:
3503
3504 * Start the application:
3505 +
3506 --
3507 [role="term"]
3508 ----
3509 $ ./app
3510 ----
3511 --
3512
3513 |
3514 The instrumented application is linked with the tracepoint provider
3515 package shared object.
3516
3517 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
3518
3519 |
3520 include::../common/ust-sit-step-tp-so.txt[]
3521
3522 To build the instrumented application:
3523
3524 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3525 +
3526 --
3527 [source,c]
3528 ----
3529 #define LTTNG_UST_TRACEPOINT_DEFINE
3530 ----
3531 --
3532
3533 . Compile the application source file:
3534 +
3535 --
3536 [role="term"]
3537 ----
3538 $ gcc -c app.c
3539 ----
3540 --
3541
3542 . Build the application:
3543 +
3544 --
3545 [role="term"]
3546 ----
3547 $ gcc -o app app.o -ldl -L. -ltpp
3548 ----
3549 --
3550
3551 To run the instrumented application:
3552
3553 * Start the application:
3554 +
3555 --
3556 [role="term"]
3557 ----
3558 $ ./app
3559 ----
3560 --
3561
3562 |
3563 The tracepoint provider package shared object is preloaded before the
3564 instrumented application starts.
3565
3566 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3567
3568 |
3569 include::../common/ust-sit-step-tp-so.txt[]
3570
3571 To build the instrumented application:
3572
3573 . In path:{app.c}, before including path:{tpp.h}, add the
3574   following lines:
3575 +
3576 --
3577 [source,c]
3578 ----
3579 #define LTTNG_UST_TRACEPOINT_DEFINE
3580 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3581 ----
3582 --
3583
3584 . Compile the application source file:
3585 +
3586 --
3587 [role="term"]
3588 ----
3589 $ gcc -c app.c
3590 ----
3591 --
3592
3593 . Build the application:
3594 +
3595 --
3596 [role="term"]
3597 ----
3598 $ gcc -o app app.o -ldl
3599 ----
3600 --
3601
3602 To run the instrumented application with tracing support:
3603
3604 * Preload the tracepoint provider package shared object and
3605   start the application:
3606 +
3607 --
3608 [role="term"]
3609 ----
3610 $ LD_PRELOAD=./libtpp.so ./app
3611 ----
3612 --
3613
3614 To run the instrumented application without tracing support:
3615
3616 * Start the application:
3617 +
3618 --
3619 [role="term"]
3620 ----
3621 $ ./app
3622 ----
3623 --
3624
3625 |
3626 The instrumented application dynamically loads the tracepoint provider
3627 package shared object.
3628
3629 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3630
3631 |
3632 include::../common/ust-sit-step-tp-so.txt[]
3633
3634 To build the instrumented application:
3635
3636 . In path:{app.c}, before including path:{tpp.h}, add the
3637   following lines:
3638 +
3639 --
3640 [source,c]
3641 ----
3642 #define LTTNG_UST_TRACEPOINT_DEFINE
3643 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3644 ----
3645 --
3646
3647 . Compile the application source file:
3648 +
3649 --
3650 [role="term"]
3651 ----
3652 $ gcc -c app.c
3653 ----
3654 --
3655
3656 . Build the application:
3657 +
3658 --
3659 [role="term"]
3660 ----
3661 $ gcc -o app app.o -ldl
3662 ----
3663 --
3664
3665 To run the instrumented application:
3666
3667 * Start the application:
3668 +
3669 --
3670 [role="term"]
3671 ----
3672 $ ./app
3673 ----
3674 --
3675
3676 |
3677 The application is linked with the instrumented user library.
3678
3679 The instrumented user library is statically linked with the tracepoint
3680 provider package object file.
3681
3682 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3683
3684 |
3685 include::../common/ust-sit-step-tp-o-fpic.txt[]
3686
3687 To build the instrumented user library:
3688
3689 . In path:{emon.c}, before including path:{tpp.h}, add the
3690   following line:
3691 +
3692 --
3693 [source,c]
3694 ----
3695 #define LTTNG_UST_TRACEPOINT_DEFINE
3696 ----
3697 --
3698
3699 . Compile the user library source file:
3700 +
3701 --
3702 [role="term"]
3703 ----
3704 $ gcc -I. -fpic -c emon.c
3705 ----
3706 --
3707
3708 . Build the user library shared object:
3709 +
3710 --
3711 [role="term"]
3712 ----
3713 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3714 ----
3715 --
3716
3717 To build the application:
3718
3719 . Compile the application source file:
3720 +
3721 --
3722 [role="term"]
3723 ----
3724 $ gcc -c app.c
3725 ----
3726 --
3727
3728 . Build the application:
3729 +
3730 --
3731 [role="term"]
3732 ----
3733 $ gcc -o app app.o -L. -lemon
3734 ----
3735 --
3736
3737 To run the application:
3738
3739 * Start the application:
3740 +
3741 --
3742 [role="term"]
3743 ----
3744 $ ./app
3745 ----
3746 --
3747
3748 |
3749 The application is linked with the instrumented user library.
3750
3751 The instrumented user library is linked with the tracepoint provider
3752 package shared object.
3753
3754 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3755
3756 |
3757 include::../common/ust-sit-step-tp-so.txt[]
3758
3759 To build the instrumented user library:
3760
3761 . In path:{emon.c}, before including path:{tpp.h}, add the
3762   following line:
3763 +
3764 --
3765 [source,c]
3766 ----
3767 #define LTTNG_UST_TRACEPOINT_DEFINE
3768 ----
3769 --
3770
3771 . Compile the user library source file:
3772 +
3773 --
3774 [role="term"]
3775 ----
3776 $ gcc -I. -fpic -c emon.c
3777 ----
3778 --
3779
3780 . Build the user library shared object:
3781 +
3782 --
3783 [role="term"]
3784 ----
3785 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3786 ----
3787 --
3788
3789 To build the application:
3790
3791 . Compile the application source file:
3792 +
3793 --
3794 [role="term"]
3795 ----
3796 $ gcc -c app.c
3797 ----
3798 --
3799
3800 . Build the application:
3801 +
3802 --
3803 [role="term"]
3804 ----
3805 $ gcc -o app app.o -L. -lemon
3806 ----
3807 --
3808
3809 To run the application:
3810
3811 * Start the application:
3812 +
3813 --
3814 [role="term"]
3815 ----
3816 $ ./app
3817 ----
3818 --
3819
3820 |
3821 The tracepoint provider package shared object is preloaded before the
3822 application starts.
3823
3824 The application is linked with the instrumented user library.
3825
3826 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3827
3828 |
3829 include::../common/ust-sit-step-tp-so.txt[]
3830
3831 To build the instrumented user library:
3832
3833 . In path:{emon.c}, before including path:{tpp.h}, add the
3834   following lines:
3835 +
3836 --
3837 [source,c]
3838 ----
3839 #define LTTNG_UST_TRACEPOINT_DEFINE
3840 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3841 ----
3842 --
3843
3844 . Compile the user library source file:
3845 +
3846 --
3847 [role="term"]
3848 ----
3849 $ gcc -I. -fpic -c emon.c
3850 ----
3851 --
3852
3853 . Build the user library shared object:
3854 +
3855 --
3856 [role="term"]
3857 ----
3858 $ gcc -shared -o libemon.so emon.o -ldl
3859 ----
3860 --
3861
3862 To build the application:
3863
3864 . Compile the application source file:
3865 +
3866 --
3867 [role="term"]
3868 ----
3869 $ gcc -c app.c
3870 ----
3871 --
3872
3873 . Build the application:
3874 +
3875 --
3876 [role="term"]
3877 ----
3878 $ gcc -o app app.o -L. -lemon
3879 ----
3880 --
3881
3882 To run the application with tracing support:
3883
3884 * Preload the tracepoint provider package shared object and
3885   start the application:
3886 +
3887 --
3888 [role="term"]
3889 ----
3890 $ LD_PRELOAD=./libtpp.so ./app
3891 ----
3892 --
3893
3894 To run the application without tracing support:
3895
3896 * Start the application:
3897 +
3898 --
3899 [role="term"]
3900 ----
3901 $ ./app
3902 ----
3903 --
3904
3905 |
3906 The application is linked with the instrumented user library.
3907
3908 The instrumented user library dynamically loads the tracepoint provider
3909 package shared object.
3910
3911 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3912
3913 |
3914 include::../common/ust-sit-step-tp-so.txt[]
3915
3916 To build the instrumented user library:
3917
3918 . In path:{emon.c}, before including path:{tpp.h}, add the
3919   following lines:
3920 +
3921 --
3922 [source,c]
3923 ----
3924 #define LTTNG_UST_TRACEPOINT_DEFINE
3925 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3926 ----
3927 --
3928
3929 . Compile the user library source file:
3930 +
3931 --
3932 [role="term"]
3933 ----
3934 $ gcc -I. -fpic -c emon.c
3935 ----
3936 --
3937
3938 . Build the user library shared object:
3939 +
3940 --
3941 [role="term"]
3942 ----
3943 $ gcc -shared -o libemon.so emon.o -ldl
3944 ----
3945 --
3946
3947 To build the application:
3948
3949 . Compile the application source file:
3950 +
3951 --
3952 [role="term"]
3953 ----
3954 $ gcc -c app.c
3955 ----
3956 --
3957
3958 . Build the application:
3959 +
3960 --
3961 [role="term"]
3962 ----
3963 $ gcc -o app app.o -L. -lemon
3964 ----
3965 --
3966
3967 To run the application:
3968
3969 * Start the application:
3970 +
3971 --
3972 [role="term"]
3973 ----
3974 $ ./app
3975 ----
3976 --
3977
3978 |
3979 The application dynamically loads the instrumented user library.
3980
3981 The instrumented user library is linked with the tracepoint provider
3982 package shared object.
3983
3984 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3985
3986 |
3987 include::../common/ust-sit-step-tp-so.txt[]
3988
3989 To build the instrumented user library:
3990
3991 . In path:{emon.c}, before including path:{tpp.h}, add the
3992   following line:
3993 +
3994 --
3995 [source,c]
3996 ----
3997 #define LTTNG_UST_TRACEPOINT_DEFINE
3998 ----
3999 --
4000
4001 . Compile the user library source file:
4002 +
4003 --
4004 [role="term"]
4005 ----
4006 $ gcc -I. -fpic -c emon.c
4007 ----
4008 --
4009
4010 . Build the user library shared object:
4011 +
4012 --
4013 [role="term"]
4014 ----
4015 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
4016 ----
4017 --
4018
4019 To build the application:
4020
4021 . Compile the application source file:
4022 +
4023 --
4024 [role="term"]
4025 ----
4026 $ gcc -c app.c
4027 ----
4028 --
4029
4030 . Build the application:
4031 +
4032 --
4033 [role="term"]
4034 ----
4035 $ gcc -o app app.o -ldl -L. -lemon
4036 ----
4037 --
4038
4039 To run the application:
4040
4041 * Start the application:
4042 +
4043 --
4044 [role="term"]
4045 ----
4046 $ ./app
4047 ----
4048 --
4049
4050 |
4051 The application dynamically loads the instrumented user library.
4052
4053 The instrumented user library dynamically loads the tracepoint provider
4054 package shared object.
4055
4056 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
4057
4058 |
4059 include::../common/ust-sit-step-tp-so.txt[]
4060
4061 To build the instrumented user library:
4062
4063 . In path:{emon.c}, before including path:{tpp.h}, add the
4064   following lines:
4065 +
4066 --
4067 [source,c]
4068 ----
4069 #define LTTNG_UST_TRACEPOINT_DEFINE
4070 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4071 ----
4072 --
4073
4074 . Compile the user library source file:
4075 +
4076 --
4077 [role="term"]
4078 ----
4079 $ gcc -I. -fpic -c emon.c
4080 ----
4081 --
4082
4083 . Build the user library shared object:
4084 +
4085 --
4086 [role="term"]
4087 ----
4088 $ gcc -shared -o libemon.so emon.o -ldl
4089 ----
4090 --
4091
4092 To build the application:
4093
4094 . Compile the application source file:
4095 +
4096 --
4097 [role="term"]
4098 ----
4099 $ gcc -c app.c
4100 ----
4101 --
4102
4103 . Build the application:
4104 +
4105 --
4106 [role="term"]
4107 ----
4108 $ gcc -o app app.o -ldl -L. -lemon
4109 ----
4110 --
4111
4112 To run the application:
4113
4114 * Start the application:
4115 +
4116 --
4117 [role="term"]
4118 ----
4119 $ ./app
4120 ----
4121 --
4122
4123 |
4124 The tracepoint provider package shared object is preloaded before the
4125 application starts.
4126
4127 The application dynamically loads the instrumented user library.
4128
4129 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
4130
4131 |
4132 include::../common/ust-sit-step-tp-so.txt[]
4133
4134 To build the instrumented user library:
4135
4136 . In path:{emon.c}, before including path:{tpp.h}, add the
4137   following lines:
4138 +
4139 --
4140 [source,c]
4141 ----
4142 #define LTTNG_UST_TRACEPOINT_DEFINE
4143 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4144 ----
4145 --
4146
4147 . Compile the user library source file:
4148 +
4149 --
4150 [role="term"]
4151 ----
4152 $ gcc -I. -fpic -c emon.c
4153 ----
4154 --
4155
4156 . Build the user library shared object:
4157 +
4158 --
4159 [role="term"]
4160 ----
4161 $ gcc -shared -o libemon.so emon.o -ldl
4162 ----
4163 --
4164
4165 To build the application:
4166
4167 . Compile the application source file:
4168 +
4169 --
4170 [role="term"]
4171 ----
4172 $ gcc -c app.c
4173 ----
4174 --
4175
4176 . Build the application:
4177 +
4178 --
4179 [role="term"]
4180 ----
4181 $ gcc -o app app.o -L. -lemon
4182 ----
4183 --
4184
4185 To run the application with tracing support:
4186
4187 * Preload the tracepoint provider package shared object and
4188   start the application:
4189 +
4190 --
4191 [role="term"]
4192 ----
4193 $ LD_PRELOAD=./libtpp.so ./app
4194 ----
4195 --
4196
4197 To run the application without tracing support:
4198
4199 * Start the application:
4200 +
4201 --
4202 [role="term"]
4203 ----
4204 $ ./app
4205 ----
4206 --
4207
4208 |
4209 The application is statically linked with the tracepoint provider
4210 package object file.
4211
4212 The application is linked with the instrumented user library.
4213
4214 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
4215
4216 |
4217 include::../common/ust-sit-step-tp-o.txt[]
4218
4219 To build the instrumented user library:
4220
4221 . In path:{emon.c}, before including path:{tpp.h}, add the
4222   following line:
4223 +
4224 --
4225 [source,c]
4226 ----
4227 #define LTTNG_UST_TRACEPOINT_DEFINE
4228 ----
4229 --
4230
4231 . Compile the user library source file:
4232 +
4233 --
4234 [role="term"]
4235 ----
4236 $ gcc -I. -fpic -c emon.c
4237 ----
4238 --
4239
4240 . Build the user library shared object:
4241 +
4242 --
4243 [role="term"]
4244 ----
4245 $ gcc -shared -o libemon.so emon.o
4246 ----
4247 --
4248
4249 To build the application:
4250
4251 . Compile the application source file:
4252 +
4253 --
4254 [role="term"]
4255 ----
4256 $ gcc -c app.c
4257 ----
4258 --
4259
4260 . Build the application:
4261 +
4262 --
4263 [role="term"]
4264 ----
4265 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
4266 ----
4267 --
4268
4269 To run the instrumented application:
4270
4271 * Start the application:
4272 +
4273 --
4274 [role="term"]
4275 ----
4276 $ ./app
4277 ----
4278 --
4279
4280 |
4281 The application is statically linked with the tracepoint provider
4282 package object file.
4283
4284 The application dynamically loads the instrumented user library.
4285
4286 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
4287
4288 |
4289 include::../common/ust-sit-step-tp-o.txt[]
4290
4291 To build the application:
4292
4293 . In path:{app.c}, before including path:{tpp.h}, add the following line:
4294 +
4295 --
4296 [source,c]
4297 ----
4298 #define LTTNG_UST_TRACEPOINT_DEFINE
4299 ----
4300 --
4301
4302 . Compile the application source file:
4303 +
4304 --
4305 [role="term"]
4306 ----
4307 $ gcc -c app.c
4308 ----
4309 --
4310
4311 . Build the application:
4312 +
4313 --
4314 [role="term"]
4315 ----
4316 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
4317       -llttng-ust -ldl
4318 ----
4319 --
4320 +
4321 The `--export-dynamic` option passed to the linker is necessary for the
4322 dynamically loaded library to ``see'' the tracepoint symbols defined in
4323 the application.
4324
4325 To build the instrumented user library:
4326
4327 . Compile the user library source file:
4328 +
4329 --
4330 [role="term"]
4331 ----
4332 $ gcc -I. -fpic -c emon.c
4333 ----
4334 --
4335
4336 . Build the user library shared object:
4337 +
4338 --
4339 [role="term"]
4340 ----
4341 $ gcc -shared -o libemon.so emon.o
4342 ----
4343 --
4344
4345 To run the application:
4346
4347 * Start the application:
4348 +
4349 --
4350 [role="term"]
4351 ----
4352 $ ./app
4353 ----
4354 --
4355 |====
4356
4357
4358 [[using-lttng-ust-with-daemons]]
4359 ===== Use noch:{LTTng-UST} with daemons
4360
4361 If your instrumented application calls man:fork(2), man:clone(2),
4362 or BSD's man:rfork(2), without a following man:exec(3)-family
4363 system call, you must preload the path:{liblttng-ust-fork.so} shared
4364 object when you start the application.
4365
4366 [role="term"]
4367 ----
4368 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
4369 ----
4370
4371 If your tracepoint provider package is
4372 a shared library which you also preload, you must put both
4373 shared objects in env:LD_PRELOAD:
4374
4375 [role="term"]
4376 ----
4377 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
4378 ----
4379
4380
4381 [role="since-2.9"]
4382 [[liblttng-ust-fd]]
4383 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
4384
4385 If your instrumented application closes one or more file descriptors
4386 which it did not open itself, you must preload the
4387 path:{liblttng-ust-fd.so} shared object when you start the application:
4388
4389 [role="term"]
4390 ----
4391 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
4392 ----
4393
4394 Typical use cases include closing all the file descriptors after
4395 man:fork(2) or man:rfork(2) and buggy applications doing
4396 ``double closes''.
4397
4398
4399 [[lttng-ust-pkg-config]]
4400 ===== Use noch:{pkg-config}
4401
4402 On some distributions, LTTng-UST ships with a
4403 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
4404 metadata file. If this is your case, then use cmd:pkg-config to
4405 build an application on the command line:
4406
4407 [role="term"]
4408 ----
4409 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
4410 ----
4411
4412
4413 [[instrumenting-32-bit-app-on-64-bit-system]]
4414 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
4415
4416 In order to trace a 32-bit application running on a 64-bit system,
4417 LTTng must use a dedicated 32-bit
4418 <<lttng-consumerd,consumer daemon>>.
4419
4420 The following steps show how to build and install a 32-bit consumer
4421 daemon, which is _not_ part of the default 64-bit LTTng build, how to
4422 build and install the 32-bit LTTng-UST libraries, and how to build and
4423 link an instrumented 32-bit application in that context.
4424
4425 To build a 32-bit instrumented application for a 64-bit target system,
4426 assuming you have a fresh target system with no installed Userspace RCU
4427 or LTTng packages:
4428
4429 . Download, build, and install a 32-bit version of Userspace RCU:
4430 +
4431 --
4432 [role="term"]
4433 ----
4434 $ cd $(mktemp -d) &&
4435   wget https://lttng.org/files/urcu/userspace-rcu-latest-0.13.tar.bz2 &&
4436   tar -xf userspace-rcu-latest-0.13.tar.bz2 &&
4437   cd userspace-rcu-0.13.* &&
4438   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
4439   make &&
4440   sudo make install &&
4441   sudo ldconfig
4442 ----
4443 --
4444
4445 . Using the package manager of your distribution, or from source,
4446   install the 32-bit versions of the following dependencies of
4447   LTTng-tools and LTTng-UST:
4448 +
4449 --
4450 * https://sourceforge.net/projects/libuuid/[libuuid]
4451 * https://directory.fsf.org/wiki/Popt[popt]
4452 * https://www.xmlsoft.org/[libxml2]
4453 * **Optional**: https://github.com/numactl/numactl[numactl]
4454 --
4455
4456 . Download, build, and install a 32-bit version of the latest
4457   LTTng-UST{nbsp}{revision}:
4458 +
4459 --
4460 [role="term"]
4461 ----
4462 $ cd $(mktemp -d) &&
4463   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
4464   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
4465   cd lttng-ust-2.13.* &&
4466   ./configure --libdir=/usr/local/lib32 \
4467               CFLAGS=-m32 CXXFLAGS=-m32 \
4468               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4469   make &&
4470   sudo make install &&
4471   sudo ldconfig
4472 ----
4473 --
4474 +
4475 Add `--disable-numa` to `./configure` if you don't have
4476 https://github.com/numactl/numactl[numactl].
4477 +
4478 [NOTE]
4479 ====
4480 Depending on your distribution, 32-bit libraries could be installed at a
4481 different location than `/usr/lib32`. For example, Debian is known to
4482 install some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
4483
4484 In this case, make sure to set `LDFLAGS` to all the
4485 relevant 32-bit library paths, for example:
4486
4487 [role="term"]
4488 ----
4489 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
4490 ----
4491 ====
4492
4493 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
4494   the 32-bit consumer daemon:
4495 +
4496 --
4497 [role="term"]
4498 ----
4499 $ cd $(mktemp -d) &&
4500   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4501   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4502   cd lttng-tools-2.13.* &&
4503   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
4504               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
4505               --disable-bin-lttng --disable-bin-lttng-crash \
4506               --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
4507   make &&
4508   cd src/bin/lttng-consumerd &&
4509   sudo make install &&
4510   sudo ldconfig
4511 ----
4512 --
4513
4514 . From your distribution or from source, <<installing-lttng,install>>
4515   the 64-bit versions of LTTng-UST and Userspace RCU.
4516
4517 . Download, build, and install the 64-bit version of the
4518   latest LTTng-tools{nbsp}{revision}:
4519 +
4520 --
4521 [role="term"]
4522 ----
4523 $ cd $(mktemp -d) &&
4524   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4525   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4526   cd lttng-tools-2.13.* &&
4527   ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4528               --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4529   make &&
4530   sudo make install &&
4531   sudo ldconfig
4532 ----
4533 --
4534
4535 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4536   when linking your 32-bit application:
4537 +
4538 ----
4539 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4540 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4541 ----
4542 +
4543 For example, let's rebuild the quick start example in
4544 ``<<tracing-your-own-user-application,Record user application events>>''
4545 as an instrumented 32-bit application:
4546 +
4547 --
4548 [role="term"]
4549 ----
4550 $ gcc -m32 -c -I. hello-tp.c
4551 $ gcc -m32 -c hello.c
4552 $ gcc -m32 -o hello hello.o hello-tp.o \
4553       -L/usr/lib32 -L/usr/local/lib32 \
4554       -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4555       -llttng-ust -ldl
4556 ----
4557 --
4558
4559 No special action is required to execute the 32-bit application and
4560 for LTTng to trace it: use the command-line man:lttng(1) tool as usual.
4561
4562
4563 [role="since-2.5"]
4564 [[tracef]]
4565 ==== Use `lttng_ust_tracef()`
4566
4567 man:lttng_ust_tracef(3) is a small LTTng-UST API designed for quick,
4568 man:printf(3)-like instrumentation without the burden of
4569 <<tracepoint-provider,creating>> and
4570 <<building-tracepoint-providers-and-user-application,building>>
4571 a tracepoint provider package.
4572
4573 To use `lttng_ust_tracef()` in your application:
4574
4575 . In the C or $$C++$$ source files where you need to use
4576   `lttng_ust_tracef()`, include `<lttng/tracef.h>`:
4577 +
4578 --
4579 [source,c]
4580 ----
4581 #include <lttng/tracef.h>
4582 ----
4583 --
4584
4585 . In the source code of the application, use `lttng_ust_tracef()` like
4586   you would use man:printf(3):
4587 +
4588 --
4589 [source,c]
4590 ----
4591     /* ... */
4592
4593     lttng_ust_tracef("my message: %d (%s)", my_integer, my_string);
4594
4595     /* ... */
4596 ----
4597 --
4598
4599 . Link your application with `liblttng-ust`:
4600 +
4601 --
4602 [role="term"]
4603 ----
4604 $ gcc -o app app.c -llttng-ust
4605 ----
4606 --
4607
4608 To record the events that `lttng_ust_tracef()` calls emit:
4609
4610 * <<enabling-disabling-events,Create a recording event rule>> which
4611   matches user space events named `lttng_ust_tracef:*`:
4612 +
4613 --
4614 [role="term"]
4615 ----
4616 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4617 ----
4618 --
4619
4620 [IMPORTANT]
4621 .Limitations of `lttng_ust_tracef()`
4622 ====
4623 The `lttng_ust_tracef()` utility function was developed to make user
4624 space tracing super simple, albeit with notable disadvantages compared
4625 to <<defining-tracepoints,user-defined tracepoints>>:
4626
4627 * All the created events have the same tracepoint provider and
4628   tracepoint names, respectively `lttng_ust_tracef` and `event`.
4629 * There's no static type checking.
4630 * The only event record field you actually get, named `msg`, is a string
4631   potentially containing the values you passed to `lttng_ust_tracef()`
4632   using your own format string. This also means that you can't filter
4633   events with a custom expression at run time because there are no
4634   isolated fields.
4635 * Since `lttng_ust_tracef()` uses the man:vasprintf(3) function of the
4636   C{nbsp}standard library behind the scenes to format the strings at run
4637   time, its expected performance is lower than with user-defined
4638   tracepoints, which don't require a conversion to a string.
4639
4640 Taking this into consideration, `lttng_ust_tracef()` is useful for some
4641 quick prototyping and debugging, but you shouldn't consider it for any
4642 permanent and serious applicative instrumentation.
4643 ====
4644
4645
4646 [role="since-2.7"]
4647 [[tracelog]]
4648 ==== Use `lttng_ust_tracelog()`
4649
4650 The man:tracelog(3) API is very similar to
4651 <<tracef,`lttng_ust_tracef()`>>, with the difference that it accepts an
4652 additional log level parameter.
4653
4654 The goal of `lttng_ust_tracelog()` is to ease the migration from logging
4655 to tracing.
4656
4657 To use `lttng_ust_tracelog()` in your application:
4658
4659 . In the C or $$C++$$ source files where you need to use `tracelog()`,
4660   include `<lttng/tracelog.h>`:
4661 +
4662 --
4663 [source,c]
4664 ----
4665 #include <lttng/tracelog.h>
4666 ----
4667 --
4668
4669 . In the source code of the application, use `lttng_ust_tracelog()` like
4670   you would use man:printf(3), except for the first parameter which is
4671   the log level:
4672 +
4673 --
4674 [source,c]
4675 ----
4676     /* ... */
4677
4678     tracelog(LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING,
4679             "my message: %d (%s)", my_integer, my_string);
4680
4681     /* ... */
4682 ----
4683 --
4684 +
4685 See man:lttng-ust(3) for a list of available log level names.
4686
4687 . Link your application with `liblttng-ust`:
4688 +
4689 --
4690 [role="term"]
4691 ----
4692 $ gcc -o app app.c -llttng-ust
4693 ----
4694 --
4695
4696 To record the events that `lttng_ust_tracelog()` calls emit with a log
4697 level _at least as severe as_ a specific log level:
4698
4699 * <<enabling-disabling-events,Create a recording event rule>> which
4700   matches user space tracepoint events named `lttng_ust_tracelog:*` and
4701   with some minimum level of severity:
4702 +
4703 --
4704 [role="term"]
4705 ----
4706 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4707                      --loglevel=WARNING
4708 ----
4709 --
4710
4711 To record the events that `lttng_ust_tracelog()` calls emit with a
4712 _specific log level_:
4713
4714 * Create a recording event rule which matches tracepoint events named
4715   `lttng_ust_tracelog:*` and with a specific log level:
4716 +
4717 --
4718 [role="term"]
4719 ----
4720 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4721                      --loglevel-only=INFO
4722 ----
4723 --
4724
4725
4726 [[prebuilt-ust-helpers]]
4727 === Load a prebuilt user space tracing helper
4728
4729 The LTTng-UST package provides a few helpers in the form of preloadable
4730 shared objects which automatically instrument system functions and
4731 calls.
4732
4733 The helper shared objects are normally found in dir:{/usr/lib}. If you
4734 built LTTng-UST <<building-from-source,from source>>, they're probably
4735 located in dir:{/usr/local/lib}.
4736
4737 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4738 are:
4739
4740 path:{liblttng-ust-libc-wrapper.so}::
4741 path:{liblttng-ust-pthread-wrapper.so}::
4742   <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4743   memory and POSIX threads function tracing>>.
4744
4745 path:{liblttng-ust-cyg-profile.so}::
4746 path:{liblttng-ust-cyg-profile-fast.so}::
4747   <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4748
4749 path:{liblttng-ust-dl.so}::
4750   <<liblttng-ust-dl,Dynamic linker tracing>>.
4751
4752 To use a user space tracing helper with any user application:
4753
4754 * Preload the helper shared object when you start the application:
4755 +
4756 --
4757 [role="term"]
4758 ----
4759 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4760 ----
4761 --
4762 +
4763 You can preload more than one helper:
4764 +
4765 --
4766 [role="term"]
4767 ----
4768 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4769 ----
4770 --
4771
4772
4773 [role="since-2.3"]
4774 [[liblttng-ust-libc-pthread-wrapper]]
4775 ==== Instrument C standard library memory and POSIX threads functions
4776
4777 The path:{liblttng-ust-libc-wrapper.so} and
4778 path:{liblttng-ust-pthread-wrapper.so} helpers
4779 add instrumentation to some C standard library and POSIX
4780 threads functions.
4781
4782 [role="growable"]
4783 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4784 |====
4785 |TP provider name |TP name |Instrumented function
4786
4787 .6+|`lttng_ust_libc` |`malloc`         |man:malloc(3)
4788                      |`calloc`         |man:calloc(3)
4789                      |`realloc`        |man:realloc(3)
4790                      |`free`           |man:free(3)
4791                      |`memalign`       |man:memalign(3)
4792                      |`posix_memalign` |man:posix_memalign(3)
4793 |====
4794
4795 [role="growable"]
4796 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4797 |====
4798 |TP provider name |TP name |Instrumented function
4799
4800 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4801                         |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4802                         |`pthread_mutex_trylock`  |man:pthread_mutex_trylock(3p)
4803                         |`pthread_mutex_unlock`   |man:pthread_mutex_unlock(3p)
4804 |====
4805
4806 When you preload the shared object, it replaces the functions listed
4807 in the previous tables by wrappers which contain tracepoints and call
4808 the replaced functions.
4809
4810
4811 [[liblttng-ust-cyg-profile]]
4812 ==== Instrument function entry and exit
4813
4814 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4815 to the entry and exit points of functions.
4816
4817 man:gcc(1) and man:clang(1) have an option named
4818 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4819 which generates instrumentation calls for entry and exit to functions.
4820 The LTTng-UST function tracing helpers,
4821 path:{liblttng-ust-cyg-profile.so} and
4822 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4823 to add tracepoints to the two generated functions (which contain
4824 `cyg_profile` in their names, hence the name of the helper).
4825
4826 To use the LTTng-UST function tracing helper, the source files to
4827 instrument must be built using the `-finstrument-functions` compiler
4828 flag.
4829
4830 There are two versions of the LTTng-UST function tracing helper:
4831
4832 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4833   that you should only use when it can be _guaranteed_ that the
4834   complete event stream is recorded without any lost event record.
4835   Any kind of duplicate information is left out.
4836 +
4837 Assuming no event record is lost, having only the function addresses on
4838 entry is enough to create a call graph, since an event record always
4839 contains the ID of the CPU that generated it.
4840 +
4841 Use a tool like man:addr2line(1) to convert function addresses back to
4842 source file names and line numbers.
4843
4844 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4845 which also works in use cases where event records might get discarded or
4846 not recorded from application startup.
4847 In these cases, the trace analyzer needs more information to be
4848 able to reconstruct the program flow.
4849
4850 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4851 points of this helper.
4852
4853 All the tracepoints that this helper provides have the log level
4854 `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4855
4856 TIP: It's sometimes a good idea to limit the number of source files that
4857 you compile with the `-finstrument-functions` option to prevent LTTng
4858 from writing an excessive amount of trace data at run time. When using
4859 man:gcc(1), use the
4860 `-finstrument-functions-exclude-function-list` option to avoid
4861 instrument entries and exits of specific function names.
4862
4863
4864 [role="since-2.4"]
4865 [[liblttng-ust-dl]]
4866 ==== Instrument the dynamic linker
4867
4868 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4869 man:dlopen(3) and man:dlclose(3) function calls.
4870
4871 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4872 of this helper.
4873
4874
4875 [role="since-2.4"]
4876 [[java-application]]
4877 === Instrument a Java application
4878
4879 You can instrument any Java application which uses one of the following
4880 logging frameworks:
4881
4882 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4883   (JUL) core logging facilities.
4884
4885 * https://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4886   LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4887
4888 [role="img-100"]
4889 .LTTng-UST Java agent imported by a Java application.
4890 image::java-app.png[]
4891
4892 Note that the methods described below are new in LTTng{nbsp}2.8.
4893 Previous LTTng versions use another technique.
4894
4895 NOTE: We use https://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4896 and https://ci.lttng.org/[continuous integration], thus this version is
4897 directly supported. However, the LTTng-UST Java agent is also tested
4898 with OpenJDK{nbsp}7.
4899
4900
4901 [role="since-2.8"]
4902 [[jul]]
4903 ==== Use the LTTng-UST Java agent for `java.util.logging`
4904
4905 To use the LTTng-UST Java agent in a Java application which uses
4906 `java.util.logging` (JUL):
4907
4908 . In the source code of the Java application, import the LTTng-UST log
4909   handler package for `java.util.logging`:
4910 +
4911 --
4912 [source,java]
4913 ----
4914 import org.lttng.ust.agent.jul.LttngLogHandler;
4915 ----
4916 --
4917
4918 . Create an LTTng-UST `java.util.logging` log handler:
4919 +
4920 --
4921 [source,java]
4922 ----
4923 Handler lttngUstLogHandler = new LttngLogHandler();
4924 ----
4925 --
4926
4927 . Add this handler to the `java.util.logging` loggers which should emit
4928   LTTng events:
4929 +
4930 --
4931 [source,java]
4932 ----
4933 Logger myLogger = Logger.getLogger("some-logger");
4934
4935 myLogger.addHandler(lttngUstLogHandler);
4936 ----
4937 --
4938
4939 . Use `java.util.logging` log statements and configuration as usual.
4940   The loggers with an attached LTTng-UST log handler can emit
4941   LTTng events.
4942
4943 . Before exiting the application, remove the LTTng-UST log handler from
4944   the loggers attached to it and call its `close()` method:
4945 +
4946 --
4947 [source,java]
4948 ----
4949 myLogger.removeHandler(lttngUstLogHandler);
4950 lttngUstLogHandler.close();
4951 ----
4952 --
4953 +
4954 This isn't strictly necessary, but it's recommended for a clean
4955 disposal of the resources of the handler.
4956
4957 . Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
4958   path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4959   in the
4960   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4961   path] when you build the Java application.
4962 +
4963 The JAR files are typically located in dir:{/usr/share/java}.
4964 +
4965 IMPORTANT: The LTTng-UST Java agent must be
4966 <<installing-lttng,installed>> for the logging framework your
4967 application uses.
4968
4969 .Use the LTTng-UST Java agent for `java.util.logging`.
4970 ====
4971 [source,java]
4972 .path:{Test.java}
4973 ----
4974 import java.io.IOException;
4975 import java.util.logging.Handler;
4976 import java.util.logging.Logger;
4977 import org.lttng.ust.agent.jul.LttngLogHandler;
4978
4979 public class Test
4980 {
4981     private static final int answer = 42;
4982
4983     public static void main(String[] argv) throws Exception
4984     {
4985         // Create a logger
4986         Logger logger = Logger.getLogger("jello");
4987
4988         // Create an LTTng-UST log handler
4989         Handler lttngUstLogHandler = new LttngLogHandler();
4990
4991         // Add the LTTng-UST log handler to our logger
4992         logger.addHandler(lttngUstLogHandler);
4993
4994         // Log at will!
4995         logger.info("some info");
4996         logger.warning("some warning");
4997         Thread.sleep(500);
4998         logger.finer("finer information; the answer is " + answer);
4999         Thread.sleep(123);
5000         logger.severe("error!");
5001
5002         // Not mandatory, but cleaner
5003         logger.removeHandler(lttngUstLogHandler);
5004         lttngUstLogHandler.close();
5005     }
5006 }
5007 ----
5008
5009 Build this example:
5010
5011 [role="term"]
5012 ----
5013 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5014 ----
5015
5016 <<creating-destroying-tracing-sessions,Create a recording session>>,
5017 <<enabling-disabling-events,create a recording event rule>> matching JUL
5018 events named `jello`, and <<basic-tracing-session-control,start
5019 recording>>:
5020
5021 [role="term"]
5022 ----
5023 $ lttng create
5024 $ lttng enable-event --jul jello
5025 $ lttng start
5026 ----
5027
5028 Run the compiled class:
5029
5030 [role="term"]
5031 ----
5032 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5033 ----
5034
5035 <<basic-tracing-session-control,Stop recording>> and inspect the
5036 recorded events:
5037
5038 [role="term"]
5039 ----
5040 $ lttng stop
5041 $ lttng view
5042 ----
5043 ====
5044
5045 In the resulting trace, an <<event,event record>> which a Java
5046 application using `java.util.logging` generated is named
5047 `lttng_jul:event` and has the following fields:
5048
5049 `msg`::
5050   Log record message.
5051
5052 `logger_name`::
5053   Logger name.
5054
5055 `class_name`::
5056   Name of the class in which the log statement was executed.
5057
5058 `method_name`::
5059   Name of the method in which the log statement was executed.
5060
5061 `long_millis`::
5062   Logging time (timestamp in milliseconds).
5063
5064 `int_loglevel`::
5065   Log level integer value.
5066
5067 `int_threadid`::
5068   ID of the thread in which the log statement was executed.
5069
5070 Use the opt:lttng-enable-event(1):--loglevel or
5071 opt:lttng-enable-event(1):--loglevel-only option of the
5072 man:lttng-enable-event(1) command to target a range of
5073 `java.util.logging` log levels or a specific `java.util.logging` log
5074 level.
5075
5076
5077 [role="since-2.8"]
5078 [[log4j]]
5079 ==== Use the LTTng-UST Java agent for Apache log4j
5080
5081 To use the LTTng-UST Java agent in a Java application which uses
5082 Apache log4j{nbsp}1.2:
5083
5084 . In the source code of the Java application, import the LTTng-UST log
5085   appender package for Apache log4j:
5086 +
5087 --
5088 [source,java]
5089 ----
5090 import org.lttng.ust.agent.log4j.LttngLogAppender;
5091 ----
5092 --
5093
5094 . Create an LTTng-UST log4j log appender:
5095 +
5096 --
5097 [source,java]
5098 ----
5099 Appender lttngUstLogAppender = new LttngLogAppender();
5100 ----
5101 --
5102
5103 . Add this appender to the log4j loggers which should emit LTTng events:
5104 +
5105 --
5106 [source,java]
5107 ----
5108 Logger myLogger = Logger.getLogger("some-logger");
5109
5110 myLogger.addAppender(lttngUstLogAppender);
5111 ----
5112 --
5113
5114 . Use Apache log4j log statements and configuration as usual. The
5115   loggers with an attached LTTng-UST log appender can emit LTTng events.
5116
5117 . Before exiting the application, remove the LTTng-UST log appender from
5118   the loggers attached to it and call its `close()` method:
5119 +
5120 --
5121 [source,java]
5122 ----
5123 myLogger.removeAppender(lttngUstLogAppender);
5124 lttngUstLogAppender.close();
5125 ----
5126 --
5127 +
5128 This isn't strictly necessary, but it's recommended for a clean
5129 disposal of the resources of the appender.
5130
5131 . Include the common and log4j-specific JAR
5132   files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
5133   path:{lttng-ust-agent-log4j.jar}, in the
5134   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
5135   path] when you build the Java application.
5136 +
5137 The JAR files are typically located in dir:{/usr/share/java}.
5138 +
5139 IMPORTANT: The LTTng-UST Java agent must be
5140 <<installing-lttng,installed>> for the logging framework your
5141 application uses.
5142
5143 .Use the LTTng-UST Java agent for Apache log4j.
5144 ====
5145 [source,java]
5146 .path:{Test.java}
5147 ----
5148 import org.apache.log4j.Appender;
5149 import org.apache.log4j.Logger;
5150 import org.lttng.ust.agent.log4j.LttngLogAppender;
5151
5152 public class Test
5153 {
5154     private static final int answer = 42;
5155
5156     public static void main(String[] argv) throws Exception
5157     {
5158         // Create a logger
5159         Logger logger = Logger.getLogger("jello");
5160
5161         // Create an LTTng-UST log appender
5162         Appender lttngUstLogAppender = new LttngLogAppender();
5163
5164         // Add the LTTng-UST log appender to our logger
5165         logger.addAppender(lttngUstLogAppender);
5166
5167         // Log at will!
5168         logger.info("some info");
5169         logger.warn("some warning");
5170         Thread.sleep(500);
5171         logger.debug("debug information; the answer is " + answer);
5172         Thread.sleep(123);
5173         logger.fatal("error!");
5174
5175         // Not mandatory, but cleaner
5176         logger.removeAppender(lttngUstLogAppender);
5177         lttngUstLogAppender.close();
5178     }
5179 }
5180
5181 ----
5182
5183 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
5184 file):
5185
5186 [role="term"]
5187 ----
5188 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
5189 ----
5190
5191 <<creating-destroying-tracing-sessions,Create a recording session>>,
5192 <<enabling-disabling-events,create a recording event rule>> matching
5193 log4j events named `jello`, and <<basic-tracing-session-control,start
5194 recording>>:
5195
5196 [role="term"]
5197 ----
5198 $ lttng create
5199 $ lttng enable-event --log4j jello
5200 $ lttng start
5201 ----
5202
5203 Run the compiled class:
5204
5205 [role="term"]
5206 ----
5207 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
5208 ----
5209
5210 <<basic-tracing-session-control,Stop recording>> and inspect the
5211 recorded events:
5212
5213 [role="term"]
5214 ----
5215 $ lttng stop
5216 $ lttng view
5217 ----
5218 ====
5219
5220 In the resulting trace, an <<event,event record>> which a Java
5221 application using log4j generated is named `lttng_log4j:event` and
5222 has the following fields:
5223
5224 `msg`::
5225   Log record message.
5226
5227 `logger_name`::
5228   Logger name.
5229
5230 `class_name`::
5231   Name of the class in which the log statement was executed.
5232
5233 `method_name`::
5234   Name of the method in which the log statement was executed.
5235
5236 `filename`::
5237   Name of the file in which the executed log statement is located.
5238
5239 `line_number`::
5240   Line number at which the log statement was executed.
5241
5242 `timestamp`::
5243   Logging timestamp.
5244
5245 `int_loglevel`::
5246   Log level integer value.
5247
5248 `thread_name`::
5249   Name of the Java thread in which the log statement was executed.
5250
5251 Use the opt:lttng-enable-event(1):--loglevel or
5252 opt:lttng-enable-event(1):--loglevel-only option of the
5253 man:lttng-enable-event(1) command to target a range of Apache log4j
5254 log levels or a specific log4j log level.
5255
5256
5257 [role="since-2.8"]
5258 [[java-application-context]]
5259 ==== Provide application-specific context fields in a Java application
5260
5261 A Java application-specific context field is a piece of state which
5262 the Java application provides. You can <<adding-context,add>> such
5263 a context field to be recorded, using the
5264 man:lttng-add-context(1) command, to each <<event,event record>>
5265 which the log statements of this application produce.
5266
5267 For example, a given object might have a current request ID variable.
5268 You can create a context information retriever for this object and
5269 assign a name to this current request ID. You can then, using the
5270 man:lttng-add-context(1) command, add this context field by name so that
5271 LTTng writes it to the event records of a given `java.util.logging` or
5272 log4j <<channel,channel>>.
5273
5274 To provide application-specific context fields in a Java application:
5275
5276 . In the source code of the Java application, import the LTTng-UST
5277   Java agent context classes and interfaces:
5278 +
5279 --
5280 [source,java]
5281 ----
5282 import org.lttng.ust.agent.context.ContextInfoManager;
5283 import org.lttng.ust.agent.context.IContextInfoRetriever;
5284 ----
5285 --
5286
5287 . Create a context information retriever class, that is, a class which
5288   implements the `IContextInfoRetriever` interface:
5289 +
5290 --
5291 [source,java]
5292 ----
5293 class MyContextInfoRetriever implements IContextInfoRetriever
5294 {
5295     @Override
5296     public Object retrieveContextInfo(String key)
5297     {
5298         if (key.equals("intCtx")) {
5299             return (short) 17;
5300         } else if (key.equals("strContext")) {
5301             return "context value!";
5302         } else {
5303             return null;
5304         }
5305     }
5306 }
5307 ----
5308 --
5309 +
5310 This `retrieveContextInfo()` method is the only member of the
5311 `IContextInfoRetriever` interface. Its role is to return the current
5312 value of a state by name to create a context field. The names of the
5313 context fields and which state variables they return depends on your
5314 specific scenario.
5315 +
5316 All primitive types and objects are supported as context fields.
5317 When `retrieveContextInfo()` returns an object, the context field
5318 serializer calls its `toString()` method to add a string field to
5319 event records. The method can also return `null`, which means that
5320 no context field is available for the required name.
5321
5322 . Register an instance of your context information retriever class to
5323   the context information manager singleton:
5324 +
5325 --
5326 [source,java]
5327 ----
5328 IContextInfoRetriever cir = new MyContextInfoRetriever();
5329 ContextInfoManager cim = ContextInfoManager.getInstance();
5330 cim.registerContextInfoRetriever("retrieverName", cir);
5331 ----
5332 --
5333
5334 . Before exiting the application, remove your context information
5335   retriever from the context information manager singleton:
5336 +
5337 --
5338 [source,java]
5339 ----
5340 ContextInfoManager cim = ContextInfoManager.getInstance();
5341 cim.unregisterContextInfoRetriever("retrieverName");
5342 ----
5343 --
5344 +
5345 This isn't strictly necessary, but it's recommended for a clean
5346 disposal of some resources of the manager.
5347
5348 . Build your Java application with LTTng-UST Java agent support as
5349   usual, following the procedure for either the
5350   <<jul,`java.util.logging`>> or <<log4j,Apache log4j>> framework.
5351
5352 .Provide application-specific context fields in a Java application.
5353 ====
5354 [source,java]
5355 .path:{Test.java}
5356 ----
5357 import java.util.logging.Handler;
5358 import java.util.logging.Logger;
5359 import org.lttng.ust.agent.jul.LttngLogHandler;
5360 import org.lttng.ust.agent.context.ContextInfoManager;
5361 import org.lttng.ust.agent.context.IContextInfoRetriever;
5362
5363 public class Test
5364 {
5365     // Our context information retriever class
5366     private static class MyContextInfoRetriever
5367     implements IContextInfoRetriever
5368     {
5369         @Override
5370         public Object retrieveContextInfo(String key) {
5371             if (key.equals("intCtx")) {
5372                 return (short) 17;
5373             } else if (key.equals("strContext")) {
5374                 return "context value!";
5375             } else {
5376                 return null;
5377             }
5378         }
5379     }
5380
5381     private static final int answer = 42;
5382
5383     public static void main(String args[]) throws Exception
5384     {
5385         // Get the context information manager instance
5386         ContextInfoManager cim = ContextInfoManager.getInstance();
5387
5388         // Create and register our context information retriever
5389         IContextInfoRetriever cir = new MyContextInfoRetriever();
5390         cim.registerContextInfoRetriever("myRetriever", cir);
5391
5392         // Create a logger
5393         Logger logger = Logger.getLogger("jello");
5394
5395         // Create an LTTng-UST log handler
5396         Handler lttngUstLogHandler = new LttngLogHandler();
5397
5398         // Add the LTTng-UST log handler to our logger
5399         logger.addHandler(lttngUstLogHandler);
5400
5401         // Log at will!
5402         logger.info("some info");
5403         logger.warning("some warning");
5404         Thread.sleep(500);
5405         logger.finer("finer information; the answer is " + answer);
5406         Thread.sleep(123);
5407         logger.severe("error!");
5408
5409         // Not mandatory, but cleaner
5410         logger.removeHandler(lttngUstLogHandler);
5411         lttngUstLogHandler.close();
5412         cim.unregisterContextInfoRetriever("myRetriever");
5413     }
5414 }
5415 ----
5416
5417 Build this example:
5418
5419 [role="term"]
5420 ----
5421 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5422 ----
5423
5424 <<creating-destroying-tracing-sessions,Create a recording session>> and
5425 <<enabling-disabling-events,create a recording event rule>> matching
5426 `java.util.logging` events named `jello`:
5427
5428 [role="term"]
5429 ----
5430 $ lttng create
5431 $ lttng enable-event --jul jello
5432 ----
5433
5434 <<adding-context,Add the application-specific context fields>> to be
5435 recorded to the event records of the `java.util.logging` channel:
5436
5437 [role="term"]
5438 ----
5439 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
5440 $ lttng add-context --jul --type='$app.myRetriever:strContext'
5441 ----
5442
5443 <<basic-tracing-session-control,Start recording>>:
5444
5445 [role="term"]
5446 ----
5447 $ lttng start
5448 ----
5449
5450 Run the compiled class:
5451
5452 [role="term"]
5453 ----
5454 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5455 ----
5456
5457 <<basic-tracing-session-control,Stop recording>> and inspect the
5458 recorded events:
5459
5460 [role="term"]
5461 ----
5462 $ lttng stop
5463 $ lttng view
5464 ----
5465 ====
5466
5467
5468 [role="since-2.7"]
5469 [[python-application]]
5470 === Instrument a Python application
5471
5472 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
5473 uses the standard
5474 https://docs.python.org/3/library/logging.html[`logging`] package.
5475
5476 Each log statement creates an LTTng event once the application module
5477 imports the <<lttng-ust-agents,LTTng-UST Python agent>> package.
5478
5479 [role="img-100"]
5480 .A Python application importing the LTTng-UST Python agent.
5481 image::python-app.png[]
5482
5483 To use the LTTng-UST Python agent:
5484
5485 . In the source code of the Python application, import the LTTng-UST
5486   Python agent:
5487 +
5488 --
5489 [source,python]
5490 ----
5491 import lttngust
5492 ----
5493 --
5494 +
5495 The LTTng-UST Python agent automatically adds its logging handler to the
5496 root logger at import time.
5497 +
5498 A log statement that the application executes before this import doesn't
5499 create an LTTng event.
5500 +
5501 IMPORTANT: The LTTng-UST Python agent must be
5502 <<installing-lttng,installed>>.
5503
5504 . Use log statements and logging configuration as usual.
5505   Since the LTTng-UST Python agent adds a handler to the _root_
5506   logger, any log statement from any logger can emit an LTTng event.
5507
5508 .Use the LTTng-UST Python agent.
5509 ====
5510 [source,python]
5511 .path:{test.py}
5512 ----
5513 import lttngust
5514 import logging
5515 import time
5516
5517
5518 def example():
5519     logging.basicConfig()
5520     logger = logging.getLogger('my-logger')
5521
5522     while True:
5523         logger.debug('debug message')
5524         logger.info('info message')
5525         logger.warn('warn message')
5526         logger.error('error message')
5527         logger.critical('critical message')
5528         time.sleep(1)
5529
5530
5531 if __name__ == '__main__':
5532     example()
5533 ----
5534
5535 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5536 logging handler which prints to the standard error stream, isn't
5537 strictly required for LTTng-UST tracing to work, but in versions of
5538 Python preceding{nbsp}3.2, you could see a warning message which
5539 indicates that no handler exists for the logger `my-logger`.
5540
5541 <<creating-destroying-tracing-sessions,Create a recording session>>,
5542 <<enabling-disabling-events,create a recording event rule>> matching
5543 Python logging events named `my-logger`, and
5544 <<basic-tracing-session-control,start recording>>:
5545
5546 [role="term"]
5547 ----
5548 $ lttng create
5549 $ lttng enable-event --python my-logger
5550 $ lttng start
5551 ----
5552
5553 Run the Python script:
5554
5555 [role="term"]
5556 ----
5557 $ python test.py
5558 ----
5559
5560 <<basic-tracing-session-control,Stop recording>> and inspect the
5561 recorded events:
5562
5563 [role="term"]
5564 ----
5565 $ lttng stop
5566 $ lttng view
5567 ----
5568 ====
5569
5570 In the resulting trace, an <<event,event record>> which a Python
5571 application generated is named `lttng_python:event` and has the
5572 following fields:
5573
5574 `asctime`::
5575   Logging time (string).
5576
5577 `msg`::
5578   Log record message.
5579
5580 `logger_name`::
5581   Logger name.
5582
5583 `funcName`::
5584   Name of the function in which the log statement was executed.
5585
5586 `lineno`::
5587   Line number at which the log statement was executed.
5588
5589 `int_loglevel`::
5590   Log level integer value.
5591
5592 `thread`::
5593   ID of the Python thread in which the log statement was executed.
5594
5595 `threadName`::
5596   Name of the Python thread in which the log statement was executed.
5597
5598 Use the opt:lttng-enable-event(1):--loglevel or
5599 opt:lttng-enable-event(1):--loglevel-only option of the
5600 man:lttng-enable-event(1) command to target a range of Python log levels
5601 or a specific Python log level.
5602
5603 When an application imports the LTTng-UST Python agent, the agent tries
5604 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5605 <<start-sessiond,start the session daemon>> _before_ you run the Python
5606 application. If a session daemon is found, the agent tries to register
5607 to it during five seconds, after which the application continues
5608 without LTTng tracing support. Override this timeout value with
5609 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5610 (milliseconds).
5611
5612 If the session daemon stops while a Python application with an imported
5613 LTTng-UST Python agent runs, the agent retries to connect and to
5614 register to a session daemon every three seconds. Override this
5615 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5616 variable.
5617
5618
5619 [role="since-2.5"]
5620 [[proc-lttng-logger-abi]]
5621 === Use the LTTng logger
5622
5623 The `lttng-tracer` Linux kernel module, part of
5624 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5625 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5626 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5627 to any of those files to create one or more LTTng events.
5628
5629 [role="img-100"]
5630 .An application writes to the LTTng logger file to create one or more LTTng events.
5631 image::lttng-logger.png[]
5632
5633 The LTTng logger is the quickest method--not the most efficient,
5634 however--to add instrumentation to an application. It's designed
5635 mostly to instrument shell scripts:
5636
5637 [role="term"]
5638 ----
5639 $ echo "Some message, some $variable" > /dev/lttng-logger
5640 ----
5641
5642 Any event that the LTTng logger creates is named `lttng_logger` and
5643 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5644 other instrumentation points in the kernel tracing domain, **any Unix
5645 user** can <<enabling-disabling-events,create a recording event rule>>
5646 which matches events named `lttng_logger`, not only the root user or
5647 users in the <<tracing-group,tracing group>>.
5648
5649 To use the LTTng logger:
5650
5651 * From any application, write text data to the path:{/dev/lttng-logger}
5652   file.
5653
5654 The `msg` field of `lttng_logger` event records contains the
5655 recorded message.
5656
5657 NOTE: The maximum message length of an LTTng logger event is
5658 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5659 than one event to contain the remaining data.
5660
5661 You shouldn't use the LTTng logger to trace a user application which you
5662 can instrument in a more efficient way, namely:
5663
5664 * <<c-application,C and $$C++$$ applications>>.
5665 * <<java-application,Java applications>>.
5666 * <<python-application,Python applications>>.
5667
5668 .Use the LTTng logger.
5669 ====
5670 [source,bash]
5671 .path:{test.bash}
5672 ----
5673 echo 'Hello, World!' > /dev/lttng-logger
5674 sleep 2
5675 df --human-readable --print-type / > /dev/lttng-logger
5676 ----
5677
5678 <<creating-destroying-tracing-sessions,Create a recording session>>,
5679 <<enabling-disabling-events,create a recording event rule>> matching
5680 Linux kernel tracepoint events named `lttng_logger`, and
5681 <<basic-tracing-session-control,start recording>>:
5682
5683 [role="term"]
5684 ----
5685 $ lttng create
5686 $ lttng enable-event --kernel lttng_logger
5687 $ lttng start
5688 ----
5689
5690 Run the Bash script:
5691
5692 [role="term"]
5693 ----
5694 $ bash test.bash
5695 ----
5696
5697 <<basic-tracing-session-control,Stop recording>> and inspect the recorded
5698 events:
5699
5700 [role="term"]
5701 ----
5702 $ lttng stop
5703 $ lttng view
5704 ----
5705 ====
5706
5707
5708 [[instrumenting-linux-kernel]]
5709 === Instrument a Linux kernel image or module
5710
5711 NOTE: This section shows how to _add_ instrumentation points to the
5712 Linux kernel. The subsystems of the kernel are already thoroughly
5713 instrumented at strategic points for LTTng when you
5714 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5715 package.
5716
5717
5718 [[linux-add-lttng-layer]]
5719 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5720
5721 This section shows how to add an LTTng layer to existing ftrace
5722 instrumentation using the `TRACE_EVENT()` API.
5723
5724 This section doesn't document the `TRACE_EVENT()` macro. Read the
5725 following articles to learn more about this API:
5726
5727 * https://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5728 * https://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5729 * https://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5730
5731 The following procedure assumes that your ftrace tracepoints are
5732 correctly defined in their own header and that they're created in
5733 one source file using the `CREATE_TRACE_POINTS` definition.
5734
5735 To add an LTTng layer over an existing ftrace tracepoint:
5736
5737 . Make sure the following kernel configuration options are
5738   enabled:
5739 +
5740 --
5741 * `CONFIG_MODULES`
5742 * `CONFIG_KALLSYMS`
5743 * `CONFIG_HIGH_RES_TIMERS`
5744 * `CONFIG_TRACEPOINTS`
5745 --
5746
5747 . Build the Linux source tree with your custom ftrace tracepoints.
5748 . Boot the resulting Linux image on your target system.
5749 +
5750 Confirm that the tracepoints exist by looking for their names in the
5751 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5752 is your subsystem name.
5753
5754 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5755 +
5756 --
5757 [role="term"]
5758 ----
5759 $ cd $(mktemp -d) &&
5760   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
5761   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
5762   cd lttng-modules-2.13.*
5763 ----
5764 --
5765
5766 . In dir:{instrumentation/events/lttng-module}, relative to the root
5767   of the LTTng-modules source tree, create a header file named
5768   +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5769   LTTng-modules tracepoint definitions using the LTTng-modules
5770   macros in it.
5771 +
5772 Start with this template:
5773 +
5774 --
5775 [source,c]
5776 .path:{instrumentation/events/lttng-module/my_subsys.h}
5777 ----
5778 #undef TRACE_SYSTEM
5779 #define TRACE_SYSTEM my_subsys
5780
5781 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5782 #define _LTTNG_MY_SUBSYS_H
5783
5784 #include "../../../probes/lttng-tracepoint-event.h"
5785 #include <linux/tracepoint.h>
5786
5787 LTTNG_TRACEPOINT_EVENT(
5788     /*
5789      * Format is identical to the TRACE_EVENT() version for the three
5790      * following macro parameters:
5791      */
5792     my_subsys_my_event,
5793     TP_PROTO(int my_int, const char *my_string),
5794     TP_ARGS(my_int, my_string),
5795
5796     /* LTTng-modules specific macros */
5797     TP_FIELDS(
5798         ctf_integer(int, my_int_field, my_int)
5799         ctf_string(my_bar_field, my_bar)
5800     )
5801 )
5802
5803 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5804
5805 #include "../../../probes/define_trace.h"
5806 ----
5807 --
5808 +
5809 The entries in the `TP_FIELDS()` section are the list of fields for the
5810 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5811 the `TRACE_EVENT()` ftrace macro.
5812 +
5813 See ``<<lttng-modules-tp-fields,Tracepoint fields macros>>'' for a
5814 complete description of the available `ctf_*()` macros.
5815
5816 . Create the kernel module C{nbsp}source file of the LTTng-modules
5817   probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5818   subsystem name:
5819 +
5820 --
5821 [source,c]
5822 .path:{probes/lttng-probe-my-subsys.c}
5823 ----
5824 #include <linux/module.h>
5825 #include "../lttng-tracer.h"
5826
5827 /*
5828  * Build-time verification of mismatch between mainline
5829  * TRACE_EVENT() arguments and the LTTng-modules adaptation
5830  * layer LTTNG_TRACEPOINT_EVENT() arguments.
5831  */
5832 #include <trace/events/my_subsys.h>
5833
5834 /* Create LTTng tracepoint probes */
5835 #define LTTNG_PACKAGE_BUILD
5836 #define CREATE_TRACE_POINTS
5837 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5838
5839 #include "../instrumentation/events/lttng-module/my_subsys.h"
5840
5841 MODULE_LICENSE("GPL and additional rights");
5842 MODULE_AUTHOR("Your name <your-email>");
5843 MODULE_DESCRIPTION("LTTng my_subsys probes");
5844 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5845     __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5846     __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5847     LTTNG_MODULES_EXTRAVERSION);
5848 ----
5849 --
5850
5851 . Edit path:{probes/KBuild} and add your new kernel module object
5852   next to the existing ones:
5853 +
5854 --
5855 [source,make]
5856 .path:{probes/KBuild}
5857 ----
5858 # ...
5859
5860 obj-m += lttng-probe-module.o
5861 obj-m += lttng-probe-power.o
5862
5863 obj-m += lttng-probe-my-subsys.o
5864
5865 # ...
5866 ----
5867 --
5868
5869 . Build and install the LTTng kernel modules:
5870 +
5871 --
5872 [role="term"]
5873 ----
5874 $ make KERNELDIR=/path/to/linux
5875 # make modules_install && depmod -a
5876 ----
5877 --
5878 +
5879 Replace `/path/to/linux` with the path to the Linux source tree where
5880 you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
5881
5882 Note that you can also use the
5883 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5884 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5885 C{nbsp}code that need to be executed before LTTng records the event
5886 fields.
5887
5888 The best way to learn how to use the previous LTTng-modules macros is to
5889 inspect the existing LTTng-modules tracepoint definitions in the
5890 dir:{instrumentation/events/lttng-module} header files. Compare them
5891 with the Linux kernel mainline versions in the
5892 dir:{include/trace/events} directory of the Linux source tree.
5893
5894
5895 [role="since-2.7"]
5896 [[lttng-tracepoint-event-code]]
5897 ===== Use custom C code to access the data for tracepoint fields
5898
5899 Although we recommended to always use the
5900 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5901 the arguments and fields of an LTTng-modules tracepoint when possible,
5902 sometimes you need a more complex process to access the data that the
5903 tracer records as event record fields. In other words, you need local
5904 variables and multiple C{nbsp}statements instead of simple
5905 argument-based expressions that you pass to the
5906 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5907
5908 Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5909 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5910 a block of C{nbsp}code to be executed before LTTng records the fields.
5911 The structure of this macro is:
5912
5913 [source,c]
5914 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5915 ----
5916 LTTNG_TRACEPOINT_EVENT_CODE(
5917     /*
5918      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5919      * version for the following three macro parameters:
5920      */
5921     my_subsys_my_event,
5922     TP_PROTO(int my_int, const char *my_string),
5923     TP_ARGS(my_int, my_string),
5924
5925     /* Declarations of custom local variables */
5926     TP_locvar(
5927         int a = 0;
5928         unsigned long b = 0;
5929         const char *name = "(undefined)";
5930         struct my_struct *my_struct;
5931     ),
5932
5933     /*
5934      * Custom code which uses both tracepoint arguments
5935      * (in TP_ARGS()) and local variables (in TP_locvar()).
5936      *
5937      * Local variables are actually members of a structure pointed
5938      * to by the special variable tp_locvar.
5939      */
5940     TP_code(
5941         if (my_int) {
5942             tp_locvar->a = my_int + 17;
5943             tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5944             tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5945             tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5946             put_my_struct(tp_locvar->my_struct);
5947
5948             if (tp_locvar->b) {
5949                 tp_locvar->a = 1;
5950             }
5951         }
5952     ),
5953
5954     /*
5955      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5956      * version for this, except that tp_locvar members can be
5957      * used in the argument expression parameters of
5958      * the ctf_*() macros.
5959      */
5960     TP_FIELDS(
5961         ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5962         ctf_integer(int, my_struct_a, tp_locvar->a)
5963         ctf_string(my_string_field, my_string)
5964         ctf_string(my_struct_name, tp_locvar->name)
5965     )
5966 )
5967 ----
5968
5969 IMPORTANT: The C code defined in `TP_code()` must not have any side
5970 effects when executed. In particular, the code must not allocate
5971 memory or get resources without deallocating this memory or putting
5972 those resources afterwards.
5973
5974
5975 [[instrumenting-linux-kernel-tracing]]
5976 ==== Load and unload a custom probe kernel module
5977
5978 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5979 kernel module>> in the kernel before it can emit LTTng events.
5980
5981 To load the default probe kernel modules and a custom probe kernel
5982 module:
5983
5984 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5985   probe modules to load when starting a root <<lttng-sessiond,session
5986   daemon>>:
5987 +
5988 --
5989 .Load the `my_subsys`, `usb`, and the default probe modules.
5990 ====
5991 [role="term"]
5992 ----
5993 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5994 ----
5995 ====
5996 --
5997 +
5998 You only need to pass the subsystem name, not the whole kernel module
5999 name.
6000
6001 To load _only_ a given custom probe kernel module:
6002
6003 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
6004   modules to load when starting a root session daemon:
6005 +
6006 --
6007 .Load only the `my_subsys` and `usb` probe modules.
6008 ====
6009 [role="term"]
6010 ----
6011 # lttng-sessiond --kmod-probes=my_subsys,usb
6012 ----
6013 ====
6014 --
6015
6016 To confirm that a probe module is loaded:
6017
6018 * Use man:lsmod(8):
6019 +
6020 --
6021 [role="term"]
6022 ----
6023 $ lsmod | grep lttng_probe_usb
6024 ----
6025 --
6026
6027 To unload the loaded probe modules:
6028
6029 * Kill the session daemon with `SIGTERM`:
6030 +
6031 --
6032 [role="term"]
6033 ----
6034 # pkill lttng-sessiond
6035 ----
6036 --
6037 +
6038 You can also use the `--remove` option of man:modprobe(8) if the session
6039 daemon terminates abnormally.
6040
6041
6042 [[controlling-tracing]]
6043 == Tracing control
6044
6045 Once an application or a Linux kernel is <<instrumenting,instrumented>>
6046 for LTTng tracing, you can _trace_ it.
6047
6048 In the LTTng context, _tracing_ means making sure that LTTng attempts to
6049 execute some action(s) when a CPU executes an instrumentation point.
6050
6051 This section is divided in topics on how to use the various
6052 <<plumbing,components of LTTng>>, in particular the
6053 <<lttng-cli,cmd:lttng command-line tool>>, to _control_ the LTTng
6054 daemons and tracers.
6055
6056 NOTE: In the following subsections, we refer to an man:lttng(1) command
6057 using its man page name. For example, instead of ``Run the `create`
6058 command to'', we write ``Run the man:lttng-create(1) command to''.
6059
6060
6061 [[start-sessiond]]
6062 === Start a session daemon
6063
6064 In some situations, you need to run a <<lttng-sessiond,session daemon>>
6065 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
6066 command-line tool.
6067
6068 You will see the following error when you run a command while no session
6069 daemon is running:
6070
6071 ----
6072 Error: No session daemon is available
6073 ----
6074
6075 The only command that automatically runs a session daemon is
6076 man:lttng-create(1), which you use to
6077 <<creating-destroying-tracing-sessions,create a recording session>>. While
6078 this could be your most used first operation, sometimes it's not. Some
6079 examples are:
6080
6081 * <<list-instrumentation-points,List the available instrumentation points>>.
6082 * <<saving-loading-tracing-session,Load a recording session configuration>>.
6083 * <<add-event-rule-matches-trigger,Add a trigger>>.
6084
6085 All the examples above don't require a recording session to operate on.
6086
6087 [[tracing-group]] Each Unix user can have its own running session daemon
6088 to use the user space LTTng tracer. The session daemon that the `root`
6089 user starts is the only one allowed to control the LTTng kernel tracer.
6090 Members of the Unix _tracing group_ may connect to and control the root
6091 session daemon, even for user space tracing. See the ``Session daemon
6092 connection'' section of man:lttng(1) to learn more about the Unix
6093 tracing group.
6094
6095 To start a user session daemon:
6096
6097 * Run man:lttng-sessiond(8):
6098 +
6099 --
6100 [role="term"]
6101 ----
6102 $ lttng-sessiond --daemonize
6103 ----
6104 --
6105
6106 To start the root session daemon:
6107
6108 * Run man:lttng-sessiond(8) as the `root` user:
6109 +
6110 --
6111 [role="term"]
6112 ----
6113 # lttng-sessiond --daemonize
6114 ----
6115 --
6116
6117 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
6118 start the session daemon in foreground.
6119
6120 To stop a session daemon, kill its process (see man:kill(1)) with the
6121 standard `TERM` signal.
6122
6123 Note that some Linux distributions could manage the LTTng session daemon
6124 as a service. In this case, we suggest that you use the service manager
6125 to start, restart, and stop session daemons.
6126
6127
6128 [[creating-destroying-tracing-sessions]]
6129 === Create and destroy a recording session
6130
6131 Many LTTng control operations happen in the scope of a
6132 <<tracing-session,recording session>>, which is the dialogue between the
6133 <<lttng-sessiond,session daemon>> and you for everything related to
6134 <<event,event recording>>.
6135
6136 To create a recording session with a generated name:
6137
6138 * Use the man:lttng-create(1) command:
6139 +
6140 --
6141 [role="term"]
6142 ----
6143 $ lttng create
6144 ----
6145 --
6146
6147 The name of the created recording session is `auto` followed by the
6148 creation date.
6149
6150 To create a recording session with a specific name:
6151
6152 * Use the optional argument of the man:lttng-create(1) command:
6153 +
6154 --
6155 [role="term"]
6156 ----
6157 $ lttng create SESSION
6158 ----
6159 --
6160 +
6161 Replace +__SESSION__+ with your specific recording session name.
6162
6163 In <<local-mode,local mode>>, LTTng writes the traces of a recording
6164 session to the +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+
6165 directory by default, where +__NAME__+ is the name of the recording
6166 session. Note that the env:LTTNG_HOME environment variable defaults to
6167 `$HOME` if not set.
6168
6169 To output LTTng traces to a non-default location:
6170
6171 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1)
6172   command:
6173 +
6174 --
6175 [role="term"]
6176 ----
6177 $ lttng create my-session --output=/tmp/some-directory
6178 ----
6179 --
6180
6181 You may create as many recording sessions as you wish.
6182
6183 To list all the existing recording sessions for your Unix user, or for
6184 all users if your Unix user is `root`:
6185
6186 * Use the man:lttng-list(1) command:
6187 +
6188 --
6189 [role="term"]
6190 ----
6191 $ lttng list
6192 ----
6193 --
6194
6195 [[cur-tracing-session]]When you create a recording session, the
6196 man:lttng-create(1) command sets it as the _current recording session_.
6197 The following man:lttng(1) commands operate on the current recording
6198 session when you don't specify one:
6199
6200 [role="list-3-cols"]
6201 * man:lttng-add-context(1)
6202 * man:lttng-clear(1)
6203 * man:lttng-destroy(1)
6204 * man:lttng-disable-channel(1)
6205 * man:lttng-disable-event(1)
6206 * man:lttng-disable-rotation(1)
6207 * man:lttng-enable-channel(1)
6208 * man:lttng-enable-event(1)
6209 * man:lttng-enable-rotation(1)
6210 * man:lttng-load(1)
6211 * man:lttng-regenerate(1)
6212 * man:lttng-rotate(1)
6213 * man:lttng-save(1)
6214 * man:lttng-snapshot(1)
6215 * man:lttng-start(1)
6216 * man:lttng-status(1)
6217 * man:lttng-stop(1)
6218 * man:lttng-track(1)
6219 * man:lttng-untrack(1)
6220 * man:lttng-view(1)
6221
6222 To change the current recording session:
6223
6224 * Use the man:lttng-set-session(1) command:
6225 +
6226 --
6227 [role="term"]
6228 ----
6229 $ lttng set-session SESSION
6230 ----
6231 --
6232 +
6233 Replace +__SESSION__+ with the name of the new current recording session.
6234
6235 When you're done recording in a given recording session, destroy it.
6236 This operation frees the resources taken by the recording session to
6237 destroy; it doesn't destroy the trace data that LTTng wrote for this
6238 recording session (see ``<<clear,Clear a recording session>>'' for one
6239 way to do this).
6240
6241 To destroy the current recording session:
6242
6243 * Use the man:lttng-destroy(1) command:
6244 +
6245 --
6246 [role="term"]
6247 ----
6248 $ lttng destroy
6249 ----
6250 --
6251
6252 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
6253 implicitly (see ``<<basic-tracing-session-control,Start and stop a
6254 recording session>>''). You need to stop recording to make LTTng flush the
6255 remaining trace data and make the trace readable.
6256
6257
6258 [[list-instrumentation-points]]
6259 === List the available instrumentation points
6260
6261 The <<lttng-sessiond,session daemon>> can query the running instrumented
6262 user applications and the Linux kernel to get a list of available
6263 instrumentation points:
6264
6265 * LTTng tracepoints and system calls for the Linux kernel
6266   <<domain,tracing domain>>.
6267
6268 * LTTng tracepoints for the user space tracing domain.
6269
6270 To list the available instrumentation points:
6271
6272 . <<start-sessiond,Make sure>> there's a running
6273   <<lttng-sessiond,session daemon>> to which your Unix user can
6274   connect.
6275
6276 . Use the man:lttng-list(1) command with the option of the requested
6277   tracing domain amongst:
6278 +
6279 --
6280 opt:lttng-list(1):--kernel::
6281     Linux kernel tracepoints.
6282 +
6283 Your Unix user must be `root`, or it must be a member of the Unix
6284 <<tracing-group,tracing group>>.
6285
6286 opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
6287     Linux kernel system calls.
6288 +
6289 Your Unix user must be `root`, or it must be a member of the Unix
6290 <<tracing-group,tracing group>>.
6291
6292 opt:lttng-list(1):--userspace::
6293     User space tracepoints.
6294
6295 opt:lttng-list(1):--jul::
6296    `java.util.logging` loggers.
6297
6298 opt:lttng-list(1):--log4j::
6299     Apache log4j loggers.
6300
6301 opt:lttng-list(1):--python::
6302     Python loggers.
6303 --
6304
6305 .List the available user space tracepoints.
6306 ====
6307 [role="term"]
6308 ----
6309 $ lttng list --userspace
6310 ----
6311 ====
6312
6313 .List the available Linux kernel system calls.
6314 ====
6315 [role="term"]
6316 ----
6317 $ lttng list --kernel --syscall
6318 ----
6319 ====
6320
6321
6322 [[enabling-disabling-events]]
6323 === Create and enable a recording event rule
6324
6325 Once you <<creating-destroying-tracing-sessions,create a recording
6326 session>>, you can create <<event,recording event rules>> with the
6327 man:lttng-enable-event(1) command.
6328
6329 The man:lttng-enable-event(1) command always attaches an event rule to a
6330 <<channel,channel>> on creation. The command can create a _default
6331 channel_, named `channel0`, for you. The man:lttng-enable-event(1)
6332 command reuses the default channel each time you run it for the same
6333 tracing domain and session.
6334
6335 A recording event rule is always enabled at creation time.
6336
6337 The following examples show how to combine the command-line arguments of
6338 the man:lttng-enable-event(1) command to create simple to more complex
6339 recording event rules within the <<cur-tracing-session,current recording
6340 session>>.
6341
6342 .Create a recording event rule matching specific Linux kernel tracepoint events (default channel).
6343 ====
6344 [role="term"]
6345 ----
6346 # lttng enable-event --kernel sched_switch
6347 ----
6348 ====
6349
6350 .Create a recording event rule matching Linux kernel system call events with four specific names (default channel).
6351 ====
6352 [role="term"]
6353 ----
6354 # lttng enable-event --kernel --syscall open,write,read,close
6355 ----
6356 ====
6357
6358 .Create recording event rules matching tracepoint events which satisfy a filter expressions (default channel).
6359 ====
6360 [role="term"]
6361 ----
6362 # lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
6363 ----
6364
6365 [role="term"]
6366 ----
6367 # lttng enable-event --kernel --all \
6368                      --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
6369 ----
6370
6371 [role="term"]
6372 ----
6373 $ lttng enable-event --jul my_logger \
6374                      --filter='$app.retriever:cur_msg_id > 3'
6375 ----
6376
6377 IMPORTANT: Make sure to always single-quote the filter string when you
6378 run man:lttng(1) from a shell.
6379
6380 See also ``<<pid-tracking,Allow specific processes to record events>>''
6381 which offers another, more efficient filtering mechanism for process ID,
6382 user ID, and group ID attributes.
6383 ====
6384
6385 .Create a recording event rule matching any user space event from the `my_app` tracepoint provider and with a log level range (default channel).
6386 ====
6387 [role="term"]
6388 ----
6389 $ lttng enable-event --userspace my_app:'*' --loglevel=INFO
6390 ----
6391
6392 IMPORTANT: Make sure to always single-quote the wildcard character when
6393 you run man:lttng(1) from a shell.
6394 ====
6395
6396 .Create a recording event rule matching user space events named specifically, but with name exclusions (default channel).
6397 ====
6398 [role="term"]
6399 ----
6400 $ lttng enable-event --userspace my_app:'*' \
6401                      --exclude=my_app:set_user,my_app:handle_sig
6402 ----
6403 ====
6404
6405 .Create a recording event rule matching any Apache log4j event with a specific log level (default channel).
6406 ====
6407 [role="term"]
6408 ----
6409 $ lttng enable-event --log4j --all --loglevel-only=WARN
6410 ----
6411 ====
6412
6413 .Create a recording event rule, attached to a specific channel, and matching user space tracepoint events named `my_app:my_tracepoint`.
6414 ====
6415 [role="term"]
6416 ----
6417 $ lttng enable-event --userspace my_app:my_tracepoint \
6418                      --channel=my-channel
6419 ----
6420 ====
6421
6422 .Create a recording event rule matching user space probe events for the `malloc` function entry in path:{/usr/lib/libc.so.6}:
6423 ====
6424 [role="term"]
6425 ----
6426 # lttng enable-event --kernel \
6427                      --userspace-probe=/usr/lib/libc.so.6:malloc \
6428                      libc_malloc
6429 ----
6430 ====
6431
6432 .Create a recording event rule matching user space probe events for the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
6433 ====
6434 [role="term"]
6435 ----
6436 # lttng enable-event --kernel \
6437                      --userspace-probe=sdt:serv:server:accept_request \
6438                      server_accept_request
6439 ----
6440 ====
6441
6442 The recording event rules of a given channel form a whitelist: as soon
6443 as an event rule matches an event, LTTng emits it _once_ and therefore
6444 <<channel-overwrite-mode-vs-discard-mode,can>> record it. For example,
6445 the following rules both match user space tracepoint events named
6446 `my_app:my_tracepoint` with an `INFO` log level:
6447
6448 [role="term"]
6449 ----
6450 $ lttng enable-event --userspace my_app:my_tracepoint
6451 $ lttng enable-event --userspace my_app:my_tracepoint \
6452                      --loglevel=INFO
6453 ----
6454
6455 The second recording event rule is redundant: the first one includes the
6456 second one.
6457
6458
6459 [[disable-event-rule]]
6460 === Disable a recording event rule
6461
6462 To disable a <<event,recording event rule>> that you
6463 <<enabling-disabling-events,created>> previously, use the
6464 man:lttng-disable-event(1) command.
6465
6466 man:lttng-disable-event(1) can only find recording event rules to
6467 disable by their <<instrumentation-point-types,instrumentation point
6468 type>> and event name conditions. Therefore, you cannot disable
6469 recording event rules having a specific instrumentation point log level
6470 condition, for example.
6471
6472 LTTng doesn't emit (and, therefore, won't record) an event which only
6473 _disabled_ recording event rules match.
6474
6475 .Disable event rules matching Python logging events from the `my-logger` logger (default <<channel,channel>>, <<cur-tracing-session,current recording session>>).
6476 ====
6477 [role="term"]
6478 ----
6479 $ lttng disable-event --python my-logger
6480 ----
6481 ====
6482
6483 .Disable event rules matching all `java.util.logging` events (default channel, recording session `my-session`).
6484 ====
6485 [role="term"]
6486 ----
6487 $ lttng disable-event --jul --session=my-session '*'
6488 ----
6489 ====
6490
6491 .Disable _all_ the Linux kernel recording event rules (channel `my-chan`, current recording session).
6492 ====
6493 The opt:lttng-disable-event(1):--all-events option isn't, like the
6494 opt:lttng-enable-event(1):--all option of the man:lttng-enable-event(1)
6495 command, an alias for the event name globbing pattern `*`: it disables
6496 _all_ the recording event rules of a given channel.
6497
6498 [role="term"]
6499 ----
6500 # lttng disable-event --kernel --channel=my-chan --all-events
6501 ----
6502 ====
6503
6504 NOTE: You can't _remove_ a recording event rule once you create it.
6505
6506
6507 [[status]]
6508 === Get the status of a recording session
6509
6510 To get the status of the <<cur-tracing-session,current recording
6511 session>>, that is, its parameters, its channels, recording event rules,
6512 and their attributes:
6513
6514 * Use the man:lttng-status(1) command:
6515 +
6516 --
6517 [role="term"]
6518 ----
6519 $ lttng status
6520 ----
6521 --
6522
6523 To get the status of any recording session:
6524
6525 * Use the man:lttng-list(1) command with the name of the recording
6526   session:
6527 +
6528 --
6529 [role="term"]
6530 ----
6531 $ lttng list SESSION
6532 ----
6533 --
6534 +
6535 Replace +__SESSION__+ with the recording session name.
6536
6537
6538 [[basic-tracing-session-control]]
6539 === Start and stop a recording session
6540
6541 Once you <<creating-destroying-tracing-sessions,create a recording
6542 session>> and <<enabling-disabling-events,create one or more recording
6543 event rules>>, you can start and stop the tracers for this recording
6544 session.
6545
6546 To start the <<cur-tracing-session,current recording session>>:
6547
6548 * Use the man:lttng-start(1) command:
6549 +
6550 --
6551 [role="term"]
6552 ----
6553 $ lttng start
6554 ----
6555 --
6556
6557 LTTng is flexible: you can launch user applications before or after you
6558 start the tracers. An LTTng tracer only <<event,records an event>> if a
6559 recording event rule matches it, which means the tracer is active.
6560
6561 The `start-session` <<trigger,trigger>> action can also start a recording
6562 session.
6563
6564 To stop the current recording session:
6565
6566 * Use the man:lttng-stop(1) command:
6567 +
6568 --
6569 [role="term"]
6570 ----
6571 $ lttng stop
6572 ----
6573 --
6574 +
6575 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6576 records>> or lost sub-buffers since the last time you ran
6577 man:lttng-start(1), the man:lttng-stop(1) command prints corresponding
6578 warnings.
6579
6580 IMPORTANT: You need to stop recording to make LTTng flush the remaining
6581 trace data and make the trace readable. Note that the
6582 man:lttng-destroy(1) command (see
6583 ``<<creating-destroying-tracing-sessions,Create and destroy a recording
6584 session>>'') also runs the man:lttng-stop(1) command implicitly.
6585
6586 The `stop-session` <<trigger,trigger>> action can also stop a recording
6587 session.
6588
6589 [role="since-2.12"]
6590 [[clear]]
6591 === Clear a recording session
6592
6593 You might need to remove all the current tracing data of one or more
6594 <<tracing-session,recording sessions>> between multiple attempts to
6595 reproduce a problem without interrupting the LTTng recording activity.
6596
6597 To clear the tracing data of the
6598 <<cur-tracing-session,current recording session>>:
6599
6600 * Use the man:lttng-clear(1) command:
6601 +
6602 --
6603 [role="term"]
6604 ----
6605 $ lttng clear
6606 ----
6607 --
6608
6609 To clear the tracing data of all the recording sessions:
6610
6611 * Use the `lttng clear` command with its opt:lttng-clear(1):--all
6612   option:
6613 +
6614 --
6615 [role="term"]
6616 ----
6617 $ lttng clear --all
6618 ----
6619 --
6620
6621
6622 [[enabling-disabling-channels]]
6623 === Create a channel
6624
6625 Once you <<creating-destroying-tracing-sessions,create a recording
6626 session>>, you can create a <<channel,channel>> with the
6627 man:lttng-enable-channel(1) command.
6628
6629 Note that LTTng can automatically create a default channel when you
6630 <<enabling-disabling-events,create a recording event rule>>.
6631 Therefore, you only need to create a channel when you need non-default
6632 attributes.
6633
6634 Specify each non-default channel attribute with a command-line
6635 option when you run the man:lttng-enable-channel(1) command.
6636
6637 You can only create a custom channel in the Linux kernel and user space
6638 <<domain,tracing domains>>: the Java/Python logging tracing domains have
6639 their own default channel which LTTng automatically creates when you
6640 <<enabling-disabling-events,create a recording event rule>>.
6641
6642 [IMPORTANT]
6643 ====
6644 As of LTTng{nbsp}{revision}, you may _not_ perform the
6645 following operations with the man:lttng-enable-channel(1) command:
6646
6647 * Change an attribute of an existing channel.
6648
6649 * Enable a disabled channel once its recording session has been
6650   <<basic-tracing-session-control,active>> at least once.
6651
6652 * Create a channel once its recording session has been active at
6653   least once.
6654
6655 * Create a user space channel with a given
6656   <<channel-buffering-schemes,buffering scheme>> and create a second
6657   user space channel with a different buffering scheme in the same
6658   recording session.
6659 ====
6660
6661 The following examples show how to combine the command-line options of
6662 the man:lttng-enable-channel(1) command to create simple to more complex
6663 channels within the <<cur-tracing-session,current recording session>>.
6664
6665 .Create a Linux kernel channel with default attributes.
6666 ====
6667 [role="term"]
6668 ----
6669 # lttng enable-channel --kernel my-channel
6670 ----
6671 ====
6672
6673 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6674 ====
6675 [role="term"]
6676 ----
6677 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6678                        --buffers-pid my-channel
6679 ----
6680 ====
6681
6682 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6683 ====
6684 <<creating-destroying-tracing-sessions,Create a recording session>>,
6685 create the channel, <<enabling-disabling-events,create a recording event
6686 rule>>, and <<basic-tracing-session-control,start recording>>:
6687
6688 [role="term"]
6689 ----
6690 $ lttng create
6691 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-chan
6692 $ lttng enable-event --userspace --channel=blocking-chan --all
6693 $ lttng start
6694 ----
6695
6696 Run an application instrumented with LTTng-UST tracepoints and allow it
6697 to block:
6698
6699 [role="term"]
6700 ----
6701 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6702 ----
6703 ====
6704
6705 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream.
6706 ====
6707 [role="term"]
6708 ----
6709 # lttng enable-channel --kernel --tracefile-count=8 \
6710                        --tracefile-size=4194304 my-channel
6711 ----
6712 ====
6713
6714 .Create a user space channel in <<overwrite-mode,overwrite>> (or ``flight recorder'') mode.
6715 ====
6716 [role="term"]
6717 ----
6718 $ lttng enable-channel --userspace --overwrite my-channel
6719 ----
6720 ====
6721
6722 .<<enabling-disabling-events,Create>> the same <<event,recording event rule>> attached to two different channels.
6723 ====
6724 [role="term"]
6725 ----
6726 $ lttng enable-event --userspace --channel=my-channel app:tp
6727 $ lttng enable-event --userspace --channel=other-channel app:tp
6728 ----
6729
6730 When a CPU executes the `app:tp` <<c-application,user space
6731 tracepoint>>, the two recording event rules above match the created
6732 event, making LTTng emit the event. Because the recording event rules
6733 are not attached to the same channel, LTTng records the event twice.
6734 ====
6735
6736
6737 [[disable-channel]]
6738 === Disable a channel
6739
6740 To disable a specific channel that you
6741 <<enabling-disabling-channels,created>> previously, use the
6742 man:lttng-disable-channel(1) command.
6743
6744 .Disable a specific Linux kernel channel (<<cur-tracing-session,current recording session>>).
6745 ====
6746 [role="term"]
6747 ----
6748 # lttng disable-channel --kernel my-channel
6749 ----
6750 ====
6751
6752 An enabled channel is an implicit <<event,recording event rule>>
6753 condition.
6754
6755 NOTE: As of LTTng{nbsp}{revision}, you may _not_ enable a disabled
6756 channel once its recording session has been
6757 <<basic-tracing-session-control,started>> at least once.
6758
6759
6760 [[adding-context]]
6761 === Add context fields to be recorded to the event records of a channel
6762
6763 <<event,Event record>> fields in trace files provide important
6764 information about previously emitted events, but sometimes some external
6765 context may help you solve a problem faster.
6766
6767 Examples of context fields are:
6768
6769 * The **process ID**, **thread ID**, **process name**, and
6770   **process priority** of the thread from which LTTng emits the event.
6771
6772 * The **hostname** of the system on which LTTng emits the event.
6773
6774 * The Linux kernel and user call stacks (since LTTng{nbsp}2.11).
6775
6776 * The current values of many possible **performance counters** using
6777   perf, for example:
6778
6779 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6780 ** Cache misses.
6781 ** Branch instructions, misses, and loads.
6782 ** CPU faults.
6783
6784 * Any state defined at the application level (supported for the
6785   `java.util.logging` and Apache log4j <<domain,tracing domains>>).
6786
6787 To get the full list of available context fields:
6788
6789 * Use the opt:lttng-add-context(1):--list option of the
6790   man:lttng-add-context(1) command:
6791 +
6792 [role="term"]
6793 ----
6794 $ lttng add-context --list
6795 ----
6796
6797 .Add context fields to be recorded to the event records of all the <<channel,channels>> of the <<cur-tracing-session,current recording session>>.
6798 ====
6799 The following command line adds the virtual process identifier and the
6800 per-thread CPU cycles count fields to all the user space channels of the
6801 current recording session.
6802
6803 [role="term"]
6804 ----
6805 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6806 ----
6807 ====
6808
6809 .Add performance counter context fields by raw ID
6810 ====
6811 See man:lttng-add-context(1) for the exact format of the context field
6812 type, which is partly compatible with the format used in
6813 man:perf-record(1).
6814
6815 [role="term"]
6816 ----
6817 # lttng add-context --userspace --type=perf:thread:raw:r0110:test
6818 # lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6819 ----
6820 ====
6821
6822 .Add context fields to be recorded to the event records of a specific channel.
6823 ====
6824 The following command line adds the thread identifier and user call
6825 stack context fields to the Linux kernel channel named `my-channel` of
6826 the <<cur-tracing-session,current recording session>>.
6827
6828 [role="term"]
6829 ----
6830 # lttng add-context --kernel --channel=my-channel \
6831                     --type=tid --type=callstack-user
6832 ----
6833 ====
6834
6835 .Add an <<java-application-context,application-specific context field>> to be recorded to the event records of a specific channel.
6836 ====
6837 The following command line makes sure LTTng writes the `cur_msg_id`
6838 context field of the `retriever` context retriever to all the Java
6839 logging <<event,event records>> of the channel named `my-channel`:
6840
6841 [role="term"]
6842 ----
6843 # lttng add-context --kernel --channel=my-channel \
6844                     --type='$app:retriever:cur_msg_id'
6845 ----
6846
6847 IMPORTANT: Make sure to always single-quote the `$` character when you
6848 run man:lttng-add-context(1) from a shell.
6849 ====
6850
6851 NOTE: You can't undo what the man:lttng-add-context(1) command does.
6852
6853
6854 [role="since-2.7"]
6855 [[pid-tracking]]
6856 === Allow specific processes to record events
6857
6858 It's often useful to only allow processes with specific attributes to
6859 record events. For example, you may wish to record all the system calls
6860 which a given process makes (à la man:strace(1)).
6861
6862 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6863 purpose. Both commands operate on _inclusion sets_ of process
6864 attributes. The available process attribute types are:
6865
6866 Linux kernel <<domain,tracing domain>>::
6867 +
6868 * Process ID (PID).
6869
6870 * Virtual process ID (VPID).
6871 +
6872 This is the PID as seen by the application.
6873
6874 * Unix user ID (UID).
6875
6876 * Virtual Unix user ID (VUID).
6877 +
6878 This is the UID as seen by the application.
6879
6880 * Unix group ID (GID).
6881
6882 * Virtual Unix group ID (VGID).
6883 +
6884 This is the GID as seen by the application.
6885
6886 User space tracing domain::
6887 +
6888 * VPID
6889 * VUID
6890 * VGID
6891
6892 A <<tracing-session,recording session>> has nine process
6893 attribute inclusion sets: six for the Linux kernel <<domain,tracing domain>>
6894 and three for the user space tracing domain.
6895
6896 For a given recording session, a process{nbsp}__P__ is allowed to record
6897 LTTng events for a given <<domain,tracing domain>>{nbsp}__D__ if _all_
6898 the attributes of{nbsp}__P__ are part of the inclusion sets
6899 of{nbsp}__D__.
6900
6901 Whether a process is allowed or not to record LTTng events is an
6902 implicit condition of all <<event,recording event rules>>. Therefore, if
6903 LTTng creates an event{nbsp}__E__ for a given process, but this process
6904 may not record events, then no recording event rule matches{nbsp}__E__,
6905 which means LTTng won't emit and record{nbsp}__E__.
6906
6907 When you <<creating-destroying-tracing-sessions,create a recording
6908 session>>, all its process attribute inclusion sets contain all the
6909 possible values. In other words, all processes are allowed to record
6910 events.
6911
6912 Add values to an inclusion set with the man:lttng-track(1) command and
6913 remove values with the man:lttng-untrack(1) command.
6914
6915 [NOTE]
6916 ====
6917 The process attribute values are _numeric_.
6918
6919 Should a process with a given ID (part of an inclusion set), for
6920 example, exit, and then a new process be given this same ID, then the
6921 latter would also be allowed to record events.
6922
6923 With the man:lttng-track(1) command, you can add Unix user and group
6924 _names_ to the user and group inclusion sets: the
6925 <<lttng-sessiond,session daemon>> finds the corresponding UID, VUID,
6926 GID, or VGID once on _addition_ to the inclusion set. This means that if
6927 you rename the user or group after you run the man:lttng-track(1)
6928 command, its user/group ID remains part of the inclusion sets.
6929 ====
6930
6931 .Allow processes to record events based on their virtual process ID (VPID).
6932 ====
6933 For the sake of the following example, assume the target system has
6934 16{nbsp}possible VPIDs.
6935
6936 When you
6937 <<creating-destroying-tracing-sessions,create a recording session>>,
6938 the user space VPID inclusion set contains _all_ the possible VPIDs:
6939
6940 [role="img-100"]
6941 .The VPID inclusion set is full.
6942 image::track-all.png[]
6943
6944 When the inclusion set is full and you run the man:lttng-track(1)
6945 command to specify some VPIDs, LTTng:
6946
6947 . Clears the inclusion set.
6948 . Adds the specific VPIDs to the inclusion set.
6949
6950 After:
6951
6952 [role="term"]
6953 ----
6954 $ lttng track --userspace --vpid=3,4,7,10,13
6955 ----
6956
6957 the VPID inclusion set is:
6958
6959 [role="img-100"]
6960 .The VPID inclusion set contains the VPIDs 3, 4, 7, 10, and 13.
6961 image::track-3-4-7-10-13.png[]
6962
6963 Add more VPIDs to the inclusion set afterwards:
6964
6965 [role="term"]
6966 ----
6967 $ lttng track --userspace --vpid=1,15,16
6968 ----
6969
6970 The result is:
6971
6972 [role="img-100"]
6973 .VPIDs 1, 15, and 16 are added to the inclusion set.
6974 image::track-1-3-4-7-10-13-15-16.png[]
6975
6976 The man:lttng-untrack(1) command removes entries from process attribute
6977 inclusion sets. Given the previous example, the following command:
6978
6979 [role="term"]
6980 ----
6981 $ lttng untrack --userspace --vpid=3,7,10,13
6982 ----
6983
6984 leads to this VPID inclusion set:
6985
6986 [role="img-100"]
6987 .VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6988 image::track-1-4-15-16.png[]
6989
6990 You can make the VPID inclusion set full again with the
6991 opt:lttng-track(1):--all option:
6992
6993 [role="term"]
6994 ----
6995 $ lttng track --userspace --vpid --all
6996 ----
6997
6998 The result is, again:
6999
7000 [role="img-100"]
7001 .The VPID inclusion set is full.
7002 image::track-all.png[]
7003 ====
7004
7005 .Allow specific processes to record events based on their user ID (UID).
7006 ====
7007 A typical use case with process attribute inclusion sets is to start
7008 with an empty inclusion set, then <<basic-tracing-session-control,start
7009 the tracers>>, and finally add values manually while the tracers are
7010 active.
7011
7012 Use the opt:lttng-untrack(1):--all option of the
7013 man:lttng-untrack(1) command to clear the inclusion set after you
7014 <<creating-destroying-tracing-sessions,create a recording session>>, for
7015 example (with UIDs):
7016
7017 [role="term"]
7018 ----
7019 # lttng untrack --kernel --uid --all
7020 ----
7021
7022 gives:
7023
7024 [role="img-100"]
7025 .The UID inclusion set is empty.
7026 image::untrack-all.png[]
7027
7028 If the LTTng tracer runs with this inclusion set configuration, it
7029 records no events within the <<cur-tracing-session,current recording
7030 session>> because no processes is allowed to do so. Use the
7031 man:lttng-track(1) command as usual to add specific values to the UID
7032 inclusion set when you need to, for example:
7033
7034 [role="term"]
7035 ----
7036 # lttng track --kernel --uid=http,11
7037 ----
7038
7039 Result:
7040
7041 [role="img-100"]
7042 .UIDs 6 (`http`) and 11 are part of the UID inclusion set.
7043 image::track-6-11.png[]
7044 ====
7045
7046
7047 [role="since-2.5"]
7048 [[saving-loading-tracing-session]]
7049 === Save and load recording session configurations
7050
7051 Configuring a <<tracing-session,recording session>> can be long. Some of
7052 the tasks involved are:
7053
7054 * <<enabling-disabling-channels,Create channels>> with
7055   specific attributes.
7056
7057 * <<adding-context,Add context fields>> to be recorded to the
7058   <<event,event records>> of specific channels.
7059
7060 * <<enabling-disabling-events,Create recording event rules>> with
7061   specific log level, filter, and other conditions.
7062
7063 If you use LTTng to solve real world problems, chances are you have to
7064 record events using the same recording session setup over and over,
7065 modifying a few variables each time in your instrumented program or
7066 environment.
7067
7068 To avoid constant recording session reconfiguration, the man:lttng(1)
7069 command-line tool can save and load recording session configurations
7070 to/from XML files.
7071
7072 To save a given recording session configuration:
7073
7074 * Use the man:lttng-save(1) command:
7075 +
7076 --
7077 [role="term"]
7078 ----
7079 $ lttng save SESSION
7080 ----
7081 --
7082 +
7083 Replace +__SESSION__+ with the name of the recording session to save.
7084
7085 LTTng saves recording session configurations to
7086 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
7087 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. See
7088 man:lttng-save(1) to learn more about the recording session configuration
7089 output path.
7090
7091 LTTng saves all configuration parameters, for example:
7092
7093 * The recording session name.
7094 * The trace data output path.
7095 * The <<channel,channels>>, with their state and all their attributes.
7096 * The context fields you added to channels.
7097 * The <<event,recording event rules>> with their state and conditions.
7098
7099 To load a recording session:
7100
7101 * Use the man:lttng-load(1) command:
7102 +
7103 --
7104 [role="term"]
7105 ----
7106 $ lttng load SESSION
7107 ----
7108 --
7109 +
7110 Replace +__SESSION__+ with the name of the recording session to load.
7111
7112 When LTTng loads a configuration, it restores your saved recording session
7113 as if you just configured it manually.
7114
7115 You can also save and load many sessions at a time; see
7116 man:lttng-save(1) and man:lttng-load(1) to learn more.
7117
7118
7119 [[sending-trace-data-over-the-network]]
7120 === Send trace data over the network
7121
7122 LTTng can send the recorded trace data of a <<tracing-session,recording
7123 session>> to a remote system over the network instead of writing it to
7124 the local file system.
7125
7126 To send the trace data over the network:
7127
7128 . On the _remote_ system (which can also be the target system),
7129   start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
7130 +
7131 --
7132 [role="term"]
7133 ----
7134 $ lttng-relayd
7135 ----
7136 --
7137
7138 . On the _target_ system, create a recording session
7139   <<net-streaming-mode,configured>> to send trace data over the network:
7140 +
7141 --
7142 [role="term"]
7143 ----
7144 $ lttng create my-session --set-url=net://remote-system
7145 ----
7146 --
7147 +
7148 Replace +__remote-system__+ with the host name or IP address of the
7149 remote system. See man:lttng-create(1) for the exact URL format.
7150
7151 . On the target system, use the man:lttng(1) command-line tool as usual.
7152 +
7153 When recording is <<basic-tracing-session-control,active>>, the
7154 <<lttng-consumerd,consumer daemon>> of the target sends the contents of
7155 <<channel,sub-buffers>> to the remote relay daemon instead of flushing
7156 them to the local file system. The relay daemon writes the received
7157 packets to its local file system.
7158
7159 See the ``Output directory'' section of man:lttng-relayd(8) to learn
7160 where a relay daemon writes its received trace data.
7161
7162
7163 [role="since-2.4"]
7164 [[lttng-live]]
7165 === View events as LTTng records them (noch:{LTTng} live)
7166
7167 _LTTng live_ is a network protocol implemented by the
7168 <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)) to allow compatible
7169 trace readers to display or analyze <<event,event records>> as LTTng
7170 records events on the target system while recording is
7171 <<basic-tracing-session-control,active>>.
7172
7173 The relay daemon creates a _tee_: it forwards the trace data to both the
7174 local file system and to connected live readers:
7175
7176 [role="img-90"]
7177 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live reader.
7178 image::live.png[]
7179
7180 To use LTTng live:
7181
7182 . On the _target system_, create a <<tracing-session,recording session>>
7183   in _live mode_:
7184 +
7185 --
7186 [role="term"]
7187 ----
7188 $ lttng create my-session --live
7189 ----
7190 --
7191 +
7192 This operation spawns a local relay daemon.
7193
7194 . Start the live reader and configure it to connect to the relay daemon.
7195 +
7196 For example, with man:babeltrace2(1):
7197 +
7198 --
7199 [role="term"]
7200 ----
7201 $ babeltrace2 net://localhost/host/HOSTNAME/my-session
7202 ----
7203 --
7204 +
7205 Replace +__HOSTNAME__+ with the host name of the target system.
7206
7207 . Configure the recording session as usual with the man:lttng(1)
7208   command-line tool, and <<basic-tracing-session-control,start recording>>.
7209
7210 List the available live recording sessions with man:babeltrace2(1):
7211
7212 [role="term"]
7213 ----
7214 $ babeltrace2 net://localhost
7215 ----
7216
7217 You can start the relay daemon on another system. In this case, you need
7218 to specify the URL of the relay daemon when you
7219 <<creating-destroying-tracing-sessions,create the recording session>> with
7220 the opt:lttng-create(1):--set-url option of the man:lttng-create(1)
7221 command. You also need to replace +__localhost__+ in the procedure above
7222 with the host name of the system on which the relay daemon runs.
7223
7224
7225 [role="since-2.3"]
7226 [[taking-a-snapshot]]
7227 === Take a snapshot of the current sub-buffers of a recording session
7228
7229 The normal behavior of LTTng is to append full sub-buffers to growing
7230 trace data files. This is ideal to keep a full history of the events
7231 which the target system emitted, but it can represent too much data in
7232 some situations.
7233
7234 For example, you may wish to have LTTng record your application
7235 continuously until some critical situation happens, in which case you
7236 only need the latest few recorded events to perform the desired
7237 analysis, not multi-gigabyte trace files.
7238
7239 With the man:lttng-snapshot(1) command, you can take a _snapshot_ of the
7240 current <<channel,sub-buffers>> of a given <<tracing-session,recording
7241 session>>. LTTng can write the snapshot to the local file system or send
7242 it over the network.
7243
7244 [role="img-100"]
7245 .A snapshot is a copy of the current sub-buffers, which LTTng does _not_ clear after the operation.
7246 image::snapshot.png[]
7247
7248 The snapshot feature of LTTng is similar to how a
7249 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder] or the
7250 ``roll'' mode of an oscilloscope work.
7251
7252 TIP: If you wish to create unmanaged, self-contained, non-overlapping
7253 trace chunk archives instead of a simple copy of the current
7254 sub-buffers, see the <<session-rotation,recording session rotation>>
7255 feature (available since LTTng{nbsp}2.11).
7256
7257 To take a snapshot of the <<cur-tracing-session,current recording
7258 session>>:
7259
7260 . Create a recording session in <<snapshot-mode,snapshot mode>>:
7261 +
7262 --
7263 [role="term"]
7264 ----
7265 $ lttng create my-session --snapshot
7266 ----
7267 --
7268 +
7269 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
7270 <<channel,channels>> created in this mode is automatically set to
7271 <<overwrite-mode,_overwrite_>>.
7272
7273 . Configure the recording session as usual with the man:lttng(1)
7274   command-line tool, and <<basic-tracing-session-control,start
7275   recording>>.
7276
7277 . **Optional**: When you need to take a snapshot,
7278   <<basic-tracing-session-control,stop recording>>.
7279 +
7280 You can take a snapshot when the tracers are active, but if you stop
7281 them first, you're guaranteed that the trace data in the sub-buffers
7282 doesn't change before you actually take the snapshot.
7283
7284 . Take a snapshot:
7285 +
7286 --
7287 [role="term"]
7288 ----
7289 $ lttng snapshot record --name=my-first-snapshot
7290 ----
7291 --
7292 +
7293 LTTng writes the current sub-buffers of all the channels of the
7294 <<cur-tracing-session,current recording session>> to
7295 trace files on the local file system. Those trace files have
7296 `my-first-snapshot` in their name.
7297
7298 There's no difference between the format of a normal trace file and the
7299 format of a snapshot: LTTng trace readers also support LTTng snapshots.
7300
7301 By default, LTTng writes snapshot files to the path shown by
7302
7303 [role="term"]
7304 ----
7305 $ lttng snapshot list-output
7306 ----
7307
7308 You can change this path or decide to send snapshots over the network
7309 using either:
7310
7311 . An output path or URL that you specify when you
7312   <<creating-destroying-tracing-sessions,create the recording session>>.
7313
7314 . A snapshot output path or URL that you add using the
7315   `add-output` action of the man:lttng-snapshot(1) command.
7316
7317 . An output path or URL that you provide directly to the
7318   `record` action of the man:lttng-snapshot(1) command.
7319
7320 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
7321 you specify a URL, a <<lttng-relayd,relay daemon>> must listen on a
7322 remote system (see ``<<sending-trace-data-over-the-network,Send trace
7323 data over the network>>'').
7324
7325 The `snapshot-session` <<trigger,trigger>> action can also take
7326 a recording session snapshot.
7327
7328
7329 [role="since-2.11"]
7330 [[session-rotation]]
7331 === Archive the current trace chunk (rotate a recording session)
7332
7333 The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
7334 current sub-buffers of a recording session to the file system or send them
7335 over the network. When you take a snapshot, LTTng doesn't clear the ring
7336 buffers of the recording session: if you take another snapshot immediately
7337 after, both snapshots could contain overlapping trace data.
7338
7339 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
7340 _recording session rotation_ is a feature which appends the content of the
7341 ring buffers to what's already on the file system or sent over the
7342 network since the creation of the recording session or since the last
7343 rotation, and then clears those ring buffers to avoid trace data
7344 overlaps.
7345
7346 What LTTng is about to write when performing a recording session rotation
7347 is called the _current trace chunk_. When LTTng writes or sends over the
7348 network this current trace chunk, it becomes a _trace chunk archive_.
7349 Therefore, a recording session rotation operation _archives_ the current
7350 trace chunk.
7351
7352 [role="img-100"]
7353 .A recording session rotation operation _archives_ the current trace chunk.
7354 image::rotation.png[]
7355
7356 A trace chunk archive is a self-contained LTTng trace which LTTng
7357 doesn't manage anymore: you can read it, modify it, move it, or remove
7358 it.
7359
7360 As of LTTng{nbsp}{revision}, there are three methods to perform a
7361 recording session rotation:
7362
7363 * <<immediate-rotation,Immediately>>.
7364
7365 * With a <<rotation-schedule,rotation schedule>>.
7366
7367 * Through the execution of a `rotate-session` <<trigger,trigger>>
7368   action.
7369
7370 [[immediate-rotation]]To perform an immediate rotation of the
7371 <<cur-tracing-session,current recording session>>:
7372
7373 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7374   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7375   mode>> (only those two recording session modes support recording session
7376   rotation):
7377 +
7378 --
7379 [role="term"]
7380 ----
7381 # lttng create my-session
7382 ----
7383 --
7384
7385 . <<enabling-disabling-events,Create one or more recording event rules>>
7386   and <<basic-tracing-session-control,start recording>>:
7387 +
7388 --
7389 [role="term"]
7390 ----
7391 # lttng enable-event --kernel sched_'*'
7392 # lttng start
7393 ----
7394 --
7395
7396 . When needed, immediately rotate the current recording session:
7397 +
7398 --
7399 [role="term"]
7400 ----
7401 # lttng rotate
7402 ----
7403 --
7404 +
7405 The man:lttng-rotate(1) command prints the path to the created trace
7406 chunk archive. See its manual page to learn about the format of trace
7407 chunk archive directory names.
7408 +
7409 Perform other immediate rotations while the recording session is active.
7410 It's guaranteed that all the trace chunk archives don't contain
7411 overlapping trace data. You can also perform an immediate rotation once
7412 you have <<basic-tracing-session-control,stopped>> the recording session.
7413
7414 . When you're done recording,
7415   <<creating-destroying-tracing-sessions,destroy the current recording
7416   session>>:
7417 +
7418 --
7419 [role="term"]
7420 ----
7421 # lttng destroy
7422 ----
7423 --
7424 +
7425 The recording session destruction operation creates one last trace chunk
7426 archive from the current trace chunk.
7427
7428 [[rotation-schedule]]A recording session rotation schedule is a planned
7429 rotation which LTTng performs automatically based on one of the
7430 following conditions:
7431
7432 * A timer with a configured period expires.
7433
7434 * The total size of the _flushed_ part of the current trace chunk
7435   becomes greater than or equal to a configured value.
7436
7437 To schedule a rotation of the <<cur-tracing-session,current recording
7438 session>>, set a _rotation schedule_:
7439
7440 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7441   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7442   mode>> (only those two creation modes support recording session
7443   rotation):
7444 +
7445 --
7446 [role="term"]
7447 ----
7448 # lttng create my-session
7449 ----
7450 --
7451
7452 . <<enabling-disabling-events,Create one or more recording event rules>>:
7453 +
7454 --
7455 [role="term"]
7456 ----
7457 # lttng enable-event --kernel sched_'*'
7458 ----
7459 --
7460
7461 . Set a recording session rotation schedule:
7462 +
7463 --
7464 [role="term"]
7465 ----
7466 # lttng enable-rotation --timer=10s
7467 ----
7468 --
7469 +
7470 In this example, we set a rotation schedule so that LTTng performs a
7471 recording session rotation every ten seconds.
7472 +
7473 See man:lttng-enable-rotation(1) to learn more about other ways to set a
7474 rotation schedule.
7475
7476 . <<basic-tracing-session-control,Start recording>>:
7477 +
7478 --
7479 [role="term"]
7480 ----
7481 # lttng start
7482 ----
7483 --
7484 +
7485 LTTng performs recording session rotations automatically while the
7486 recording session is active thanks to the rotation schedule.
7487
7488 . When you're done recording,
7489   <<creating-destroying-tracing-sessions,destroy the current recording
7490   session>>:
7491 +
7492 --
7493 [role="term"]
7494 ----
7495 # lttng destroy
7496 ----
7497 --
7498 +
7499 The recording session destruction operation creates one last trace chunk
7500 archive from the current trace chunk.
7501
7502 Unset a recording session rotation schedule with the
7503 man:lttng-disable-rotation(1) command.
7504
7505
7506 [role="since-2.13"]
7507 [[add-event-rule-matches-trigger]]
7508 === Add an ``event rule matches'' trigger to a session daemon
7509
7510 With the man:lttng-add-trigger(1) command, you can add a
7511 <<trigger,trigger>> to a <<lttng-sessiond,session daemon>>.
7512
7513 A trigger associates an LTTng tracing condition to one or more actions:
7514 when the condition is satisfied, LTTng attempts to execute the actions.
7515
7516 A trigger doesn't need any <<tracing-session,recording session>> to exist:
7517 it belongs to a session daemon.
7518
7519 As of LTTng{nbsp}{revision}, many condition types are available through
7520 the <<liblttng-ctl-lttng,`liblttng-ctl`>> C{nbsp}API, but the
7521 man:lttng-add-trigger(1) command only accepts the ``event rule matches''
7522 condition.
7523
7524 An ``event rule matches'' condition is satisfied when its event rule
7525 matches an event.
7526
7527 Unlike a <<event,recording event rule>>, the event rule of an
7528 ``event rule matches'' trigger condition has no implicit conditions,
7529 that is:
7530
7531 * It has no enabled/disabled state.
7532 * It has no attached <<channel,channel>>.
7533 * It doesn't belong to a <<tracing-session,recording session>>.
7534
7535 Both the man:lttng-add-trigger(1) and man:lttng-enable-event(1) commands
7536 accept command-line arguments to specify an <<event-rule,event rule>>.
7537 That being said, the former is a more recent command and therefore
7538 follows the common event rule specification format (see
7539 man:lttng-event-rule(7)).
7540
7541 .Start a <<tracing-session,recording session>> when an event rule matches.
7542 ====
7543 This example shows how to add the following trigger to the root
7544 <<lttng-sessiond,session daemon>>:
7545
7546 Condition::
7547     An event rule matches a Linux kernel system call event of which the
7548     name starts with `exec` and `*/ls` matches the `filename` payload
7549     field.
7550 +
7551 With such an event rule, LTTng emits an event when the cmd:ls program
7552 starts.
7553
7554 Action::
7555     <<basic-tracing-session-control,Start the recording session>>
7556     named `pitou`.
7557
7558 To add such a trigger to the root session daemon:
7559
7560 . **If there's no currently running LTTng root session daemon**, start
7561   one:
7562 +
7563 [role="term"]
7564 ----
7565 # lttng-sessiond --daemonize
7566 ----
7567
7568 . <<creating-destroying-tracing-sessions,Create a recording session>>
7569   named `pitou` and
7570   <<enabling-disabling-events,create a recording event rule>> matching
7571   all the system call events:
7572 +
7573 [role="term"]
7574 ----
7575 # lttng create pitou
7576 # lttng enable-event --kernel --syscall --all
7577 ----
7578
7579 . Add the trigger to the root session daemon:
7580 +
7581 [role="term"]
7582 ----
7583 # lttng add-trigger --condition=event-rule-matches \
7584                     --type=syscall --name='exec*' \
7585                     --filter='filename == "*/ls"' \
7586                     --action=start-session pitou
7587 ----
7588 +
7589 Confirm that the trigger exists with the man:lttng-list-triggers(1)
7590 command:
7591 +
7592 [role="term"]
7593 ----
7594 # lttng list-triggers
7595 ----
7596
7597 . Make sure the `pitou` recording session is still inactive (stopped):
7598 +
7599 [role="term"]
7600 ----
7601 # lttng list pitou
7602 ----
7603 +
7604 The first line should be something like:
7605 +
7606 ----
7607 Recording session pitou: [inactive]
7608 ----
7609
7610 Run the cmd:ls program to fire the LTTng trigger above:
7611
7612 [role="term"]
7613 ----
7614 $ ls ~
7615 ----
7616
7617 At this point, the `pitou` recording session should be active
7618 (started). Confirm this with the man:lttng-list(1) command again:
7619
7620 [role="term"]
7621 ----
7622 # lttng list pitou
7623 ----
7624
7625 The first line should now look like:
7626
7627 ----
7628 Recording session pitou: [active]
7629 ----
7630
7631 This line confirms that the LTTng trigger you added fired, therefore
7632 starting the `pitou` recording session.
7633 ====
7634
7635 .[[trigger-event-notif]]Send a notification to a user application when an event rule matches.
7636 ====
7637 This example shows how to add the following trigger to the root
7638 <<lttng-sessiond,session daemon>>:
7639
7640 Condition::
7641     An event rule matches a Linux kernel tracepoint event named
7642     `sched_switch` and of which the value of the `next_comm` payload
7643     field is `bash`.
7644 +
7645 With such an event rule, LTTng emits an event when Linux gives access to
7646 the processor to a process named `bash`.
7647
7648 Action::
7649     Send an LTTng notification to a user application.
7650
7651 Moreover, we'll specify a _capture descriptor_ with the
7652 `event-rule-matches` trigger condition so that the user application can
7653 get the value of a specific `sched_switch` event payload field.
7654
7655 First, write and build the user application:
7656
7657 . Create the C{nbsp}source file of the application:
7658 +
7659 --
7660 [source,c]
7661 .path:{notif-app.c}
7662 ----
7663 #include <stdlib.h>
7664 #include <stdio.h>
7665 #include <stdbool.h>
7666 #include <assert.h>
7667 #include <string.h>
7668 #include <lttng/lttng.h>
7669
7670 /*
7671  * Subscribes to notifications, through the notification channel
7672  * `notification_channel`, which match the condition of the trigger
7673  * named `trigger_name`.
7674  *
7675  * Returns `true` on success.
7676  */
7677 static bool subscribe(struct lttng_notification_channel *notification_channel,
7678                       const char *trigger_name)
7679 {
7680     const struct lttng_condition *condition = NULL;
7681     struct lttng_triggers *triggers = NULL;
7682     unsigned int trigger_count;
7683     unsigned int i;
7684     enum lttng_error_code error_code;
7685     enum lttng_trigger_status trigger_status;
7686     bool ret = false;
7687
7688     /* Get all LTTng triggers */
7689     error_code = lttng_list_triggers(&triggers);
7690     assert(error_code == LTTNG_OK);
7691
7692     /* Get the number of triggers */
7693     trigger_status = lttng_triggers_get_count(triggers, &trigger_count);
7694     assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7695
7696     /* Find the trigger named `trigger_name` */
7697     for (i = 0; i < trigger_count; i++) {
7698         const struct lttng_trigger *trigger;
7699         const char *this_trigger_name;
7700
7701         trigger = lttng_triggers_get_at_index(triggers, i);
7702         trigger_status = lttng_trigger_get_name(trigger, &this_trigger_name);
7703         assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7704
7705         if (strcmp(this_trigger_name, trigger_name) == 0) {
7706             /* Trigger found: subscribe with its condition */
7707             enum lttng_notification_channel_status notification_channel_status;
7708
7709             notification_channel_status = lttng_notification_channel_subscribe(
7710                 notification_channel,
7711                 lttng_trigger_get_const_condition(trigger));
7712             assert(notification_channel_status ==
7713                    LTTNG_NOTIFICATION_CHANNEL_STATUS_OK);
7714             ret = true;
7715             break;
7716         }
7717     }
7718
7719     lttng_triggers_destroy(triggers);
7720     return ret;
7721 }
7722
7723 /*
7724  * Handles the evaluation `evaluation` of a single notification.
7725  */
7726 static void handle_evaluation(const struct lttng_evaluation *evaluation)
7727 {
7728     enum lttng_evaluation_status evaluation_status;
7729     const struct lttng_event_field_value *array_field_value;
7730     const struct lttng_event_field_value *string_field_value;
7731     enum lttng_event_field_value_status event_field_value_status;
7732     const char *string_field_string_value;
7733
7734     /* Get the value of the first captured (string) field */
7735     evaluation_status = lttng_evaluation_event_rule_matches_get_captured_values(
7736         evaluation, &array_field_value);
7737     assert(evaluation_status == LTTNG_EVALUATION_STATUS_OK);
7738     event_field_value_status =
7739         lttng_event_field_value_array_get_element_at_index(
7740             array_field_value, 0, &string_field_value);
7741     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7742     assert(lttng_event_field_value_get_type(string_field_value) ==
7743            LTTNG_EVENT_FIELD_VALUE_TYPE_STRING);
7744     event_field_value_status = lttng_event_field_value_string_get_value(
7745             string_field_value, &string_field_string_value);
7746     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7747
7748     /* Print the string value of the field */
7749     puts(string_field_string_value);
7750 }
7751
7752 int main(int argc, char *argv[])
7753 {
7754     int exit_status = EXIT_SUCCESS;
7755     struct lttng_notification_channel *notification_channel;
7756     enum lttng_notification_channel_status notification_channel_status;
7757     const struct lttng_condition *condition;
7758     const char *trigger_name;
7759     bool subscribe_res;
7760
7761     assert(argc >= 2);
7762     trigger_name = argv[1];
7763
7764     /*
7765      * Create a notification channel.
7766      *
7767      * A notification channel connects the user application to the LTTng
7768      * session daemon.
7769      *
7770      * You can use this notification channel to listen to various types
7771      * of notifications.
7772      */
7773     notification_channel = lttng_notification_channel_create(
7774         lttng_session_daemon_notification_endpoint);
7775     assert(notification_channel);
7776
7777     /*
7778      * Subscribe to notifications which match the condition of the
7779      * trigger named `trigger_name`.
7780      */
7781     if (!subscribe(notification_channel, trigger_name)) {
7782         fprintf(stderr,
7783                 "Error: Failed to subscribe to notifications (trigger `%s`).\n",
7784                 trigger_name);
7785         exit_status = EXIT_FAILURE;
7786         goto end;
7787     }
7788
7789     /*
7790      * Notification loop.
7791      *
7792      * Put this in a dedicated thread to avoid blocking the main thread.
7793      */
7794     while (true) {
7795         struct lttng_notification *notification;
7796         enum lttng_notification_channel_status status;
7797         const struct lttng_evaluation *notification_evaluation;
7798
7799         /* Receive the next notification */
7800         status = lttng_notification_channel_get_next_notification(
7801             notification_channel, &notification);
7802
7803         switch (status) {
7804         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7805             break;
7806         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7807             /*
7808              * The session daemon can drop notifications if a receiving
7809              * application doesn't consume the notifications fast
7810              * enough.
7811              */
7812             continue;
7813         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7814             /*
7815              * The session daemon closed the notification channel.
7816              *
7817              * This is typically caused by a session daemon shutting
7818              * down.
7819              */
7820             goto end;
7821         default:
7822             /* Unhandled conditions or errors */
7823             exit_status = EXIT_FAILURE;
7824             goto end;
7825         }
7826
7827         /*
7828          * Handle the condition evaluation.
7829          *
7830          * A notification provides, amongst other things:
7831          *
7832          * * The condition that caused LTTng to send this notification.
7833          *
7834          * * The condition evaluation, which provides more specific
7835          *   information on the evaluation of the condition.
7836          */
7837         handle_evaluation(lttng_notification_get_evaluation(notification));
7838
7839         /* Destroy the notification object */
7840         lttng_notification_destroy(notification);
7841     }
7842
7843 end:
7844     lttng_notification_channel_destroy(notification_channel);
7845     return exit_status;
7846 }
7847 ----
7848 --
7849 +
7850 This application prints the first captured string field value of the
7851 condition evaluation of each LTTng notification it receives.
7852
7853 . Build the `notif-app` application,
7854   using https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
7855   to provide the right compiler and linker flags:
7856 +
7857 --
7858 [role="term"]
7859 ----
7860 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
7861 ----
7862 --
7863
7864 Now, to add the trigger to the root session daemon:
7865
7866 [start=3]
7867 . **If there's no currently running LTTng root session daemon**, start
7868   one:
7869 +
7870 [role="term"]
7871 ----
7872 # lttng-sessiond --daemonize
7873 ----
7874
7875 . Add the trigger, naming it `sched-switch-notif`, to the root
7876   session daemon:
7877 +
7878 [role="term"]
7879 ----
7880 # lttng add-trigger --name=sched-switch-notif \
7881                     --condition=event-rule-matches \
7882                     --type=kernel --name=sched_switch \
7883                     --filter='next_comm == "bash"' --capture=prev_comm \
7884                     --action=notify
7885 ----
7886 +
7887 Confirm that the `sched-switch-notif` trigger exists with the
7888 man:lttng-list-triggers(1) command:
7889 +
7890 [role="term"]
7891 ----
7892 # lttng list-triggers
7893 ----
7894
7895 Run the cmd:notif-app application, passing the name of the trigger
7896 of which to watch the notifications:
7897
7898 [role="term"]
7899 ----
7900 # ./notif-app sched-switch-notif
7901 ----
7902
7903 Now, in an interactive Bash, type a few keys to fire the
7904 `sched-switch-notif` trigger. Watch the `notif-app` application print
7905 the previous process names.
7906 ====
7907
7908 [role="since-2.6"]
7909 [[mi]]
7910 === Use the machine interface
7911
7912 With any command of the man:lttng(1) command-line tool, set the
7913 opt:lttng(1):--mi option to `xml` (before the command name) to get an
7914 XML machine interface output, for example:
7915
7916 [role="term"]
7917 ----
7918 $ lttng --mi=xml list my-session
7919 ----
7920
7921 A schema definition (XSD) is
7922 https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/mi-lttng-4.0.xsd[available]
7923 to ease the integration with external tools as much as possible.
7924
7925
7926 [role="since-2.8"]
7927 [[metadata-regenerate]]
7928 === Regenerate the metadata of an LTTng trace
7929
7930 An LTTng trace, which is a https://diamon.org/ctf[CTF] trace, has both
7931 data stream files and a metadata stream file. This metadata file
7932 contains, amongst other things, information about the offset of the
7933 clock sources which LTTng uses to assign timestamps to <<event,event
7934 records>> when recording.
7935
7936 If, once a <<tracing-session,recording session>> is
7937 <<basic-tracing-session-control,started>>, a major
7938 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
7939 happens, the clock offset of the trace also needs to be updated. Use
7940 the `metadata` item of the man:lttng-regenerate(1) command to do so.
7941
7942 The main use case of this command is to allow a system to boot with
7943 an incorrect wall time and have LTTng trace it before its wall time
7944 is corrected. Once the system is known to be in a state where its
7945 wall time is correct, you can run `lttng regenerate metadata`.
7946
7947 To regenerate the metadata stream files of the
7948 <<cur-tracing-session,current recording session>>:
7949
7950 * Use the `metadata` item of the man:lttng-regenerate(1) command:
7951 +
7952 --
7953 [role="term"]
7954 ----
7955 $ lttng regenerate metadata
7956 ----
7957 --
7958
7959
7960 [role="since-2.9"]
7961 [[regenerate-statedump]]
7962 === Regenerate the state dump event records of a recording session
7963
7964 The LTTng kernel and user space tracers generate state dump
7965 <<event,event records>> when the application starts or when you
7966 <<basic-tracing-session-control,start a recording session>>.
7967
7968 An analysis can use the state dump event records to set an initial state
7969 before it builds the rest of the state from the subsequent event
7970 records. http://tracecompass.org/[Trace Compass] is a notable
7971 example of an application which uses the state dump of an LTTng trace.
7972
7973 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7974 state dump event records aren't included in the snapshot trace files
7975 because they were recorded to a <<channel,sub-buffer>> that has been
7976 consumed or <<overwrite-mode,overwritten>> already.
7977
7978 Use the `statedump` item of the man:lttng-regenerate(1) command to emit
7979 and record the state dump events again.
7980
7981 To regenerate the state dump of the <<cur-tracing-session,current
7982 recording session>>, provided you created it in <<snapshot-mode,snapshot
7983 mode>>, before you take a snapshot:
7984
7985 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7986 +
7987 --
7988 [role="term"]
7989 ----
7990 $ lttng regenerate statedump
7991 ----
7992 --
7993
7994 . <<basic-tracing-session-control,Stop the recording session>>:
7995 +
7996 --
7997 [role="term"]
7998 ----
7999 $ lttng stop
8000 ----
8001 --
8002
8003 . <<taking-a-snapshot,Take a snapshot>>:
8004 +
8005 --
8006 [role="term"]
8007 ----
8008 $ lttng snapshot record --name=my-snapshot
8009 ----
8010 --
8011
8012 Depending on the event throughput, you should run steps{nbsp}1
8013 and{nbsp}2 as closely as possible.
8014
8015 [NOTE]
8016 ====
8017 To record the state dump events, you need to
8018 <<enabling-disabling-events,create recording event rules>> which enable
8019 them:
8020
8021 * The names of LTTng-UST state dump tracepoints start with
8022   `lttng_ust_statedump:`.
8023
8024 * The names of LTTng-modules state dump tracepoints start with
8025   `lttng_statedump_`.
8026 ====
8027
8028
8029 [role="since-2.7"]
8030 [[persistent-memory-file-systems]]
8031 === Record trace data on persistent memory file systems
8032
8033 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile
8034 random-access memory] (NVRAM) is random-access memory that retains its
8035 information when power is turned off (non-volatile). Systems with such
8036 memory can store data structures in RAM and retrieve them after a
8037 reboot, without flushing to typical _storage_.
8038
8039 Linux supports NVRAM file systems thanks to either
8040 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
8041 (requires Linux{nbsp}4.1+) or http://pramfs.sourceforge.net/[PRAMFS] (requires Linux{nbsp}<{nbsp}4).
8042
8043 This section doesn't describe how to operate such file systems; we
8044 assume that you have a working persistent memory file system.
8045
8046 When you <<creating-destroying-tracing-sessions,create a recording
8047 session>>, you can specify the path of the shared memory holding the
8048 sub-buffers. If you specify a location on an NVRAM file system, then you
8049 can retrieve the latest recorded trace data when the system reboots
8050 after a crash.
8051
8052 To record trace data on a persistent memory file system and retrieve the
8053 trace data after a system crash:
8054
8055 . Create a recording session with a <<channel,sub-buffer>> shared memory
8056   path located on an NVRAM file system:
8057 +
8058 --
8059 [role="term"]
8060 ----
8061 $ lttng create my-session --shm-path=/path/to/shm/on/nvram
8062 ----
8063 --
8064
8065 . Configure the recording session as usual with the man:lttng(1)
8066   command-line tool, and <<basic-tracing-session-control,start
8067   recording>>.
8068
8069 . After a system crash, use the man:lttng-crash(1) command-line tool to
8070   read the trace data recorded on the NVRAM file system:
8071 +
8072 --
8073 [role="term"]
8074 ----
8075 $ lttng-crash /path/to/shm/on/nvram
8076 ----
8077 --
8078
8079 The binary layout of the ring buffer files isn't exactly the same as the
8080 trace files layout. This is why you need to use man:lttng-crash(1)
8081 instead of some standard LTTng trace reader.
8082
8083 To convert the ring buffer files to LTTng trace files:
8084
8085 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
8086 +
8087 --
8088 [role="term"]
8089 ----
8090 $ lttng-crash --extract=/path/to/trace /path/to/shm/on/nvram
8091 ----
8092 --
8093
8094
8095 [role="since-2.10"]
8096 [[notif-trigger-api]]
8097 === Get notified when the buffer usage of a channel is too high or too low
8098
8099 With the notification and <<trigger,trigger>> C{nbsp}API of
8100 <<liblttng-ctl-lttng,`liblttng-ctl`>>, LTTng can notify your user
8101 application when the buffer usage of one or more <<channel,channels>>
8102 becomes too low or too high.
8103
8104 Use this API and enable or disable <<event,recording event rules>> while
8105 a recording session <<basic-tracing-session-control,is active>> to avoid
8106 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>, for
8107 example.
8108
8109 .Send a notification to a user application when the buffer usage of an LTTng channel is too high.
8110 ====
8111 In this example, we create and build an application which gets notified
8112 when the buffer usage of a specific LTTng channel is higher than
8113 75{nbsp}%.
8114
8115 We only print that it's the case in this example, but we could as well
8116 use the `liblttng-ctl` C{nbsp}API to <<enabling-disabling-events,disable
8117 recording event rules>> when this happens, for example.
8118
8119 . Create the C{nbsp}source file of the application:
8120 +
8121 --
8122 [source,c]
8123 .path:{notif-app.c}
8124 ----
8125 #include <stdlib.h>
8126 #include <stdio.h>
8127 #include <assert.h>
8128 #include <lttng/lttng.h>
8129
8130 int main(int argc, char *argv[])
8131 {
8132     int exit_status = EXIT_SUCCESS;
8133     struct lttng_notification_channel *notification_channel;
8134     struct lttng_condition *condition;
8135     struct lttng_action *action;
8136     struct lttng_trigger *trigger;
8137     const char *recording_session_name;
8138     const char *channel_name;
8139
8140     assert(argc >= 3);
8141     recording_session_name = argv[1];
8142     channel_name = argv[2];
8143
8144     /*
8145      * Create a notification channel.
8146      *
8147      * A notification channel connects the user application to the LTTng
8148      * session daemon.
8149      *
8150      * You can use this notification channel to listen to various types
8151      * of notifications.
8152      */
8153     notification_channel = lttng_notification_channel_create(
8154         lttng_session_daemon_notification_endpoint);
8155
8156     /*
8157      * Create a "buffer usage becomes greater than" condition.
8158      *
8159      * In this case, the condition is satisfied when the buffer usage
8160      * becomes greater than or equal to 75 %.
8161      *
8162      * We create the condition for a specific recording session name,
8163      * channel name, and for the user space tracing domain.
8164      *
8165      * The following condition types also exist:
8166      *
8167      * * The buffer usage of a channel becomes less than a given value.
8168      *
8169      * * The consumed data size of a recording session becomes greater
8170      *   than a given value.
8171      *
8172      * * A recording session rotation becomes ongoing.
8173      *
8174      * * A recording session rotation becomes completed.
8175      *
8176      * * A given event rule matches an event.
8177      */
8178     condition = lttng_condition_buffer_usage_high_create();
8179     lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
8180     lttng_condition_buffer_usage_set_session_name(condition,
8181                                                   recording_session_name);
8182     lttng_condition_buffer_usage_set_channel_name(condition,
8183                                                   channel_name);
8184     lttng_condition_buffer_usage_set_domain_type(condition,
8185                                                  LTTNG_DOMAIN_UST);
8186
8187     /*
8188      * Create an action (receive a notification) to execute when the
8189      * condition created above is satisfied.
8190      */
8191     action = lttng_action_notify_create();
8192
8193     /*
8194      * Create a trigger.
8195      *
8196      * A trigger associates a condition to an action: LTTng executes
8197      * the action when the condition is satisfied.
8198      */
8199     trigger = lttng_trigger_create(condition, action);
8200
8201     /* Register the trigger to the LTTng session daemon. */
8202     lttng_register_trigger(trigger);
8203
8204     /*
8205      * Now that we have registered a trigger, LTTng will send a
8206      * notification every time its condition is met through a
8207      * notification channel.
8208      *
8209      * To receive this notification, we must subscribe to notifications
8210      * which match the same condition.
8211      */
8212     lttng_notification_channel_subscribe(notification_channel,
8213                                          condition);
8214
8215     /*
8216      * Notification loop.
8217      *
8218      * Put this in a dedicated thread to avoid blocking the main thread.
8219      */
8220     for (;;) {
8221         struct lttng_notification *notification;
8222         enum lttng_notification_channel_status status;
8223         const struct lttng_evaluation *notification_evaluation;
8224         const struct lttng_condition *notification_condition;
8225         double buffer_usage;
8226
8227         /* Receive the next notification. */
8228         status = lttng_notification_channel_get_next_notification(
8229             notification_channel, &notification);
8230
8231         switch (status) {
8232         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
8233             break;
8234         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
8235             /*
8236              * The session daemon can drop notifications if a monitoring
8237              * application isn't consuming the notifications fast
8238              * enough.
8239              */
8240             continue;
8241         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
8242             /*
8243              * The session daemon closed the notification channel.
8244              *
8245              * This is typically caused by a session daemon shutting
8246              * down.
8247              */
8248             goto end;
8249         default:
8250             /* Unhandled conditions or errors. */
8251             exit_status = EXIT_FAILURE;
8252             goto end;
8253         }
8254
8255         /*
8256          * A notification provides, amongst other things:
8257          *
8258          * * The condition that caused LTTng to send this notification.
8259          *
8260          * * The condition evaluation, which provides more specific
8261          *   information on the evaluation of the condition.
8262          *
8263          * The condition evaluation provides the buffer usage
8264          * value at the moment the condition was satisfied.
8265          */
8266         notification_condition = lttng_notification_get_condition(
8267             notification);
8268         notification_evaluation = lttng_notification_get_evaluation(
8269             notification);
8270
8271         /* We're subscribed to only one condition. */
8272         assert(lttng_condition_get_type(notification_condition) ==
8273             LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
8274
8275         /*
8276          * Get the exact sampled buffer usage from the condition
8277          * evaluation.
8278          */
8279         lttng_evaluation_buffer_usage_get_usage_ratio(
8280             notification_evaluation, &buffer_usage);
8281
8282         /*
8283          * At this point, instead of printing a message, we could do
8284          * something to reduce the buffer usage of the channel, like
8285          * disable specific events, for example.
8286          */
8287         printf("Buffer usage is %f %% in recording session \"%s\", "
8288                "user space channel \"%s\".\n", buffer_usage * 100,
8289                recording_session_name, channel_name);
8290
8291         /* Destroy the notification object. */
8292         lttng_notification_destroy(notification);
8293     }
8294
8295 end:
8296     lttng_action_destroy(action);
8297     lttng_condition_destroy(condition);
8298     lttng_trigger_destroy(trigger);
8299     lttng_notification_channel_destroy(notification_channel);
8300     return exit_status;
8301 }
8302 ----
8303 --
8304
8305 . Build the `notif-app` application, linking it with `liblttng-ctl`:
8306 +
8307 --
8308 [role="term"]
8309 ----
8310 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
8311 ----
8312 --
8313
8314 . <<creating-destroying-tracing-sessions,Create a recording session>>,
8315   <<enabling-disabling-events,create a recording event rule>> matching
8316   all the user space tracepoint events, and
8317   <<basic-tracing-session-control,start recording>>:
8318 +
8319 --
8320 [role="term"]
8321 ----
8322 $ lttng create my-session
8323 $ lttng enable-event --userspace --all
8324 $ lttng start
8325 ----
8326 --
8327 +
8328 If you create the channel manually with the man:lttng-enable-channel(1)
8329 command, you can set its <<channel-monitor-timer,monitor timer>> to
8330 control how frequently LTTng samples the current values of the channel
8331 properties to evaluate user conditions.
8332
8333 . Run the `notif-app` application.
8334 +
8335 This program accepts the <<tracing-session,recording session>> and
8336 user space channel names as its two first arguments. The channel
8337 which LTTng automatically creates with the man:lttng-enable-event(1)
8338 command above is named `channel0`:
8339 +
8340 --
8341 [role="term"]
8342 ----
8343 $ ./notif-app my-session channel0
8344 ----
8345 --
8346
8347 . In another terminal, run an application with a very high event
8348   throughput so that the 75{nbsp}% buffer usage condition is reached.
8349 +
8350 In the first terminal, the application should print lines like this:
8351 +
8352 ----
8353 Buffer usage is 81.45197 % in recording session "my-session", user space
8354 channel "channel0".
8355 ----
8356 +
8357 If you don't see anything, try to make the threshold of the condition in
8358 path:{notif-app.c} lower (0.1{nbsp}%, for example), and then rebuild the
8359 `notif-app` application (step{nbsp}2) and run it again (step{nbsp}4).
8360 ====
8361
8362
8363 [[reference]]
8364 == Reference
8365
8366 [[lttng-modules-ref]]
8367 === noch:{LTTng-modules}
8368
8369
8370 [role="since-2.9"]
8371 [[lttng-tracepoint-enum]]
8372 ==== `LTTNG_TRACEPOINT_ENUM()` usage
8373
8374 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
8375
8376 [source,c]
8377 ----
8378 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
8379 ----
8380
8381 Replace:
8382
8383 * `name` with the name of the enumeration (C identifier, unique
8384   amongst all the defined enumerations).
8385 * `entries` with a list of enumeration entries.
8386
8387 The available enumeration entry macros are:
8388
8389 +ctf_enum_value(__name__, __value__)+::
8390   Entry named +__name__+ mapped to the integral value +__value__+.
8391
8392 +ctf_enum_range(__name__, __begin__, __end__)+::
8393   Entry named +__name__+ mapped to the range of integral values between
8394   +__begin__+ (included) and +__end__+ (included).
8395
8396 +ctf_enum_auto(__name__)+::
8397   Entry named +__name__+ mapped to the integral value following the
8398   last mapping value.
8399 +
8400 The last value of a `ctf_enum_value()` entry is its +__value__+
8401 parameter.
8402 +
8403 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
8404 +
8405 If `ctf_enum_auto()` is the first entry in the list, its integral
8406 value is 0.
8407
8408 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
8409 to use a defined enumeration as a tracepoint field.
8410
8411 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
8412 ====
8413 [source,c]
8414 ----
8415 LTTNG_TRACEPOINT_ENUM(
8416     my_enum,
8417     TP_ENUM_VALUES(
8418         ctf_enum_auto("AUTO: EXPECT 0")
8419         ctf_enum_value("VALUE: 23", 23)
8420         ctf_enum_value("VALUE: 27", 27)
8421         ctf_enum_auto("AUTO: EXPECT 28")
8422         ctf_enum_range("RANGE: 101 TO 303", 101, 303)
8423         ctf_enum_auto("AUTO: EXPECT 304")
8424     )
8425 )
8426 ----
8427 ====
8428
8429
8430 [role="since-2.7"]
8431 [[lttng-modules-tp-fields]]
8432 ==== Tracepoint fields macros (for `TP_FIELDS()`)
8433
8434 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
8435 tracepoint fields, which must be listed within `TP_FIELDS()` in
8436 `LTTNG_TRACEPOINT_EVENT()`, are:
8437
8438 [role="func-desc growable",cols="asciidoc,asciidoc"]
8439 .Available macros to define LTTng-modules tracepoint fields
8440 |====
8441 |Macro |Description and parameters
8442
8443 |
8444 +ctf_integer(__t__, __n__, __e__)+
8445
8446 +ctf_integer_nowrite(__t__, __n__, __e__)+
8447
8448 +ctf_user_integer(__t__, __n__, __e__)+
8449
8450 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
8451 |
8452 Standard integer, displayed in base{nbsp}10.
8453
8454 +__t__+::
8455   Integer C type (`int`, `long`, `size_t`, ...).
8456
8457 +__n__+::
8458   Field name.
8459
8460 +__e__+::
8461   Argument expression.
8462
8463 |
8464 +ctf_integer_hex(__t__, __n__, __e__)+
8465
8466 +ctf_user_integer_hex(__t__, __n__, __e__)+
8467 |
8468 Standard integer, displayed in base{nbsp}16.
8469
8470 +__t__+::
8471   Integer C type.
8472
8473 +__n__+::
8474   Field name.
8475
8476 +__e__+::
8477   Argument expression.
8478
8479 |+ctf_integer_oct(__t__, __n__, __e__)+
8480 |
8481 Standard integer, displayed in base{nbsp}8.
8482
8483 +__t__+::
8484   Integer C type.
8485
8486 +__n__+::
8487   Field name.
8488
8489 +__e__+::
8490   Argument expression.
8491
8492 |
8493 +ctf_integer_network(__t__, __n__, __e__)+
8494
8495 +ctf_user_integer_network(__t__, __n__, __e__)+
8496 |
8497 Integer in network byte order (big-endian), displayed in base{nbsp}10.
8498
8499 +__t__+::
8500   Integer C type.
8501
8502 +__n__+::
8503   Field name.
8504
8505 +__e__+::
8506   Argument expression.
8507
8508 |
8509 +ctf_integer_network_hex(__t__, __n__, __e__)+
8510
8511 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
8512 |
8513 Integer in network byte order, displayed in base{nbsp}16.
8514
8515 +__t__+::
8516   Integer C type.
8517
8518 +__n__+::
8519   Field name.
8520
8521 +__e__+::
8522   Argument expression.
8523
8524 |
8525 +ctf_enum(__N__, __t__, __n__, __e__)+
8526
8527 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
8528
8529 +ctf_user_enum(__N__, __t__, __n__, __e__)+
8530
8531 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
8532 |
8533 Enumeration.
8534
8535 +__N__+::
8536   Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
8537
8538 +__t__+::
8539   Integer C type (`int`, `long`, `size_t`, ...).
8540
8541 +__n__+::
8542   Field name.
8543
8544 +__e__+::
8545   Argument expression.
8546
8547 |
8548 +ctf_string(__n__, __e__)+
8549
8550 +ctf_string_nowrite(__n__, __e__)+
8551
8552 +ctf_user_string(__n__, __e__)+
8553
8554 +ctf_user_string_nowrite(__n__, __e__)+
8555 |
8556 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
8557
8558 +__n__+::
8559   Field name.
8560
8561 +__e__+::
8562   Argument expression.
8563
8564 |
8565 +ctf_array(__t__, __n__, __e__, __s__)+
8566
8567 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
8568
8569 +ctf_user_array(__t__, __n__, __e__, __s__)+
8570
8571 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
8572 |
8573 Statically-sized array of integers.
8574
8575 +__t__+::
8576   Array element C type.
8577
8578 +__n__+::
8579   Field name.
8580
8581 +__e__+::
8582   Argument expression.
8583
8584 +__s__+::
8585   Number of elements.
8586
8587 |
8588 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
8589
8590 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8591
8592 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
8593
8594 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8595 |
8596 Statically-sized array of bits.
8597
8598 The type of +__e__+ must be an integer type. +__s__+ is the number
8599 of elements of such type in +__e__+, not the number of bits.
8600
8601 +__t__+::
8602   Array element C type.
8603
8604 +__n__+::
8605   Field name.
8606
8607 +__e__+::
8608   Argument expression.
8609
8610 +__s__+::
8611   Number of elements.
8612
8613 |
8614 +ctf_array_text(__t__, __n__, __e__, __s__)+
8615
8616 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
8617
8618 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
8619
8620 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
8621 |
8622 Statically-sized array, printed as text.
8623
8624 The string doesn't need to be null-terminated.
8625
8626 +__t__+::
8627   Array element C type (always `char`).
8628
8629 +__n__+::
8630   Field name.
8631
8632 +__e__+::
8633   Argument expression.
8634
8635 +__s__+::
8636   Number of elements.
8637
8638 |
8639 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
8640
8641 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8642
8643 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
8644
8645 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8646 |
8647 Dynamically-sized array of integers.
8648
8649 The type of +__E__+ must be unsigned.
8650
8651 +__t__+::
8652   Array element C type.
8653
8654 +__n__+::
8655   Field name.
8656
8657 +__e__+::
8658   Argument expression.
8659
8660 +__T__+::
8661   Length expression C type.
8662
8663 +__E__+::
8664   Length expression.
8665
8666 |
8667 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8668
8669 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8670 |
8671 Dynamically-sized array of integers, displayed in base{nbsp}16.
8672
8673 The type of +__E__+ must be unsigned.
8674
8675 +__t__+::
8676   Array element C type.
8677
8678 +__n__+::
8679   Field name.
8680
8681 +__e__+::
8682   Argument expression.
8683
8684 +__T__+::
8685   Length expression C type.
8686
8687 +__E__+::
8688   Length expression.
8689
8690 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
8691 |
8692 Dynamically-sized array of integers in network byte order (big-endian),
8693 displayed in base{nbsp}10.
8694
8695 The type of +__E__+ must be unsigned.
8696
8697 +__t__+::
8698   Array element C type.
8699
8700 +__n__+::
8701   Field name.
8702
8703 +__e__+::
8704   Argument expression.
8705
8706 +__T__+::
8707   Length expression C type.
8708
8709 +__E__+::
8710   Length expression.
8711
8712 |
8713 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8714
8715 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8716
8717 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8718
8719 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8720 |
8721 Dynamically-sized array of bits.
8722
8723 The type of +__e__+ must be an integer type. +__s__+ is the number
8724 of elements of such type in +__e__+, not the number of bits.
8725
8726 The type of +__E__+ must be unsigned.
8727
8728 +__t__+::
8729   Array element C type.
8730
8731 +__n__+::
8732   Field name.
8733
8734 +__e__+::
8735   Argument expression.
8736
8737 +__T__+::
8738   Length expression C type.
8739
8740 +__E__+::
8741   Length expression.
8742
8743 |
8744 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8745
8746 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8747
8748 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8749
8750 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8751 |
8752 Dynamically-sized array, displayed as text.
8753
8754 The string doesn't need to be null-terminated.
8755
8756 The type of +__E__+ must be unsigned.
8757
8758 The behaviour is undefined if +__e__+ is `NULL`.
8759
8760 +__t__+::
8761   Sequence element C type (always `char`).
8762
8763 +__n__+::
8764   Field name.
8765
8766 +__e__+::
8767   Argument expression.
8768
8769 +__T__+::
8770   Length expression C type.
8771
8772 +__E__+::
8773   Length expression.
8774 |====
8775
8776 Use the `_user` versions when the argument expression, `e`, is
8777 a user space address. In the cases of `ctf_user_integer*()` and
8778 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
8779 be addressable.
8780
8781 The `_nowrite` versions omit themselves from the trace data, but are
8782 otherwise identical. This means LTTng won't write the `_nowrite` fields
8783 to the recorded trace. Their primary purpose is to make some of the
8784 event context available to the <<enabling-disabling-events,recording
8785 event rule filters>> without having to commit the data to
8786 <<channel,sub-buffers>>.
8787
8788
8789 [[glossary]]
8790 == Glossary
8791
8792 Terms related to LTTng and to tracing in general:
8793
8794 [[def-action]]action::
8795   The part of a <<def-trigger,trigger>> which LTTng executes when the
8796   trigger <<def-condition,condition>> is satisfied.
8797
8798 Babeltrace::
8799   The https://diamon.org/babeltrace[Babeltrace] project, which includes:
8800 +
8801 * The
8802   https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
8803   command-line interface.
8804 * The libbabeltrace2 library which offers a
8805   https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
8806 * https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
8807 * Plugins.
8808
8809 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
8810   A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
8811
8812 [[def-channel]]<<channel,channel>>::
8813   An entity which is responsible for a set of
8814   <<def-ring-buffer,ring buffers>>.
8815 +
8816 <<def-recording-event-rule,Recording event rules>> are always attached
8817 to a specific channel.
8818
8819 clock::
8820   A source of time for a <<def-tracer,tracer>>.
8821
8822 [[def-condition]]condition::
8823   The part of a <<def-trigger,trigger>> which must be satisfied for
8824   LTTng to attempt to execute the trigger <<def-action,actions>>.
8825
8826 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
8827   A program which is responsible for consuming the full
8828   <<def-sub-buffer,sub-buffers>> and write them to a file system or
8829   send them over the network.
8830
8831 [[def-current-trace-chunk]]current trace chunk::
8832   A <<def-trace-chunk,trace chunk>> which includes the current content
8833   of all the <<def-sub-buffer,sub-buffers>> of the
8834   <<def-tracing-session,recording session>> and the stream files
8835   produced since the latest event amongst:
8836 +
8837 * The creation of the recording session.
8838 * The last <<def-tracing-session-rotation,recording session rotation>>, if
8839   any.
8840
8841 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
8842   The <<def-event-record-loss-mode,event record loss mode>> in which
8843   the <<def-tracer,tracer>> _discards_ new <<def-event-record,event
8844   records>> when there's no <<def-sub-buffer,sub-buffer>> space left to
8845   store them.
8846
8847 [[def-event]]event::
8848   The execution of an <<def-instrumentation-point,instrumentation
8849   point>>, like a <<def-tracepoint,tracepoint>> that you manually place
8850   in some source code, or a Linux kprobe.
8851 +
8852 When an instrumentation point is executed, LTTng creates an event.
8853 +
8854 When an <<def-event-rule,event rule>> matches the event,
8855 <<def-lttng,LTTng>> executes some action, for example:
8856 +
8857 * Record its payload to a <<def-sub-buffer,sub-buffer>> as an
8858   <<def-event-record,event record>>.
8859 * Attempt to execute the user-defined actions of a
8860   <<def-trigger,trigger>> with an
8861   <<add-event-rule-matches-trigger,``event rule matches''>> condition.
8862
8863 [[def-event-name]]event name::
8864   The name of an <<def-event,event>>, which is also the name of the
8865   <<def-event-record,event record>>.
8866 +
8867 This is also called the _instrumentation point name_.
8868
8869 [[def-event-record]]event record::
8870   A record (binary serialization), in a <<def-trace,trace>>, of the
8871   payload of an <<def-event,event>>.
8872 +
8873 The payload of an event record has zero or more _fields_.
8874
8875 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
8876   The mechanism by which event records of a given
8877   <<def-channel,channel>> are lost (not recorded) when there's no
8878   <<def-sub-buffer,sub-buffer>> space left to store them.
8879
8880 [[def-event-rule]]<<event-rule,event rule>>::
8881   Set of conditions which an <<def-event,event>> must satisfy
8882   for LTTng to execute some action.
8883 +
8884 An event rule is said to _match_ events, like a
8885 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
8886 matches strings.
8887 +
8888 A <<def-recording-event-rule,recording event rule>> is a specific type
8889 of event rule of which the action is to <<def-record,record>> the event
8890 to a <<def-sub-buffer,sub-buffer>>.
8891
8892 [[def-incl-set]]inclusion set::
8893   In the <<pid-tracking,process attribute inclusion set>> context: a
8894   set of <<def-proc-attr,process attributes>> of a given type.
8895
8896 <<instrumenting,instrumentation>>::
8897   The use of <<def-lttng,LTTng>> probes to make a kernel or
8898   <<def-user-application,user application>> traceable.
8899
8900 [[def-instrumentation-point]]instrumentation point::
8901   A point in the execution path of a kernel or
8902   <<def-user-application,user application>> which, when executed,
8903   create an <<def-event,event>>.
8904
8905 instrumentation point name::
8906   See _<<def-event-name,event name>>_.
8907
8908 `java.util.logging`::
8909   The
8910   https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
8911   of the Java platform.
8912
8913 log4j::
8914   A https://logging.apache.org/log4j/1.2/[logging library] for Java
8915   developed by the Apache Software Foundation.
8916
8917 log level::
8918   Level of severity of a log statement or user space
8919   <<def-instrumentation-point,instrumentation point>>.
8920
8921 [[def-lttng]]LTTng::
8922   The _Linux Trace Toolkit: next generation_ project.
8923
8924 <<lttng-cli,cmd:lttng>>::
8925   A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
8926   project which you can use to send and receive control messages to and
8927   from a <<def-session-daemon,session daemon>>.
8928
8929 cmd:lttng-consumerd::
8930   The name of the <<def-consumer-daemon,consumer daemon>> program.
8931
8932 cmd:lttng-crash::
8933   A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8934   which can convert <<def-ring-buffer,ring buffer>> files (usually
8935   <<persistent-memory-file-systems,saved on a persistent memory file
8936   system>>) to <<def-trace,trace>> files.
8937 +
8938 See man:lttng-crash(1).
8939
8940 LTTng Documentation::
8941   This document.
8942
8943 <<lttng-live,LTTng live>>::
8944   A communication protocol between the <<lttng-relayd,relay daemon>> and
8945   live readers which makes it possible to show or analyze
8946   <<def-event-record,event records>> ``live'', as they're received by
8947   the <<def-relay-daemon,relay daemon>>.
8948
8949 <<lttng-modules,LTTng-modules>>::
8950   The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8951   which contains the Linux kernel modules to make the Linux kernel
8952   <<def-instrumentation-point,instrumentation points>> available for
8953   <<def-lttng,LTTng>> tracing.
8954
8955 cmd:lttng-relayd::
8956   The name of the <<def-relay-daemon,relay daemon>> program.
8957
8958 cmd:lttng-sessiond::
8959   The name of the <<def-session-daemon,session daemon>> program.
8960
8961 [[def-lttng-tools]]LTTng-tools::
8962   The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8963   contains the various programs and libraries used to
8964   <<controlling-tracing,control tracing>>.
8965
8966 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8967   The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8968   contains libraries to instrument
8969   <<def-user-application,user applications>>.
8970
8971 <<lttng-ust-agents,LTTng-UST Java agent>>::
8972   A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8973   allow the LTTng instrumentation of `java.util.logging` and Apache
8974   log4j{nbsp}1.2 logging statements.
8975
8976 <<lttng-ust-agents,LTTng-UST Python agent>>::
8977   A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8978   to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8979   statements.
8980
8981 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8982   The <<def-event-record-loss-mode,event record loss mode>> in which new
8983   <<def-event-record,event records>> _overwrite_ older event records
8984   when there's no <<def-sub-buffer,sub-buffer>> space left to store
8985   them.
8986
8987 <<channel-buffering-schemes,per-process buffering>>::
8988   A <<def-buffering-scheme,buffering scheme>> in which each instrumented
8989   process has its own <<def-sub-buffer,sub-buffers>> for a given user
8990   space <<def-channel,channel>>.
8991
8992 <<channel-buffering-schemes,per-user buffering>>::
8993   A <<def-buffering-scheme,buffering scheme>> in which all the processes
8994   of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
8995   given user space <<def-channel,channel>>.
8996
8997 [[def-proc-attr]]process attribute::
8998   In the <<pid-tracking,process attribute inclusion set>> context:
8999 +
9000 * A process ID.
9001 * A virtual process ID.
9002 * A Unix user ID.
9003 * A virtual Unix user ID.
9004 * A Unix group ID.
9005 * A virtual Unix group ID.
9006
9007 record (_noun_)::
9008   See <<def-event-record,_event record_>>.
9009
9010 [[def-record]]record (_verb_)::
9011   Serialize the binary payload of an <<def-event,event>> to a
9012   <<def-sub-buffer,sub-buffer>>.
9013
9014 [[def-recording-event-rule]]<<event,recording event rule>>::
9015   Specific type of <<def-event-rule,event rule>> of which the action is
9016   to <<def-record,record>> the matched event to a
9017   <<def-sub-buffer,sub-buffer>>.
9018
9019 [[def-tracing-session]][[def-recording-session]]<<tracing-session,recording session>>::
9020   A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
9021
9022 [[def-tracing-session-rotation]]<<session-rotation,recording session rotation>>::
9023   The action of archiving the
9024   <<def-current-trace-chunk,current trace chunk>> of a
9025   <<def-tracing-session,recording session>>.
9026
9027 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
9028   A process which is responsible for receiving the <<def-trace,trace>>
9029   data which a distant <<def-consumer-daemon,consumer daemon>> sends.
9030
9031 [[def-ring-buffer]]ring buffer::
9032   A set of <<def-sub-buffer,sub-buffers>>.
9033
9034 rotation::
9035   See _<<def-tracing-session-rotation,recording session rotation>>_.
9036
9037 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
9038   A process which receives control commands from you and orchestrates
9039   the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
9040
9041 <<taking-a-snapshot,snapshot>>::
9042   A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
9043   of a given <<def-tracing-session,recording session>>, saved as
9044   <<def-trace,trace>> files.
9045
9046 [[def-sub-buffer]]sub-buffer::
9047   One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
9048   which contains <<def-event-record,event records>>.
9049
9050 timestamp::
9051   The time information attached to an <<def-event,event>> when LTTng
9052   creates it.
9053
9054 [[def-trace]]trace (_noun_)::
9055   A set of:
9056 +
9057 * One https://diamon.org/ctf/[CTF] metadata stream file.
9058 * One or more CTF data stream files which are the concatenations of one
9059   or more flushed <<def-sub-buffer,sub-buffers>>.
9060
9061 [[def-trace-verb]]trace (_verb_)::
9062   From the perspective of a <<def-tracer,tracer>>: attempt to execute
9063   one or more actions when emitting an <<def-event,event>> in an
9064   application or in a system.
9065
9066 [[def-trace-chunk]]trace chunk::
9067   A self-contained <<def-trace,trace>> which is part of a
9068   <<def-tracing-session,recording session>>. Each
9069   <<def-tracing-session-rotation, recording session rotation>> produces a
9070   <<def-trace-chunk-archive,trace chunk archive>>.
9071
9072 [[def-trace-chunk-archive]]trace chunk archive::
9073   The result of a <<def-tracing-session-rotation, recording session
9074   rotation>>.
9075 +
9076 <<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
9077 containing <<def-tracing-session,recording session>> is still active: you
9078 are free to read it, modify it, move it, or remove it.
9079
9080 Trace Compass::
9081   The http://tracecompass.org[Trace Compass] project and application.
9082
9083 [[def-tracepoint]]tracepoint::
9084   An instrumentation point using the tracepoint mechanism of the Linux
9085   kernel or of <<def-lttng-ust,LTTng-UST>>.
9086
9087 tracepoint definition::
9088   The definition of a single <<def-tracepoint,tracepoint>>.
9089
9090 tracepoint name::
9091   The name of a <<def-tracepoint,tracepoint>>.
9092
9093 [[def-tracepoint-provider]]tracepoint provider::
9094   A set of functions providing <<def-tracepoint,tracepoints>> to an
9095   instrumented <<def-user-application,user application>>.
9096 +
9097 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
9098 provider package>>: many tracepoint providers can exist within a
9099 tracepoint provider package.
9100
9101 [[def-tracepoint-provider-package]]tracepoint provider package::
9102   One or more <<def-tracepoint-provider,tracepoint providers>> compiled
9103   as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
9104   link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
9105   library].
9106
9107 [[def-tracer]]tracer::
9108   A piece of software which executes some action when it emits
9109   an <<def-event,event>>, like <<def-record,record>> it to some
9110   buffer.
9111
9112 <<domain,tracing domain>>::
9113   A type of LTTng <<def-tracer,tracer>>.
9114
9115 <<tracing-group,tracing group>>::
9116   The Unix group which a Unix user can be part of to be allowed to
9117   control the Linux kernel LTTng <<def-tracer,tracer>>.
9118
9119 [[def-trigger]]<<trigger,trigger>>::
9120   A <<def-condition,condition>>-<<def-action,actions>> pair; when the
9121   condition of a trigger is satisfied, LTTng attempts to execute its
9122   actions.
9123
9124 [[def-user-application]]user application::
9125   An application (program or library) running in user space, as opposed
9126   to a Linux kernel module, for example.