2.13/lttng-docs-2.13.txt

   1 The LTTng Documentation
   2 =======================
   3 Philippe Proulx <pproulx@efficios.com>
   4 v2.13, 28 November 2023
   5
   6
   7 include::../common/copyright.txt[]
   8
   9
  10 include::../common/welcome.txt[]
  11
  12
  13 include::../common/audience.txt[]
  14
  15
  16 [[chapters]]
  17 === What's in this documentation?
  18
  19 The LTTng Documentation is divided into the following sections:
  20
  21 * ``**<<nuts-and-bolts,Nuts and bolts>>**'' explains the
  22   rudiments of software tracing and the rationale behind the
  23   LTTng project.
  24 +
  25 Skip this section if you’re familiar with software tracing and with the
  26 LTTng project.
  27
  28 * ``**<<installing-lttng,Installation>>**'' describes the steps to
  29   install the LTTng packages on common Linux distributions and from
  30   their sources.
  31 +
  32 Skip this section if you already properly installed LTTng on your target
  33 system.
  34
  35 * ``**<<getting-started,Quick start>>**'' is a concise guide to
  36   get started quickly with LTTng kernel and user space tracing.
  37 +
  38 We recommend this section if you're new to LTTng or to software tracing
  39 in general.
  40 +
  41 Skip this section if you're not new to LTTng.
  42
  43 * ``**<<core-concepts,Core concepts>>**'' explains the concepts at
  44   the heart of LTTng.
  45 +
  46 It's a good idea to become familiar with the core concepts
  47 before attempting to use the toolkit.
  48
  49 * ``**<<plumbing,Components of LTTng>>**'' describes the various
  50   components of the LTTng machinery, like the daemons, the libraries,
  51   and the command-line interface.
  52
  53 * ``**<<instrumenting,Instrumentation>>**'' shows different ways to
  54   instrument user applications and the Linux kernel for LTTng tracing.
  55 +
  56 Instrumenting source code is essential to provide a meaningful
  57 source of events.
  58 +
  59 Skip this section if you don't have a programming background.
  60
  61 * ``**<<controlling-tracing,Tracing control>>**'' is divided into topics
  62   which demonstrate how to use the vast array of features that
  63   LTTng{nbsp}{revision} offers.
  64
  65 * ``**<<reference,Reference>>**'' contains API reference tables.
  66
  67 * ``**<<glossary,Glossary>>**'' is a specialized dictionary of terms
  68   related to LTTng or to the field of software tracing.
  69
  70
  71 include::../common/convention.txt[]
  72
  73
  74 include::../common/acknowledgements.txt[]
  75
  76
  77 [[whats-new]]
  78 == What's new in LTTng{nbsp}{revision}?
  79
  80 LTTng{nbsp}{revision} bears the name _Nordicité_, the product of a
  81 collaboration between https://champlibre.co/[Champ Libre] and
  82 https://www.boreale.com/[Boréale]. This farmhouse IPA is brewed with
  83 https://en.wikipedia.org/wiki/Kveik[Kveik] yeast and Québec-grown
  84 barley, oats, and juniper branches. The result is a remarkable, fruity,
  85 hazy golden IPA that offers a balanced touch of resinous and woodsy
  86 bitterness.
  87
  88 New features and changes in LTTng{nbsp}{revision}:
  89
  90 General::
  91 +
  92 * The LTTng trigger API of <<liblttng-ctl-lttng,`liblttng-ctl`>> now
  93   offers the ``__event rule matches__'' condition (an <<event-rule,event
  94   rule>> matches an event) as well as the following new actions:
  95 +
  96 --
  97 * <<basic-tracing-session-control,Start or stop>> a recording session.
  98 * <<session-rotation,Archive the current trace chunk>> of a
  99   recording session (rotate).
 100 * <<taking-a-snapshot,Take a snapshot>> of a recording session.
 101 --
 102 +
 103 As a reminder, a <<trigger,trigger>> is a condition-actions pair. When
 104 the condition of a trigger is satisfied, LTTng attempts to execute its
 105 actions.
 106 +
 107 This feature is also available with the new man:lttng-add-trigger(1),
 108 man:lttng-remove-trigger(1), and man:lttng-list-triggers(1)
 109 <<lttng-cli,cmd:lttng>> commands.
 110 +
 111 Starting from LTTng{nbsp}{revision}, a trigger may have more than one
 112 action.
 113 +
 114 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
 115 trigger to a session daemon>>” to learn more.
 116
 117 * The LTTng <<lttng-ust,user space>> and <<lttng-modules,kernel>>
 118   tracers offer the new namespace context field `time_ns`, which is the
 119   inode number, in the proc file system, of the current clock namespace.
 120 +
 121 See man:lttng-add-context(1), man:lttng-ust(3), and
 122 man:time_namespaces(7).
 123
 124 * The link:/man[manual pages] of LTTng-tools now have a terminology and
 125   style which match the LTTng Documentation, many fixes, more internal
 126   and manual page links, clearer lists and procedures, superior
 127   consistency, and usage examples.
 128 +
 129 The new man:lttng-event-rule(7) manual page explains the new, common
 130 way to specify an event rule on the command line.
 131 +
 132 The new man:lttng-concepts(7) manual page explains the core concepts of
 133 LTTng. Its contents is essentially the ``<<core-concepts,Core
 134 concepts>>'' section of this documentation, but more adapted to the
 135 manual page style.
 136
 137 User space tracing::
 138 +
 139 [IMPORTANT]
 140 ====
 141 The major version part of the `liblttng-ust`
 142 https://en.wikipedia.org/wiki/Soname[soname] is bumped, which means you
 143 **must recompile** your instrumented applications/libraries and
 144 <<tracepoint-provider,tracepoint provider packages>> to use
 145 LTTng-UST{nbsp}{revision}.
 146
 147 This change became a necessity to clean up the library and for
 148 `liblttng-ust` to stop exporting private symbols.
 149
 150 Also, LTTng{nbsp}{revision} prepends the `lttng_ust_` and `LTTNG_UST_`
 151 prefix to all public macro/definition/function names to offer a
 152 consistent API namespace. The LTTng{nbsp}2.12 API is still available;
 153 see the ``Compatibility with previous APIs'' section of
 154 man:lttng-ust(3).
 155 ====
 156 +
 157 Other notable changes:
 158 +
 159 * The `liblttng-ust` C{nbsp}API offers the new man:lttng_ust_vtracef(3)
 160   and man:lttng_ust_vtracelog(3) macros which are to
 161   man:lttng_ust_tracef(3) and man:lttng_ust_tracelog(3) what
 162   man:vprintf(3) is to man:printf(3).
 163
 164 * LTTng-UST now only depends on https://liburcu.org/[`liburcu`] at build
 165   time, not at run time.
 166
 167 Kernel tracing::
 168 +
 169 * The preferred display base of event record integer fields which
 170   contain memory addresses is now hexadecimal instead of decimal.
 171
 172 * The `pid` field is removed from `lttng_statedump_file_descriptor`
 173   event records and the `file_table_address` field is added.
 174 +
 175 This new field is the address of the `files_struct` structure which
 176 contains the file descriptor.
 177 +
 178 See the
 179 ``https://github.com/lttng/lttng-modules/commit/e7a0ca7205fd4be7c829d171baa8823fe4784c90[statedump: introduce `file_table_address`]''
 180 patch to learn more.
 181
 182 * The `flags` field of `syscall_entry_clone` event records is now a
 183   structure containing two enumerations (exit signal and options).
 184 +
 185 This change makes the flag values more readable and meaningful.
 186 +
 187 See the
 188 ``https://github.com/lttng/lttng-modules/commit/d775625e2ba4825b73b5897e7701ad6e2bdba115[syscalls: Make `clone()`'s `flags` field a 2 enum struct]''
 189 patch to learn more.
 190
 191 * The memory footprint of the kernel tracer is improved: the latter only
 192   generates metadata for the specific system call recording event rules
 193   that you <<enabling-disabling-events,create>>.
 194
 195
 196 [[nuts-and-bolts]]
 197 == Nuts and bolts
 198
 199 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
 200 generation_ is a modern toolkit for tracing Linux systems and
 201 applications. So your first question might be:
 202 **what is tracing?**
 203
 204
 205 [[what-is-tracing]]
 206 === What is tracing?
 207
 208 As the history of software engineering progressed and led to what
 209 we now take for granted--complex, numerous and
 210 interdependent software applications running in parallel on
 211 sophisticated operating systems like Linux--the authors of such
 212 components, software developers, began feeling a natural
 213 urge to have tools that would ensure the robustness and good performance
 214 of their masterpieces.
 215
 216 One major achievement in this field is, inarguably, the
 217 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
 218 an essential tool for developers to find and fix bugs. But even the best
 219 debugger won't help make your software run faster, and nowadays, faster
 220 software means either more work done by the same hardware, or cheaper
 221 hardware for the same work.
 222
 223 A _profiler_ is often the tool of choice to identify performance
 224 bottlenecks. Profiling is suitable to identify _where_ performance is
 225 lost in a given piece of software. The profiler outputs a profile, a
 226 statistical summary of observed events, which you may use to discover
 227 which functions took the most time to execute. However, a profiler won't
 228 report _why_ some identified functions are the bottleneck. Bottlenecks
 229 might only occur when specific conditions are met, conditions that are
 230 sometimes impossible to capture by a statistical profiler, or impossible
 231 to reproduce with an application altered by the overhead of an
 232 event-based profiler. For a thorough investigation of software
 233 performance issues, a history of execution is essential, with the
 234 recorded values of variables and context fields you choose, and with as
 235 little influence as possible on the instrumented application. This is
 236 where tracing comes in handy.
 237
 238 _Tracing_ is a technique used to understand what goes on in a running
 239 software system. The piece of software used for tracing is called a
 240 _tracer_, which is conceptually similar to a tape recorder. When
 241 recording, specific instrumentation points placed in the software source
 242 code generate events that are saved on a giant tape: a _trace_ file. You
 243 can record user application and operating system events at the same
 244 time, opening the possibility of resolving a wide range of problems that
 245 would otherwise be extremely challenging.
 246
 247 Tracing is often compared to _logging_. However, tracers and loggers are
 248 two different tools, serving two different purposes. Tracers are
 249 designed to record much lower-level events that occur much more
 250 frequently than log messages, often in the range of thousands per
 251 second, with very little execution overhead. Logging is more appropriate
 252 for a very high-level analysis of less frequent events: user accesses,
 253 exceptional conditions (errors and warnings, for example), database
 254 transactions, instant messaging communications, and such. Simply put,
 255 logging is one of the many use cases that can be satisfied with tracing.
 256
 257 The list of recorded events inside a trace file can be read manually
 258 like a log file for the maximum level of detail, but it's generally
 259 much more interesting to perform application-specific analyses to
 260 produce reduced statistics and graphs that are useful to resolve a
 261 given problem. Trace viewers and analyzers are specialized tools
 262 designed to do this.
 263
 264 In the end, this is what LTTng is: a powerful, open source set of
 265 tools to trace the Linux kernel and user applications at the same time.
 266 LTTng is composed of several components actively maintained and
 267 developed by its link:/community/#where[community].
 268
 269
 270 [[lttng-alternatives]]
 271 === Alternatives to noch:{LTTng}
 272
 273 Excluding proprietary solutions, a few competing software tracers
 274 exist for Linux:
 275
 276 https://github.com/dtrace4linux/linux[dtrace4linux]::
 277     A port of Sun Microsystems' DTrace to Linux.
 278 +
 279 The cmd:dtrace tool interprets user scripts and is responsible for
 280 loading code into the Linux kernel for further execution and collecting
 281 the outputted data.
 282
 283 https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
 284     A subsystem in the Linux kernel in which a virtual machine can
 285     execute programs passed from the user space to the kernel.
 286 +
 287 You can attach such programs to tracepoints and kprobes thanks to a
 288 system call, and they can output data to the user space when executed
 289 thanks to different mechanisms (pipe, VM register values, and eBPF maps,
 290 to name a few).
 291
 292 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
 293     The de facto function tracer of the Linux kernel.
 294 +
 295 Its user interface is a set of special files in sysfs.
 296
 297 https://perf.wiki.kernel.org/[perf]::
 298     A performance analysis tool for Linux which supports hardware
 299     performance counters, tracepoints, as well as other counters and
 300     types of probes.
 301 +
 302 The controlling utility of perf is the cmd:perf command line/text UI
 303 tool.
 304
 305 https://linux.die.net/man/1/strace[strace]::
 306     A command-line utility which records system calls made by a
 307     user process, as well as signal deliveries and changes of process
 308     state.
 309 +
 310 strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
 311 fulfill its function.
 312
 313 https://www.sysdig.org/[sysdig]::
 314     Like SystemTap, uses scripts to analyze Linux kernel events.
 315 +
 316 You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
 317 sysdig executes them while it traces the system or afterwards. The
 318 interface of sysdig is the cmd:sysdig command-line tool as well as the
 319 text UI-based cmd:csysdig tool.
 320
 321 https://sourceware.org/systemtap/[SystemTap]::
 322     A Linux kernel and user space tracer which uses custom user scripts
 323     to produce plain text traces.
 324 +
 325 SystemTap converts the scripts to the C language, and then compiles them
 326 as Linux kernel modules which are loaded to produce trace data. The
 327 primary user interface of SystemTap is the cmd:stap command-line tool.
 328
 329 The main distinctive features of LTTng is that it produces correlated
 330 kernel and user space traces, as well as doing so with the lowest
 331 overhead amongst other solutions. It produces trace files in the
 332 https://diamon.org/ctf[CTF] format, a file format optimized
 333 for the production and analyses of multi-gigabyte data.
 334
 335 LTTng is the result of more than 10{nbsp}years of active open source
 336 development by a community of passionate developers. LTTng is currently
 337 available on major desktop and server Linux distributions.
 338
 339 The main interface for tracing control is a single command-line tool
 340 named cmd:lttng. The latter can create several recording sessions, enable
 341 and disable recording event rules on the fly, filter events efficiently
 342 with custom user expressions, start and stop tracing, and much more.
 343 LTTng can write the traces on the file system or send them over the
 344 network, and keep them totally or partially. You can make LTTng execute
 345 user-defined actions when LTTng emits an event. You can view the traces
 346 once tracing becomes inactive or as LTTng records events.
 347
 348 <<installing-lttng,Install LTTng now>> and
 349 <<getting-started,start tracing>>!
 350
 351
 352 [[installing-lttng]]
 353 == Installation
 354
 355 **LTTng** is a set of software <<plumbing,components>> which interact to
 356 <<instrumenting,instrument>> the Linux kernel and user applications, and
 357 to <<controlling-tracing,control tracing>> (start and stop
 358 recording, create recording event rules, and the rest). Those
 359 components are bundled into the following packages:
 360
 361 LTTng-tools::
 362     Libraries and command-line interface to control tracing.
 363
 364 LTTng-modules::
 365     Linux kernel modules to instrument and trace the kernel.
 366
 367 LTTng-UST::
 368     Libraries and Java/Python packages to instrument and trace user
 369     applications.
 370
 371 Most distributions mark the LTTng-modules and LTTng-UST packages as
 372 optional when installing LTTng-tools (which is always required). In the
 373 following sections, we always provide the steps to install all three,
 374 but note that:
 375
 376 * You only need to install LTTng-modules if you intend to use
 377   the Linux kernel LTTng tracer.
 378
 379 * You only need to install LTTng-UST if you intend to use the user
 380   space LTTng tracer.
 381
 382 [role="growable"]
 383 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 17{nbsp}October{nbsp}2023.
 384
 385 |===
 386 |Distribution |Available in releases
 387
 388 |https://www.ubuntu.com/[Ubuntu]
 389 |xref:ubuntu[Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_].
 390
 391 Ubuntu{nbsp}18.04 LTS _Bionic Beaver_ and Ubuntu{nbsp}20.04 LTS _Focal Fossa_:
 392 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 393
 394 |https://www.debian.org/[Debian]
 395 |<<debian,Debian{nbsp}12 _bookworm_>>.
 396
 397 |https://getfedora.org/[Fedora]
 398 |xref:fedora[Fedora{nbsp}37, Fedora{nbsp}38, and Fedora{nbsp}39].
 399
 400 |https://www.archlinux.org/[Arch Linux]
 401 |<<arch-linux,_extra_ repository and AUR>>.
 402
 403 |https://alpinelinux.org/[Alpine Linux]
 404 |xref:alpine-linux[Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, and Alpine Linux{nbsp}3.18].
 405
 406 |https://buildroot.org/[Buildroot]
 407 |xref:buildroot[Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 408 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 409 Buildroot{nbsp}2023.05, and Buildroot{nbsp}2023.08].
 410
 411 |https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
 412 https://www.yoctoproject.org/[Yocto]
 413 |xref:oe-yocto[Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 414 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 415 Yocto Project{nbsp}4.3 _Nanbield_].
 416
 417 |====
 418
 419 [NOTE]
 420 ====
 421 For https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
 422 packages, see https://packages.efficios.com/[EfficiOS Enterprise
 423 Packages].
 424
 425 For other distributions, <<building-from-source,build LTTng from
 426 source>>.
 427 ====
 428
 429 [[ubuntu]]
 430 === [[ubuntu-official-repository]]Ubuntu
 431
 432 LTTng{nbsp}{revision} is available on Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_. For previous supported releases of Ubuntu, <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 433
 434 To install LTTng{nbsp}{revision} on Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_:
 435
 436 . Install the main LTTng{nbsp}{revision} packages:
 437 +
 438 --
 439 [role="term"]
 440 ----
 441 # apt-get install lttng-tools
 442 # apt-get install lttng-modules-dkms
 443 # apt-get install liblttng-ust-dev
 444 ----
 445 --
 446
 447 . **If you need to instrument and trace <<java-application,Java applications>>**,
 448 install the LTTng-UST Java agent:
 449 +
 450 --
 451 [role="term"]
 452 ----
 453 # apt-get install liblttng-ust-agent-java
 454 ----
 455 --
 456
 457 . **If you need to instrument and trace <<python-application,Python{nbsp}3
 458 applications>>**, install the LTTng-UST Python agent:
 459 +
 460 --
 461 [role="term"]
 462 ----
 463 # apt-get install python3-lttngust
 464 ----
 465 --
 466
 467 [[ubuntu-ppa]]
 468 === Ubuntu: noch:{LTTng} Stable {revision} PPA
 469
 470 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
 471 Stable{nbsp}{revision} PPA] offers the latest stable LTTng{nbsp}{revision}
 472 packages for Ubuntu{nbsp}18.04 LTS _Bionic Beaver_, Ubuntu{nbsp}20.04 LTS _Focal Fossa_,
 473 and Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_.
 474
 475 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
 476 PPA:
 477
 478 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
 479   list of packages:
 480 +
 481 --
 482 [role="term",subs="attributes"]
 483 ----
 484 # apt-add-repository ppa:lttng/stable-{revision}
 485 # apt-get update
 486 ----
 487 --
 488
 489 . Install the main LTTng{nbsp}{revision} packages:
 490 +
 491 --
 492 [role="term"]
 493 ----
 494 # apt-get install lttng-tools
 495 # apt-get install lttng-modules-dkms
 496 # apt-get install liblttng-ust-dev
 497 ----
 498 --
 499
 500 . **If you need to instrument and trace
 501   <<java-application,Java applications>>**, install the LTTng-UST
 502   Java agent:
 503 +
 504 --
 505 [role="term"]
 506 ----
 507 # apt-get install liblttng-ust-agent-java
 508 ----
 509 --
 510
 511 . **If you need to instrument and trace
 512   <<python-application,Python{nbsp}3 applications>>**, install the
 513   LTTng-UST Python agent:
 514 +
 515 --
 516 [role="term"]
 517 ----
 518 # apt-get install python3-lttngust
 519 ----
 520 --
 521
 522 [[debian]]
 523 === Debian
 524
 525 To install LTTng{nbsp}{revision} on Debian{nbsp}12 _bookworm_:
 526
 527 . Install the main LTTng{nbsp}{revision} packages:
 528 +
 529 --
 530 [role="term"]
 531 ----
 532 # apt install lttng-modules-dkms
 533 # apt install liblttng-ust-dev
 534 # apt install lttng-tools
 535 ----
 536 --
 537
 538 . **If you need to instrument and trace <<java-application,Java
 539   applications>>**, install the LTTng-UST Java agent:
 540 +
 541 --
 542 [role="term"]
 543 ----
 544 # apt install liblttng-ust-agent-java
 545 ----
 546 --
 547
 548 . **If you need to instrument and trace <<python-application,Python
 549   applications>>**, install the LTTng-UST Python agent:
 550 +
 551 --
 552 [role="term"]
 553 ----
 554 # apt install python3-lttngust
 555 ----
 556 --
 557
 558 [[fedora]]
 559 === Fedora
 560
 561 To install LTTng{nbsp}{revision} on Fedora{nbsp}37, Fedora{nbsp}38, or
 562 Fedora{nbsp}39:
 563
 564 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
 565   packages:
 566 +
 567 --
 568 [role="term"]
 569 ----
 570 # yum install lttng-tools
 571 # yum install lttng-ust
 572 ----
 573 --
 574
 575 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 576 +
 577 --
 578 [role="term",subs="attributes,specialcharacters"]
 579 ----
 580 $ cd $(mktemp -d) &&
 581   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 582   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 583   cd lttng-modules-{revision}.* &&
 584   make &&
 585   sudo make modules_install &&
 586   sudo depmod -a
 587 ----
 588 --
 589
 590 [IMPORTANT]
 591 .Java and Python application instrumentation and tracing
 592 ====
 593 If you need to instrument and trace <<java-application,Java
 594 applications>> on Fedora, you need to build and install
 595 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
 596 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
 597 `--enable-java-agent-all` options to the `configure` script, depending
 598 on which Java logging framework you use.
 599
 600 If you need to instrument and trace <<python-application,Python
 601 applications>> on Fedora, you need to build and install
 602 LTTng-UST{nbsp}{revision} from source and pass the
 603 `--enable-python-agent` option to the `configure` script.
 604 ====
 605
 606
 607 [[arch-linux]]
 608 === Arch Linux
 609
 610 LTTng-UST{nbsp}{revision} is available in the _extra_
 611 repository of Arch Linux, while LTTng-tools{nbsp}{revision} and
 612 LTTng-modules{nbsp}{revision} are available in the
 613 https://aur.archlinux.org/[AUR].
 614
 615 To install LTTng{nbsp}{revision} on Arch Linux, using
 616 https://github.com/Jguer/yay[yay] for the AUR packages:
 617
 618 . Install the main LTTng{nbsp}{revision} packages:
 619 +
 620 --
 621 [role="term"]
 622 ----
 623 # pacman -Sy lttng-ust
 624 $ yay -Sy lttng-tools
 625 $ yay -Sy lttng-modules
 626 ----
 627 --
 628
 629 . **If you need to instrument and trace <<python-application,Python
 630   applications>>**, install the LTTng-UST Python agent:
 631 +
 632 --
 633 [role="term"]
 634 ----
 635 # pacman -Sy python-lttngust
 636 ----
 637 --
 638
 639
 640 [[alpine-linux]]
 641 === Alpine Linux
 642
 643 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
 644 Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, or Alpine Linux{nbsp}3.18:
 645
 646 . Add the LTTng packages:
 647 +
 648 --
 649 [role="term"]
 650 ----
 651 # apk add lttng-tools
 652 # apk add lttng-ust-dev
 653 ----
 654 --
 655
 656 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 657 +
 658 --
 659 [role="term",subs="attributes,specialcharacters"]
 660 ----
 661 $ cd $(mktemp -d) &&
 662   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 663   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 664   cd lttng-modules-{revision}.* &&
 665   make &&
 666   sudo make modules_install &&
 667   sudo depmod -a
 668 ----
 669 --
 670
 671
 672 [[buildroot]]
 673 === Buildroot
 674
 675 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 676 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 677 Buildroot{nbsp}2023.05, or Buildroot{nbsp}2023.08:
 678
 679 . Launch the Buildroot configuration tool:
 680 +
 681 --
 682 [role="term"]
 683 ----
 684 $ make menuconfig
 685 ----
 686 --
 687
 688 . In **Kernel**, check **Linux kernel**.
 689 . In **Toolchain**, check **Enable WCHAR support**.
 690 . In **Target packages**{nbsp}&#8594; **Debugging, profiling and benchmark**,
 691   check **lttng-modules** and **lttng-tools**.
 692 . In **Target packages**{nbsp}&#8594; **Libraries**{nbsp}&#8594;
 693   **Other**, check **lttng-libust**.
 694
 695
 696 [[oe-yocto]]
 697 === OpenEmbedded and Yocto
 698
 699 LTTng{nbsp}{revision} recipes are available in the
 700 https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
 701 layer for Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 702 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 703 Yocto Project{nbsp}4.3 _Nanbield_ under the following names:
 704
 705 * `lttng-tools`
 706 * `lttng-modules`
 707 * `lttng-ust`
 708
 709 With BitBake, the simplest way to include LTTng recipes in your target
 710 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
 711
 712 ----
 713 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
 714 ----
 715
 716 If you use Hob:
 717
 718 . Select a machine and an image recipe.
 719 . Click **Edit image recipe**.
 720 . Under the **All recipes** tab, search for **lttng**.
 721 . Check the desired LTTng recipes.
 722
 723
 724 [[building-from-source]]
 725 === Build from source
 726
 727 To build and install LTTng{nbsp}{revision} from source:
 728
 729 . Using the package manager of your distribution, or from source,
 730   install the following dependencies of LTTng-tools and LTTng-UST:
 731 +
 732 --
 733 * https://sourceforge.net/projects/libuuid/[libuuid]
 734 * https://directory.fsf.org/wiki/Popt[popt]
 735 * https://liburcu.org/[Userspace RCU]
 736 * http://www.xmlsoft.org/[libxml2]
 737 * **Optional**: https://github.com/numactl/numactl[numactl]
 738 --
 739
 740 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 741 +
 742 --
 743 [role="term"]
 744 ----
 745 $ cd $(mktemp -d) &&
 746   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
 747   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
 748   cd lttng-modules-2.13.* &&
 749   make &&
 750   sudo make modules_install &&
 751   sudo depmod -a
 752 ----
 753 --
 754
 755 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
 756 +
 757 --
 758 [role="term"]
 759 ----
 760 $ cd $(mktemp -d) &&
 761   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
 762   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
 763   cd lttng-ust-2.13.* &&
 764   ./configure &&
 765   make &&
 766   sudo make install &&
 767   sudo ldconfig
 768 ----
 769 --
 770 +
 771 Add `--disable-numa` to `./configure` if you don't have
 772 https://github.com/numactl/numactl[numactl].
 773 +
 774 --
 775 [IMPORTANT]
 776 .Java and Python application tracing
 777 ====
 778 If you need to instrument and have LTTng trace <<java-application,Java
 779 applications>>, pass the `--enable-java-agent-jul`,
 780 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
 781 `configure` script, depending on which Java logging framework you use.
 782
 783 If you need to instrument and have LTTng trace
 784 <<python-application,Python applications>>, pass the
 785 `--enable-python-agent` option to the `configure` script. You can set
 786 the env:PYTHON environment variable to the path to the Python interpreter
 787 for which to install the LTTng-UST Python agent package.
 788 ====
 789 --
 790 +
 791 --
 792 [NOTE]
 793 ====
 794 By default, LTTng-UST libraries are installed to
 795 dir:{/usr/local/lib}, which is the de facto directory in which to
 796 keep self-compiled and third-party libraries.
 797
 798 When <<building-tracepoint-providers-and-user-application,linking an
 799 instrumented user application with `liblttng-ust`>>:
 800
 801 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
 802   variable.
 803
 804 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
 805   man:gcc(1), man:g++(1), or man:clang(1).
 806 ====
 807 --
 808
 809 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
 810 +
 811 --
 812 [role="term"]
 813 ----
 814 $ cd $(mktemp -d) &&
 815   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
 816   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
 817   cd lttng-tools-2.13.* &&
 818   ./configure &&
 819   make &&
 820   sudo make install &&
 821   sudo ldconfig
 822 ----
 823 --
 824
 825 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
 826 previous steps automatically for a given version of LTTng and confine
 827 the installed files to a specific directory. This can be useful to try
 828 LTTng without installing it on your system.
 829
 830 [[linux-kernel-sig]]
 831 === Linux kernel module signature
 832
 833 Linux kernel modules require trusted signatures in order to be loaded
 834 when any of the following is true:
 835
 836 * The system boots with
 837   https://uefi.org/specs/UEFI/2.10/32_Secure_Boot_and_Driver_Signing.html#secure-boot-and-driver-signing[Secure Boot]
 838   enabled.
 839
 840 * The Linux kernel which boots is configured with
 841   `CONFIG_MODULE_SIG_FORCE`.
 842
 843 * The Linux kernel boots with a command line containing
 844   `module.sig_enforce=1`.
 845
 846 .`root` user running <<lttng-sessiond,`lttng-sessiond`>> which fails to load a required <<lttng-modules,kernel module>> due to the signature enforcement policies.
 847 ====
 848 [role="term"]
 849 ----
 850 # lttng-sessiond
 851 Warning: No tracing group detected
 852 modprobe: ERROR: could not insert 'lttng_ring_buffer_client_discard': Key was rejected by service
 853 Error: Unable to load required module lttng-ring-buffer-client-discard
 854 Warning: No kernel tracer available
 855 ----
 856 ====
 857
 858 There are several methods to enroll trusted keys for signing modules
 859 that are built from source. The precise details vary from one Linux
 860 version to another, and distributions may have their own mechanisms. For
 861 example, https://github.com/dell/dkms[DKMS] may autogenerate a key and
 862 sign modules, but the key isn't automatically enrolled.
 863
 864 See
 865 https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html[Kernel
 866 module signing facility] and the documentation of your distribution
 867 to learn more about signing Linux kernel modules.
 868
 869 [[getting-started]]
 870 == Quick start
 871
 872 This is a short guide to get started quickly with LTTng kernel and user
 873 space tracing.
 874
 875 Before you follow this guide, make sure to <<installing-lttng,install>>
 876 LTTng.
 877
 878 This tutorial walks you through the steps to:
 879
 880 . <<tracing-the-linux-kernel,Record Linux kernel events>>.
 881
 882 . <<tracing-your-own-user-application,Record the events of a user
 883   application>> written in C.
 884
 885 . <<viewing-and-analyzing-your-traces,View and analyze the
 886   recorded events>>.
 887
 888
 889 [[tracing-the-linux-kernel]]
 890 === Record Linux kernel events
 891
 892 NOTE: The following command lines start with the `#` prompt because you
 893 need root privileges to control the Linux kernel LTTng tracer. You can
 894 also control the kernel tracer as a regular user if your Unix user is a
 895 member of the <<tracing-group,tracing group>>.
 896
 897 . Create a <<tracing-session,recording session>> to write LTTng traces
 898   to dir:{/tmp/my-kernel-trace}:
 899 +
 900 --
 901 [role="term"]
 902 ----
 903 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
 904 ----
 905 --
 906
 907 . List the available kernel tracepoints and system calls:
 908 +
 909 --
 910 [role="term"]
 911 ----
 912 # lttng list --kernel
 913 # lttng list --kernel --syscall
 914 ----
 915 --
 916
 917 . Create <<event,recording event rules>> which match events having
 918   the desired names, for example the `sched_switch` and
 919   `sched_process_fork` tracepoints, and the man:open(2) and man:close(2)
 920   system calls:
 921 +
 922 --
 923 [role="term"]
 924 ----
 925 # lttng enable-event --kernel sched_switch,sched_process_fork
 926 # lttng enable-event --kernel --syscall open,close
 927 ----
 928 --
 929 +
 930 Create a recording event rule which matches _all_ the Linux kernel
 931 tracepoint events with the opt:lttng-enable-event(1):--all option
 932 (recording with such a recording event rule generates a lot of data):
 933 +
 934 --
 935 [role="term"]
 936 ----
 937 # lttng enable-event --kernel --all
 938 ----
 939 --
 940
 941 . <<basic-tracing-session-control,Start recording>>:
 942 +
 943 --
 944 [role="term"]
 945 ----
 946 # lttng start
 947 ----
 948 --
 949
 950 . Do some operation on your system for a few seconds. For example,
 951   load a website, or list the files of a directory.
 952
 953 . <<creating-destroying-tracing-sessions,Destroy>> the current
 954   recording session:
 955 +
 956 --
 957 [role="term"]
 958 ----
 959 # lttng destroy
 960 ----
 961 --
 962 +
 963 The man:lttng-destroy(1) command doesn't destroy the trace data; it
 964 only destroys the state of the recording session.
 965 +
 966 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
 967 implicitly (see ``<<basic-tracing-session-control,Start and stop a
 968 recording session>>''). You need to stop recording to make LTTng flush
 969 the remaining trace data and make the trace readable.
 970
 971 . For the sake of this example, make the recorded trace accessible to
 972   the non-root users:
 973 +
 974 --
 975 [role="term"]
 976 ----
 977 # chown -R $(whoami) /tmp/my-kernel-trace
 978 ----
 979 --
 980
 981 See ``<<viewing-and-analyzing-your-traces,View and analyze the
 982 recorded events>>'' to view the recorded events.
 983
 984
 985 [[tracing-your-own-user-application]]
 986 === Record user application events
 987
 988 This section walks you through a simple example to record the events of
 989 a _Hello world_ program written in{nbsp}C.
 990
 991 To create the traceable user application:
 992
 993 . Create the tracepoint provider header file, which defines the
 994   tracepoints and the events they can generate:
 995 +
 996 --
 997 [source,c]
 998 .path:{hello-tp.h}
 999 ----
1000 #undef LTTNG_UST_TRACEPOINT_PROVIDER
1001 #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
1002
1003 #undef LTTNG_UST_TRACEPOINT_INCLUDE
1004 #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
1005
1006 #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
1007 #define _HELLO_TP_H
1008
1009 #include <lttng/tracepoint.h>
1010
1011 LTTNG_UST_TRACEPOINT_EVENT(
1012     hello_world,
1013     my_first_tracepoint,
1014     LTTNG_UST_TP_ARGS(
1015         int, my_integer_arg,
1016         char *, my_string_arg
1017     ),
1018     LTTNG_UST_TP_FIELDS(
1019         lttng_ust_field_string(my_string_field, my_string_arg)
1020         lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
1021     )
1022 )
1023
1024 #endif /* _HELLO_TP_H */
1025
1026 #include <lttng/tracepoint-event.h>
1027 ----
1028 --
1029
1030 . Create the tracepoint provider package source file:
1031 +
1032 --
1033 [source,c]
1034 .path:{hello-tp.c}
1035 ----
1036 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
1037 #define LTTNG_UST_TRACEPOINT_DEFINE
1038
1039 #include "hello-tp.h"
1040 ----
1041 --
1042
1043 . Build the tracepoint provider package:
1044 +
1045 --
1046 [role="term"]
1047 ----
1048 $ gcc -c -I. hello-tp.c
1049 ----
1050 --
1051
1052 . Create the _Hello World_ application source file:
1053 +
1054 --
1055 [source,c]
1056 .path:{hello.c}
1057 ----
1058 #include <stdio.h>
1059 #include "hello-tp.h"
1060
1061 int main(int argc, char *argv[])
1062 {
1063     unsigned int i;
1064
1065     puts("Hello, World!\nPress Enter to continue...");
1066
1067     /*
1068      * The following getchar() call only exists for the purpose of this
1069      * demonstration, to pause the application in order for you to have
1070      * time to list its tracepoints. You don't need it otherwise.
1071      */
1072     getchar();
1073
1074     /*
1075      * An lttng_ust_tracepoint() call.
1076      *
1077      * Arguments, as defined in `hello-tp.h`:
1078      *
1079      * 1. Tracepoint provider name (required)
1080      * 2. Tracepoint name (required)
1081      * 3. `my_integer_arg` (first user-defined argument)
1082      * 4. `my_string_arg` (second user-defined argument)
1083      *
1084      * Notice the tracepoint provider and tracepoint names are
1085      * C identifiers, NOT strings: they're in fact parts of variables
1086      * that the macros in `hello-tp.h` create.
1087      */
1088     lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,
1089                          "hi there!");
1090
1091     for (i = 0; i < argc; i++) {
1092         lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1093                              i, argv[i]);
1094     }
1095
1096     puts("Quitting now!");
1097     lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1098                          i * i, "i^2");
1099     return 0;
1100 }
1101 ----
1102 --
1103
1104 . Build the application:
1105 +
1106 --
1107 [role="term"]
1108 ----
1109 $ gcc -c hello.c
1110 ----
1111 --
1112
1113 . Link the application with the tracepoint provider package,
1114   `liblttng-ust` and `libdl`:
1115 +
1116 --
1117 [role="term"]
1118 ----
1119 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1120 ----
1121 --
1122
1123 Here's the whole build process:
1124
1125 [role="img-100"]
1126 .Build steps of the user space tracing tutorial.
1127 image::ust-flow.png[]
1128
1129 To record the events of the user application:
1130
1131 . Run the application with a few arguments:
1132 +
1133 --
1134 [role="term"]
1135 ----
1136 $ ./hello world and beyond
1137 ----
1138 --
1139 +
1140 You see:
1141 +
1142 --
1143 ----
1144 Hello, World!
1145 Press Enter to continue...
1146 ----
1147 --
1148
1149 . Start an LTTng <<lttng-sessiond,session daemon>>:
1150 +
1151 --
1152 [role="term"]
1153 ----
1154 $ lttng-sessiond --daemonize
1155 ----
1156 --
1157 +
1158 NOTE: A session daemon might already be running, for example as a
1159 service that the service manager of your distribution started.
1160
1161 . List the available user space tracepoints:
1162 +
1163 --
1164 [role="term"]
1165 ----
1166 $ lttng list --userspace
1167 ----
1168 --
1169 +
1170 You see the `hello_world:my_first_tracepoint` tracepoint listed
1171 under the `./hello` process.
1172
1173 . Create a <<tracing-session,recording session>>:
1174 +
1175 --
1176 [role="term"]
1177 ----
1178 $ lttng create my-user-space-session
1179 ----
1180 --
1181
1182 . Create a <<event,recording event rule>> which matches user space
1183   tracepoint events named `hello_world:my_first_tracepoint`:
1184 +
1185 --
1186 [role="term"]
1187 ----
1188 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1189 ----
1190 --
1191
1192 . <<basic-tracing-session-control,Start recording>>:
1193 +
1194 --
1195 [role="term"]
1196 ----
1197 $ lttng start
1198 ----
1199 --
1200
1201 . Go back to the running `hello` application and press **Enter**.
1202 +
1203 The program executes all `lttng_ust_tracepoint()` instrumentation
1204 points, emitting events as the event rule you created in step{nbsp}5
1205 matches them, and
1206 exits.
1207
1208 . <<creating-destroying-tracing-sessions,Destroy>> the current
1209   recording session:
1210 +
1211 --
1212 [role="term"]
1213 ----
1214 $ lttng destroy
1215 ----
1216 --
1217 +
1218 The man:lttng-destroy(1) command doesn't destroy the trace data; it
1219 only destroys the state of the recording session.
1220 +
1221 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1222 implicitly (see ``<<basic-tracing-session-control,Start and stop a
1223 recording session>>''). You need to stop recording to make LTTng flush
1224 the remaining trace data and make the trace readable.
1225
1226 By default, LTTng saves the traces to the
1227 +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+ directory, where
1228 +__NAME__+ is the recording session name. The env:LTTNG_HOME environment
1229 variable defaults to `$HOME` if not set.
1230
1231
1232 [[viewing-and-analyzing-your-traces]]
1233 === View and analyze the recorded events
1234
1235 Once you have completed the <<tracing-the-linux-kernel,Record Linux
1236 kernel events>> and <<tracing-your-own-user-application,Record user
1237 application events>> tutorials, you can inspect the recorded events.
1238
1239 There are many tools you can use to read LTTng traces:
1240
1241 https://babeltrace.org/[Babeltrace{nbsp}2]::
1242     A rich, flexible trace manipulation toolkit which includes
1243     a versatile command-line interface
1244     (man:babeltrace2(1)),
1245     a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C{nbsp}library],
1246     and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1247     so that you can easily process or convert an LTTng trace with
1248     your own script.
1249 +
1250 The Babeltrace{nbsp}2 project ships with a plugin
1251 (man:babeltrace2-plugin-ctf(7)) which supports the format of the traces
1252 which LTTng produces, https://diamon.org/ctf/[CTF].
1253
1254 http://tracecompass.org/[Trace Compass]::
1255     A graphical user interface for viewing and analyzing any type of
1256     logs or traces, including those of LTTng.
1257
1258 https://github.com/lttng/lttng-analyses[LTTng analyses]::
1259     An experimental project which includes many high-level analyses of
1260     LTTng kernel traces, like scheduling statistics, interrupt
1261     frequency distribution, top CPU usage, and more.
1262
1263 NOTE: This section assumes that LTTng wrote the traces it recorded
1264 during the previous tutorials to their default location, in the
1265 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1266 environment variable defaults to `$HOME` if not set.
1267
1268
1269 [[viewing-and-analyzing-your-traces-bt]]
1270 ==== Use the cmd:babeltrace2 command-line tool
1271
1272 The simplest way to list all the recorded events of an LTTng trace is to
1273 pass its path to man:babeltrace2(1), without options:
1274
1275 [role="term"]
1276 ----
1277 $ babeltrace2 ~/lttng-traces/my-user-space-session*
1278 ----
1279
1280 The cmd:babeltrace2 command finds all traces recursively within the
1281 given path and prints all their events, sorting them chronologically.
1282
1283 Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1284 further filtering:
1285
1286 [role="term"]
1287 ----
1288 $ babeltrace2 /tmp/my-kernel-trace | grep _switch
1289 ----
1290
1291 Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1292 the recorded events:
1293
1294 [role="term"]
1295 ----
1296 $ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1297 ----
1298
1299
1300 [[viewing-and-analyzing-your-traces-bt-python]]
1301 ==== Use the Babeltrace{nbsp}2 Python bindings
1302
1303 The <<viewing-and-analyzing-your-traces-bt,text output of
1304 cmd:babeltrace2>> is useful to isolate event records by simple matching
1305 using man:grep(1) and similar utilities. However, more elaborate
1306 filters, such as keeping only event records with a field value falling
1307 within a specific range, are not trivial to write using a shell.
1308 Moreover, reductions and even the most basic computations involving
1309 multiple event records are virtually impossible to implement.
1310
1311 Fortunately, Babeltrace{nbsp}2 ships with
1312 https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1313 which make it easy to read the event records of an LTTng trace
1314 sequentially and compute the desired information.
1315
1316 The following script accepts an LTTng Linux kernel trace path as its
1317 first argument and prints the short names of the top five running
1318 processes on CPU{nbsp}0 during the whole trace:
1319
1320 [source,python]
1321 .path:{top5proc.py}
1322 ----
1323 import bt2
1324 import sys
1325 import collections
1326
1327
1328 def top5proc():
1329     # Get the trace path from the first command-line argument
1330     it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1331
1332     # This counter dictionary will hold execution times:
1333     #
1334     #     Task command name -> Total execution time (ns)
1335     exec_times = collections.Counter()
1336
1337     # This holds the last `sched_switch` timestamp
1338     last_ts = None
1339
1340     for msg in it:
1341         # We only care about event messages
1342         if type(msg) is not bt2._EventMessageConst:
1343             continue
1344
1345         # Event of the event message
1346         event = msg.event
1347
1348         # Keep only `sched_switch` events
1349         if event.cls.name != 'sched_switch':
1350             continue
1351
1352         # Keep only records of events which LTTng emitted from CPU 0
1353         if event.packet.context_field['cpu_id'] != 0:
1354             continue
1355
1356         # Event timestamp (ns)
1357         cur_ts = msg.default_clock_snapshot.ns_from_origin
1358
1359         if last_ts is None:
1360             # Start here
1361             last_ts = cur_ts
1362
1363         # (Short) name of the previous task command
1364         prev_comm = str(event.payload_field['prev_comm'])
1365
1366         # Initialize an entry in our dictionary if not done yet
1367         if prev_comm not in exec_times:
1368             exec_times[prev_comm] = 0
1369
1370         # Compute previous command execution time
1371         diff = cur_ts - last_ts
1372
1373         # Update execution time of this command
1374         exec_times[prev_comm] += diff
1375
1376         # Update last timestamp
1377         last_ts = cur_ts
1378
1379     # Print top 5
1380     for name, ns in exec_times.most_common(5):
1381         print('{:20}{} s'.format(name, ns / 1e9))
1382
1383
1384 if __name__ == '__main__':
1385     top5proc()
1386 ----
1387
1388 Run this script:
1389
1390 [role="term"]
1391 ----
1392 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1393 ----
1394
1395 Output example:
1396
1397 ----
1398 swapper/0           48.607245889 s
1399 chromium            7.192738188 s
1400 pavucontrol         0.709894415 s
1401 Compositor          0.660867933 s
1402 Xorg.bin            0.616753786 s
1403 ----
1404
1405 Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1406 since we weren't using the CPU that much when recording, its first
1407 position in the list makes sense.
1408
1409
1410 [[core-concepts]]
1411 == [[understanding-lttng]]Core concepts
1412
1413 From a user's perspective, the LTTng system is built on a few concepts,
1414 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1415 operates by sending commands to the <<lttng-sessiond,session daemon>>
1416 (through <<liblttng-ctl-lttng,`liblttng-ctl`>>).
1417
1418 Understanding how those objects relate to each other is key to master
1419 the toolkit.
1420
1421 The core concepts of LTTng are:
1422
1423 * <<"event-rule","Instrumentation point, event rule, and event">>
1424 * <<trigger,Trigger>>
1425 * <<tracing-session,Recording session>>
1426 * <<domain,Tracing domain>>
1427 * <<channel,Channel and ring buffer>>
1428 * <<event,Recording event rule and event record>>
1429
1430 NOTE: The man:lttng-concepts(7) manual page also documents the core
1431 concepts of LTTng, with more links to other LTTng-tools manual pages.
1432
1433
1434 [[event-rule]]
1435 === Instrumentation point, event rule, and event
1436
1437 An _instrumentation point_ is a point, within a piece of software,
1438 which, when executed, creates an LTTng _event_.
1439
1440 LTTng offers various <<instrumentation-point-types,types of
1441 instrumentation>>.
1442
1443 An _event rule_ is a set of conditions to match a set of events.
1444
1445 When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is
1446 said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies _all_ the
1447 conditions of{nbsp}__ER__. This concept is similar to a
1448 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
1449 which matches a set of strings.
1450
1451 When an event rule matches an event, LTTng _emits_ the event, therefore
1452 attempting to execute one or more actions.
1453
1454 [IMPORTANT]
1455 ====
1456 [[event-creation-emission-opti]]The event creation and emission
1457 processes are documentation concepts to help understand the journey from
1458 an instrumentation point to the execution of actions.
1459
1460 The actual creation of an event can be costly because LTTng needs to
1461 evaluate the arguments of the instrumentation point.
1462
1463 In practice, LTTng implements various optimizations for the Linux kernel
1464 and user space <<domain,tracing domains>> to avoid actually creating an
1465 event when the tracer knows, thanks to properties which are independent
1466 from the event payload and current context, that it would never emit
1467 such an event. Those properties are:
1468
1469 * The <<instrumentation-point-types,instrumentation point type>>.
1470
1471 * The instrumentation point name.
1472
1473 * The instrumentation point log level.
1474
1475 * For a <<event,recording event rule>>:
1476 ** The status of the rule itself.
1477 ** The status of the <<channel,channel>>.
1478 ** The activity of the <<tracing-session,recording session>>.
1479 ** Whether or not the process for which LTTng would create the event is
1480    <<pid-tracking,allowed to record events>>.
1481
1482 In other words: if, for a given instrumentation point{nbsp}__IP__, the
1483 LTTng tracer knows that it would never emit an event,
1484 executing{nbsp}__IP__ represents a simple boolean variable check and,
1485 for a Linux kernel recording event rule, a few process attribute checks.
1486 ====
1487
1488 As of LTTng{nbsp}{revision}, there are two places where you can find an
1489 event rule:
1490
1491 <<event,Recording event rule>>::
1492     A specific type of event rule of which the action is to record the
1493     matched event as an event record.
1494 +
1495 See ``<<enabling-disabling-events,Create and enable a recording event
1496 rule>>'' to learn more.
1497
1498 ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13)::
1499     When the event rule of the trigger condition matches an event, LTTng
1500     can execute user-defined actions such as sending an LTTng
1501     notification,
1502     <<basic-tracing-session-control,starting a recording session>>,
1503     and more.
1504 +
1505 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
1506 trigger to a session daemon>>” to learn more.
1507
1508 For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy _all_ the
1509 basic conditions of an event rule{nbsp}__ER__, that is:
1510
1511 * The instrumentation point from which LTTng
1512   creates{nbsp}__E__ has a specific
1513   <<instrumentation-point-types,type>>.
1514
1515 * A pattern matches the name of{nbsp}__E__ while another pattern
1516   doesn't.
1517
1518 * The log level of the instrumentation point from which LTTng
1519   creates{nbsp}__E__ is at least as severe as some value, or is exactly
1520   some value.
1521
1522 * The fields of the payload of{nbsp}__E__ and the current context fields
1523   satisfy a filter expression.
1524
1525 A <<event,recording event rule>> has additional, implicit conditions to
1526 satisfy.
1527
1528
1529 [[instrumentation-point-types]]
1530 ==== Instrumentation point types
1531
1532 As of LTTng{nbsp}{revision}, the available instrumentation point
1533 types are, depending on the <<domain,tracing domain>>:
1534
1535 Linux kernel::
1536     LTTng tracepoint:::
1537         A statically defined point in the source code of the kernel
1538         image or of a kernel module using the
1539         <<lttng-modules,LTTng-modules>> macros.
1540
1541     Linux kernel system call:::
1542         Entry, exit, or both of a Linux kernel system call.
1543
1544     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kprobe]:::
1545         A single probe dynamically placed in the compiled kernel code.
1546 +
1547 When you create such an instrumentation point, you set its memory
1548 address or symbol name.
1549
1550     Linux user space probe:::
1551         A single probe dynamically placed at the entry of a compiled
1552         user space application/library function through the kernel.
1553 +
1554 When you create such an instrumentation point, you set:
1555 +
1556 --
1557 With the ELF method::
1558     Its application/library path and its symbol name.
1559
1560 With the USDT method::
1561     Its application/library path, its provider name, and its probe name.
1562 +
1563 ``USDT'' stands for _SystemTap User-level Statically Defined Tracing_,
1564 a http://dtrace.org/blogs/about/[DTrace]-style marker.
1565 --
1566 +
1567 As of LTTng{nbsp}{revision}, LTTng only supports USDT probes which
1568 are _not_ reference-counted.
1569
1570     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kretprobe]:::
1571         Entry, exit, or both of a Linux kernel function.
1572 +
1573 When you create such an instrumentation point, you set the memory
1574 address or symbol name of its function.
1575
1576 User space::
1577     LTTng tracepoint:::
1578         A statically defined point in the source code of a C/$$C++$$
1579         application/library using the
1580         <<lttng-ust,LTTng-UST>> macros.
1581
1582 `java.util.logging`, Apache log4j, and Python::
1583     Java or Python logging statement:::
1584         A method call on a Java or Python logger attached to an
1585         LTTng-UST handler.
1586
1587 See ``<<list-instrumentation-points,List the available instrumentation
1588 points>>'' to learn how to list available Linux kernel, user space, and
1589 logging instrumentation points.
1590
1591
1592 [[trigger]]
1593 === Trigger
1594
1595 A _trigger_ associates a condition to one or more actions.
1596
1597 When the condition of a trigger is satisfied, LTTng attempts to execute
1598 its actions.
1599
1600 As of LTTng{nbsp}{revision}, the available trigger conditions and
1601 actions are:
1602
1603 Conditions::
1604 +
1605 * The consumed buffer size of a given <<tracing-session,recording
1606   session>> becomes greater than some value.
1607
1608 * The buffer usage of a given <<channel,channel>> becomes greater than
1609   some value.
1610
1611 * The buffer usage of a given channel becomes less than some value.
1612
1613 * There's an ongoing <<session-rotation,recording session rotation>>.
1614
1615 * A recording session rotation becomes completed.
1616
1617 * An <<add-event-rule-matches-trigger,event rule matches>> an event.
1618
1619 Actions::
1620 +
1621 * <<trigger-event-notif,Send a notification>> to a user application.
1622 * <<basic-tracing-session-control,Start>> a given recording session.
1623 * <<basic-tracing-session-control,Stop>> a given recording session.
1624 * <<session-rotation,Archive the current trace chunk>> of a given
1625   recording session (rotate).
1626 * <<taking-a-snapshot,Take a snapshot>> of a given recording session.
1627
1628 A trigger belongs to a <<lttng-sessiond,session daemon>>, not to a
1629 specific recording session. For a given session daemon, each Unix user has
1630 its own, private triggers. Note, however, that the `root` Unix user may,
1631 for the root session daemon:
1632
1633 * Add a trigger as another Unix user.
1634
1635 * List all the triggers, regardless of their owner.
1636
1637 * Remove a trigger which belongs to another Unix user.
1638
1639 For a given session daemon and Unix user, a trigger has a unique name.
1640
1641
1642 [[tracing-session]]
1643 === Recording session
1644
1645 A _recording session_ (named ``tracing session'' prior to
1646 LTTng{nbsp}2.13) is a stateful dialogue between you and a
1647 <<lttng-sessiond,session daemon>> for everything related to
1648 <<event,event recording>>.
1649
1650 Everything that you do when you control LTTng tracers to record events
1651 happens within a recording session. In particular, a recording session:
1652
1653 * Has its own name, unique for a given session daemon.
1654
1655 * Has its own set of trace files, if any.
1656
1657 * Has its own state of activity (started or stopped).
1658 +
1659 An active recording session is an implicit <<event,recording event rule>>
1660 condition.
1661
1662 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1663   snapshot, or live).
1664
1665 * Has its own <<channel,channels>> to which are attached their own
1666   recording event rules.
1667
1668 * Has its own <<pid-tracking,process attribute inclusion sets>>.
1669
1670 [role="img-100"]
1671 .A _recording session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,recording event rules>>.
1672 image::concepts.png[]
1673
1674 Those attributes and objects are completely isolated between different
1675 recording sessions.
1676
1677 A recording session is like an
1678 https://en.wikipedia.org/wiki/Automated_teller_machine[ATM] session: the
1679 operations you do on the banking system through the ATM don't alter the
1680 data of other users of the same system. In the case of the ATM, a
1681 session lasts as long as your bank card is inside. In the case of LTTng,
1682 a recording session lasts from the man:lttng-create(1) command to the
1683 man:lttng-destroy(1) command.
1684
1685 [role="img-100"]
1686 .Each Unix user has its own set of recording sessions.
1687 image::many-sessions.png[]
1688
1689 A recording session belongs to a <<lttng-sessiond,session daemon>>. For a
1690 given session daemon, each Unix user has its own, private recording
1691 sessions. Note, however, that the `root` Unix user may operate on or
1692 destroy another user's recording session.
1693
1694
1695 [[tracing-session-mode]]
1696 ==== Recording session mode
1697
1698 LTTng offers four recording session modes:
1699
1700 [[local-mode]]Local mode::
1701     Write the trace data to the local file system.
1702
1703 [[net-streaming-mode]]Network streaming mode::
1704     Send the trace data over the network to a listening
1705     <<lttng-relayd,relay daemon>>.
1706
1707 [[snapshot-mode]]Snapshot mode::
1708     Only write the trace data to the local file system or send it to a
1709     listening relay daemon when LTTng <<taking-a-snapshot,takes a
1710     snapshot>>.
1711 +
1712 LTTng forces all the <<channel,channels>>
1713 to be created to be configured to be snapshot-ready.
1714 +
1715 LTTng takes a snapshot of such a recording session when:
1716 +
1717 --
1718 * You run the man:lttng-snapshot(1) command.
1719
1720 * LTTng executes a `snapshot-session` <<trigger,trigger>> action.
1721 --
1722
1723 [[live-mode]]Live mode::
1724     Send the trace data over the network to a listening relay daemon
1725     for <<lttng-live,live reading>>.
1726 +
1727 An LTTng live reader (for example, man:babeltrace2(1)) can connect to
1728 the same relay daemon to receive trace data while the recording session is
1729 active.
1730
1731
1732 [[domain]]
1733 === Tracing domain
1734
1735 A _tracing domain_ identifies a type of LTTng tracer.
1736
1737 A tracing domain has its own properties and features.
1738
1739 There are currently five available tracing domains:
1740
1741 * Linux kernel
1742 * User space
1743 * `java.util.logging` (JUL)
1744 * log4j
1745 * Python
1746
1747 You must specify a tracing domain to target a type of LTTng tracer when
1748 using some <<lttng-cli,cmd:lttng>> commands to avoid ambiguity. For
1749 example, because the Linux kernel and user space tracing domains support
1750 named tracepoints as <<event-rule,instrumentation points>>, you need to
1751 specify a tracing domain when you <<enabling-disabling-events,create
1752 an event rule>> because both tracing domains could have tracepoints
1753 sharing the same name.
1754
1755 You can create <<channel,channels>> in the Linux kernel and user space
1756 tracing domains. The other tracing domains have a single, default
1757 channel.
1758
1759
1760 [[channel]]
1761 === Channel and ring buffer
1762
1763 A _channel_ is an object which is responsible for a set of
1764 _ring buffers_.
1765
1766 Each ring buffer is divided into multiple _sub-buffers_. When a
1767 <<event,recording event rule>>
1768 matches an event, LTTng can record it to one or more sub-buffers of one
1769 or more channels.
1770
1771 When you <<enabling-disabling-channels,create a channel>>, you set its
1772 final attributes, that is:
1773
1774 * Its <<channel-buffering-schemes,buffering scheme>>.
1775
1776 * What to do <<channel-overwrite-mode-vs-discard-mode,when there's no
1777   space left>> for a new event record because all sub-buffers are full.
1778
1779 * The <<channel-subbuf-size-vs-subbuf-count,size of each ring buffer and
1780   how many sub-buffers>> a ring buffer has.
1781
1782 * The <<tracefile-rotation,size of each trace file LTTng writes for this
1783   channel and the maximum count>> of trace files.
1784
1785 * The periods of its <<channel-read-timer,read>>,
1786   <<channel-switch-timer,switch>>, and <<channel-monitor-timer,monitor>>
1787   timers.
1788
1789 * For a Linux kernel channel: its output type.
1790 +
1791 See the opt:lttng-enable-channel(1):--output option of the
1792 man:lttng-enable-channel(1) command.
1793
1794 * For a user space channel: the value of its
1795   <<blocking-timeout-example,blocking timeout>>.
1796
1797 A channel is always associated to a <<domain,tracing domain>>. The
1798 `java.util.logging` (JUL), log4j, and Python tracing domains each have a
1799 default channel which you can't configure.
1800
1801 A channel owns <<event,recording event rules>>.
1802
1803
1804 [[channel-buffering-schemes]]
1805 ==== Buffering scheme
1806
1807 A channel has at least one ring buffer _per CPU_. LTTng always records
1808 an event to the ring buffer dedicated to the CPU which emits it.
1809
1810 The buffering scheme of a user space channel determines what has its own
1811 set of per-CPU ring buffers:
1812
1813 Per-user buffering::
1814   Allocate one set of ring buffers--one per CPU--shared by all the
1815   instrumented processes of:
1816     If your Unix user is `root`:::
1817       Each Unix user.
1818 +
1819 --
1820 [role="img-100"]
1821 .Per-user buffering scheme (recording session belongs to the `root` Unix user).
1822 image::per-user-buffering-root.png[]
1823 --
1824
1825     Otherwise:::
1826       Your Unix user.
1827 +
1828 --
1829 [role="img-100"]
1830 .Per-user buffering scheme (recording session belongs to the `Bob` Unix user).
1831 image::per-user-buffering.png[]
1832 --
1833
1834 Per-process buffering::
1835   Allocate one set of ring buffers--one per CPU--for each
1836   instrumented process of:
1837     If your Unix user is `root`:::
1838       All Unix users.
1839 +
1840 --
1841 [role="img-100"]
1842 .Per-process buffering scheme (recording session belongs to the `root` Unix user).
1843 image::per-process-buffering-root.png[]
1844 --
1845
1846     Otherwise:::
1847       Your Unix user.
1848 +
1849 --
1850 [role="img-100"]
1851 .Per-process buffering scheme (recording session belongs to the `Bob` Unix user).
1852 image::per-process-buffering.png[]
1853 --
1854
1855 The per-process buffering scheme tends to consume more memory than the
1856 per-user option because systems generally have more instrumented
1857 processes than Unix users running instrumented processes. However, the
1858 per-process buffering scheme ensures that one process having a high
1859 event throughput won't fill all the shared sub-buffers of the same Unix
1860 user, only its own.
1861
1862 The buffering scheme of a Linux kernel channel is always to allocate a
1863 single set of ring buffers for the whole system. This scheme is similar
1864 to the per-user option, but with a single, global user ``running'' the
1865 kernel.
1866
1867
1868 [[channel-overwrite-mode-vs-discard-mode]]
1869 ==== Event record loss mode
1870
1871 When LTTng emits an event, LTTng can record it to a specific, available
1872 sub-buffer within the ring buffers of specific channels. When there's no
1873 space left in a sub-buffer, the tracer marks it as consumable and
1874 another, available sub-buffer starts receiving the following event
1875 records. An LTTng <<lttng-consumerd,consumer daemon>> eventually
1876 consumes the marked sub-buffer, which returns to the available state.
1877
1878 [NOTE]
1879 [role="docsvg-channel-subbuf-anim"]
1880 ====
1881 {note-no-anim}
1882 ====
1883
1884 In an ideal world, sub-buffers are consumed faster than they're filled.
1885 In the real world, however, all sub-buffers can be full at some point,
1886 leaving no space to record the following events.
1887
1888 In an ideal world, sub-buffers are consumed faster than they're filled,
1889 as it's the case in the previous animation. In the real world,
1890 however, all sub-buffers can be full at some point, leaving no space to
1891 record the following events.
1892
1893 By default, <<lttng-modules,LTTng-modules>> and <<lttng-ust,LTTng-UST>>
1894 are _non-blocking_ tracers: when there's no available sub-buffer to
1895 record an event, it's acceptable to lose event records when the
1896 alternative would be to cause substantial delays in the execution of the
1897 instrumented application. LTTng privileges performance over integrity;
1898 it aims at perturbing the instrumented application as little as possible
1899 in order to make the detection of subtle race conditions and rare
1900 interrupt cascades possible.
1901
1902 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1903 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1904 example>> to learn how to use the blocking mode.
1905
1906 When it comes to losing event records because there's no available
1907 sub-buffer, or because the blocking timeout of
1908 the channel is reached, the _event record loss mode_ of the channel
1909 determines what to do. The available event record loss modes are:
1910
1911 [[discard-mode]]Discard mode::
1912     Drop the newest event records until a sub-buffer becomes available.
1913 +
1914 This is the only available mode when you specify a blocking timeout.
1915 +
1916 With this mode, LTTng increments a count of lost event records when an
1917 event record is lost and saves this count to the trace. A trace reader
1918 can use the saved discarded event record count of the trace to decide
1919 whether or not to perform some analysis even if trace data is known to
1920 be missing.
1921
1922 [[overwrite-mode]]Overwrite mode::
1923     Clear the sub-buffer containing the oldest event records and start
1924     writing the newest event records there.
1925 +
1926 This mode is sometimes called _flight recorder mode_ because it's
1927 similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight
1928 recorder]: always keep a fixed amount of the latest data. It's also
1929 similar to the roll mode of an oscilloscope.
1930 +
1931 Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer
1932 its sequence number within its data stream. With a <<local-mode,local>>,
1933 <<net-streaming-mode,network streaming>>, or <<live-mode,live>> recording
1934 session, a trace reader can use such sequence numbers to report lost
1935 packets. A trace reader can use the saved discarded sub-buffer (packet)
1936 count of the trace to decide whether or not to perform some analysis
1937 even if trace data is known to be missing.
1938 +
1939 With this mode, LTTng doesn't write to the trace the exact number of
1940 lost event records in the lost sub-buffers.
1941
1942 Which mechanism you should choose depends on your context: prioritize
1943 the newest or the oldest event records in the ring buffer?
1944
1945 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1946 as soon as a there's no space left for a new event record, whereas in
1947 discard mode, the tracer only discards the event record that doesn't
1948 fit.
1949
1950 There are a few ways to decrease your probability of losing event
1951 records. The ``<<channel-subbuf-size-vs-subbuf-count,Sub-buffer size and
1952 count>>'' section shows how to fine-tune the sub-buffer size and count
1953 of a channel to virtually stop losing event records, though at the cost
1954 of greater memory usage.
1955
1956
1957 [[channel-subbuf-size-vs-subbuf-count]]
1958 ==== Sub-buffer size and count
1959
1960 A channel has one or more ring buffer for each CPU of the target system.
1961
1962 See the ``<<channel-buffering-schemes,Buffering scheme>>'' section to
1963 learn how many ring buffers of a given channel are dedicated to each CPU
1964 depending on its buffering scheme.
1965
1966 Set the size of each sub-buffer the ring buffers of a channel contain
1967 and how many there are
1968 when you <<enabling-disabling-channels,create it>>.
1969
1970 Note that LTTng switching the current sub-buffer of a ring buffer
1971 (marking a full one as consumable and switching to an available one for
1972 LTTng to record the next events) introduces noticeable CPU overhead.
1973 Knowing this, the following list presents a few practical situations
1974 along with how to configure the sub-buffer size and count for them:
1975
1976 High event throughput::
1977     In general, prefer large sub-buffers to lower the risk of losing
1978     event records.
1979 +
1980 Having larger sub-buffers also ensures a lower sub-buffer switching
1981 frequency.
1982 +
1983 The sub-buffer count is only meaningful if you create the channel in
1984 <<overwrite-mode,overwrite mode>>: in this case, if LTTng overwrites a
1985 sub-buffer, then the other sub-buffers are left unaltered.
1986
1987 Low event throughput::
1988     In general, prefer smaller sub-buffers since the risk of losing
1989     event records is low.
1990 +
1991 Because LTTng emits events less frequently, the sub-buffer switching
1992 frequency should remain low and therefore the overhead of the tracer
1993 shouldn't be a problem.
1994
1995 Low memory system::
1996     If your target system has a low memory limit, prefer fewer first,
1997     then smaller sub-buffers.
1998 +
1999 Even if the system is limited in memory, you want to keep the
2000 sub-buffers as large as possible to avoid a high sub-buffer switching
2001 frequency.
2002
2003 Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format,
2004 which means event record data is very compact. For example, the average
2005 LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a
2006 sub-buffer size of 1{nbsp}MiB is considered large.
2007
2008 The previous scenarios highlight the major trade-off between a few large
2009 sub-buffers and more, smaller sub-buffers: sub-buffer switching
2010 frequency vs. how many event records are lost in overwrite mode.
2011 Assuming a constant event throughput and using the overwrite mode, the
2012 two following configurations have the same ring buffer total size:
2013
2014 [NOTE]
2015 [role="docsvg-channel-subbuf-size-vs-count-anim"]
2016 ====
2017 {note-no-anim}
2018 ====
2019
2020 Two sub-buffers of 4{nbsp}MiB each::
2021     Expect a very low sub-buffer switching frequency, but if LTTng
2022     ever needs to overwrite a sub-buffer, half of the event records so
2023     far (4{nbsp}MiB) are definitely lost.
2024
2025 Eight sub-buffers of 1{nbsp}MiB each::
2026     Expect four times the tracer overhead of the configuration above,
2027     but if LTTng needs to overwrite a sub-buffer, only the eighth of
2028     event records so far (1{nbsp}MiB) are definitely lost.
2029
2030 In <<discard-mode,discard mode>>, the sub-buffer count parameter is
2031 pointless: use two sub-buffers and set their size according to your
2032 requirements.
2033
2034
2035 [[tracefile-rotation]]
2036 ==== Maximum trace file size and count (trace file rotation)
2037
2038 By default, trace files can grow as large as needed.
2039
2040 Set the maximum size of each trace file that LTTng writes of a given
2041 channel when you <<enabling-disabling-channels,create it>>.
2042
2043 When the size of a trace file reaches the fixed maximum size of the
2044 channel, LTTng creates another file to contain the next event records.
2045 LTTng appends a file count to each trace file name in this case.
2046
2047 If you set the trace file size attribute when you create a channel, the
2048 maximum number of trace files that LTTng creates is _unlimited_ by
2049 default. To limit them, set a maximum number of trace files. When the
2050 number of trace files reaches the fixed maximum count of the channel,
2051 LTTng overwrites the oldest trace file. This mechanism is called _trace
2052 file rotation_.
2053
2054 [IMPORTANT]
2055 ====
2056 Even if you don't limit the trace file count, always assume that LTTng
2057 manages all the trace files of the recording session.
2058
2059 In other words, there's no safe way to know if LTTng still holds a given
2060 trace file open with the trace file rotation feature.
2061
2062 The only way to obtain an unmanaged, self-contained LTTng trace before
2063 you <<creating-destroying-tracing-sessions,destroy the recording session>>
2064 is with the <<session-rotation,recording session rotation>> feature, which
2065 is available since LTTng{nbsp}2.11.
2066 ====
2067
2068
2069 [[channel-timers]]
2070 ==== Timers
2071
2072 Each channel can have up to three optional timers:
2073
2074 [[channel-switch-timer]]Switch timer::
2075     When this timer expires, a sub-buffer switch happens: for each ring
2076     buffer of the channel, LTTng marks the current sub-buffer as
2077     consumable and _switches_ to an available one to record the next
2078     events.
2079 +
2080 [NOTE]
2081 [role="docsvg-channel-switch-timer"]
2082 ====
2083 {note-no-anim}
2084 ====
2085 +
2086 A switch timer is useful to ensure that LTTng consumes and commits trace
2087 data to trace files or to a distant <<lttng-relayd,relay daemon>>
2088 periodically in case of a low event throughput.
2089 +
2090 Such a timer is also convenient when you use large
2091 <<channel-subbuf-size-vs-subbuf-count,sub-buffers>> to cope with a
2092 sporadic high event throughput, even if the throughput is otherwise low.
2093 +
2094 Set the period of the switch timer of a channel when you
2095 <<enabling-disabling-channels,create it>> with
2096 the opt:lttng-enable-channel(1):--switch-timer option.
2097
2098 [[channel-read-timer]]Read timer::
2099     When this timer expires, LTTng checks for full, consumable
2100     sub-buffers.
2101 +
2102 By default, the LTTng tracers use an asynchronous message mechanism to
2103 signal a full sub-buffer so that a <<lttng-consumerd,consumer daemon>>
2104 can consume it.
2105 +
2106 When such messages must be avoided, for example in real-time
2107 applications, use this timer instead.
2108 +
2109 Set the period of the read timer of a channel when you
2110 <<enabling-disabling-channels,create it>> with the
2111 opt:lttng-enable-channel(1):--read-timer option.
2112
2113 [[channel-monitor-timer]]Monitor timer::
2114     When this timer expires, the consumer daemon samples some channel
2115     statistics to evaluate the following <<trigger,trigger>>
2116     conditions:
2117 +
2118 --
2119 . The consumed buffer size of a given <<tracing-session,recording
2120   session>> becomes greater than some value.
2121 . The buffer usage of a given channel becomes greater than some value.
2122 . The buffer usage of a given channel becomes less than some value.
2123 --
2124 +
2125 If you disable the monitor timer of a channel{nbsp}__C__:
2126 +
2127 --
2128 * The consumed buffer size value of the recording session of{nbsp}__C__
2129   could be wrong for trigger condition type{nbsp}1: the consumed buffer
2130   size of{nbsp}__C__ won't be part of the grand total.
2131
2132 * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3)
2133   for{nbsp}__C__ will never be satisfied.
2134 --
2135 +
2136 Set the period of the monitor timer of a channel when you
2137 <<enabling-disabling-channels,create it>> with the
2138 opt:lttng-enable-channel(1):--monitor-timer option.
2139
2140
2141 [[event]]
2142 === Recording event rule and event record
2143
2144 A _recording event rule_ is a specific type of <<event-rule,event rule>>
2145 of which the action is to serialize and record the matched event as an
2146 _event record_.
2147
2148 Set the explicit conditions of a recording event rule when you
2149 <<enabling-disabling-events,create it>>. A recording event rule also has
2150 the following implicit conditions:
2151
2152 * The recording event rule itself is enabled.
2153 +
2154 A recording event rule is enabled on creation.
2155
2156 * The <<channel,channel>> to which the recording event rule is attached
2157   is enabled.
2158 +
2159 A channel is enabled on creation.
2160
2161 * The <<tracing-session,recording session>> of the recording event rule is
2162   <<basic-tracing-session-control,active>> (started).
2163 +
2164 A recording session is inactive (stopped) on creation.
2165
2166 * The process for which LTTng creates an event to match is
2167   <<pid-tracking,allowed to record events>>.
2168 +
2169 All processes are allowed to record events on recording session
2170 creation.
2171
2172 You always attach a recording event rule to a channel, which belongs to
2173 a recording session, when you create it.
2174
2175 When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__,
2176 LTTng attempts to serialize and record{nbsp}__E__ to one of the
2177 available sub-buffers of the channel to which{nbsp}__E__ is attached.
2178
2179 When multiple matching recording event rules are attached to the same
2180 channel, LTTng attempts to serialize and record the matched event
2181 _once_. In the following example, the second recording event rule is
2182 redundant when both are enabled:
2183
2184 [role="term"]
2185 ----
2186 $ lttng enable-event --userspace hello:world
2187 $ lttng enable-event --userspace hello:world --loglevel=INFO
2188 ----
2189
2190 [role="img-100"]
2191 .Logical path from an instrumentation point to an event record.
2192 image::event-rule.png[]
2193
2194 As of LTTng{nbsp}{revision}, you cannot remove a recording event
2195 rule: it exists as long as its recording session exists.
2196
2197
2198 [[plumbing]]
2199 == Components of noch:{LTTng}
2200
2201 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
2202 to call LTTng a simple _tool_ since it's composed of multiple
2203 interacting components.
2204
2205 This section describes those components, explains their respective
2206 roles, and shows how they connect together to form the LTTng ecosystem.
2207
2208 The following diagram shows how the most important components of LTTng
2209 interact with user applications, the Linux kernel, and you:
2210
2211 [role="img-100"]
2212 .Control and trace data paths between LTTng components.
2213 image::plumbing.png[]
2214
2215 The LTTng project integrates:
2216
2217 LTTng-tools::
2218     Libraries and command-line interface to control recording sessions:
2219 +
2220 * <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
2221 * <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
2222 * <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
2223 * <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
2224 * <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
2225 * <<persistent-memory-file-systems,`lttng-crash` command-line tool>>
2226   (man:lttng-crash(1)).
2227
2228 LTTng-UST::
2229     Libraries and Java/Python packages to instrument and trace user
2230     applications:
2231 +
2232 * <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
2233   headers to instrument and trace any native user application.
2234 * <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
2235 ** `liblttng-ust-libc-wrapper`
2236 ** `liblttng-ust-pthread-wrapper`
2237 ** `liblttng-ust-cyg-profile`
2238 ** `liblttng-ust-cyg-profile-fast`
2239 ** `liblttng-ust-dl`
2240 * <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
2241   Java applications using `java.util.logging` or
2242   Apache log4j{nbsp}1.2 logging.
2243 * <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
2244   Python applications using the standard `logging` package.
2245
2246 LTTng-modules::
2247     <<lttng-modules,Linux kernel modules>> to instrument and trace the
2248     kernel:
2249 +
2250 * LTTng kernel tracer module.
2251 * Recording ring buffer kernel modules.
2252 * Probe kernel modules.
2253 * LTTng logger kernel module.
2254
2255
2256 [[lttng-cli]]
2257 === Tracing control command-line interface
2258
2259 The _man:lttng(1) command-line tool_ is the standard user interface to
2260 control LTTng <<tracing-session,recording sessions>>.
2261
2262 The cmd:lttng tool is part of LTTng-tools.
2263
2264 The cmd:lttng tool is linked with
2265 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
2266 one or more <<lttng-sessiond,session daemons>> behind the scenes.
2267
2268 The cmd:lttng tool has a Git-like interface:
2269
2270 [role="term"]
2271 ----
2272 $ lttng [GENERAL OPTIONS] <COMMAND> [COMMAND OPTIONS]
2273 ----
2274
2275 The ``<<controlling-tracing,Tracing control>>'' section explores the
2276 available features of LTTng through its cmd:lttng tool.
2277
2278
2279 [[liblttng-ctl-lttng]]
2280 === Tracing control library
2281
2282 [role="img-100"]
2283 .The tracing control library.
2284 image::plumbing-liblttng-ctl.png[]
2285
2286 The _LTTng control library_, `liblttng-ctl`, is used to communicate with
2287 a <<lttng-sessiond,session daemon>> using a C{nbsp}API that hides the
2288 underlying details of the protocol.
2289
2290 `liblttng-ctl` is part of LTTng-tools.
2291
2292 The <<lttng-cli,cmd:lttng command-line tool>> is linked with
2293 `liblttng-ctl`.
2294
2295 Use `liblttng-ctl` in C or $$C++$$ source code by including its
2296 ``master'' header:
2297
2298 [source,c]
2299 ----
2300 #include <lttng/lttng.h>
2301 ----
2302
2303 As of LTTng{nbsp}{revision}, the best available developer documentation
2304 for `liblttng-ctl` is its installed header files. Functions and
2305 structures are documented with header comments.
2306
2307
2308 [[lttng-ust]]
2309 === User space tracing library
2310
2311 [role="img-100"]
2312 .The user space tracing library.
2313 image::plumbing-liblttng-ust.png[]
2314
2315 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
2316 is the LTTng user space tracer.
2317
2318 `liblttng-ust` receives commands from a <<lttng-sessiond,session
2319 daemon>>, for example to allow specific instrumentation points to emit
2320 LTTng <<event-rule,events>>, and writes event records to <<channel,ring
2321 buffers>> shared with a <<lttng-consumerd,consumer daemon>>.
2322
2323 `liblttng-ust` is part of LTTng-UST.
2324
2325 `liblttng-ust` can also send asynchronous messages to the session daemon
2326 when it emits an event. This supports the ``event rule matches''
2327 <<trigger,trigger>> condition feature (see
2328 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2329 to a session daemon>>”).
2330
2331 Public C{nbsp}header files are installed beside `liblttng-ust` to
2332 instrument any <<c-application,C or $$C++$$ application>>.
2333
2334 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
2335 packages, use their own <<tracepoint-provider,tracepoint provider
2336 package>> which is linked with `liblttng-ust`.
2337
2338 An application or library doesn't have to initialize `liblttng-ust`
2339 manually: its constructor does the necessary tasks to register the
2340 application to a session daemon. The initialization phase also
2341 configures instrumentation points depending on the <<event-rule,event
2342 rules>> that you already created.
2343
2344
2345 [[lttng-ust-agents]]
2346 === User space tracing agents
2347
2348 [role="img-100"]
2349 .The user space tracing agents.
2350 image::plumbing-lttng-ust-agents.png[]
2351
2352 The _LTTng-UST Java and Python agents_ are regular Java and Python
2353 packages which add LTTng tracing capabilities to the
2354 native logging frameworks.
2355
2356 The LTTng-UST agents are part of LTTng-UST.
2357
2358 In the case of Java, the
2359 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
2360 core logging facilities] and
2361 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
2362 Note that Apache Log4j{nbsp}2 isn't supported.
2363
2364 In the case of Python, the standard
2365 https://docs.python.org/3/library/logging.html[`logging`] package
2366 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
2367 LTTng-UST Python agent package.
2368
2369 The applications using the LTTng-UST agents are in the
2370 `java.util.logging` (JUL), log4j, and Python <<domain,tracing domains>>.
2371
2372 Both agents use the same mechanism to convert log statements to LTTng
2373 events. When an agent initializes, it creates a log handler that
2374 attaches to the root logger. The agent also registers to a
2375 <<lttng-sessiond,session daemon>>. When the user application executes a
2376 log statement, the root logger passes it to the log handler of the
2377 agent. The custom log handler of the agent calls a native function in a
2378 tracepoint provider package shared library linked with
2379 <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
2380 other fields, like its logger name and its log level. This native
2381 function contains a user space instrumentation point, therefore tracing
2382 the log statement.
2383
2384 The log level condition of a <<event,recording event rule>> is
2385 considered when tracing a Java or a Python application, and it's
2386 compatible with the standard `java.util.logging`, log4j, and Python log
2387 levels.
2388
2389
2390 [[lttng-modules]]
2391 === LTTng kernel modules
2392
2393 [role="img-100"]
2394 .The LTTng kernel modules.
2395 image::plumbing-lttng-modules.png[]
2396
2397 The _LTTng kernel modules_ are a set of Linux kernel modules
2398 which implement the kernel tracer of the LTTng project.
2399
2400 The LTTng kernel modules are part of LTTng-modules.
2401
2402 The LTTng kernel modules include:
2403
2404 * A set of _probe_ modules.
2405 +
2406 Each module attaches to a specific subsystem
2407 of the Linux kernel using its tracepoint instrument points.
2408 +
2409 There are also modules to attach to the entry and return points of the
2410 Linux system call functions.
2411
2412 * _Ring buffer_ modules.
2413 +
2414 A ring buffer implementation is provided as kernel modules. The LTTng
2415 kernel tracer writes to ring buffers; a
2416 <<lttng-consumerd,consumer daemon>> reads from ring buffers.
2417
2418 * The _LTTng kernel tracer_ module.
2419 * The <<proc-lttng-logger-abi,_LTTng logger_>> module.
2420 +
2421 The LTTng logger module implements the special path:{/proc/lttng-logger}
2422 (and path:{/dev/lttng-logger}, since LTTng{nbsp}2.11) files so that any
2423 executable can generate LTTng events by opening those files and
2424 writing to them.
2425
2426 The LTTng kernel tracer can also send asynchronous messages to the
2427 <<lttng-sessiond,session daemon>> when it emits an event.
2428 This supports the ``event rule matches''
2429 <<trigger,trigger>> condition feature (see
2430 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2431 to a session daemon>>”).
2432
2433 Generally, you don't have to load the LTTng kernel modules manually
2434 (using man:modprobe(8), for example): a root session daemon loads the
2435 necessary modules when starting. If you have extra probe modules, you
2436 can specify to load them to the session daemon on the command line
2437 (see the opt:lttng-sessiond(8):--extra-kmod-probes option). See also
2438 <<linux-kernel-sig,Linux kernel module signature>>.
2439
2440 The LTTng kernel modules are installed in
2441 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
2442 the kernel release (output of `uname --kernel-release`).
2443
2444
2445 [[lttng-sessiond]]
2446 === Session daemon
2447
2448 [role="img-100"]
2449 .The session daemon.
2450 image::plumbing-sessiond.png[]
2451
2452 The _session daemon_, man:lttng-sessiond(8), is a
2453 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which:
2454
2455 * Manages <<tracing-session,recording sessions>>.
2456
2457 * Controls the various components (like tracers and
2458   <<lttng-consumerd,consumer daemons>>) of LTTng.
2459
2460 * Sends <<notif-trigger-api,asynchronous notifications>> to user
2461   applications.
2462
2463 The session daemon is part of LTTng-tools.
2464
2465 The session daemon sends control requests to and receives control
2466 responses from:
2467
2468 * The <<lttng-ust,user space tracing library>>.
2469 +
2470 Any instance of the user space tracing library first registers to
2471 a session daemon. Then, the session daemon can send requests to
2472 this instance, such as:
2473 +
2474 --
2475 ** Get the list of tracepoints.
2476 ** Share a <<event,recording event rule>> so that the user space tracing
2477    library can decide whether or not a given tracepoint can emit events.
2478    Amongst the possible conditions of a recording event rule is a filter
2479    expression which `liblttng-ust` evaluates before it emits an event.
2480 ** Share <<channel,channel>> attributes and ring buffer locations.
2481 --
2482 +
2483 The session daemon and the user space tracing library use a Unix
2484 domain socket to communicate.
2485
2486 * The <<lttng-ust-agents,user space tracing agents>>.
2487 +
2488 Any instance of a user space tracing agent first registers to
2489 a session daemon. Then, the session daemon can send requests to
2490 this instance, such as:
2491 +
2492 --
2493 ** Get the list of loggers.
2494 ** Enable or disable a specific logger.
2495 --
2496 +
2497 The session daemon and the user space tracing agent use a TCP connection
2498 to communicate.
2499
2500 * The <<lttng-modules,LTTng kernel tracer>>.
2501 * The <<lttng-consumerd,consumer daemon>>.
2502 +
2503 The session daemon sends requests to the consumer daemon to instruct
2504 it where to send the trace data streams, amongst other information.
2505
2506 * The <<lttng-relayd,relay daemon>>.
2507
2508 The session daemon receives commands from the
2509 <<liblttng-ctl-lttng,tracing control library>>.
2510
2511 The session daemon can receive asynchronous messages from the
2512 <<lttng-ust,user space>> and <<lttng-modules,kernel>> tracers
2513 when they emit events. This supports the ``event rule matches''
2514 <<trigger,trigger>> condition feature (see
2515 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2516 to a session daemon>>”).
2517
2518 The root session daemon loads the appropriate
2519 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2520 one or more <<lttng-consumerd,consumer daemons>> as soon as you create
2521 a <<event,recording event rule>>.
2522
2523 The session daemon doesn't send and receive trace data: this is the
2524 role of the <<lttng-consumerd,consumer daemon>> and
2525 <<lttng-relayd,relay daemon>>. It does, however, generate the
2526 https://diamon.org/ctf/[CTF] metadata stream.
2527
2528 Each Unix user can have its own session daemon instance. The
2529 recording sessions which different session daemons manage are completely
2530 independent.
2531
2532 The root user's session daemon is the only one which is
2533 allowed to control the LTTng kernel tracer, and its spawned consumer
2534 daemon is the only one which is allowed to consume trace data from the
2535 LTTng kernel tracer. Note, however, that any Unix user which is a member
2536 of the <<tracing-group,tracing group>> is allowed
2537 to create <<channel,channels>> in the
2538 Linux kernel <<domain,tracing domain>>, and therefore to use the Linux
2539 kernel LTTng tracer.
2540
2541 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2542 session daemon when using its `create` command if none is currently
2543 running. You can also start the session daemon manually.
2544
2545
2546 [[lttng-consumerd]]
2547 === Consumer daemon
2548
2549 [role="img-100"]
2550 .The consumer daemon.
2551 image::plumbing-consumerd.png[]
2552
2553 The _consumer daemon_, cmd:lttng-consumerd, is a
2554 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which shares
2555 ring buffers with user applications or with the LTTng kernel modules to
2556 collect trace data and send it to some location (file system or to a
2557 <<lttng-relayd,relay daemon>> over the network).
2558
2559 The consumer daemon is part of LTTng-tools.
2560
2561 You don't start a consumer daemon manually: a consumer daemon is always
2562 spawned by a <<lttng-sessiond,session daemon>> as soon as you create a
2563 <<event,recording event rule>>, that is, before you start recording. When
2564 you kill its owner session daemon, the consumer daemon also exits
2565 because it's the child process of the session daemon. Command-line
2566 options of man:lttng-sessiond(8) target the consumer daemon process.
2567
2568 There are up to two running consumer daemons per Unix user, whereas only
2569 one session daemon can run per user. This is because each process can be
2570 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2571 and 64-bit processes, it's more efficient to have separate
2572 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2573 exception: it can have up to _three_ running consumer daemons: 32-bit
2574 and 64-bit instances for its user applications, and one more
2575 reserved for collecting kernel trace data.
2576
2577
2578 [[lttng-relayd]]
2579 === Relay daemon
2580
2581 [role="img-100"]
2582 .The relay daemon.
2583 image::plumbing-relayd.png[]
2584
2585 The _relay daemon_, man:lttng-relayd(8), is a
2586 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] acting as a bridge
2587 between remote session and consumer daemons, local trace files, and a
2588 remote live trace reader.
2589
2590 The relay daemon is part of LTTng-tools.
2591
2592 The main purpose of the relay daemon is to implement a receiver of
2593 <<sending-trace-data-over-the-network,trace data over the network>>.
2594 This is useful when the target system doesn't have much file system
2595 space to write trace files locally.
2596
2597 The relay daemon is also a server to which a
2598 <<lttng-live,live trace reader>> can
2599 connect. The live trace reader sends requests to the relay daemon to
2600 receive trace data as the target system records events. The
2601 communication protocol is named _LTTng live_; it's used over TCP
2602 connections.
2603
2604 Note that you can start the relay daemon on the target system directly.
2605 This is the setup of choice when the use case is to view/analyze events
2606 as the target system records them without the need of a remote system.
2607
2608
2609 [[instrumenting]]
2610 == [[using-lttng]]Instrumentation
2611
2612 There are many examples of tracing and monitoring in our everyday life:
2613
2614 * You have access to real-time and historical weather reports and
2615   forecasts thanks to weather stations installed around the country.
2616 * You know your heart is safe thanks to an electrocardiogram.
2617 * You make sure not to drive your car too fast and to have enough fuel
2618   to reach your destination thanks to gauges visible on your dashboard.
2619
2620 All the previous examples have something in common: they rely on
2621 **instruments**. Without the electrodes attached to the surface of your
2622 body skin, cardiac monitoring is futile.
2623
2624 LTTng, as a tracer, is no different from those real life examples. If
2625 you're about to trace a software system or, in other words, record its
2626 history of execution, you better have **instrumentation points** in the
2627 subject you're tracing, that is, the actual software system.
2628
2629 <<instrumentation-point-types,Various ways>> were developed to
2630 instrument a piece of software for LTTng tracing. The most
2631 straightforward one is to manually place static instrumentation points,
2632 called _tracepoints_, in the source code of the application. The Linux
2633 kernel <<domain,tracing domain>> also makes it possible to dynamically
2634 add instrumentation points.
2635
2636 If you're only interested in tracing the Linux kernel, your
2637 instrumentation needs are probably already covered by the built-in
2638 <<lttng-modules,Linux kernel instrumentation points>> of LTTng. You may
2639 also wish to have LTTng trace a user application which is already
2640 instrumented for LTTng tracing. In such cases, skip this whole section
2641 and read the topics of the ``<<controlling-tracing,Tracing control>>''
2642 section.
2643
2644 Many methods are available to instrument a piece of software for LTTng
2645 tracing:
2646
2647 * <<c-application,Instrument a C/$$C++$$ user application>>.
2648 * <<prebuilt-ust-helpers,Load a prebuilt user space tracing helper>>.
2649 * <<java-application,Instrument a Java application>>.
2650 * <<python-application,Instrument a Python application>>.
2651 * <<proc-lttng-logger-abi,Use the LTTng logger>>.
2652 * <<instrumenting-linux-kernel,Instrument a Linux kernel image or module>>.
2653
2654
2655 [[c-application]]
2656 === [[cxx-application]]Instrument a C/$$C++$$ user application
2657
2658 The high level procedure to instrument a C or $$C++$$ user application
2659 with the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`,
2660 is:
2661
2662 . <<tracepoint-provider,Create the source files of a tracepoint provider
2663   package>>.
2664
2665 . <<probing-the-application-source-code,Add tracepoints to
2666   the source code of the application>>.
2667
2668 . <<building-tracepoint-providers-and-user-application,Build and link
2669   a tracepoint provider package and the user application>>.
2670
2671 If you need quick, man:printf(3)-like instrumentation, skip those steps
2672 and use <<tracef,`lttng_ust_tracef()`>> or
2673 <<tracelog,`lttng_ust_tracelog()`>> instead.
2674
2675 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2676 instrument a user application with `liblttng-ust`.
2677
2678
2679 [[tracepoint-provider]]
2680 ==== Create the source files of a tracepoint provider package
2681
2682 A _tracepoint provider_ is a set of compiled functions which provide
2683 **tracepoints** to an application, the type of instrumentation point
2684 which LTTng-UST provides.
2685
2686 Those functions can make LTTng emit events with user-defined fields and
2687 serialize those events as event records to one or more LTTng-UST
2688 <<channel,channel>> sub-buffers. The `lttng_ust_tracepoint()` macro,
2689 which you <<probing-the-application-source-code,insert in the source
2690 code of a user application>>, calls those functions.
2691
2692 A _tracepoint provider package_ is an object file (`.o`) or a shared
2693 library (`.so`) which contains one or more tracepoint providers. Its
2694 source files are:
2695
2696 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2697 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2698
2699 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2700 the LTTng user space tracer, at run time.
2701
2702 [role="img-100"]
2703 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2704 image::ust-app.png[]
2705
2706 NOTE: If you need quick, man:printf(3)-like instrumentation, skip
2707 creating and using a tracepoint provider and use
2708 <<tracef,`lttng_ust_tracef()`>> or <<tracelog,`lttng_ust_tracelog()`>>
2709 instead.
2710
2711
2712 [[tpp-header]]
2713 ===== Create a tracepoint provider header file template
2714
2715 A _tracepoint provider header file_ contains the tracepoint definitions
2716 of a tracepoint provider.
2717
2718 To create a tracepoint provider header file:
2719
2720 . Start from this template:
2721 +
2722 --
2723 [source,c]
2724 .Tracepoint provider header file template (`.h` file extension).
2725 ----
2726 #undef LTTNG_UST_TRACEPOINT_PROVIDER
2727 #define LTTNG_UST_TRACEPOINT_PROVIDER provider_name
2728
2729 #undef LTTNG_UST_TRACEPOINT_INCLUDE
2730 #define LTTNG_UST_TRACEPOINT_INCLUDE "./tp.h"
2731
2732 #if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
2733 #define _TP_H
2734
2735 #include <lttng/tracepoint.h>
2736
2737 /*
2738  * Use LTTNG_UST_TRACEPOINT_EVENT(), LTTNG_UST_TRACEPOINT_EVENT_CLASS(),
2739  * LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(), and
2740  * LTTNG_UST_TRACEPOINT_LOGLEVEL() here.
2741  */
2742
2743 #endif /* _TP_H */
2744
2745 #include <lttng/tracepoint-event.h>
2746 ----
2747 --
2748 +
2749 Replace:
2750 +
2751 * +__provider_name__+ with the name of your tracepoint provider.
2752 * `"tp.h"` with the name of your tracepoint provider header file.
2753
2754 . Below the `#include <lttng/tracepoint.h>` line, put your
2755   <<defining-tracepoints,tracepoint definitions>>.
2756
2757 Your tracepoint provider name must be unique amongst all the possible
2758 tracepoint provider names used on the same target system. We suggest to
2759 include the name of your project or company in the name, for example,
2760 `org_lttng_my_project_tpp`.
2761
2762
2763 [[defining-tracepoints]]
2764 ===== Create a tracepoint definition
2765
2766 A _tracepoint definition_ defines, for a given tracepoint:
2767
2768 * Its **input arguments**.
2769 +
2770 They're the macro parameters that the `lttng_ust_tracepoint()` macro
2771 accepts for this particular tracepoint in the source code of the user
2772 application.
2773
2774 * Its **output event fields**.
2775 +
2776 They're the sources of event fields that form the payload of any event
2777 that the execution of the `lttng_ust_tracepoint()` macro emits for this
2778 particular tracepoint.
2779
2780 Create a tracepoint definition with the
2781 `LTTNG_UST_TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2782 line in the
2783 <<tpp-header,tracepoint provider header file template>>.
2784
2785 The syntax of the `LTTNG_UST_TRACEPOINT_EVENT()` macro is:
2786
2787 [source,c]
2788 .`LTTNG_UST_TRACEPOINT_EVENT()` macro syntax.
2789 ----
2790 LTTNG_UST_TRACEPOINT_EVENT(
2791     /* Tracepoint provider name */
2792     provider_name,
2793
2794     /* Tracepoint name */
2795     tracepoint_name,
2796
2797     /* Input arguments */
2798     LTTNG_UST_TP_ARGS(
2799         arguments
2800     ),
2801
2802     /* Output event fields */
2803     LTTNG_UST_TP_FIELDS(
2804         fields
2805     )
2806 )
2807 ----
2808
2809 Replace:
2810
2811 * +__provider_name__+ with your tracepoint provider name.
2812 * +__tracepoint_name__+ with your tracepoint name.
2813 * +__arguments__+ with the <<tpp-def-input-args,input arguments>>.
2814 * +__fields__+ with the <<tpp-def-output-fields,output event field>>
2815   definitions.
2816
2817 The full name of this tracepoint is `provider_name:tracepoint_name`.
2818
2819 [IMPORTANT]
2820 .Event name length limitation
2821 ====
2822 The concatenation of the tracepoint provider name and the tracepoint
2823 name must not exceed **254{nbsp}characters**. If it does, the
2824 instrumented application compiles and runs, but LTTng throws multiple
2825 warnings and you could experience serious issues.
2826 ====
2827
2828 [[tpp-def-input-args]]The syntax of the `LTTNG_UST_TP_ARGS()` macro is:
2829
2830 [source,c]
2831 .`LTTNG_UST_TP_ARGS()` macro syntax.
2832 ----
2833 LTTNG_UST_TP_ARGS(
2834     type, arg_name
2835 )
2836 ----
2837
2838 Replace:
2839
2840 * +__type__+ with the C{nbsp}type of the argument.
2841 * +__arg_name__+ with the argument name.
2842
2843 You can repeat +__type__+ and +__arg_name__+ up to 10{nbsp}times to have
2844 more than one argument.
2845
2846 .`LTTNG_UST_TP_ARGS()` usage with three arguments.
2847 ====
2848 [source,c]
2849 ----
2850 LTTNG_UST_TP_ARGS(
2851     int, count,
2852     float, ratio,
2853     const char*, query
2854 )
2855 ----
2856 ====
2857
2858 The `LTTNG_UST_TP_ARGS()` and `LTTNG_UST_TP_ARGS(void)` forms are valid
2859 to create a tracepoint definition with no input arguments.
2860
2861 [[tpp-def-output-fields]]The `LTTNG_UST_TP_FIELDS()` macro contains a
2862 list of `lttng_ust_field_*()` macros. Each `lttng_ust_field_*()` macro
2863 defines one event field. See man:lttng-ust(3) for a complete description
2864 of the available `lttng_ust_field_*()` macros. A `lttng_ust_field_*()`
2865 macro specifies the type, size, and byte order of one event field.
2866
2867 Each `lttng_ust_field_*()` macro takes an _argument expression_
2868 parameter. This is a C{nbsp}expression that the tracer evaluates at the
2869 `lttng_ust_tracepoint()` macro site in the source code of the
2870 application. This expression provides the source of data of a field. The
2871 argument expression can include input argument names listed in the
2872 `LTTNG_UST_TP_ARGS()` macro.
2873
2874 Each `lttng_ust_field_*()` macro also takes a _field name_ parameter.
2875 Field names must be unique within a given tracepoint definition.
2876
2877 Here's a complete tracepoint definition example:
2878
2879 .Tracepoint definition.
2880 ====
2881 The following tracepoint definition defines a tracepoint which takes
2882 three input arguments and has four output event fields.
2883
2884 [source,c]
2885 ----
2886 #include "my-custom-structure.h"
2887
2888 LTTNG_UST_TRACEPOINT_EVENT(
2889     my_provider,
2890     my_tracepoint,
2891     LTTNG_UST_TP_ARGS(
2892         const struct my_custom_structure *, my_custom_structure,
2893         float, ratio,
2894         const char *, query
2895     ),
2896     LTTNG_UST_TP_FIELDS(
2897         lttng_ust_field_string(query_field, query)
2898         lttng_ust_field_float(double, ratio_field, ratio)
2899         lttng_ust_field_integer(int, recv_size,
2900                                 my_custom_structure->recv_size)
2901         lttng_ust_field_integer(int, send_size,
2902                                 my_custom_structure->send_size)
2903     )
2904 )
2905 ----
2906
2907 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
2908 macro in the source code of your application like this:
2909
2910 [source,c]
2911 ----
2912 lttng_ust_tracepoint(my_provider, my_tracepoint,
2913                      my_structure, some_ratio, the_query);
2914 ----
2915 ====
2916
2917 NOTE: The LTTng-UST tracer only evaluates the arguments of a tracepoint
2918 at run time when such a tracepoint _could_ emit an event. See
2919 <<event-creation-emission-opti,this note>> to learn more.
2920
2921
2922 [[using-tracepoint-classes]]
2923 ===== Use a tracepoint class
2924
2925 A _tracepoint class_ is a class of tracepoints which share the same
2926 output event field definitions. A _tracepoint instance_ is one
2927 instance of such a defined tracepoint class, with its own tracepoint
2928 name.
2929
2930 The <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()` macro>> is
2931 actually a shorthand which defines both a tracepoint class and a
2932 tracepoint instance at the same time.
2933
2934 When you build a tracepoint provider package, the C or $$C++$$ compiler
2935 creates one serialization function for each **tracepoint class**. A
2936 serialization function is responsible for serializing the event fields
2937 of a tracepoint to a sub-buffer when recording.
2938
2939 For various performance reasons, when your situation requires multiple
2940 tracepoint definitions with different names, but with the same event
2941 fields, we recommend that you manually create a tracepoint class and
2942 instantiate as many tracepoint instances as needed. One positive effect
2943 of such a design, amongst other advantages, is that all tracepoint
2944 instances of the same tracepoint class reuse the same serialization
2945 function, thus reducing
2946 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2947
2948 .Use a tracepoint class and tracepoint instances.
2949 ====
2950 Consider the following three tracepoint definitions:
2951
2952 [source,c]
2953 ----
2954 LTTNG_UST_TRACEPOINT_EVENT(
2955     my_app,
2956     get_account,
2957     LTTNG_UST_TP_ARGS(
2958         int, userid,
2959         size_t, len
2960     ),
2961     LTTNG_UST_TP_FIELDS(
2962         lttng_ust_field_integer(int, userid, userid)
2963         lttng_ust_field_integer(size_t, len, len)
2964     )
2965 )
2966
2967 LTTNG_UST_TRACEPOINT_EVENT(
2968     my_app,
2969     get_settings,
2970     LTTNG_UST_TP_ARGS(
2971         int, userid,
2972         size_t, len
2973     ),
2974     LTTNG_UST_TP_FIELDS(
2975         lttng_ust_field_integer(int, userid, userid)
2976         lttng_ust_field_integer(size_t, len, len)
2977     )
2978 )
2979
2980 LTTNG_UST_TRACEPOINT_EVENT(
2981     my_app,
2982     get_transaction,
2983     LTTNG_UST_TP_ARGS(
2984         int, userid,
2985         size_t, len
2986     ),
2987     LTTNG_UST_TP_FIELDS(
2988         lttng_ust_field_integer(int, userid, userid)
2989         lttng_ust_field_integer(size_t, len, len)
2990     )
2991 )
2992 ----
2993
2994 In this case, we create three tracepoint classes, with one implicit
2995 tracepoint instance for each of them: `get_account`, `get_settings`, and
2996 `get_transaction`. However, they all share the same event field names
2997 and types. Hence three identical, yet independent serialization
2998 functions are created when you build the tracepoint provider package.
2999
3000 A better design choice is to define a single tracepoint class and three
3001 tracepoint instances:
3002
3003 [source,c]
3004 ----
3005 /* The tracepoint class */
3006 LTTNG_UST_TRACEPOINT_EVENT_CLASS(
3007     /* Tracepoint class provider name */
3008     my_app,
3009
3010     /* Tracepoint class name */
3011     my_class,
3012
3013     /* Input arguments */
3014     LTTNG_UST_TP_ARGS(
3015         int, userid,
3016         size_t, len
3017     ),
3018
3019     /* Output event fields */
3020     LTTNG_UST_TP_FIELDS(
3021         lttng_ust_field_integer(int, userid, userid)
3022         lttng_ust_field_integer(size_t, len, len)
3023     )
3024 )
3025
3026 /* The tracepoint instances */
3027 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3028     /* Tracepoint class provider name */
3029     my_app,
3030
3031     /* Tracepoint class name */
3032     my_class,
3033
3034     /* Instance provider name */
3035     my_app,
3036
3037     /* Tracepoint name */
3038     get_account,
3039
3040     /* Input arguments */
3041     LTTNG_UST_TP_ARGS(
3042         int, userid,
3043         size_t, len
3044     )
3045 )
3046 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3047     my_app,
3048     my_class,
3049     get_settings,
3050     LTTNG_UST_TP_ARGS(
3051         int, userid,
3052         size_t, len
3053     )
3054 )
3055 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3056     my_app,
3057     my_class,
3058     get_transaction,
3059     LTTNG_UST_TP_ARGS(
3060         int, userid,
3061         size_t, len
3062     )
3063 )
3064 ----
3065 ====
3066
3067 The tracepoint class and instance provider names must be the same if the
3068 `LTTNG_UST_TRACEPOINT_EVENT_CLASS()` and
3069 `LTTNG_UST_TRACEPOINT_EVENT_INSTANCE()` expansions are part of the same
3070 translation unit. See man:lttng-ust(3) to learn more.
3071
3072
3073 [[assigning-log-levels]]
3074 ===== Assign a log level to a tracepoint definition
3075
3076 Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
3077 with the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro.
3078
3079 Assigning different levels of severity to tracepoint definitions can be
3080 useful: when you <<enabling-disabling-events,create a recording event
3081 rule>>, you can target tracepoints having a log level at least as severe
3082 as a specific value.
3083
3084 The concept of LTTng-UST log levels is similar to the levels found
3085 in typical logging frameworks:
3086
3087 * In a logging framework, the log level is given by the function
3088   or method name you use at the log statement site: `debug()`,
3089   `info()`, `warn()`, `error()`, and so on.
3090
3091 * In LTTng-UST, you statically assign the log level to a tracepoint
3092   definition; any `lttng_ust_tracepoint()` macro invocation which refers
3093   to this definition has this log level.
3094
3095 You must use `LTTNG_UST_TRACEPOINT_LOGLEVEL()` _after_ the
3096 <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()`>> or
3097 <<using-tracepoint-classes,`LTTNG_UST_TRACEPOINT_INSTANCE()`>> macro for
3098 a given tracepoint.
3099
3100 The syntax of the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro is:
3101
3102 [source,c]
3103 .`LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro syntax.
3104 ----
3105 LTTNG_UST_TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
3106 ----
3107
3108 Replace:
3109
3110 * +__provider_name__+ with the tracepoint provider name.
3111 * +__tracepoint_name__+ with the tracepoint name.
3112 * +__log_level__+ with the log level to assign to the tracepoint
3113   definition named +__tracepoint_name__+ in the +__provider_name__+
3114   tracepoint provider.
3115 +
3116 See man:lttng-ust(3) for a list of available log level names.
3117
3118 .Assign the `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT` log level to a tracepoint definition.
3119 ====
3120 [source,c]
3121 ----
3122 /* Tracepoint definition */
3123 LTTNG_UST_TRACEPOINT_EVENT(
3124     my_app,
3125     get_transaction,
3126     LTTNG_UST_TP_ARGS(
3127         int, userid,
3128         size_t, len
3129     ),
3130     LTTNG_UST_TP_FIELDS(
3131         lttng_ust_field_integer(int, userid, userid)
3132         lttng_ust_field_integer(size_t, len, len)
3133     )
3134 )
3135
3136 /* Log level assignment */
3137 LTTNG_UST_TRACEPOINT_LOGLEVEL(my_app, get_transaction,
3138                               LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT)
3139 ----
3140 ====
3141
3142
3143 [[tpp-source]]
3144 ===== Create a tracepoint provider package source file
3145
3146 A _tracepoint provider package source file_ is a C source file which
3147 includes a <<tpp-header,tracepoint provider header file>> to expand its
3148 macros into event serialization and other functions.
3149
3150 Use the following tracepoint provider package source file template:
3151
3152 [source,c]
3153 .Tracepoint provider package source file template.
3154 ----
3155 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
3156
3157 #include "tp.h"
3158 ----
3159
3160 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
3161 header file>> name. You may also include more than one tracepoint
3162 provider header file here to create a tracepoint provider package
3163 holding more than one tracepoint providers.
3164
3165
3166 [[probing-the-application-source-code]]
3167 ==== Add tracepoints to the source code of an application
3168
3169 Once you <<tpp-header,create a tracepoint provider header file>>, use
3170 the `lttng_ust_tracepoint()` macro in the source code of your
3171 application to insert the tracepoints that this header
3172 <<defining-tracepoints,defines>>.
3173
3174 The `lttng_ust_tracepoint()` macro takes at least two parameters: the
3175 tracepoint provider name and the tracepoint name. The corresponding
3176 tracepoint definition defines the other parameters.
3177
3178 .`lttng_ust_tracepoint()` usage.
3179 ====
3180 The following <<defining-tracepoints,tracepoint definition>> defines a
3181 tracepoint which takes two input arguments and has two output event
3182 fields.
3183
3184 [source,c]
3185 .Tracepoint provider header file.
3186 ----
3187 #include "my-custom-structure.h"
3188
3189 LTTNG_UST_TRACEPOINT_EVENT(
3190     my_provider,
3191     my_tracepoint,
3192     LTTNG_UST_TP_ARGS(
3193         int, argc,
3194         const char *, cmd_name
3195     ),
3196     LTTNG_UST_TP_FIELDS(
3197         lttng_ust_field_string(cmd_name, cmd_name)
3198         lttng_ust_field_integer(int, number_of_args, argc)
3199     )
3200 )
3201 ----
3202
3203 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3204 macro in the source code of your application like this:
3205
3206 [source,c]
3207 .Application source file.
3208 ----
3209 #include "tp.h"
3210
3211 int main(int argc, char* argv[])
3212 {
3213     lttng_ust_tracepoint(my_provider, my_tracepoint, argc, argv[0]);
3214     return 0;
3215 }
3216 ----
3217
3218 Note how the source code of the application includes
3219 the tracepoint provider header file containing the tracepoint
3220 definitions to use, path:{tp.h}.
3221 ====
3222
3223 .`lttng_ust_tracepoint()` usage with a complex tracepoint definition.
3224 ====
3225 Consider this complex tracepoint definition, where multiple event
3226 fields refer to the same input arguments in their argument expression
3227 parameter:
3228
3229 [source,c]
3230 .Tracepoint provider header file.
3231 ----
3232 /* For `struct stat` */
3233 #include <sys/types.h>
3234 #include <sys/stat.h>
3235 #include <unistd.h>
3236
3237 LTTNG_UST_TRACEPOINT_EVENT(
3238     my_provider,
3239     my_tracepoint,
3240     LTTNG_UST_TP_ARGS(
3241         int, my_int_arg,
3242         char *, my_str_arg,
3243         struct stat *, st
3244     ),
3245     LTTNG_UST_TP_FIELDS(
3246         lttng_ust_field_integer(int, my_constant_field, 23 + 17)
3247         lttng_ust_field_integer(int, my_int_arg_field, my_int_arg)
3248         lttng_ust_field_integer(int, my_int_arg_field2,
3249                                 my_int_arg * my_int_arg)
3250         lttng_ust_field_integer(int, sum4_field,
3251                                 my_str_arg[0] + my_str_arg[1] +
3252                                 my_str_arg[2] + my_str_arg[3])
3253         lttng_ust_field_string(my_str_arg_field, my_str_arg)
3254         lttng_ust_field_integer_hex(off_t, size_field, st->st_size)
3255         lttng_ust_field_float(double, size_dbl_field, (double) st->st_size)
3256         lttng_ust_field_sequence_text(char, half_my_str_arg_field,
3257                                       my_str_arg, size_t,
3258                                       strlen(my_str_arg) / 2)
3259     )
3260 )
3261 ----
3262
3263 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3264 macro in the source code of your application like this:
3265
3266 [source,c]
3267 .Application source file.
3268 ----
3269 #define LTTNG_UST_TRACEPOINT_DEFINE
3270 #include "tp.h"
3271
3272 int main(void)
3273 {
3274     struct stat s;
3275
3276     stat("/etc/fstab", &s);
3277     lttng_ust_tracepoint(my_provider, my_tracepoint, 23,
3278                          "Hello, World!", &s);
3279
3280     return 0;
3281 }
3282 ----
3283
3284 If you look at the event record that LTTng writes when recording this
3285 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
3286 it should look like this:
3287
3288 .Event record fields
3289 |====
3290 |Field name              |Field value
3291 |`my_constant_field`     |40
3292 |`my_int_arg_field`      |23
3293 |`my_int_arg_field2`     |529
3294 |`sum4_field`            |389
3295 |`my_str_arg_field`      |`Hello, World!`
3296 |`size_field`            |0x12d
3297 |`size_dbl_field`        |301.0
3298 |`half_my_str_arg_field` |`Hello,`
3299 |====
3300 ====
3301
3302 Sometimes, the arguments you pass to `lttng_ust_tracepoint()` are
3303 expensive to evaluate--they use the call stack, for example. To avoid
3304 this computation when LTTng wouldn't emit any event anyway, use the
3305 `lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros.
3306
3307 The syntax of the `lttng_ust_tracepoint_enabled()` and
3308 `lttng_ust_do_tracepoint()` macros is:
3309
3310 [source,c]
3311 .`lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros syntax.
3312 ----
3313 lttng_ust_tracepoint_enabled(provider_name, tracepoint_name)
3314
3315 lttng_ust_do_tracepoint(provider_name, tracepoint_name, ...)
3316 ----
3317
3318 Replace:
3319
3320 * +__provider_name__+ with the tracepoint provider name.
3321 * +__tracepoint_name__+ with the tracepoint name.
3322
3323 `lttng_ust_tracepoint_enabled()` returns a non-zero value if executing
3324 the tracepoint named `tracepoint_name` from the provider named
3325 `provider_name` _could_ make LTTng emit an event, depending on the
3326 payload of said event.
3327
3328 `lttng_ust_do_tracepoint()` is like `lttng_ust_tracepoint()`, except
3329 that it doesn't check what `lttng_ust_tracepoint_enabled()` checks.
3330 Using `lttng_ust_tracepoint()` with `lttng_ust_tracepoint_enabled()` is
3331 dangerous because `lttng_ust_tracepoint()` also contains the
3332 `lttng_ust_tracepoint_enabled()` check; therefore, a race condition is
3333 possible in this situation:
3334
3335 [source,c]
3336 .Possible race condition when using `lttng_ust_tracepoint_enabled()` with `lttng_ust_tracepoint()`.
3337 ----
3338 if (lttng_ust_tracepoint_enabled(my_provider, my_tracepoint)) {
3339     stuff = prepare_stuff();
3340 }
3341
3342 lttng_ust_tracepoint(my_provider, my_tracepoint, stuff);
3343 ----
3344
3345 If `lttng_ust_tracepoint_enabled()` is false, but would be true after
3346 the conditional block, then `stuff` isn't prepared: the emitted event
3347 will either contain wrong data, or the whole application could crash
3348 (with a segmentation fault, for example).
3349
3350 NOTE: Neither `lttng_ust_tracepoint_enabled()` nor
3351 `lttng_ust_do_tracepoint()` have an `STAP_PROBEV()` call. If you need
3352 it, you must emit this call yourself.
3353
3354
3355 [[building-tracepoint-providers-and-user-application]]
3356 ==== Build and link a tracepoint provider package and an application
3357
3358 Once you have one or more <<tpp-header,tracepoint provider header
3359 files>> and a <<tpp-source,tracepoint provider package source file>>,
3360 create the tracepoint provider package by compiling its source
3361 file. From here, multiple build and run scenarios are possible. The
3362 following table shows common application and library configurations
3363 along with the required command lines to achieve them.
3364
3365 In the following diagrams, we use the following file names:
3366
3367 `app`::
3368   Executable application.
3369
3370 `app.o`::
3371   Application object file.
3372
3373 `tpp.o`::
3374   Tracepoint provider package object file.
3375
3376 `tpp.a`::
3377   Tracepoint provider package archive file.
3378
3379 `libtpp.so`::
3380   Tracepoint provider package shared object file.
3381
3382 `emon.o`::
3383   User library object file.
3384
3385 `libemon.so`::
3386   User library shared object file.
3387
3388 We use the following symbols in the diagrams of table below:
3389
3390 [role="img-100"]
3391 .Symbols used in the build scenario diagrams.
3392 image::ust-sit-symbols.png[]
3393
3394 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
3395 variable in the following instructions.
3396
3397 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
3398 .Common tracepoint provider package scenarios.
3399 |====
3400 |Scenario |Instructions
3401
3402 |
3403 The instrumented application is statically linked with
3404 the tracepoint provider package object.
3405
3406 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
3407
3408 |
3409 include::../common/ust-sit-step-tp-o.txt[]
3410
3411 To build the instrumented application:
3412
3413 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3414 +
3415 --
3416 [source,c]
3417 ----
3418 #define LTTNG_UST_TRACEPOINT_DEFINE
3419 ----
3420 --
3421
3422 . Compile the application source file:
3423 +
3424 --
3425 [role="term"]
3426 ----
3427 $ gcc -c app.c
3428 ----
3429 --
3430
3431 . Build the application:
3432 +
3433 --
3434 [role="term"]
3435 ----
3436 $ gcc -o app app.o tpp.o -llttng-ust -ldl
3437 ----
3438 --
3439
3440 To run the instrumented application:
3441
3442 * Start the application:
3443 +
3444 --
3445 [role="term"]
3446 ----
3447 $ ./app
3448 ----
3449 --
3450
3451 |
3452 The instrumented application is statically linked with the
3453 tracepoint provider package archive file.
3454
3455 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
3456
3457 |
3458 To create the tracepoint provider package archive file:
3459
3460 . Compile the <<tpp-source,tracepoint provider package source file>>:
3461 +
3462 --
3463 [role="term"]
3464 ----
3465 $ gcc -I. -c tpp.c
3466 ----
3467 --
3468
3469 . Create the tracepoint provider package archive file:
3470 +
3471 --
3472 [role="term"]
3473 ----
3474 $ ar rcs tpp.a tpp.o
3475 ----
3476 --
3477
3478 To build the instrumented application:
3479
3480 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3481 +
3482 --
3483 [source,c]
3484 ----
3485 #define LTTNG_UST_TRACEPOINT_DEFINE
3486 ----
3487 --
3488
3489 . Compile the application source file:
3490 +
3491 --
3492 [role="term"]
3493 ----
3494 $ gcc -c app.c
3495 ----
3496 --
3497
3498 . Build the application:
3499 +
3500 --
3501 [role="term"]
3502 ----
3503 $ gcc -o app app.o tpp.a -llttng-ust -ldl
3504 ----
3505 --
3506
3507 To run the instrumented application:
3508
3509 * Start the application:
3510 +
3511 --
3512 [role="term"]
3513 ----
3514 $ ./app
3515 ----
3516 --
3517
3518 |
3519 The instrumented application is linked with the tracepoint provider
3520 package shared object.
3521
3522 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
3523
3524 |
3525 include::../common/ust-sit-step-tp-so.txt[]
3526
3527 To build the instrumented application:
3528
3529 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3530 +
3531 --
3532 [source,c]
3533 ----
3534 #define LTTNG_UST_TRACEPOINT_DEFINE
3535 ----
3536 --
3537
3538 . Compile the application source file:
3539 +
3540 --
3541 [role="term"]
3542 ----
3543 $ gcc -c app.c
3544 ----
3545 --
3546
3547 . Build the application:
3548 +
3549 --
3550 [role="term"]
3551 ----
3552 $ gcc -o app app.o -ldl -L. -ltpp
3553 ----
3554 --
3555
3556 To run the instrumented application:
3557
3558 * Start the application:
3559 +
3560 --
3561 [role="term"]
3562 ----
3563 $ ./app
3564 ----
3565 --
3566
3567 |
3568 The tracepoint provider package shared object is preloaded before the
3569 instrumented application starts.
3570
3571 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3572
3573 |
3574 include::../common/ust-sit-step-tp-so.txt[]
3575
3576 To build the instrumented application:
3577
3578 . In path:{app.c}, before including path:{tpp.h}, add the
3579   following lines:
3580 +
3581 --
3582 [source,c]
3583 ----
3584 #define LTTNG_UST_TRACEPOINT_DEFINE
3585 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3586 ----
3587 --
3588
3589 . Compile the application source file:
3590 +
3591 --
3592 [role="term"]
3593 ----
3594 $ gcc -c app.c
3595 ----
3596 --
3597
3598 . Build the application:
3599 +
3600 --
3601 [role="term"]
3602 ----
3603 $ gcc -o app app.o -ldl
3604 ----
3605 --
3606
3607 To run the instrumented application with tracing support:
3608
3609 * Preload the tracepoint provider package shared object and
3610   start the application:
3611 +
3612 --
3613 [role="term"]
3614 ----
3615 $ LD_PRELOAD=./libtpp.so ./app
3616 ----
3617 --
3618
3619 To run the instrumented application without tracing support:
3620
3621 * Start the application:
3622 +
3623 --
3624 [role="term"]
3625 ----
3626 $ ./app
3627 ----
3628 --
3629
3630 |
3631 The instrumented application dynamically loads the tracepoint provider
3632 package shared object.
3633
3634 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3635
3636 |
3637 include::../common/ust-sit-step-tp-so.txt[]
3638
3639 To build the instrumented application:
3640
3641 . In path:{app.c}, before including path:{tpp.h}, add the
3642   following lines:
3643 +
3644 --
3645 [source,c]
3646 ----
3647 #define LTTNG_UST_TRACEPOINT_DEFINE
3648 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3649 ----
3650 --
3651
3652 . Compile the application source file:
3653 +
3654 --
3655 [role="term"]
3656 ----
3657 $ gcc -c app.c
3658 ----
3659 --
3660
3661 . Build the application:
3662 +
3663 --
3664 [role="term"]
3665 ----
3666 $ gcc -o app app.o -ldl
3667 ----
3668 --
3669
3670 To run the instrumented application:
3671
3672 * Start the application:
3673 +
3674 --
3675 [role="term"]
3676 ----
3677 $ ./app
3678 ----
3679 --
3680
3681 |
3682 The application is linked with the instrumented user library.
3683
3684 The instrumented user library is statically linked with the tracepoint
3685 provider package object file.
3686
3687 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3688
3689 |
3690 include::../common/ust-sit-step-tp-o-fpic.txt[]
3691
3692 To build the instrumented user library:
3693
3694 . In path:{emon.c}, before including path:{tpp.h}, add the
3695   following line:
3696 +
3697 --
3698 [source,c]
3699 ----
3700 #define LTTNG_UST_TRACEPOINT_DEFINE
3701 ----
3702 --
3703
3704 . Compile the user library source file:
3705 +
3706 --
3707 [role="term"]
3708 ----
3709 $ gcc -I. -fpic -c emon.c
3710 ----
3711 --
3712
3713 . Build the user library shared object:
3714 +
3715 --
3716 [role="term"]
3717 ----
3718 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3719 ----
3720 --
3721
3722 To build the application:
3723
3724 . Compile the application source file:
3725 +
3726 --
3727 [role="term"]
3728 ----
3729 $ gcc -c app.c
3730 ----
3731 --
3732
3733 . Build the application:
3734 +
3735 --
3736 [role="term"]
3737 ----
3738 $ gcc -o app app.o -L. -lemon
3739 ----
3740 --
3741
3742 To run the application:
3743
3744 * Start the application:
3745 +
3746 --
3747 [role="term"]
3748 ----
3749 $ ./app
3750 ----
3751 --
3752
3753 |
3754 The application is linked with the instrumented user library.
3755
3756 The instrumented user library is linked with the tracepoint provider
3757 package shared object.
3758
3759 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3760
3761 |
3762 include::../common/ust-sit-step-tp-so.txt[]
3763
3764 To build the instrumented user library:
3765
3766 . In path:{emon.c}, before including path:{tpp.h}, add the
3767   following line:
3768 +
3769 --
3770 [source,c]
3771 ----
3772 #define LTTNG_UST_TRACEPOINT_DEFINE
3773 ----
3774 --
3775
3776 . Compile the user library source file:
3777 +
3778 --
3779 [role="term"]
3780 ----
3781 $ gcc -I. -fpic -c emon.c
3782 ----
3783 --
3784
3785 . Build the user library shared object:
3786 +
3787 --
3788 [role="term"]
3789 ----
3790 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3791 ----
3792 --
3793
3794 To build the application:
3795
3796 . Compile the application source file:
3797 +
3798 --
3799 [role="term"]
3800 ----
3801 $ gcc -c app.c
3802 ----
3803 --
3804
3805 . Build the application:
3806 +
3807 --
3808 [role="term"]
3809 ----
3810 $ gcc -o app app.o -L. -lemon
3811 ----
3812 --
3813
3814 To run the application:
3815
3816 * Start the application:
3817 +
3818 --
3819 [role="term"]
3820 ----
3821 $ ./app
3822 ----
3823 --
3824
3825 |
3826 The tracepoint provider package shared object is preloaded before the
3827 application starts.
3828
3829 The application is linked with the instrumented user library.
3830
3831 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3832
3833 |
3834 include::../common/ust-sit-step-tp-so.txt[]
3835
3836 To build the instrumented user library:
3837
3838 . In path:{emon.c}, before including path:{tpp.h}, add the
3839   following lines:
3840 +
3841 --
3842 [source,c]
3843 ----
3844 #define LTTNG_UST_TRACEPOINT_DEFINE
3845 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3846 ----
3847 --
3848
3849 . Compile the user library source file:
3850 +
3851 --
3852 [role="term"]
3853 ----
3854 $ gcc -I. -fpic -c emon.c
3855 ----
3856 --
3857
3858 . Build the user library shared object:
3859 +
3860 --
3861 [role="term"]
3862 ----
3863 $ gcc -shared -o libemon.so emon.o -ldl
3864 ----
3865 --
3866
3867 To build the application:
3868
3869 . Compile the application source file:
3870 +
3871 --
3872 [role="term"]
3873 ----
3874 $ gcc -c app.c
3875 ----
3876 --
3877
3878 . Build the application:
3879 +
3880 --
3881 [role="term"]
3882 ----
3883 $ gcc -o app app.o -L. -lemon
3884 ----
3885 --
3886
3887 To run the application with tracing support:
3888
3889 * Preload the tracepoint provider package shared object and
3890   start the application:
3891 +
3892 --
3893 [role="term"]
3894 ----
3895 $ LD_PRELOAD=./libtpp.so ./app
3896 ----
3897 --
3898
3899 To run the application without tracing support:
3900
3901 * Start the application:
3902 +
3903 --
3904 [role="term"]
3905 ----
3906 $ ./app
3907 ----
3908 --
3909
3910 |
3911 The application is linked with the instrumented user library.
3912
3913 The instrumented user library dynamically loads the tracepoint provider
3914 package shared object.
3915
3916 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3917
3918 |
3919 include::../common/ust-sit-step-tp-so.txt[]
3920
3921 To build the instrumented user library:
3922
3923 . In path:{emon.c}, before including path:{tpp.h}, add the
3924   following lines:
3925 +
3926 --
3927 [source,c]
3928 ----
3929 #define LTTNG_UST_TRACEPOINT_DEFINE
3930 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3931 ----
3932 --
3933
3934 . Compile the user library source file:
3935 +
3936 --
3937 [role="term"]
3938 ----
3939 $ gcc -I. -fpic -c emon.c
3940 ----
3941 --
3942
3943 . Build the user library shared object:
3944 +
3945 --
3946 [role="term"]
3947 ----
3948 $ gcc -shared -o libemon.so emon.o -ldl
3949 ----
3950 --
3951
3952 To build the application:
3953
3954 . Compile the application source file:
3955 +
3956 --
3957 [role="term"]
3958 ----
3959 $ gcc -c app.c
3960 ----
3961 --
3962
3963 . Build the application:
3964 +
3965 --
3966 [role="term"]
3967 ----
3968 $ gcc -o app app.o -L. -lemon
3969 ----
3970 --
3971
3972 To run the application:
3973
3974 * Start the application:
3975 +
3976 --
3977 [role="term"]
3978 ----
3979 $ ./app
3980 ----
3981 --
3982
3983 |
3984 The application dynamically loads the instrumented user library.
3985
3986 The instrumented user library is linked with the tracepoint provider
3987 package shared object.
3988
3989 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3990
3991 |
3992 include::../common/ust-sit-step-tp-so.txt[]
3993
3994 To build the instrumented user library:
3995
3996 . In path:{emon.c}, before including path:{tpp.h}, add the
3997   following line:
3998 +
3999 --
4000 [source,c]
4001 ----
4002 #define LTTNG_UST_TRACEPOINT_DEFINE
4003 ----
4004 --
4005
4006 . Compile the user library source file:
4007 +
4008 --
4009 [role="term"]
4010 ----
4011 $ gcc -I. -fpic -c emon.c
4012 ----
4013 --
4014
4015 . Build the user library shared object:
4016 +
4017 --
4018 [role="term"]
4019 ----
4020 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
4021 ----
4022 --
4023
4024 To build the application:
4025
4026 . Compile the application source file:
4027 +
4028 --
4029 [role="term"]
4030 ----
4031 $ gcc -c app.c
4032 ----
4033 --
4034
4035 . Build the application:
4036 +
4037 --
4038 [role="term"]
4039 ----
4040 $ gcc -o app app.o -ldl -L. -lemon
4041 ----
4042 --
4043
4044 To run the application:
4045
4046 * Start the application:
4047 +
4048 --
4049 [role="term"]
4050 ----
4051 $ ./app
4052 ----
4053 --
4054
4055 |
4056 The application dynamically loads the instrumented user library.
4057
4058 The instrumented user library dynamically loads the tracepoint provider
4059 package shared object.
4060
4061 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
4062
4063 |
4064 include::../common/ust-sit-step-tp-so.txt[]
4065
4066 To build the instrumented user library:
4067
4068 . In path:{emon.c}, before including path:{tpp.h}, add the
4069   following lines:
4070 +
4071 --
4072 [source,c]
4073 ----
4074 #define LTTNG_UST_TRACEPOINT_DEFINE
4075 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4076 ----
4077 --
4078
4079 . Compile the user library source file:
4080 +
4081 --
4082 [role="term"]
4083 ----
4084 $ gcc -I. -fpic -c emon.c
4085 ----
4086 --
4087
4088 . Build the user library shared object:
4089 +
4090 --
4091 [role="term"]
4092 ----
4093 $ gcc -shared -o libemon.so emon.o -ldl
4094 ----
4095 --
4096
4097 To build the application:
4098
4099 . Compile the application source file:
4100 +
4101 --
4102 [role="term"]
4103 ----
4104 $ gcc -c app.c
4105 ----
4106 --
4107
4108 . Build the application:
4109 +
4110 --
4111 [role="term"]
4112 ----
4113 $ gcc -o app app.o -ldl -L. -lemon
4114 ----
4115 --
4116
4117 To run the application:
4118
4119 * Start the application:
4120 +
4121 --
4122 [role="term"]
4123 ----
4124 $ ./app
4125 ----
4126 --
4127
4128 |
4129 The tracepoint provider package shared object is preloaded before the
4130 application starts.
4131
4132 The application dynamically loads the instrumented user library.
4133
4134 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
4135
4136 |
4137 include::../common/ust-sit-step-tp-so.txt[]
4138
4139 To build the instrumented user library:
4140
4141 . In path:{emon.c}, before including path:{tpp.h}, add the
4142   following lines:
4143 +
4144 --
4145 [source,c]
4146 ----
4147 #define LTTNG_UST_TRACEPOINT_DEFINE
4148 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4149 ----
4150 --
4151
4152 . Compile the user library source file:
4153 +
4154 --
4155 [role="term"]
4156 ----
4157 $ gcc -I. -fpic -c emon.c
4158 ----
4159 --
4160
4161 . Build the user library shared object:
4162 +
4163 --
4164 [role="term"]
4165 ----
4166 $ gcc -shared -o libemon.so emon.o -ldl
4167 ----
4168 --
4169
4170 To build the application:
4171
4172 . Compile the application source file:
4173 +
4174 --
4175 [role="term"]
4176 ----
4177 $ gcc -c app.c
4178 ----
4179 --
4180
4181 . Build the application:
4182 +
4183 --
4184 [role="term"]
4185 ----
4186 $ gcc -o app app.o -L. -lemon
4187 ----
4188 --
4189
4190 To run the application with tracing support:
4191
4192 * Preload the tracepoint provider package shared object and
4193   start the application:
4194 +
4195 --
4196 [role="term"]
4197 ----
4198 $ LD_PRELOAD=./libtpp.so ./app
4199 ----
4200 --
4201
4202 To run the application without tracing support:
4203
4204 * Start the application:
4205 +
4206 --
4207 [role="term"]
4208 ----
4209 $ ./app
4210 ----
4211 --
4212
4213 |
4214 The application is statically linked with the tracepoint provider
4215 package object file.
4216
4217 The application is linked with the instrumented user library.
4218
4219 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
4220
4221 |
4222 include::../common/ust-sit-step-tp-o.txt[]
4223
4224 To build the instrumented user library:
4225
4226 . In path:{emon.c}, before including path:{tpp.h}, add the
4227   following line:
4228 +
4229 --
4230 [source,c]
4231 ----
4232 #define LTTNG_UST_TRACEPOINT_DEFINE
4233 ----
4234 --
4235
4236 . Compile the user library source file:
4237 +
4238 --
4239 [role="term"]
4240 ----
4241 $ gcc -I. -fpic -c emon.c
4242 ----
4243 --
4244
4245 . Build the user library shared object:
4246 +
4247 --
4248 [role="term"]
4249 ----
4250 $ gcc -shared -o libemon.so emon.o
4251 ----
4252 --
4253
4254 To build the application:
4255
4256 . Compile the application source file:
4257 +
4258 --
4259 [role="term"]
4260 ----
4261 $ gcc -c app.c
4262 ----
4263 --
4264
4265 . Build the application:
4266 +
4267 --
4268 [role="term"]
4269 ----
4270 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
4271 ----
4272 --
4273
4274 To run the instrumented application:
4275
4276 * Start the application:
4277 +
4278 --
4279 [role="term"]
4280 ----
4281 $ ./app
4282 ----
4283 --
4284
4285 |
4286 The application is statically linked with the tracepoint provider
4287 package object file.
4288
4289 The application dynamically loads the instrumented user library.
4290
4291 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
4292
4293 |
4294 include::../common/ust-sit-step-tp-o.txt[]
4295
4296 To build the application:
4297
4298 . In path:{app.c}, before including path:{tpp.h}, add the following line:
4299 +
4300 --
4301 [source,c]
4302 ----
4303 #define LTTNG_UST_TRACEPOINT_DEFINE
4304 ----
4305 --
4306
4307 . Compile the application source file:
4308 +
4309 --
4310 [role="term"]
4311 ----
4312 $ gcc -c app.c
4313 ----
4314 --
4315
4316 . Build the application:
4317 +
4318 --
4319 [role="term"]
4320 ----
4321 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
4322       -llttng-ust -ldl
4323 ----
4324 --
4325 +
4326 The `--export-dynamic` option passed to the linker is necessary for the
4327 dynamically loaded library to ``see'' the tracepoint symbols defined in
4328 the application.
4329
4330 To build the instrumented user library:
4331
4332 . Compile the user library source file:
4333 +
4334 --
4335 [role="term"]
4336 ----
4337 $ gcc -I. -fpic -c emon.c
4338 ----
4339 --
4340
4341 . Build the user library shared object:
4342 +
4343 --
4344 [role="term"]
4345 ----
4346 $ gcc -shared -o libemon.so emon.o
4347 ----
4348 --
4349
4350 To run the application:
4351
4352 * Start the application:
4353 +
4354 --
4355 [role="term"]
4356 ----
4357 $ ./app
4358 ----
4359 --
4360 |====
4361
4362
4363 [[using-lttng-ust-with-daemons]]
4364 ===== Use noch:{LTTng-UST} with daemons
4365
4366 If your instrumented application calls man:fork(2), man:clone(2),
4367 or BSD's man:rfork(2), without a following man:exec(3)-family
4368 system call, you must preload the path:{liblttng-ust-fork.so} shared
4369 object when you start the application.
4370
4371 [role="term"]
4372 ----
4373 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
4374 ----
4375
4376 If your tracepoint provider package is
4377 a shared library which you also preload, you must put both
4378 shared objects in env:LD_PRELOAD:
4379
4380 [role="term"]
4381 ----
4382 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
4383 ----
4384
4385
4386 [role="since-2.9"]
4387 [[liblttng-ust-fd]]
4388 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
4389
4390 If your instrumented application closes one or more file descriptors
4391 which it did not open itself, you must preload the
4392 path:{liblttng-ust-fd.so} shared object when you start the application:
4393
4394 [role="term"]
4395 ----
4396 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
4397 ----
4398
4399 Typical use cases include closing all the file descriptors after
4400 man:fork(2) or man:rfork(2) and buggy applications doing
4401 ``double closes''.
4402
4403
4404 [[lttng-ust-pkg-config]]
4405 ===== Use noch:{pkg-config}
4406
4407 On some distributions, LTTng-UST ships with a
4408 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
4409 metadata file. If this is your case, then use cmd:pkg-config to
4410 build an application on the command line:
4411
4412 [role="term"]
4413 ----
4414 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
4415 ----
4416
4417
4418 [[instrumenting-32-bit-app-on-64-bit-system]]
4419 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
4420
4421 In order to trace a 32-bit application running on a 64-bit system,
4422 LTTng must use a dedicated 32-bit
4423 <<lttng-consumerd,consumer daemon>>.
4424
4425 The following steps show how to build and install a 32-bit consumer
4426 daemon, which is _not_ part of the default 64-bit LTTng build, how to
4427 build and install the 32-bit LTTng-UST libraries, and how to build and
4428 link an instrumented 32-bit application in that context.
4429
4430 To build a 32-bit instrumented application for a 64-bit target system,
4431 assuming you have a fresh target system with no installed Userspace RCU
4432 or LTTng packages:
4433
4434 . Download, build, and install a 32-bit version of Userspace RCU:
4435 +
4436 --
4437 [role="term"]
4438 ----
4439 $ cd $(mktemp -d) &&
4440   wget https://lttng.org/files/urcu/userspace-rcu-latest-0.13.tar.bz2 &&
4441   tar -xf userspace-rcu-latest-0.13.tar.bz2 &&
4442   cd userspace-rcu-0.13.* &&
4443   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
4444   make &&
4445   sudo make install &&
4446   sudo ldconfig
4447 ----
4448 --
4449
4450 . Using the package manager of your distribution, or from source,
4451   install the 32-bit versions of the following dependencies of
4452   LTTng-tools and LTTng-UST:
4453 +
4454 --
4455 * https://sourceforge.net/projects/libuuid/[libuuid]
4456 * https://directory.fsf.org/wiki/Popt[popt]
4457 * https://www.xmlsoft.org/[libxml2]
4458 * **Optional**: https://github.com/numactl/numactl[numactl]
4459 --
4460
4461 . Download, build, and install a 32-bit version of the latest
4462   LTTng-UST{nbsp}{revision}:
4463 +
4464 --
4465 [role="term"]
4466 ----
4467 $ cd $(mktemp -d) &&
4468   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
4469   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
4470   cd lttng-ust-2.13.* &&
4471   ./configure --libdir=/usr/local/lib32 \
4472               CFLAGS=-m32 CXXFLAGS=-m32 \
4473               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4474   make &&
4475   sudo make install &&
4476   sudo ldconfig
4477 ----
4478 --
4479 +
4480 Add `--disable-numa` to `./configure` if you don't have
4481 https://github.com/numactl/numactl[numactl].
4482 +
4483 [NOTE]
4484 ====
4485 Depending on your distribution, 32-bit libraries could be installed at a
4486 different location than `/usr/lib32`. For example, Debian is known to
4487 install some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
4488
4489 In this case, make sure to set `LDFLAGS` to all the
4490 relevant 32-bit library paths, for example:
4491
4492 [role="term"]
4493 ----
4494 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
4495 ----
4496 ====
4497
4498 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
4499   the 32-bit consumer daemon:
4500 +
4501 --
4502 [role="term"]
4503 ----
4504 $ cd $(mktemp -d) &&
4505   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4506   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4507   cd lttng-tools-2.13.* &&
4508   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
4509               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
4510               --disable-bin-lttng --disable-bin-lttng-crash \
4511               --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
4512   make &&
4513   cd src/bin/lttng-consumerd &&
4514   sudo make install &&
4515   sudo ldconfig
4516 ----
4517 --
4518
4519 . From your distribution or from source, <<installing-lttng,install>>
4520   the 64-bit versions of LTTng-UST and Userspace RCU.
4521
4522 . Download, build, and install the 64-bit version of the
4523   latest LTTng-tools{nbsp}{revision}:
4524 +
4525 --
4526 [role="term"]
4527 ----
4528 $ cd $(mktemp -d) &&
4529   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4530   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4531   cd lttng-tools-2.13.* &&
4532   ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4533               --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4534   make &&
4535   sudo make install &&
4536   sudo ldconfig
4537 ----
4538 --
4539
4540 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4541   when linking your 32-bit application:
4542 +
4543 ----
4544 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4545 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4546 ----
4547 +
4548 For example, let's rebuild the quick start example in
4549 ``<<tracing-your-own-user-application,Record user application events>>''
4550 as an instrumented 32-bit application:
4551 +
4552 --
4553 [role="term"]
4554 ----
4555 $ gcc -m32 -c -I. hello-tp.c
4556 $ gcc -m32 -c hello.c
4557 $ gcc -m32 -o hello hello.o hello-tp.o \
4558       -L/usr/lib32 -L/usr/local/lib32 \
4559       -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4560       -llttng-ust -ldl
4561 ----
4562 --
4563
4564 No special action is required to execute the 32-bit application and
4565 for LTTng to trace it: use the command-line man:lttng(1) tool as usual.
4566
4567
4568 [role="since-2.5"]
4569 [[tracef]]
4570 ==== Use `lttng_ust_tracef()`
4571
4572 man:lttng_ust_tracef(3) is a small LTTng-UST API designed for quick,
4573 man:printf(3)-like instrumentation without the burden of
4574 <<tracepoint-provider,creating>> and
4575 <<building-tracepoint-providers-and-user-application,building>>
4576 a tracepoint provider package.
4577
4578 To use `lttng_ust_tracef()` in your application:
4579
4580 . In the C or $$C++$$ source files where you need to use
4581   `lttng_ust_tracef()`, include `<lttng/tracef.h>`:
4582 +
4583 --
4584 [source,c]
4585 ----
4586 #include <lttng/tracef.h>
4587 ----
4588 --
4589
4590 . In the source code of the application, use `lttng_ust_tracef()` like
4591   you would use man:printf(3):
4592 +
4593 --
4594 [source,c]
4595 ----
4596     /* ... */
4597
4598     lttng_ust_tracef("my message: %d (%s)", my_integer, my_string);
4599
4600     /* ... */
4601 ----
4602 --
4603
4604 . Link your application with `liblttng-ust`:
4605 +
4606 --
4607 [role="term"]
4608 ----
4609 $ gcc -o app app.c -llttng-ust
4610 ----
4611 --
4612
4613 To record the events that `lttng_ust_tracef()` calls emit:
4614
4615 * <<enabling-disabling-events,Create a recording event rule>> which
4616   matches user space events named `lttng_ust_tracef:*`:
4617 +
4618 --
4619 [role="term"]
4620 ----
4621 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4622 ----
4623 --
4624
4625 [IMPORTANT]
4626 .Limitations of `lttng_ust_tracef()`
4627 ====
4628 The `lttng_ust_tracef()` utility function was developed to make user
4629 space tracing super simple, albeit with notable disadvantages compared
4630 to <<defining-tracepoints,user-defined tracepoints>>:
4631
4632 * All the created events have the same tracepoint provider and
4633   tracepoint names, respectively `lttng_ust_tracef` and `event`.
4634 * There's no static type checking.
4635 * The only event record field you actually get, named `msg`, is a string
4636   potentially containing the values you passed to `lttng_ust_tracef()`
4637   using your own format string. This also means that you can't filter
4638   events with a custom expression at run time because there are no
4639   isolated fields.
4640 * Since `lttng_ust_tracef()` uses the man:vasprintf(3) function of the
4641   C{nbsp}standard library behind the scenes to format the strings at run
4642   time, its expected performance is lower than with user-defined
4643   tracepoints, which don't require a conversion to a string.
4644
4645 Taking this into consideration, `lttng_ust_tracef()` is useful for some
4646 quick prototyping and debugging, but you shouldn't consider it for any
4647 permanent and serious applicative instrumentation.
4648 ====
4649
4650
4651 [role="since-2.7"]
4652 [[tracelog]]
4653 ==== Use `lttng_ust_tracelog()`
4654
4655 The man:tracelog(3) API is very similar to
4656 <<tracef,`lttng_ust_tracef()`>>, with the difference that it accepts an
4657 additional log level parameter.
4658
4659 The goal of `lttng_ust_tracelog()` is to ease the migration from logging
4660 to tracing.
4661
4662 To use `lttng_ust_tracelog()` in your application:
4663
4664 . In the C or $$C++$$ source files where you need to use `tracelog()`,
4665   include `<lttng/tracelog.h>`:
4666 +
4667 --
4668 [source,c]
4669 ----
4670 #include <lttng/tracelog.h>
4671 ----
4672 --
4673
4674 . In the source code of the application, use `lttng_ust_tracelog()` like
4675   you would use man:printf(3), except for the first parameter which is
4676   the log level:
4677 +
4678 --
4679 [source,c]
4680 ----
4681     /* ... */
4682
4683     tracelog(LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING,
4684             "my message: %d (%s)", my_integer, my_string);
4685
4686     /* ... */
4687 ----
4688 --
4689 +
4690 See man:lttng-ust(3) for a list of available log level names.
4691
4692 . Link your application with `liblttng-ust`:
4693 +
4694 --
4695 [role="term"]
4696 ----
4697 $ gcc -o app app.c -llttng-ust
4698 ----
4699 --
4700
4701 To record the events that `lttng_ust_tracelog()` calls emit with a log
4702 level _at least as severe as_ a specific log level:
4703
4704 * <<enabling-disabling-events,Create a recording event rule>> which
4705   matches user space tracepoint events named `lttng_ust_tracelog:*` and
4706   with some minimum level of severity:
4707 +
4708 --
4709 [role="term"]
4710 ----
4711 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4712                      --loglevel=WARNING
4713 ----
4714 --
4715
4716 To record the events that `lttng_ust_tracelog()` calls emit with a
4717 _specific log level_:
4718
4719 * Create a recording event rule which matches tracepoint events named
4720   `lttng_ust_tracelog:*` and with a specific log level:
4721 +
4722 --
4723 [role="term"]
4724 ----
4725 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4726                      --loglevel-only=INFO
4727 ----
4728 --
4729
4730
4731 [[prebuilt-ust-helpers]]
4732 === Load a prebuilt user space tracing helper
4733
4734 The LTTng-UST package provides a few helpers in the form of preloadable
4735 shared objects which automatically instrument system functions and
4736 calls.
4737
4738 The helper shared objects are normally found in dir:{/usr/lib}. If you
4739 built LTTng-UST <<building-from-source,from source>>, they're probably
4740 located in dir:{/usr/local/lib}.
4741
4742 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4743 are:
4744
4745 path:{liblttng-ust-libc-wrapper.so}::
4746 path:{liblttng-ust-pthread-wrapper.so}::
4747   <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4748   memory and POSIX threads function tracing>>.
4749
4750 path:{liblttng-ust-cyg-profile.so}::
4751 path:{liblttng-ust-cyg-profile-fast.so}::
4752   <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4753
4754 path:{liblttng-ust-dl.so}::
4755   <<liblttng-ust-dl,Dynamic linker tracing>>.
4756
4757 To use a user space tracing helper with any user application:
4758
4759 * Preload the helper shared object when you start the application:
4760 +
4761 --
4762 [role="term"]
4763 ----
4764 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4765 ----
4766 --
4767 +
4768 You can preload more than one helper:
4769 +
4770 --
4771 [role="term"]
4772 ----
4773 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4774 ----
4775 --
4776
4777
4778 [role="since-2.3"]
4779 [[liblttng-ust-libc-pthread-wrapper]]
4780 ==== Instrument C standard library memory and POSIX threads functions
4781
4782 The path:{liblttng-ust-libc-wrapper.so} and
4783 path:{liblttng-ust-pthread-wrapper.so} helpers
4784 add instrumentation to some C standard library and POSIX
4785 threads functions.
4786
4787 [role="growable"]
4788 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4789 |====
4790 |TP provider name |TP name |Instrumented function
4791
4792 .6+|`lttng_ust_libc` |`malloc`         |man:malloc(3)
4793                      |`calloc`         |man:calloc(3)
4794                      |`realloc`        |man:realloc(3)
4795                      |`free`           |man:free(3)
4796                      |`memalign`       |man:memalign(3)
4797                      |`posix_memalign` |man:posix_memalign(3)
4798 |====
4799
4800 [role="growable"]
4801 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4802 |====
4803 |TP provider name |TP name |Instrumented function
4804
4805 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4806                         |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4807                         |`pthread_mutex_trylock`  |man:pthread_mutex_trylock(3p)
4808                         |`pthread_mutex_unlock`   |man:pthread_mutex_unlock(3p)
4809 |====
4810
4811 When you preload the shared object, it replaces the functions listed
4812 in the previous tables by wrappers which contain tracepoints and call
4813 the replaced functions.
4814
4815
4816 [[liblttng-ust-cyg-profile]]
4817 ==== Instrument function entry and exit
4818
4819 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4820 to the entry and exit points of functions.
4821
4822 man:gcc(1) and man:clang(1) have an option named
4823 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4824 which generates instrumentation calls for entry and exit to functions.
4825 The LTTng-UST function tracing helpers,
4826 path:{liblttng-ust-cyg-profile.so} and
4827 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4828 to add tracepoints to the two generated functions (which contain
4829 `cyg_profile` in their names, hence the name of the helper).
4830
4831 To use the LTTng-UST function tracing helper, the source files to
4832 instrument must be built using the `-finstrument-functions` compiler
4833 flag.
4834
4835 There are two versions of the LTTng-UST function tracing helper:
4836
4837 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4838   that you should only use when it can be _guaranteed_ that the
4839   complete event stream is recorded without any lost event record.
4840   Any kind of duplicate information is left out.
4841 +
4842 Assuming no event record is lost, having only the function addresses on
4843 entry is enough to create a call graph, since an event record always
4844 contains the ID of the CPU that generated it.
4845 +
4846 Use a tool like man:addr2line(1) to convert function addresses back to
4847 source file names and line numbers.
4848
4849 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4850 which also works in use cases where event records might get discarded or
4851 not recorded from application startup.
4852 In these cases, the trace analyzer needs more information to be
4853 able to reconstruct the program flow.
4854
4855 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4856 points of this helper.
4857
4858 All the tracepoints that this helper provides have the log level
4859 `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4860
4861 TIP: It's sometimes a good idea to limit the number of source files that
4862 you compile with the `-finstrument-functions` option to prevent LTTng
4863 from writing an excessive amount of trace data at run time. When using
4864 man:gcc(1), use the
4865 `-finstrument-functions-exclude-function-list` option to avoid
4866 instrument entries and exits of specific function names.
4867
4868
4869 [role="since-2.4"]
4870 [[liblttng-ust-dl]]
4871 ==== Instrument the dynamic linker
4872
4873 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4874 man:dlopen(3) and man:dlclose(3) function calls.
4875
4876 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4877 of this helper.
4878
4879
4880 [role="since-2.4"]
4881 [[java-application]]
4882 === Instrument a Java application
4883
4884 You can instrument any Java application which uses one of the following
4885 logging frameworks:
4886
4887 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4888   (JUL) core logging facilities.
4889
4890 * https://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4891   LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4892
4893 [role="img-100"]
4894 .LTTng-UST Java agent imported by a Java application.
4895 image::java-app.png[]
4896
4897 Note that the methods described below are new in LTTng{nbsp}2.8.
4898 Previous LTTng versions use another technique.
4899
4900 NOTE: We use https://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4901 and https://ci.lttng.org/[continuous integration], thus this version is
4902 directly supported. However, the LTTng-UST Java agent is also tested
4903 with OpenJDK{nbsp}7.
4904
4905
4906 [role="since-2.8"]
4907 [[jul]]
4908 ==== Use the LTTng-UST Java agent for `java.util.logging`
4909
4910 To use the LTTng-UST Java agent in a Java application which uses
4911 `java.util.logging` (JUL):
4912
4913 . In the source code of the Java application, import the LTTng-UST log
4914   handler package for `java.util.logging`:
4915 +
4916 --
4917 [source,java]
4918 ----
4919 import org.lttng.ust.agent.jul.LttngLogHandler;
4920 ----
4921 --
4922
4923 . Create an LTTng-UST `java.util.logging` log handler:
4924 +
4925 --
4926 [source,java]
4927 ----
4928 Handler lttngUstLogHandler = new LttngLogHandler();
4929 ----
4930 --
4931
4932 . Add this handler to the `java.util.logging` loggers which should emit
4933   LTTng events:
4934 +
4935 --
4936 [source,java]
4937 ----
4938 Logger myLogger = Logger.getLogger("some-logger");
4939
4940 myLogger.addHandler(lttngUstLogHandler);
4941 ----
4942 --
4943
4944 . Use `java.util.logging` log statements and configuration as usual.
4945   The loggers with an attached LTTng-UST log handler can emit
4946   LTTng events.
4947
4948 . Before exiting the application, remove the LTTng-UST log handler from
4949   the loggers attached to it and call its `close()` method:
4950 +
4951 --
4952 [source,java]
4953 ----
4954 myLogger.removeHandler(lttngUstLogHandler);
4955 lttngUstLogHandler.close();
4956 ----
4957 --
4958 +
4959 This isn't strictly necessary, but it's recommended for a clean
4960 disposal of the resources of the handler.
4961
4962 . Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
4963   path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4964   in the
4965   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4966   path] when you build the Java application.
4967 +
4968 The JAR files are typically located in dir:{/usr/share/java}.
4969 +
4970 IMPORTANT: The LTTng-UST Java agent must be
4971 <<installing-lttng,installed>> for the logging framework your
4972 application uses.
4973
4974 .Use the LTTng-UST Java agent for `java.util.logging`.
4975 ====
4976 [source,java]
4977 .path:{Test.java}
4978 ----
4979 import java.io.IOException;
4980 import java.util.logging.Handler;
4981 import java.util.logging.Logger;
4982 import org.lttng.ust.agent.jul.LttngLogHandler;
4983
4984 public class Test
4985 {
4986     private static final int answer = 42;
4987
4988     public static void main(String[] argv) throws Exception
4989     {
4990         // Create a logger
4991         Logger logger = Logger.getLogger("jello");
4992
4993         // Create an LTTng-UST log handler
4994         Handler lttngUstLogHandler = new LttngLogHandler();
4995
4996         // Add the LTTng-UST log handler to our logger
4997         logger.addHandler(lttngUstLogHandler);
4998
4999         // Log at will!
5000         logger.info("some info");
5001         logger.warning("some warning");
5002         Thread.sleep(500);
5003         logger.finer("finer information; the answer is " + answer);
5004         Thread.sleep(123);
5005         logger.severe("error!");
5006
5007         // Not mandatory, but cleaner
5008         logger.removeHandler(lttngUstLogHandler);
5009         lttngUstLogHandler.close();
5010     }
5011 }
5012 ----
5013
5014 Build this example:
5015
5016 [role="term"]
5017 ----
5018 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5019 ----
5020
5021 <<creating-destroying-tracing-sessions,Create a recording session>>,
5022 <<enabling-disabling-events,create a recording event rule>> matching JUL
5023 events named `jello`, and <<basic-tracing-session-control,start
5024 recording>>:
5025
5026 [role="term"]
5027 ----
5028 $ lttng create
5029 $ lttng enable-event --jul jello
5030 $ lttng start
5031 ----
5032
5033 Run the compiled class:
5034
5035 [role="term"]
5036 ----
5037 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5038 ----
5039
5040 <<basic-tracing-session-control,Stop recording>> and inspect the
5041 recorded events:
5042
5043 [role="term"]
5044 ----
5045 $ lttng stop
5046 $ lttng view
5047 ----
5048 ====
5049
5050 In the resulting trace, an <<event,event record>> which a Java
5051 application using `java.util.logging` generated is named
5052 `lttng_jul:event` and has the following fields:
5053
5054 `msg`::
5055   Log record message.
5056
5057 `logger_name`::
5058   Logger name.
5059
5060 `class_name`::
5061   Name of the class in which the log statement was executed.
5062
5063 `method_name`::
5064   Name of the method in which the log statement was executed.
5065
5066 `long_millis`::
5067   Logging time (timestamp in milliseconds).
5068
5069 `int_loglevel`::
5070   Log level integer value.
5071
5072 `int_threadid`::
5073   ID of the thread in which the log statement was executed.
5074
5075 Use the opt:lttng-enable-event(1):--loglevel or
5076 opt:lttng-enable-event(1):--loglevel-only option of the
5077 man:lttng-enable-event(1) command to target a range of
5078 `java.util.logging` log levels or a specific `java.util.logging` log
5079 level.
5080
5081
5082 [role="since-2.8"]
5083 [[log4j]]
5084 ==== Use the LTTng-UST Java agent for Apache log4j
5085
5086 To use the LTTng-UST Java agent in a Java application which uses
5087 Apache log4j{nbsp}1.2:
5088
5089 . In the source code of the Java application, import the LTTng-UST log
5090   appender package for Apache log4j:
5091 +
5092 --
5093 [source,java]
5094 ----
5095 import org.lttng.ust.agent.log4j.LttngLogAppender;
5096 ----
5097 --
5098
5099 . Create an LTTng-UST log4j log appender:
5100 +
5101 --
5102 [source,java]
5103 ----
5104 Appender lttngUstLogAppender = new LttngLogAppender();
5105 ----
5106 --
5107
5108 . Add this appender to the log4j loggers which should emit LTTng events:
5109 +
5110 --
5111 [source,java]
5112 ----
5113 Logger myLogger = Logger.getLogger("some-logger");
5114
5115 myLogger.addAppender(lttngUstLogAppender);
5116 ----
5117 --
5118
5119 . Use Apache log4j log statements and configuration as usual. The
5120   loggers with an attached LTTng-UST log appender can emit LTTng events.
5121
5122 . Before exiting the application, remove the LTTng-UST log appender from
5123   the loggers attached to it and call its `close()` method:
5124 +
5125 --
5126 [source,java]
5127 ----
5128 myLogger.removeAppender(lttngUstLogAppender);
5129 lttngUstLogAppender.close();
5130 ----
5131 --
5132 +
5133 This isn't strictly necessary, but it's recommended for a clean
5134 disposal of the resources of the appender.
5135
5136 . Include the common and log4j-specific JAR
5137   files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
5138   path:{lttng-ust-agent-log4j.jar}, in the
5139   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
5140   path] when you build the Java application.
5141 +
5142 The JAR files are typically located in dir:{/usr/share/java}.
5143 +
5144 IMPORTANT: The LTTng-UST Java agent must be
5145 <<installing-lttng,installed>> for the logging framework your
5146 application uses.
5147
5148 .Use the LTTng-UST Java agent for Apache log4j.
5149 ====
5150 [source,java]
5151 .path:{Test.java}
5152 ----
5153 import org.apache.log4j.Appender;
5154 import org.apache.log4j.Logger;
5155 import org.lttng.ust.agent.log4j.LttngLogAppender;
5156
5157 public class Test
5158 {
5159     private static final int answer = 42;
5160
5161     public static void main(String[] argv) throws Exception
5162     {
5163         // Create a logger
5164         Logger logger = Logger.getLogger("jello");
5165
5166         // Create an LTTng-UST log appender
5167         Appender lttngUstLogAppender = new LttngLogAppender();
5168
5169         // Add the LTTng-UST log appender to our logger
5170         logger.addAppender(lttngUstLogAppender);
5171
5172         // Log at will!
5173         logger.info("some info");
5174         logger.warn("some warning");
5175         Thread.sleep(500);
5176         logger.debug("debug information; the answer is " + answer);
5177         Thread.sleep(123);
5178         logger.fatal("error!");
5179
5180         // Not mandatory, but cleaner
5181         logger.removeAppender(lttngUstLogAppender);
5182         lttngUstLogAppender.close();
5183     }
5184 }
5185
5186 ----
5187
5188 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
5189 file):
5190
5191 [role="term"]
5192 ----
5193 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
5194 ----
5195
5196 <<creating-destroying-tracing-sessions,Create a recording session>>,
5197 <<enabling-disabling-events,create a recording event rule>> matching
5198 log4j events named `jello`, and <<basic-tracing-session-control,start
5199 recording>>:
5200
5201 [role="term"]
5202 ----
5203 $ lttng create
5204 $ lttng enable-event --log4j jello
5205 $ lttng start
5206 ----
5207
5208 Run the compiled class:
5209
5210 [role="term"]
5211 ----
5212 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
5213 ----
5214
5215 <<basic-tracing-session-control,Stop recording>> and inspect the
5216 recorded events:
5217
5218 [role="term"]
5219 ----
5220 $ lttng stop
5221 $ lttng view
5222 ----
5223 ====
5224
5225 In the resulting trace, an <<event,event record>> which a Java
5226 application using log4j generated is named `lttng_log4j:event` and
5227 has the following fields:
5228
5229 `msg`::
5230   Log record message.
5231
5232 `logger_name`::
5233   Logger name.
5234
5235 `class_name`::
5236   Name of the class in which the log statement was executed.
5237
5238 `method_name`::
5239   Name of the method in which the log statement was executed.
5240
5241 `filename`::
5242   Name of the file in which the executed log statement is located.
5243
5244 `line_number`::
5245   Line number at which the log statement was executed.
5246
5247 `timestamp`::
5248   Logging timestamp.
5249
5250 `int_loglevel`::
5251   Log level integer value.
5252
5253 `thread_name`::
5254   Name of the Java thread in which the log statement was executed.
5255
5256 Use the opt:lttng-enable-event(1):--loglevel or
5257 opt:lttng-enable-event(1):--loglevel-only option of the
5258 man:lttng-enable-event(1) command to target a range of Apache log4j
5259 log levels or a specific log4j log level.
5260
5261
5262 [role="since-2.8"]
5263 [[java-application-context]]
5264 ==== Provide application-specific context fields in a Java application
5265
5266 A Java application-specific context field is a piece of state which
5267 the Java application provides. You can <<adding-context,add>> such
5268 a context field to be recorded, using the
5269 man:lttng-add-context(1) command, to each <<event,event record>>
5270 which the log statements of this application produce.
5271
5272 For example, a given object might have a current request ID variable.
5273 You can create a context information retriever for this object and
5274 assign a name to this current request ID. You can then, using the
5275 man:lttng-add-context(1) command, add this context field by name so that
5276 LTTng writes it to the event records of a given `java.util.logging` or
5277 log4j <<channel,channel>>.
5278
5279 To provide application-specific context fields in a Java application:
5280
5281 . In the source code of the Java application, import the LTTng-UST
5282   Java agent context classes and interfaces:
5283 +
5284 --
5285 [source,java]
5286 ----
5287 import org.lttng.ust.agent.context.ContextInfoManager;
5288 import org.lttng.ust.agent.context.IContextInfoRetriever;
5289 ----
5290 --
5291
5292 . Create a context information retriever class, that is, a class which
5293   implements the `IContextInfoRetriever` interface:
5294 +
5295 --
5296 [source,java]
5297 ----
5298 class MyContextInfoRetriever implements IContextInfoRetriever
5299 {
5300     @Override
5301     public Object retrieveContextInfo(String key)
5302     {
5303         if (key.equals("intCtx")) {
5304             return (short) 17;
5305         } else if (key.equals("strContext")) {
5306             return "context value!";
5307         } else {
5308             return null;
5309         }
5310     }
5311 }
5312 ----
5313 --
5314 +
5315 This `retrieveContextInfo()` method is the only member of the
5316 `IContextInfoRetriever` interface. Its role is to return the current
5317 value of a state by name to create a context field. The names of the
5318 context fields and which state variables they return depends on your
5319 specific scenario.
5320 +
5321 All primitive types and objects are supported as context fields.
5322 When `retrieveContextInfo()` returns an object, the context field
5323 serializer calls its `toString()` method to add a string field to
5324 event records. The method can also return `null`, which means that
5325 no context field is available for the required name.
5326
5327 . Register an instance of your context information retriever class to
5328   the context information manager singleton:
5329 +
5330 --
5331 [source,java]
5332 ----
5333 IContextInfoRetriever cir = new MyContextInfoRetriever();
5334 ContextInfoManager cim = ContextInfoManager.getInstance();
5335 cim.registerContextInfoRetriever("retrieverName", cir);
5336 ----
5337 --
5338
5339 . Before exiting the application, remove your context information
5340   retriever from the context information manager singleton:
5341 +
5342 --
5343 [source,java]
5344 ----
5345 ContextInfoManager cim = ContextInfoManager.getInstance();
5346 cim.unregisterContextInfoRetriever("retrieverName");
5347 ----
5348 --
5349 +
5350 This isn't strictly necessary, but it's recommended for a clean
5351 disposal of some resources of the manager.
5352
5353 . Build your Java application with LTTng-UST Java agent support as
5354   usual, following the procedure for either the
5355   <<jul,`java.util.logging`>> or <<log4j,Apache log4j>> framework.
5356
5357 .Provide application-specific context fields in a Java application.
5358 ====
5359 [source,java]
5360 .path:{Test.java}
5361 ----
5362 import java.util.logging.Handler;
5363 import java.util.logging.Logger;
5364 import org.lttng.ust.agent.jul.LttngLogHandler;
5365 import org.lttng.ust.agent.context.ContextInfoManager;
5366 import org.lttng.ust.agent.context.IContextInfoRetriever;
5367
5368 public class Test
5369 {
5370     // Our context information retriever class
5371     private static class MyContextInfoRetriever
5372     implements IContextInfoRetriever
5373     {
5374         @Override
5375         public Object retrieveContextInfo(String key) {
5376             if (key.equals("intCtx")) {
5377                 return (short) 17;
5378             } else if (key.equals("strContext")) {
5379                 return "context value!";
5380             } else {
5381                 return null;
5382             }
5383         }
5384     }
5385
5386     private static final int answer = 42;
5387
5388     public static void main(String args[]) throws Exception
5389     {
5390         // Get the context information manager instance
5391         ContextInfoManager cim = ContextInfoManager.getInstance();
5392
5393         // Create and register our context information retriever
5394         IContextInfoRetriever cir = new MyContextInfoRetriever();
5395         cim.registerContextInfoRetriever("myRetriever", cir);
5396
5397         // Create a logger
5398         Logger logger = Logger.getLogger("jello");
5399
5400         // Create an LTTng-UST log handler
5401         Handler lttngUstLogHandler = new LttngLogHandler();
5402
5403         // Add the LTTng-UST log handler to our logger
5404         logger.addHandler(lttngUstLogHandler);
5405
5406         // Log at will!
5407         logger.info("some info");
5408         logger.warning("some warning");
5409         Thread.sleep(500);
5410         logger.finer("finer information; the answer is " + answer);
5411         Thread.sleep(123);
5412         logger.severe("error!");
5413
5414         // Not mandatory, but cleaner
5415         logger.removeHandler(lttngUstLogHandler);
5416         lttngUstLogHandler.close();
5417         cim.unregisterContextInfoRetriever("myRetriever");
5418     }
5419 }
5420 ----
5421
5422 Build this example:
5423
5424 [role="term"]
5425 ----
5426 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5427 ----
5428
5429 <<creating-destroying-tracing-sessions,Create a recording session>> and
5430 <<enabling-disabling-events,create a recording event rule>> matching
5431 `java.util.logging` events named `jello`:
5432
5433 [role="term"]
5434 ----
5435 $ lttng create
5436 $ lttng enable-event --jul jello
5437 ----
5438
5439 <<adding-context,Add the application-specific context fields>> to be
5440 recorded to the event records of the `java.util.logging` channel:
5441
5442 [role="term"]
5443 ----
5444 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
5445 $ lttng add-context --jul --type='$app.myRetriever:strContext'
5446 ----
5447
5448 <<basic-tracing-session-control,Start recording>>:
5449
5450 [role="term"]
5451 ----
5452 $ lttng start
5453 ----
5454
5455 Run the compiled class:
5456
5457 [role="term"]
5458 ----
5459 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5460 ----
5461
5462 <<basic-tracing-session-control,Stop recording>> and inspect the
5463 recorded events:
5464
5465 [role="term"]
5466 ----
5467 $ lttng stop
5468 $ lttng view
5469 ----
5470 ====
5471
5472
5473 [role="since-2.7"]
5474 [[python-application]]
5475 === Instrument a Python application
5476
5477 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
5478 uses the standard
5479 https://docs.python.org/3/library/logging.html[`logging`] package.
5480
5481 Each log statement creates an LTTng event once the application module
5482 imports the <<lttng-ust-agents,LTTng-UST Python agent>> package.
5483
5484 [role="img-100"]
5485 .A Python application importing the LTTng-UST Python agent.
5486 image::python-app.png[]
5487
5488 To use the LTTng-UST Python agent:
5489
5490 . In the source code of the Python application, import the LTTng-UST
5491   Python agent:
5492 +
5493 --
5494 [source,python]
5495 ----
5496 import lttngust
5497 ----
5498 --
5499 +
5500 The LTTng-UST Python agent automatically adds its logging handler to the
5501 root logger at import time.
5502 +
5503 A log statement that the application executes before this import doesn't
5504 create an LTTng event.
5505 +
5506 IMPORTANT: The LTTng-UST Python agent must be
5507 <<installing-lttng,installed>>.
5508
5509 . Use log statements and logging configuration as usual.
5510   Since the LTTng-UST Python agent adds a handler to the _root_
5511   logger, any log statement from any logger can emit an LTTng event.
5512
5513 .Use the LTTng-UST Python agent.
5514 ====
5515 [source,python]
5516 .path:{test.py}
5517 ----
5518 import lttngust
5519 import logging
5520 import time
5521
5522
5523 def example():
5524     logging.basicConfig()
5525     logger = logging.getLogger('my-logger')
5526
5527     while True:
5528         logger.debug('debug message')
5529         logger.info('info message')
5530         logger.warn('warn message')
5531         logger.error('error message')
5532         logger.critical('critical message')
5533         time.sleep(1)
5534
5535
5536 if __name__ == '__main__':
5537     example()
5538 ----
5539
5540 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5541 logging handler which prints to the standard error stream, isn't
5542 strictly required for LTTng-UST tracing to work, but in versions of
5543 Python preceding{nbsp}3.2, you could see a warning message which
5544 indicates that no handler exists for the logger `my-logger`.
5545
5546 <<creating-destroying-tracing-sessions,Create a recording session>>,
5547 <<enabling-disabling-events,create a recording event rule>> matching
5548 Python logging events named `my-logger`, and
5549 <<basic-tracing-session-control,start recording>>:
5550
5551 [role="term"]
5552 ----
5553 $ lttng create
5554 $ lttng enable-event --python my-logger
5555 $ lttng start
5556 ----
5557
5558 Run the Python script:
5559
5560 [role="term"]
5561 ----
5562 $ python test.py
5563 ----
5564
5565 <<basic-tracing-session-control,Stop recording>> and inspect the
5566 recorded events:
5567
5568 [role="term"]
5569 ----
5570 $ lttng stop
5571 $ lttng view
5572 ----
5573 ====
5574
5575 In the resulting trace, an <<event,event record>> which a Python
5576 application generated is named `lttng_python:event` and has the
5577 following fields:
5578
5579 `asctime`::
5580   Logging time (string).
5581
5582 `msg`::
5583   Log record message.
5584
5585 `logger_name`::
5586   Logger name.
5587
5588 `funcName`::
5589   Name of the function in which the log statement was executed.
5590
5591 `lineno`::
5592   Line number at which the log statement was executed.
5593
5594 `int_loglevel`::
5595   Log level integer value.
5596
5597 `thread`::
5598   ID of the Python thread in which the log statement was executed.
5599
5600 `threadName`::
5601   Name of the Python thread in which the log statement was executed.
5602
5603 Use the opt:lttng-enable-event(1):--loglevel or
5604 opt:lttng-enable-event(1):--loglevel-only option of the
5605 man:lttng-enable-event(1) command to target a range of Python log levels
5606 or a specific Python log level.
5607
5608 When an application imports the LTTng-UST Python agent, the agent tries
5609 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5610 <<start-sessiond,start the session daemon>> _before_ you run the Python
5611 application. If a session daemon is found, the agent tries to register
5612 to it during five seconds, after which the application continues
5613 without LTTng tracing support. Override this timeout value with
5614 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5615 (milliseconds).
5616
5617 If the session daemon stops while a Python application with an imported
5618 LTTng-UST Python agent runs, the agent retries to connect and to
5619 register to a session daemon every three seconds. Override this
5620 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5621 variable.
5622
5623
5624 [role="since-2.5"]
5625 [[proc-lttng-logger-abi]]
5626 === Use the LTTng logger
5627
5628 The `lttng-tracer` Linux kernel module, part of
5629 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5630 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5631 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5632 to any of those files to create one or more LTTng events.
5633
5634 [role="img-100"]
5635 .An application writes to the LTTng logger file to create one or more LTTng events.
5636 image::lttng-logger.png[]
5637
5638 The LTTng logger is the quickest method--not the most efficient,
5639 however--to add instrumentation to an application. It's designed
5640 mostly to instrument shell scripts:
5641
5642 [role="term"]
5643 ----
5644 $ echo "Some message, some $variable" > /dev/lttng-logger
5645 ----
5646
5647 Any event that the LTTng logger creates is named `lttng_logger` and
5648 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5649 other instrumentation points in the kernel tracing domain, **any Unix
5650 user** can <<enabling-disabling-events,create a recording event rule>>
5651 which matches events named `lttng_logger`, not only the root user or
5652 users in the <<tracing-group,tracing group>>.
5653
5654 To use the LTTng logger:
5655
5656 * From any application, write text data to the path:{/dev/lttng-logger}
5657   file.
5658
5659 The `msg` field of `lttng_logger` event records contains the
5660 recorded message.
5661
5662 NOTE: The maximum message length of an LTTng logger event is
5663 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5664 than one event to contain the remaining data.
5665
5666 You shouldn't use the LTTng logger to trace a user application which you
5667 can instrument in a more efficient way, namely:
5668
5669 * <<c-application,C and $$C++$$ applications>>.
5670 * <<java-application,Java applications>>.
5671 * <<python-application,Python applications>>.
5672
5673 .Use the LTTng logger.
5674 ====
5675 [source,bash]
5676 .path:{test.bash}
5677 ----
5678 echo 'Hello, World!' > /dev/lttng-logger
5679 sleep 2
5680 df --human-readable --print-type / > /dev/lttng-logger
5681 ----
5682
5683 <<creating-destroying-tracing-sessions,Create a recording session>>,
5684 <<enabling-disabling-events,create a recording event rule>> matching
5685 Linux kernel tracepoint events named `lttng_logger`, and
5686 <<basic-tracing-session-control,start recording>>:
5687
5688 [role="term"]
5689 ----
5690 $ lttng create
5691 $ lttng enable-event --kernel lttng_logger
5692 $ lttng start
5693 ----
5694
5695 Run the Bash script:
5696
5697 [role="term"]
5698 ----
5699 $ bash test.bash
5700 ----
5701
5702 <<basic-tracing-session-control,Stop recording>> and inspect the recorded
5703 events:
5704
5705 [role="term"]
5706 ----
5707 $ lttng stop
5708 $ lttng view
5709 ----
5710 ====
5711
5712
5713 [[instrumenting-linux-kernel]]
5714 === Instrument a Linux kernel image or module
5715
5716 NOTE: This section shows how to _add_ instrumentation points to the
5717 Linux kernel. The subsystems of the kernel are already thoroughly
5718 instrumented at strategic points for LTTng when you
5719 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5720 package.
5721
5722
5723 [[linux-add-lttng-layer]]
5724 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5725
5726 This section shows how to add an LTTng layer to existing ftrace
5727 instrumentation using the `TRACE_EVENT()` API.
5728
5729 This section doesn't document the `TRACE_EVENT()` macro. Read the
5730 following articles to learn more about this API:
5731
5732 * https://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5733 * https://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5734 * https://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5735
5736 The following procedure assumes that your ftrace tracepoints are
5737 correctly defined in their own header and that they're created in
5738 one source file using the `CREATE_TRACE_POINTS` definition.
5739
5740 To add an LTTng layer over an existing ftrace tracepoint:
5741
5742 . Make sure the following kernel configuration options are
5743   enabled:
5744 +
5745 --
5746 * `CONFIG_MODULES`
5747 * `CONFIG_KALLSYMS`
5748 * `CONFIG_HIGH_RES_TIMERS`
5749 * `CONFIG_TRACEPOINTS`
5750 --
5751
5752 . Build the Linux source tree with your custom ftrace tracepoints.
5753 . Boot the resulting Linux image on your target system.
5754 +
5755 Confirm that the tracepoints exist by looking for their names in the
5756 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5757 is your subsystem name.
5758
5759 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5760 +
5761 --
5762 [role="term"]
5763 ----
5764 $ cd $(mktemp -d) &&
5765   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
5766   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
5767   cd lttng-modules-2.13.*
5768 ----
5769 --
5770
5771 . In dir:{instrumentation/events/lttng-module}, relative to the root
5772   of the LTTng-modules source tree, create a header file named
5773   +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5774   LTTng-modules tracepoint definitions using the LTTng-modules
5775   macros in it.
5776 +
5777 Start with this template:
5778 +
5779 --
5780 [source,c]
5781 .path:{instrumentation/events/lttng-module/my_subsys.h}
5782 ----
5783 #undef TRACE_SYSTEM
5784 #define TRACE_SYSTEM my_subsys
5785
5786 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5787 #define _LTTNG_MY_SUBSYS_H
5788
5789 #include "../../../probes/lttng-tracepoint-event.h"
5790 #include <linux/tracepoint.h>
5791
5792 LTTNG_TRACEPOINT_EVENT(
5793     /*
5794      * Format is identical to the TRACE_EVENT() version for the three
5795      * following macro parameters:
5796      */
5797     my_subsys_my_event,
5798     TP_PROTO(int my_int, const char *my_string),
5799     TP_ARGS(my_int, my_string),
5800
5801     /* LTTng-modules specific macros */
5802     TP_FIELDS(
5803         ctf_integer(int, my_int_field, my_int)
5804         ctf_string(my_bar_field, my_bar)
5805     )
5806 )
5807
5808 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5809
5810 #include "../../../probes/define_trace.h"
5811 ----
5812 --
5813 +
5814 The entries in the `TP_FIELDS()` section are the list of fields for the
5815 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5816 the `TRACE_EVENT()` ftrace macro.
5817 +
5818 See ``<<lttng-modules-tp-fields,Tracepoint fields macros>>'' for a
5819 complete description of the available `ctf_*()` macros.
5820
5821 . Create the kernel module C{nbsp}source file of the LTTng-modules
5822   probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5823   subsystem name:
5824 +
5825 --
5826 [source,c]
5827 .path:{probes/lttng-probe-my-subsys.c}
5828 ----
5829 #include <linux/module.h>
5830 #include "../lttng-tracer.h"
5831
5832 /*
5833  * Build-time verification of mismatch between mainline
5834  * TRACE_EVENT() arguments and the LTTng-modules adaptation
5835  * layer LTTNG_TRACEPOINT_EVENT() arguments.
5836  */
5837 #include <trace/events/my_subsys.h>
5838
5839 /* Create LTTng tracepoint probes */
5840 #define LTTNG_PACKAGE_BUILD
5841 #define CREATE_TRACE_POINTS
5842 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5843
5844 #include "../instrumentation/events/lttng-module/my_subsys.h"
5845
5846 MODULE_LICENSE("GPL and additional rights");
5847 MODULE_AUTHOR("Your name <your-email>");
5848 MODULE_DESCRIPTION("LTTng my_subsys probes");
5849 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5850     __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5851     __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5852     LTTNG_MODULES_EXTRAVERSION);
5853 ----
5854 --
5855
5856 . Edit path:{probes/KBuild} and add your new kernel module object
5857   next to the existing ones:
5858 +
5859 --
5860 [source,make]
5861 .path:{probes/KBuild}
5862 ----
5863 # ...
5864
5865 obj-m += lttng-probe-module.o
5866 obj-m += lttng-probe-power.o
5867
5868 obj-m += lttng-probe-my-subsys.o
5869
5870 # ...
5871 ----
5872 --
5873
5874 . Build and install the LTTng kernel modules:
5875 +
5876 --
5877 [role="term"]
5878 ----
5879 $ make KERNELDIR=/path/to/linux
5880 # make modules_install && depmod -a
5881 ----
5882 --
5883 +
5884 Replace `/path/to/linux` with the path to the Linux source tree where
5885 you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
5886
5887 Note that you can also use the
5888 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5889 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5890 C{nbsp}code that need to be executed before LTTng records the event
5891 fields.
5892
5893 The best way to learn how to use the previous LTTng-modules macros is to
5894 inspect the existing LTTng-modules tracepoint definitions in the
5895 dir:{instrumentation/events/lttng-module} header files. Compare them
5896 with the Linux kernel mainline versions in the
5897 dir:{include/trace/events} directory of the Linux source tree.
5898
5899
5900 [role="since-2.7"]
5901 [[lttng-tracepoint-event-code]]
5902 ===== Use custom C code to access the data for tracepoint fields
5903
5904 Although we recommended to always use the
5905 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5906 the arguments and fields of an LTTng-modules tracepoint when possible,
5907 sometimes you need a more complex process to access the data that the
5908 tracer records as event record fields. In other words, you need local
5909 variables and multiple C{nbsp}statements instead of simple
5910 argument-based expressions that you pass to the
5911 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5912
5913 Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5914 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5915 a block of C{nbsp}code to be executed before LTTng records the fields.
5916 The structure of this macro is:
5917
5918 [source,c]
5919 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5920 ----
5921 LTTNG_TRACEPOINT_EVENT_CODE(
5922     /*
5923      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5924      * version for the following three macro parameters:
5925      */
5926     my_subsys_my_event,
5927     TP_PROTO(int my_int, const char *my_string),
5928     TP_ARGS(my_int, my_string),
5929
5930     /* Declarations of custom local variables */
5931     TP_locvar(
5932         int a = 0;
5933         unsigned long b = 0;
5934         const char *name = "(undefined)";
5935         struct my_struct *my_struct;
5936     ),
5937
5938     /*
5939      * Custom code which uses both tracepoint arguments
5940      * (in TP_ARGS()) and local variables (in TP_locvar()).
5941      *
5942      * Local variables are actually members of a structure pointed
5943      * to by the special variable tp_locvar.
5944      */
5945     TP_code(
5946         if (my_int) {
5947             tp_locvar->a = my_int + 17;
5948             tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5949             tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5950             tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5951             put_my_struct(tp_locvar->my_struct);
5952
5953             if (tp_locvar->b) {
5954                 tp_locvar->a = 1;
5955             }
5956         }
5957     ),
5958
5959     /*
5960      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5961      * version for this, except that tp_locvar members can be
5962      * used in the argument expression parameters of
5963      * the ctf_*() macros.
5964      */
5965     TP_FIELDS(
5966         ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5967         ctf_integer(int, my_struct_a, tp_locvar->a)
5968         ctf_string(my_string_field, my_string)
5969         ctf_string(my_struct_name, tp_locvar->name)
5970     )
5971 )
5972 ----
5973
5974 IMPORTANT: The C code defined in `TP_code()` must not have any side
5975 effects when executed. In particular, the code must not allocate
5976 memory or get resources without deallocating this memory or putting
5977 those resources afterwards.
5978
5979
5980 [[instrumenting-linux-kernel-tracing]]
5981 ==== Load and unload a custom probe kernel module
5982
5983 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5984 kernel module>> in the kernel before it can emit LTTng events.
5985
5986 To load the default probe kernel modules and a custom probe kernel
5987 module:
5988
5989 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5990   probe modules to load when starting a root <<lttng-sessiond,session
5991   daemon>>:
5992 +
5993 --
5994 .Load the `my_subsys`, `usb`, and the default probe modules.
5995 ====
5996 [role="term"]
5997 ----
5998 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5999 ----
6000 ====
6001 --
6002 +
6003 You only need to pass the subsystem name, not the whole kernel module
6004 name.
6005
6006 To load _only_ a given custom probe kernel module:
6007
6008 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
6009   modules to load when starting a root session daemon:
6010 +
6011 --
6012 .Load only the `my_subsys` and `usb` probe modules.
6013 ====
6014 [role="term"]
6015 ----
6016 # lttng-sessiond --kmod-probes=my_subsys,usb
6017 ----
6018 ====
6019 --
6020
6021 To confirm that a probe module is loaded:
6022
6023 * Use man:lsmod(8):
6024 +
6025 --
6026 [role="term"]
6027 ----
6028 $ lsmod | grep lttng_probe_usb
6029 ----
6030 --
6031
6032 To unload the loaded probe modules:
6033
6034 * Kill the session daemon with `SIGTERM`:
6035 +
6036 --
6037 [role="term"]
6038 ----
6039 # pkill lttng-sessiond
6040 ----
6041 --
6042 +
6043 You can also use the `--remove` option of man:modprobe(8) if the session
6044 daemon terminates abnormally.
6045
6046
6047 [[controlling-tracing]]
6048 == Tracing control
6049
6050 Once an application or a Linux kernel is <<instrumenting,instrumented>>
6051 for LTTng tracing, you can _trace_ it.
6052
6053 In the LTTng context, _tracing_ means making sure that LTTng attempts to
6054 execute some action(s) when a CPU executes an instrumentation point.
6055
6056 This section is divided in topics on how to use the various
6057 <<plumbing,components of LTTng>>, in particular the
6058 <<lttng-cli,cmd:lttng command-line tool>>, to _control_ the LTTng
6059 daemons and tracers.
6060
6061 NOTE: In the following subsections, we refer to an man:lttng(1) command
6062 using its man page name. For example, instead of ``Run the `create`
6063 command to'', we write ``Run the man:lttng-create(1) command to''.
6064
6065
6066 [[start-sessiond]]
6067 === Start a session daemon
6068
6069 In some situations, you need to run a <<lttng-sessiond,session daemon>>
6070 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
6071 command-line tool.
6072
6073 You will see the following error when you run a command while no session
6074 daemon is running:
6075
6076 ----
6077 Error: No session daemon is available
6078 ----
6079
6080 The only command that automatically runs a session daemon is
6081 man:lttng-create(1), which you use to
6082 <<creating-destroying-tracing-sessions,create a recording session>>. While
6083 this could be your most used first operation, sometimes it's not. Some
6084 examples are:
6085
6086 * <<list-instrumentation-points,List the available instrumentation points>>.
6087 * <<saving-loading-tracing-session,Load a recording session configuration>>.
6088 * <<add-event-rule-matches-trigger,Add a trigger>>.
6089
6090 All the examples above don't require a recording session to operate on.
6091
6092 [[tracing-group]] Each Unix user can have its own running session daemon
6093 to use the user space LTTng tracer. The session daemon that the `root`
6094 user starts is the only one allowed to control the LTTng kernel tracer.
6095 Members of the Unix _tracing group_ may connect to and control the root
6096 session daemon, even for user space tracing. See the ``Session daemon
6097 connection'' section of man:lttng(1) to learn more about the Unix
6098 tracing group.
6099
6100 To start a user session daemon:
6101
6102 * Run man:lttng-sessiond(8):
6103 +
6104 --
6105 [role="term"]
6106 ----
6107 $ lttng-sessiond --daemonize
6108 ----
6109 --
6110
6111 To start the root session daemon:
6112
6113 * Run man:lttng-sessiond(8) as the `root` user:
6114 +
6115 --
6116 [role="term"]
6117 ----
6118 # lttng-sessiond --daemonize
6119 ----
6120 --
6121
6122 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
6123 start the session daemon in foreground.
6124
6125 To stop a session daemon, kill its process (see man:kill(1)) with the
6126 standard `TERM` signal.
6127
6128 Note that some Linux distributions could manage the LTTng session daemon
6129 as a service. In this case, we suggest that you use the service manager
6130 to start, restart, and stop session daemons.
6131
6132
6133 [[creating-destroying-tracing-sessions]]
6134 === Create and destroy a recording session
6135
6136 Many LTTng control operations happen in the scope of a
6137 <<tracing-session,recording session>>, which is the dialogue between the
6138 <<lttng-sessiond,session daemon>> and you for everything related to
6139 <<event,event recording>>.
6140
6141 To create a recording session with a generated name:
6142
6143 * Use the man:lttng-create(1) command:
6144 +
6145 --
6146 [role="term"]
6147 ----
6148 $ lttng create
6149 ----
6150 --
6151
6152 The name of the created recording session is `auto` followed by the
6153 creation date.
6154
6155 To create a recording session with a specific name:
6156
6157 * Use the optional argument of the man:lttng-create(1) command:
6158 +
6159 --
6160 [role="term"]
6161 ----
6162 $ lttng create SESSION
6163 ----
6164 --
6165 +
6166 Replace +__SESSION__+ with your specific recording session name.
6167
6168 In <<local-mode,local mode>>, LTTng writes the traces of a recording
6169 session to the +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+
6170 directory by default, where +__NAME__+ is the name of the recording
6171 session. Note that the env:LTTNG_HOME environment variable defaults to
6172 `$HOME` if not set.
6173
6174 To output LTTng traces to a non-default location:
6175
6176 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1)
6177   command:
6178 +
6179 --
6180 [role="term"]
6181 ----
6182 $ lttng create my-session --output=/tmp/some-directory
6183 ----
6184 --
6185
6186 You may create as many recording sessions as you wish.
6187
6188 To list all the existing recording sessions for your Unix user, or for
6189 all users if your Unix user is `root`:
6190
6191 * Use the man:lttng-list(1) command:
6192 +
6193 --
6194 [role="term"]
6195 ----
6196 $ lttng list
6197 ----
6198 --
6199
6200 [[cur-tracing-session]]When you create a recording session, the
6201 man:lttng-create(1) command sets it as the _current recording session_.
6202 The following man:lttng(1) commands operate on the current recording
6203 session when you don't specify one:
6204
6205 [role="list-3-cols"]
6206 * man:lttng-add-context(1)
6207 * man:lttng-clear(1)
6208 * man:lttng-destroy(1)
6209 * man:lttng-disable-channel(1)
6210 * man:lttng-disable-event(1)
6211 * man:lttng-disable-rotation(1)
6212 * man:lttng-enable-channel(1)
6213 * man:lttng-enable-event(1)
6214 * man:lttng-enable-rotation(1)
6215 * man:lttng-load(1)
6216 * man:lttng-regenerate(1)
6217 * man:lttng-rotate(1)
6218 * man:lttng-save(1)
6219 * man:lttng-snapshot(1)
6220 * man:lttng-start(1)
6221 * man:lttng-status(1)
6222 * man:lttng-stop(1)
6223 * man:lttng-track(1)
6224 * man:lttng-untrack(1)
6225 * man:lttng-view(1)
6226
6227 To change the current recording session:
6228
6229 * Use the man:lttng-set-session(1) command:
6230 +
6231 --
6232 [role="term"]
6233 ----
6234 $ lttng set-session SESSION
6235 ----
6236 --
6237 +
6238 Replace +__SESSION__+ with the name of the new current recording session.
6239
6240 When you're done recording in a given recording session, destroy it.
6241 This operation frees the resources taken by the recording session to
6242 destroy; it doesn't destroy the trace data that LTTng wrote for this
6243 recording session (see ``<<clear,Clear a recording session>>'' for one
6244 way to do this).
6245
6246 To destroy the current recording session:
6247
6248 * Use the man:lttng-destroy(1) command:
6249 +
6250 --
6251 [role="term"]
6252 ----
6253 $ lttng destroy
6254 ----
6255 --
6256
6257 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
6258 implicitly (see ``<<basic-tracing-session-control,Start and stop a
6259 recording session>>''). You need to stop recording to make LTTng flush the
6260 remaining trace data and make the trace readable.
6261
6262
6263 [[list-instrumentation-points]]
6264 === List the available instrumentation points
6265
6266 The <<lttng-sessiond,session daemon>> can query the running instrumented
6267 user applications and the Linux kernel to get a list of available
6268 instrumentation points:
6269
6270 * LTTng tracepoints and system calls for the Linux kernel
6271   <<domain,tracing domain>>.
6272
6273 * LTTng tracepoints for the user space tracing domain.
6274
6275 To list the available instrumentation points:
6276
6277 . <<start-sessiond,Make sure>> there's a running
6278   <<lttng-sessiond,session daemon>> to which your Unix user can
6279   connect.
6280
6281 . Use the man:lttng-list(1) command with the option of the requested
6282   tracing domain amongst:
6283 +
6284 --
6285 opt:lttng-list(1):--kernel::
6286     Linux kernel tracepoints.
6287 +
6288 Your Unix user must be `root`, or it must be a member of the Unix
6289 <<tracing-group,tracing group>>.
6290
6291 opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
6292     Linux kernel system calls.
6293 +
6294 Your Unix user must be `root`, or it must be a member of the Unix
6295 <<tracing-group,tracing group>>.
6296
6297 opt:lttng-list(1):--userspace::
6298     User space tracepoints.
6299
6300 opt:lttng-list(1):--jul::
6301    `java.util.logging` loggers.
6302
6303 opt:lttng-list(1):--log4j::
6304     Apache log4j loggers.
6305
6306 opt:lttng-list(1):--python::
6307     Python loggers.
6308 --
6309
6310 .List the available user space tracepoints.
6311 ====
6312 [role="term"]
6313 ----
6314 $ lttng list --userspace
6315 ----
6316 ====
6317
6318 .List the available Linux kernel system calls.
6319 ====
6320 [role="term"]
6321 ----
6322 $ lttng list --kernel --syscall
6323 ----
6324 ====
6325
6326
6327 [[enabling-disabling-events]]
6328 === Create and enable a recording event rule
6329
6330 Once you <<creating-destroying-tracing-sessions,create a recording
6331 session>>, you can create <<event,recording event rules>> with the
6332 man:lttng-enable-event(1) command.
6333
6334 The man:lttng-enable-event(1) command always attaches an event rule to a
6335 <<channel,channel>> on creation. The command can create a _default
6336 channel_, named `channel0`, for you. The man:lttng-enable-event(1)
6337 command reuses the default channel each time you run it for the same
6338 tracing domain and session.
6339
6340 A recording event rule is always enabled at creation time.
6341
6342 The following examples show how to combine the command-line arguments of
6343 the man:lttng-enable-event(1) command to create simple to more complex
6344 recording event rules within the <<cur-tracing-session,current recording
6345 session>>.
6346
6347 .Create a recording event rule matching specific Linux kernel tracepoint events (default channel).
6348 ====
6349 [role="term"]
6350 ----
6351 # lttng enable-event --kernel sched_switch
6352 ----
6353 ====
6354
6355 .Create a recording event rule matching Linux kernel system call events with four specific names (default channel).
6356 ====
6357 [role="term"]
6358 ----
6359 # lttng enable-event --kernel --syscall open,write,read,close
6360 ----
6361 ====
6362
6363 .Create recording event rules matching tracepoint events which satisfy a filter expressions (default channel).
6364 ====
6365 [role="term"]
6366 ----
6367 # lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
6368 ----
6369
6370 [role="term"]
6371 ----
6372 # lttng enable-event --kernel --all \
6373                      --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
6374 ----
6375
6376 [role="term"]
6377 ----
6378 $ lttng enable-event --jul my_logger \
6379                      --filter='$app.retriever:cur_msg_id > 3'
6380 ----
6381
6382 IMPORTANT: Make sure to always single-quote the filter string when you
6383 run man:lttng(1) from a shell.
6384
6385 See also ``<<pid-tracking,Allow specific processes to record events>>''
6386 which offers another, more efficient filtering mechanism for process ID,
6387 user ID, and group ID attributes.
6388 ====
6389
6390 .Create a recording event rule matching any user space event from the `my_app` tracepoint provider and with a log level range (default channel).
6391 ====
6392 [role="term"]
6393 ----
6394 $ lttng enable-event --userspace my_app:'*' --loglevel=INFO
6395 ----
6396
6397 IMPORTANT: Make sure to always single-quote the wildcard character when
6398 you run man:lttng(1) from a shell.
6399 ====
6400
6401 .Create a recording event rule matching user space events named specifically, but with name exclusions (default channel).
6402 ====
6403 [role="term"]
6404 ----
6405 $ lttng enable-event --userspace my_app:'*' \
6406                      --exclude=my_app:set_user,my_app:handle_sig
6407 ----
6408 ====
6409
6410 .Create a recording event rule matching any Apache log4j event with a specific log level (default channel).
6411 ====
6412 [role="term"]
6413 ----
6414 $ lttng enable-event --log4j --all --loglevel-only=WARN
6415 ----
6416 ====
6417
6418 .Create a recording event rule, attached to a specific channel, and matching user space tracepoint events named `my_app:my_tracepoint`.
6419 ====
6420 [role="term"]
6421 ----
6422 $ lttng enable-event --userspace my_app:my_tracepoint \
6423                      --channel=my-channel
6424 ----
6425 ====
6426
6427 .Create a recording event rule matching user space probe events for the `malloc` function entry in path:{/usr/lib/libc.so.6}:
6428 ====
6429 [role="term"]
6430 ----
6431 # lttng enable-event --kernel \
6432                      --userspace-probe=/usr/lib/libc.so.6:malloc \
6433                      libc_malloc
6434 ----
6435 ====
6436
6437 .Create a recording event rule matching user space probe events for the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
6438 ====
6439 [role="term"]
6440 ----
6441 # lttng enable-event --kernel \
6442                      --userspace-probe=sdt:serv:server:accept_request \
6443                      server_accept_request
6444 ----
6445 ====
6446
6447 The recording event rules of a given channel form a whitelist: as soon
6448 as an event rule matches an event, LTTng emits it _once_ and therefore
6449 <<channel-overwrite-mode-vs-discard-mode,can>> record it. For example,
6450 the following rules both match user space tracepoint events named
6451 `my_app:my_tracepoint` with an `INFO` log level:
6452
6453 [role="term"]
6454 ----
6455 $ lttng enable-event --userspace my_app:my_tracepoint
6456 $ lttng enable-event --userspace my_app:my_tracepoint \
6457                      --loglevel=INFO
6458 ----
6459
6460 The second recording event rule is redundant: the first one includes the
6461 second one.
6462
6463
6464 [[disable-event-rule]]
6465 === Disable a recording event rule
6466
6467 To disable a <<event,recording event rule>> that you
6468 <<enabling-disabling-events,created>> previously, use the
6469 man:lttng-disable-event(1) command.
6470
6471 man:lttng-disable-event(1) can only find recording event rules to
6472 disable by their <<instrumentation-point-types,instrumentation point
6473 type>> and event name conditions. Therefore, you cannot disable
6474 recording event rules having a specific instrumentation point log level
6475 condition, for example.
6476
6477 LTTng doesn't emit (and, therefore, won't record) an event which only
6478 _disabled_ recording event rules match.
6479
6480 .Disable event rules matching Python logging events from the `my-logger` logger (default <<channel,channel>>, <<cur-tracing-session,current recording session>>).
6481 ====
6482 [role="term"]
6483 ----
6484 $ lttng disable-event --python my-logger
6485 ----
6486 ====
6487
6488 .Disable event rules matching all `java.util.logging` events (default channel, recording session `my-session`).
6489 ====
6490 [role="term"]
6491 ----
6492 $ lttng disable-event --jul --session=my-session '*'
6493 ----
6494 ====
6495
6496 .Disable _all_ the Linux kernel recording event rules (channel `my-chan`, current recording session).
6497 ====
6498 The opt:lttng-disable-event(1):--all-events option isn't, like the
6499 opt:lttng-enable-event(1):--all option of the man:lttng-enable-event(1)
6500 command, an alias for the event name globbing pattern `*`: it disables
6501 _all_ the recording event rules of a given channel.
6502
6503 [role="term"]
6504 ----
6505 # lttng disable-event --kernel --channel=my-chan --all-events
6506 ----
6507 ====
6508
6509 NOTE: You can't _remove_ a recording event rule once you create it.
6510
6511
6512 [[status]]
6513 === Get the status of a recording session
6514
6515 To get the status of the <<cur-tracing-session,current recording
6516 session>>, that is, its parameters, its channels, recording event rules,
6517 and their attributes:
6518
6519 * Use the man:lttng-status(1) command:
6520 +
6521 --
6522 [role="term"]
6523 ----
6524 $ lttng status
6525 ----
6526 --
6527
6528 To get the status of any recording session:
6529
6530 * Use the man:lttng-list(1) command with the name of the recording
6531   session:
6532 +
6533 --
6534 [role="term"]
6535 ----
6536 $ lttng list SESSION
6537 ----
6538 --
6539 +
6540 Replace +__SESSION__+ with the recording session name.
6541
6542
6543 [[basic-tracing-session-control]]
6544 === Start and stop a recording session
6545
6546 Once you <<creating-destroying-tracing-sessions,create a recording
6547 session>> and <<enabling-disabling-events,create one or more recording
6548 event rules>>, you can start and stop the tracers for this recording
6549 session.
6550
6551 To start the <<cur-tracing-session,current recording session>>:
6552
6553 * Use the man:lttng-start(1) command:
6554 +
6555 --
6556 [role="term"]
6557 ----
6558 $ lttng start
6559 ----
6560 --
6561
6562 LTTng is flexible: you can launch user applications before or after you
6563 start the tracers. An LTTng tracer only <<event,records an event>> if a
6564 recording event rule matches it, which means the tracer is active.
6565
6566 The `start-session` <<trigger,trigger>> action can also start a recording
6567 session.
6568
6569 To stop the current recording session:
6570
6571 * Use the man:lttng-stop(1) command:
6572 +
6573 --
6574 [role="term"]
6575 ----
6576 $ lttng stop
6577 ----
6578 --
6579 +
6580 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6581 records>> or lost sub-buffers since the last time you ran
6582 man:lttng-start(1), the man:lttng-stop(1) command prints corresponding
6583 warnings.
6584
6585 IMPORTANT: You need to stop recording to make LTTng flush the remaining
6586 trace data and make the trace readable. Note that the
6587 man:lttng-destroy(1) command (see
6588 ``<<creating-destroying-tracing-sessions,Create and destroy a recording
6589 session>>'') also runs the man:lttng-stop(1) command implicitly.
6590
6591 The `stop-session` <<trigger,trigger>> action can also stop a recording
6592 session.
6593
6594 [role="since-2.12"]
6595 [[clear]]
6596 === Clear a recording session
6597
6598 You might need to remove all the current tracing data of one or more
6599 <<tracing-session,recording sessions>> between multiple attempts to
6600 reproduce a problem without interrupting the LTTng recording activity.
6601
6602 To clear the tracing data of the
6603 <<cur-tracing-session,current recording session>>:
6604
6605 * Use the man:lttng-clear(1) command:
6606 +
6607 --
6608 [role="term"]
6609 ----
6610 $ lttng clear
6611 ----
6612 --
6613
6614 To clear the tracing data of all the recording sessions:
6615
6616 * Use the `lttng clear` command with its opt:lttng-clear(1):--all
6617   option:
6618 +
6619 --
6620 [role="term"]
6621 ----
6622 $ lttng clear --all
6623 ----
6624 --
6625
6626
6627 [[enabling-disabling-channels]]
6628 === Create a channel
6629
6630 Once you <<creating-destroying-tracing-sessions,create a recording
6631 session>>, you can create a <<channel,channel>> with the
6632 man:lttng-enable-channel(1) command.
6633
6634 Note that LTTng can automatically create a default channel when you
6635 <<enabling-disabling-events,create a recording event rule>>.
6636 Therefore, you only need to create a channel when you need non-default
6637 attributes.
6638
6639 Specify each non-default channel attribute with a command-line
6640 option when you run the man:lttng-enable-channel(1) command.
6641
6642 You can only create a custom channel in the Linux kernel and user space
6643 <<domain,tracing domains>>: the Java/Python logging tracing domains have
6644 their own default channel which LTTng automatically creates when you
6645 <<enabling-disabling-events,create a recording event rule>>.
6646
6647 [IMPORTANT]
6648 ====
6649 As of LTTng{nbsp}{revision}, you may _not_ perform the
6650 following operations with the man:lttng-enable-channel(1) command:
6651
6652 * Change an attribute of an existing channel.
6653
6654 * Enable a disabled channel once its recording session has been
6655   <<basic-tracing-session-control,active>> at least once.
6656
6657 * Create a channel once its recording session has been active at
6658   least once.
6659
6660 * Create a user space channel with a given
6661   <<channel-buffering-schemes,buffering scheme>> and create a second
6662   user space channel with a different buffering scheme in the same
6663   recording session.
6664 ====
6665
6666 The following examples show how to combine the command-line options of
6667 the man:lttng-enable-channel(1) command to create simple to more complex
6668 channels within the <<cur-tracing-session,current recording session>>.
6669
6670 .Create a Linux kernel channel with default attributes.
6671 ====
6672 [role="term"]
6673 ----
6674 # lttng enable-channel --kernel my-channel
6675 ----
6676 ====
6677
6678 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6679 ====
6680 [role="term"]
6681 ----
6682 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6683                        --buffers-pid my-channel
6684 ----
6685 ====
6686
6687 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6688 ====
6689 <<creating-destroying-tracing-sessions,Create a recording session>>,
6690 create the channel, <<enabling-disabling-events,create a recording event
6691 rule>>, and <<basic-tracing-session-control,start recording>>:
6692
6693 [role="term"]
6694 ----
6695 $ lttng create
6696 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-chan
6697 $ lttng enable-event --userspace --channel=blocking-chan --all
6698 $ lttng start
6699 ----
6700
6701 Run an application instrumented with LTTng-UST tracepoints and allow it
6702 to block:
6703
6704 [role="term"]
6705 ----
6706 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6707 ----
6708 ====
6709
6710 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream.
6711 ====
6712 [role="term"]
6713 ----
6714 # lttng enable-channel --kernel --tracefile-count=8 \
6715                        --tracefile-size=4194304 my-channel
6716 ----
6717 ====
6718
6719 .Create a user space channel in <<overwrite-mode,overwrite>> (or ``flight recorder'') mode.
6720 ====
6721 [role="term"]
6722 ----
6723 $ lttng enable-channel --userspace --overwrite my-channel
6724 ----
6725 ====
6726
6727 .<<enabling-disabling-events,Create>> the same <<event,recording event rule>> attached to two different channels.
6728 ====
6729 [role="term"]
6730 ----
6731 $ lttng enable-event --userspace --channel=my-channel app:tp
6732 $ lttng enable-event --userspace --channel=other-channel app:tp
6733 ----
6734
6735 When a CPU executes the `app:tp` <<c-application,user space
6736 tracepoint>>, the two recording event rules above match the created
6737 event, making LTTng emit the event. Because the recording event rules
6738 are not attached to the same channel, LTTng records the event twice.
6739 ====
6740
6741
6742 [[disable-channel]]
6743 === Disable a channel
6744
6745 To disable a specific channel that you
6746 <<enabling-disabling-channels,created>> previously, use the
6747 man:lttng-disable-channel(1) command.
6748
6749 .Disable a specific Linux kernel channel (<<cur-tracing-session,current recording session>>).
6750 ====
6751 [role="term"]
6752 ----
6753 # lttng disable-channel --kernel my-channel
6754 ----
6755 ====
6756
6757 An enabled channel is an implicit <<event,recording event rule>>
6758 condition.
6759
6760 NOTE: As of LTTng{nbsp}{revision}, you may _not_ enable a disabled
6761 channel once its recording session has been
6762 <<basic-tracing-session-control,started>> at least once.
6763
6764
6765 [[adding-context]]
6766 === Add context fields to be recorded to the event records of a channel
6767
6768 <<event,Event record>> fields in trace files provide important
6769 information about previously emitted events, but sometimes some external
6770 context may help you solve a problem faster.
6771
6772 Examples of context fields are:
6773
6774 * The **process ID**, **thread ID**, **process name**, and
6775   **process priority** of the thread from which LTTng emits the event.
6776
6777 * The **hostname** of the system on which LTTng emits the event.
6778
6779 * The Linux kernel and user call stacks (since LTTng{nbsp}2.11).
6780
6781 * The current values of many possible **performance counters** using
6782   perf, for example:
6783
6784 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6785 ** Cache misses.
6786 ** Branch instructions, misses, and loads.
6787 ** CPU faults.
6788
6789 * Any state defined at the application level (supported for the
6790   `java.util.logging` and Apache log4j <<domain,tracing domains>>).
6791
6792 To get the full list of available context fields:
6793
6794 * Use the opt:lttng-add-context(1):--list option of the
6795   man:lttng-add-context(1) command:
6796 +
6797 [role="term"]
6798 ----
6799 $ lttng add-context --list
6800 ----
6801
6802 .Add context fields to be recorded to the event records of all the <<channel,channels>> of the <<cur-tracing-session,current recording session>>.
6803 ====
6804 The following command line adds the virtual process identifier and the
6805 per-thread CPU cycles count fields to all the user space channels of the
6806 current recording session.
6807
6808 [role="term"]
6809 ----
6810 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6811 ----
6812 ====
6813
6814 .Add performance counter context fields by raw ID
6815 ====
6816 See man:lttng-add-context(1) for the exact format of the context field
6817 type, which is partly compatible with the format used in
6818 man:perf-record(1).
6819
6820 [role="term"]
6821 ----
6822 # lttng add-context --userspace --type=perf:thread:raw:r0110:test
6823 # lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6824 ----
6825 ====
6826
6827 .Add context fields to be recorded to the event records of a specific channel.
6828 ====
6829 The following command line adds the thread identifier and user call
6830 stack context fields to the Linux kernel channel named `my-channel` of
6831 the <<cur-tracing-session,current recording session>>.
6832
6833 [role="term"]
6834 ----
6835 # lttng add-context --kernel --channel=my-channel \
6836                     --type=tid --type=callstack-user
6837 ----
6838 ====
6839
6840 .Add an <<java-application-context,application-specific context field>> to be recorded to the event records of a specific channel.
6841 ====
6842 The following command line makes sure LTTng writes the `cur_msg_id`
6843 context field of the `retriever` context retriever to all the Java
6844 logging <<event,event records>> of the channel named `my-channel`:
6845
6846 [role="term"]
6847 ----
6848 # lttng add-context --kernel --channel=my-channel \
6849                     --type='$app:retriever:cur_msg_id'
6850 ----
6851
6852 IMPORTANT: Make sure to always single-quote the `$` character when you
6853 run man:lttng-add-context(1) from a shell.
6854 ====
6855
6856 NOTE: You can't undo what the man:lttng-add-context(1) command does.
6857
6858
6859 [role="since-2.7"]
6860 [[pid-tracking]]
6861 === Allow specific processes to record events
6862
6863 It's often useful to only allow processes with specific attributes to
6864 record events. For example, you may wish to record all the system calls
6865 which a given process makes (à la man:strace(1)).
6866
6867 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6868 purpose. Both commands operate on _inclusion sets_ of process
6869 attributes. The available process attribute types are:
6870
6871 Linux kernel <<domain,tracing domain>>::
6872 +
6873 * Process ID (PID).
6874
6875 * Virtual process ID (VPID).
6876 +
6877 This is the PID as seen by the application.
6878
6879 * Unix user ID (UID).
6880
6881 * Virtual Unix user ID (VUID).
6882 +
6883 This is the UID as seen by the application.
6884
6885 * Unix group ID (GID).
6886
6887 * Virtual Unix group ID (VGID).
6888 +
6889 This is the GID as seen by the application.
6890
6891 User space tracing domain::
6892 +
6893 * VPID
6894 * VUID
6895 * VGID
6896
6897 A <<tracing-session,recording session>> has nine process
6898 attribute inclusion sets: six for the Linux kernel <<domain,tracing domain>>
6899 and three for the user space tracing domain.
6900
6901 For a given recording session, a process{nbsp}__P__ is allowed to record
6902 LTTng events for a given <<domain,tracing domain>>{nbsp}__D__ if _all_
6903 the attributes of{nbsp}__P__ are part of the inclusion sets
6904 of{nbsp}__D__.
6905
6906 Whether a process is allowed or not to record LTTng events is an
6907 implicit condition of all <<event,recording event rules>>. Therefore, if
6908 LTTng creates an event{nbsp}__E__ for a given process, but this process
6909 may not record events, then no recording event rule matches{nbsp}__E__,
6910 which means LTTng won't emit and record{nbsp}__E__.
6911
6912 When you <<creating-destroying-tracing-sessions,create a recording
6913 session>>, all its process attribute inclusion sets contain all the
6914 possible values. In other words, all processes are allowed to record
6915 events.
6916
6917 Add values to an inclusion set with the man:lttng-track(1) command and
6918 remove values with the man:lttng-untrack(1) command.
6919
6920 [NOTE]
6921 ====
6922 The process attribute values are _numeric_.
6923
6924 Should a process with a given ID (part of an inclusion set), for
6925 example, exit, and then a new process be given this same ID, then the
6926 latter would also be allowed to record events.
6927
6928 With the man:lttng-track(1) command, you can add Unix user and group
6929 _names_ to the user and group inclusion sets: the
6930 <<lttng-sessiond,session daemon>> finds the corresponding UID, VUID,
6931 GID, or VGID once on _addition_ to the inclusion set. This means that if
6932 you rename the user or group after you run the man:lttng-track(1)
6933 command, its user/group ID remains part of the inclusion sets.
6934 ====
6935
6936 .Allow processes to record events based on their virtual process ID (VPID).
6937 ====
6938 For the sake of the following example, assume the target system has
6939 16{nbsp}possible VPIDs.
6940
6941 When you
6942 <<creating-destroying-tracing-sessions,create a recording session>>,
6943 the user space VPID inclusion set contains _all_ the possible VPIDs:
6944
6945 [role="img-100"]
6946 .The VPID inclusion set is full.
6947 image::track-all.png[]
6948
6949 When the inclusion set is full and you run the man:lttng-track(1)
6950 command to specify some VPIDs, LTTng:
6951
6952 . Clears the inclusion set.
6953 . Adds the specific VPIDs to the inclusion set.
6954
6955 After:
6956
6957 [role="term"]
6958 ----
6959 $ lttng track --userspace --vpid=3,4,7,10,13
6960 ----
6961
6962 the VPID inclusion set is:
6963
6964 [role="img-100"]
6965 .The VPID inclusion set contains the VPIDs 3, 4, 7, 10, and 13.
6966 image::track-3-4-7-10-13.png[]
6967
6968 Add more VPIDs to the inclusion set afterwards:
6969
6970 [role="term"]
6971 ----
6972 $ lttng track --userspace --vpid=1,15,16
6973 ----
6974
6975 The result is:
6976
6977 [role="img-100"]
6978 .VPIDs 1, 15, and 16 are added to the inclusion set.
6979 image::track-1-3-4-7-10-13-15-16.png[]
6980
6981 The man:lttng-untrack(1) command removes entries from process attribute
6982 inclusion sets. Given the previous example, the following command:
6983
6984 [role="term"]
6985 ----
6986 $ lttng untrack --userspace --vpid=3,7,10,13
6987 ----
6988
6989 leads to this VPID inclusion set:
6990
6991 [role="img-100"]
6992 .VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6993 image::track-1-4-15-16.png[]
6994
6995 You can make the VPID inclusion set full again with the
6996 opt:lttng-track(1):--all option:
6997
6998 [role="term"]
6999 ----
7000 $ lttng track --userspace --vpid --all
7001 ----
7002
7003 The result is, again:
7004
7005 [role="img-100"]
7006 .The VPID inclusion set is full.
7007 image::track-all.png[]
7008 ====
7009
7010 .Allow specific processes to record events based on their user ID (UID).
7011 ====
7012 A typical use case with process attribute inclusion sets is to start
7013 with an empty inclusion set, then <<basic-tracing-session-control,start
7014 the tracers>>, and finally add values manually while the tracers are
7015 active.
7016
7017 Use the opt:lttng-untrack(1):--all option of the
7018 man:lttng-untrack(1) command to clear the inclusion set after you
7019 <<creating-destroying-tracing-sessions,create a recording session>>, for
7020 example (with UIDs):
7021
7022 [role="term"]
7023 ----
7024 # lttng untrack --kernel --uid --all
7025 ----
7026
7027 gives:
7028
7029 [role="img-100"]
7030 .The UID inclusion set is empty.
7031 image::untrack-all.png[]
7032
7033 If the LTTng tracer runs with this inclusion set configuration, it
7034 records no events within the <<cur-tracing-session,current recording
7035 session>> because no processes is allowed to do so. Use the
7036 man:lttng-track(1) command as usual to add specific values to the UID
7037 inclusion set when you need to, for example:
7038
7039 [role="term"]
7040 ----
7041 # lttng track --kernel --uid=http,11
7042 ----
7043
7044 Result:
7045
7046 [role="img-100"]
7047 .UIDs 6 (`http`) and 11 are part of the UID inclusion set.
7048 image::track-6-11.png[]
7049 ====
7050
7051
7052 [role="since-2.5"]
7053 [[saving-loading-tracing-session]]
7054 === Save and load recording session configurations
7055
7056 Configuring a <<tracing-session,recording session>> can be long. Some of
7057 the tasks involved are:
7058
7059 * <<enabling-disabling-channels,Create channels>> with
7060   specific attributes.
7061
7062 * <<adding-context,Add context fields>> to be recorded to the
7063   <<event,event records>> of specific channels.
7064
7065 * <<enabling-disabling-events,Create recording event rules>> with
7066   specific log level, filter, and other conditions.
7067
7068 If you use LTTng to solve real world problems, chances are you have to
7069 record events using the same recording session setup over and over,
7070 modifying a few variables each time in your instrumented program or
7071 environment.
7072
7073 To avoid constant recording session reconfiguration, the man:lttng(1)
7074 command-line tool can save and load recording session configurations
7075 to/from XML files.
7076
7077 To save a given recording session configuration:
7078
7079 * Use the man:lttng-save(1) command:
7080 +
7081 --
7082 [role="term"]
7083 ----
7084 $ lttng save SESSION
7085 ----
7086 --
7087 +
7088 Replace +__SESSION__+ with the name of the recording session to save.
7089
7090 LTTng saves recording session configurations to
7091 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
7092 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. See
7093 man:lttng-save(1) to learn more about the recording session configuration
7094 output path.
7095
7096 LTTng saves all configuration parameters, for example:
7097
7098 * The recording session name.
7099 * The trace data output path.
7100 * The <<channel,channels>>, with their state and all their attributes.
7101 * The context fields you added to channels.
7102 * The <<event,recording event rules>> with their state and conditions.
7103
7104 To load a recording session:
7105
7106 * Use the man:lttng-load(1) command:
7107 +
7108 --
7109 [role="term"]
7110 ----
7111 $ lttng load SESSION
7112 ----
7113 --
7114 +
7115 Replace +__SESSION__+ with the name of the recording session to load.
7116
7117 When LTTng loads a configuration, it restores your saved recording session
7118 as if you just configured it manually.
7119
7120 You can also save and load many sessions at a time; see
7121 man:lttng-save(1) and man:lttng-load(1) to learn more.
7122
7123
7124 [[sending-trace-data-over-the-network]]
7125 === Send trace data over the network
7126
7127 LTTng can send the recorded trace data of a <<tracing-session,recording
7128 session>> to a remote system over the network instead of writing it to
7129 the local file system.
7130
7131 To send the trace data over the network:
7132
7133 . On the _remote_ system (which can also be the target system),
7134   start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
7135 +
7136 --
7137 [role="term"]
7138 ----
7139 $ lttng-relayd
7140 ----
7141 --
7142
7143 . On the _target_ system, create a recording session
7144   <<net-streaming-mode,configured>> to send trace data over the network:
7145 +
7146 --
7147 [role="term"]
7148 ----
7149 $ lttng create my-session --set-url=net://remote-system
7150 ----
7151 --
7152 +
7153 Replace +__remote-system__+ with the host name or IP address of the
7154 remote system. See man:lttng-create(1) for the exact URL format.
7155
7156 . On the target system, use the man:lttng(1) command-line tool as usual.
7157 +
7158 When recording is <<basic-tracing-session-control,active>>, the
7159 <<lttng-consumerd,consumer daemon>> of the target sends the contents of
7160 <<channel,sub-buffers>> to the remote relay daemon instead of flushing
7161 them to the local file system. The relay daemon writes the received
7162 packets to its local file system.
7163
7164 See the ``Output directory'' section of man:lttng-relayd(8) to learn
7165 where a relay daemon writes its received trace data.
7166
7167
7168 [role="since-2.4"]
7169 [[lttng-live]]
7170 === View events as LTTng records them (noch:{LTTng} live)
7171
7172 _LTTng live_ is a network protocol implemented by the
7173 <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)) to allow compatible
7174 trace readers to display or analyze <<event,event records>> as LTTng
7175 records events on the target system while recording is
7176 <<basic-tracing-session-control,active>>.
7177
7178 The relay daemon creates a _tee_: it forwards the trace data to both the
7179 local file system and to connected live readers:
7180
7181 [role="img-90"]
7182 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live reader.
7183 image::live.png[]
7184
7185 To use LTTng live:
7186
7187 . On the _target system_, create a <<tracing-session,recording session>>
7188   in _live mode_:
7189 +
7190 --
7191 [role="term"]
7192 ----
7193 $ lttng create my-session --live
7194 ----
7195 --
7196 +
7197 This operation spawns a local relay daemon.
7198
7199 . Start the live reader and configure it to connect to the relay daemon.
7200 +
7201 For example, with man:babeltrace2(1):
7202 +
7203 --
7204 [role="term"]
7205 ----
7206 $ babeltrace2 net://localhost/host/HOSTNAME/my-session
7207 ----
7208 --
7209 +
7210 Replace +__HOSTNAME__+ with the host name of the target system.
7211
7212 . Configure the recording session as usual with the man:lttng(1)
7213   command-line tool, and <<basic-tracing-session-control,start recording>>.
7214
7215 List the available live recording sessions with man:babeltrace2(1):
7216
7217 [role="term"]
7218 ----
7219 $ babeltrace2 net://localhost
7220 ----
7221
7222 You can start the relay daemon on another system. In this case, you need
7223 to specify the URL of the relay daemon when you
7224 <<creating-destroying-tracing-sessions,create the recording session>> with
7225 the opt:lttng-create(1):--set-url option of the man:lttng-create(1)
7226 command. You also need to replace +__localhost__+ in the procedure above
7227 with the host name of the system on which the relay daemon runs.
7228
7229
7230 [role="since-2.3"]
7231 [[taking-a-snapshot]]
7232 === Take a snapshot of the current sub-buffers of a recording session
7233
7234 The normal behavior of LTTng is to append full sub-buffers to growing
7235 trace data files. This is ideal to keep a full history of the events
7236 which the target system emitted, but it can represent too much data in
7237 some situations.
7238
7239 For example, you may wish to have LTTng record your application
7240 continuously until some critical situation happens, in which case you
7241 only need the latest few recorded events to perform the desired
7242 analysis, not multi-gigabyte trace files.
7243
7244 With the man:lttng-snapshot(1) command, you can take a _snapshot_ of the
7245 current <<channel,sub-buffers>> of a given <<tracing-session,recording
7246 session>>. LTTng can write the snapshot to the local file system or send
7247 it over the network.
7248
7249 [role="img-100"]
7250 .A snapshot is a copy of the current sub-buffers, which LTTng does _not_ clear after the operation.
7251 image::snapshot.png[]
7252
7253 The snapshot feature of LTTng is similar to how a
7254 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder] or the
7255 ``roll'' mode of an oscilloscope work.
7256
7257 TIP: If you wish to create unmanaged, self-contained, non-overlapping
7258 trace chunk archives instead of a simple copy of the current
7259 sub-buffers, see the <<session-rotation,recording session rotation>>
7260 feature (available since LTTng{nbsp}2.11).
7261
7262 To take a snapshot of the <<cur-tracing-session,current recording
7263 session>>:
7264
7265 . Create a recording session in <<snapshot-mode,snapshot mode>>:
7266 +
7267 --
7268 [role="term"]
7269 ----
7270 $ lttng create my-session --snapshot
7271 ----
7272 --
7273 +
7274 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
7275 <<channel,channels>> created in this mode is automatically set to
7276 <<overwrite-mode,_overwrite_>>.
7277
7278 . Configure the recording session as usual with the man:lttng(1)
7279   command-line tool, and <<basic-tracing-session-control,start
7280   recording>>.
7281
7282 . **Optional**: When you need to take a snapshot,
7283   <<basic-tracing-session-control,stop recording>>.
7284 +
7285 You can take a snapshot when the tracers are active, but if you stop
7286 them first, you're guaranteed that the trace data in the sub-buffers
7287 doesn't change before you actually take the snapshot.
7288
7289 . Take a snapshot:
7290 +
7291 --
7292 [role="term"]
7293 ----
7294 $ lttng snapshot record --name=my-first-snapshot
7295 ----
7296 --
7297 +
7298 LTTng writes the current sub-buffers of all the channels of the
7299 <<cur-tracing-session,current recording session>> to
7300 trace files on the local file system. Those trace files have
7301 `my-first-snapshot` in their name.
7302
7303 There's no difference between the format of a normal trace file and the
7304 format of a snapshot: LTTng trace readers also support LTTng snapshots.
7305
7306 By default, LTTng writes snapshot files to the path shown by
7307
7308 [role="term"]
7309 ----
7310 $ lttng snapshot list-output
7311 ----
7312
7313 You can change this path or decide to send snapshots over the network
7314 using either:
7315
7316 . An output path or URL that you specify when you
7317   <<creating-destroying-tracing-sessions,create the recording session>>.
7318
7319 . A snapshot output path or URL that you add using the
7320   `add-output` action of the man:lttng-snapshot(1) command.
7321
7322 . An output path or URL that you provide directly to the
7323   `record` action of the man:lttng-snapshot(1) command.
7324
7325 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
7326 you specify a URL, a <<lttng-relayd,relay daemon>> must listen on a
7327 remote system (see ``<<sending-trace-data-over-the-network,Send trace
7328 data over the network>>'').
7329
7330 The `snapshot-session` <<trigger,trigger>> action can also take
7331 a recording session snapshot.
7332
7333
7334 [role="since-2.11"]
7335 [[session-rotation]]
7336 === Archive the current trace chunk (rotate a recording session)
7337
7338 The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
7339 current sub-buffers of a recording session to the file system or send them
7340 over the network. When you take a snapshot, LTTng doesn't clear the ring
7341 buffers of the recording session: if you take another snapshot immediately
7342 after, both snapshots could contain overlapping trace data.
7343
7344 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
7345 _recording session rotation_ is a feature which appends the content of the
7346 ring buffers to what's already on the file system or sent over the
7347 network since the creation of the recording session or since the last
7348 rotation, and then clears those ring buffers to avoid trace data
7349 overlaps.
7350
7351 What LTTng is about to write when performing a recording session rotation
7352 is called the _current trace chunk_. When LTTng writes or sends over the
7353 network this current trace chunk, it becomes a _trace chunk archive_.
7354 Therefore, a recording session rotation operation _archives_ the current
7355 trace chunk.
7356
7357 [role="img-100"]
7358 .A recording session rotation operation _archives_ the current trace chunk.
7359 image::rotation.png[]
7360
7361 A trace chunk archive is a self-contained LTTng trace which LTTng
7362 doesn't manage anymore: you can read it, modify it, move it, or remove
7363 it.
7364
7365 As of LTTng{nbsp}{revision}, there are three methods to perform a
7366 recording session rotation:
7367
7368 * <<immediate-rotation,Immediately>>.
7369
7370 * With a <<rotation-schedule,rotation schedule>>.
7371
7372 * Through the execution of a `rotate-session` <<trigger,trigger>>
7373   action.
7374
7375 [[immediate-rotation]]To perform an immediate rotation of the
7376 <<cur-tracing-session,current recording session>>:
7377
7378 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7379   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7380   mode>> (only those two recording session modes support recording session
7381   rotation):
7382 +
7383 --
7384 [role="term"]
7385 ----
7386 # lttng create my-session
7387 ----
7388 --
7389
7390 . <<enabling-disabling-events,Create one or more recording event rules>>
7391   and <<basic-tracing-session-control,start recording>>:
7392 +
7393 --
7394 [role="term"]
7395 ----
7396 # lttng enable-event --kernel sched_'*'
7397 # lttng start
7398 ----
7399 --
7400
7401 . When needed, immediately rotate the current recording session:
7402 +
7403 --
7404 [role="term"]
7405 ----
7406 # lttng rotate
7407 ----
7408 --
7409 +
7410 The man:lttng-rotate(1) command prints the path to the created trace
7411 chunk archive. See its manual page to learn about the format of trace
7412 chunk archive directory names.
7413 +
7414 Perform other immediate rotations while the recording session is active.
7415 It's guaranteed that all the trace chunk archives don't contain
7416 overlapping trace data. You can also perform an immediate rotation once
7417 you have <<basic-tracing-session-control,stopped>> the recording session.
7418
7419 . When you're done recording,
7420   <<creating-destroying-tracing-sessions,destroy the current recording
7421   session>>:
7422 +
7423 --
7424 [role="term"]
7425 ----
7426 # lttng destroy
7427 ----
7428 --
7429 +
7430 The recording session destruction operation creates one last trace chunk
7431 archive from the current trace chunk.
7432
7433 [[rotation-schedule]]A recording session rotation schedule is a planned
7434 rotation which LTTng performs automatically based on one of the
7435 following conditions:
7436
7437 * A timer with a configured period expires.
7438
7439 * The total size of the _flushed_ part of the current trace chunk
7440   becomes greater than or equal to a configured value.
7441
7442 To schedule a rotation of the <<cur-tracing-session,current recording
7443 session>>, set a _rotation schedule_:
7444
7445 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7446   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7447   mode>> (only those two creation modes support recording session
7448   rotation):
7449 +
7450 --
7451 [role="term"]
7452 ----
7453 # lttng create my-session
7454 ----
7455 --
7456
7457 . <<enabling-disabling-events,Create one or more recording event rules>>:
7458 +
7459 --
7460 [role="term"]
7461 ----
7462 # lttng enable-event --kernel sched_'*'
7463 ----
7464 --
7465
7466 . Set a recording session rotation schedule:
7467 +
7468 --
7469 [role="term"]
7470 ----
7471 # lttng enable-rotation --timer=10s
7472 ----
7473 --
7474 +
7475 In this example, we set a rotation schedule so that LTTng performs a
7476 recording session rotation every ten seconds.
7477 +
7478 See man:lttng-enable-rotation(1) to learn more about other ways to set a
7479 rotation schedule.
7480
7481 . <<basic-tracing-session-control,Start recording>>:
7482 +
7483 --
7484 [role="term"]
7485 ----
7486 # lttng start
7487 ----
7488 --
7489 +
7490 LTTng performs recording session rotations automatically while the
7491 recording session is active thanks to the rotation schedule.
7492
7493 . When you're done recording,
7494   <<creating-destroying-tracing-sessions,destroy the current recording
7495   session>>:
7496 +
7497 --
7498 [role="term"]
7499 ----
7500 # lttng destroy
7501 ----
7502 --
7503 +
7504 The recording session destruction operation creates one last trace chunk
7505 archive from the current trace chunk.
7506
7507 Unset a recording session rotation schedule with the
7508 man:lttng-disable-rotation(1) command.
7509
7510
7511 [role="since-2.13"]
7512 [[add-event-rule-matches-trigger]]
7513 === Add an ``event rule matches'' trigger to a session daemon
7514
7515 With the man:lttng-add-trigger(1) command, you can add a
7516 <<trigger,trigger>> to a <<lttng-sessiond,session daemon>>.
7517
7518 A trigger associates an LTTng tracing condition to one or more actions:
7519 when the condition is satisfied, LTTng attempts to execute the actions.
7520
7521 A trigger doesn't need any <<tracing-session,recording session>> to exist:
7522 it belongs to a session daemon.
7523
7524 As of LTTng{nbsp}{revision}, many condition types are available through
7525 the <<liblttng-ctl-lttng,`liblttng-ctl`>> C{nbsp}API, but the
7526 man:lttng-add-trigger(1) command only accepts the ``event rule matches''
7527 condition.
7528
7529 An ``event rule matches'' condition is satisfied when its event rule
7530 matches an event.
7531
7532 Unlike a <<event,recording event rule>>, the event rule of an
7533 ``event rule matches'' trigger condition has no implicit conditions,
7534 that is:
7535
7536 * It has no enabled/disabled state.
7537 * It has no attached <<channel,channel>>.
7538 * It doesn't belong to a <<tracing-session,recording session>>.
7539
7540 Both the man:lttng-add-trigger(1) and man:lttng-enable-event(1) commands
7541 accept command-line arguments to specify an <<event-rule,event rule>>.
7542 That being said, the former is a more recent command and therefore
7543 follows the common event rule specification format (see
7544 man:lttng-event-rule(7)).
7545
7546 .Start a <<tracing-session,recording session>> when an event rule matches.
7547 ====
7548 This example shows how to add the following trigger to the root
7549 <<lttng-sessiond,session daemon>>:
7550
7551 Condition::
7552     An event rule matches a Linux kernel system call event of which the
7553     name starts with `exec` and `*/ls` matches the `filename` payload
7554     field.
7555 +
7556 With such an event rule, LTTng emits an event when the cmd:ls program
7557 starts.
7558
7559 Action::
7560     <<basic-tracing-session-control,Start the recording session>>
7561     named `pitou`.
7562
7563 To add such a trigger to the root session daemon:
7564
7565 . **If there's no currently running LTTng root session daemon**, start
7566   one:
7567 +
7568 [role="term"]
7569 ----
7570 # lttng-sessiond --daemonize
7571 ----
7572
7573 . <<creating-destroying-tracing-sessions,Create a recording session>>
7574   named `pitou` and
7575   <<enabling-disabling-events,create a recording event rule>> matching
7576   all the system call events:
7577 +
7578 [role="term"]
7579 ----
7580 # lttng create pitou
7581 # lttng enable-event --kernel --syscall --all
7582 ----
7583
7584 . Add the trigger to the root session daemon:
7585 +
7586 [role="term"]
7587 ----
7588 # lttng add-trigger --condition=event-rule-matches \
7589                     --type=syscall --name='exec*' \
7590                     --filter='filename == "*/ls"' \
7591                     --action=start-session pitou
7592 ----
7593 +
7594 Confirm that the trigger exists with the man:lttng-list-triggers(1)
7595 command:
7596 +
7597 [role="term"]
7598 ----
7599 # lttng list-triggers
7600 ----
7601
7602 . Make sure the `pitou` recording session is still inactive (stopped):
7603 +
7604 [role="term"]
7605 ----
7606 # lttng list pitou
7607 ----
7608 +
7609 The first line should be something like:
7610 +
7611 ----
7612 Recording session pitou: [inactive]
7613 ----
7614
7615 Run the cmd:ls program to fire the LTTng trigger above:
7616
7617 [role="term"]
7618 ----
7619 $ ls ~
7620 ----
7621
7622 At this point, the `pitou` recording session should be active
7623 (started). Confirm this with the man:lttng-list(1) command again:
7624
7625 [role="term"]
7626 ----
7627 # lttng list pitou
7628 ----
7629
7630 The first line should now look like:
7631
7632 ----
7633 Recording session pitou: [active]
7634 ----
7635
7636 This line confirms that the LTTng trigger you added fired, therefore
7637 starting the `pitou` recording session.
7638 ====
7639
7640 .[[trigger-event-notif]]Send a notification to a user application when an event rule matches.
7641 ====
7642 This example shows how to add the following trigger to the root
7643 <<lttng-sessiond,session daemon>>:
7644
7645 Condition::
7646     An event rule matches a Linux kernel tracepoint event named
7647     `sched_switch` and of which the value of the `next_comm` payload
7648     field is `bash`.
7649 +
7650 With such an event rule, LTTng emits an event when Linux gives access to
7651 the processor to a process named `bash`.
7652
7653 Action::
7654     Send an LTTng notification to a user application.
7655
7656 Moreover, we'll specify a _capture descriptor_ with the
7657 `event-rule-matches` trigger condition so that the user application can
7658 get the value of a specific `sched_switch` event payload field.
7659
7660 First, write and build the user application:
7661
7662 . Create the C{nbsp}source file of the application:
7663 +
7664 --
7665 [source,c]
7666 .path:{notif-app.c}
7667 ----
7668 #include <stdlib.h>
7669 #include <stdio.h>
7670 #include <stdbool.h>
7671 #include <assert.h>
7672 #include <string.h>
7673 #include <lttng/lttng.h>
7674
7675 /*
7676  * Subscribes to notifications, through the notification channel
7677  * `notification_channel`, which match the condition of the trigger
7678  * named `trigger_name`.
7679  *
7680  * Returns `true` on success.
7681  */
7682 static bool subscribe(struct lttng_notification_channel *notification_channel,
7683                       const char *trigger_name)
7684 {
7685     const struct lttng_condition *condition = NULL;
7686     struct lttng_triggers *triggers = NULL;
7687     unsigned int trigger_count;
7688     unsigned int i;
7689     enum lttng_error_code error_code;
7690     enum lttng_trigger_status trigger_status;
7691     bool ret = false;
7692
7693     /* Get all LTTng triggers */
7694     error_code = lttng_list_triggers(&triggers);
7695     assert(error_code == LTTNG_OK);
7696
7697     /* Get the number of triggers */
7698     trigger_status = lttng_triggers_get_count(triggers, &trigger_count);
7699     assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7700
7701     /* Find the trigger named `trigger_name` */
7702     for (i = 0; i < trigger_count; i++) {
7703         const struct lttng_trigger *trigger;
7704         const char *this_trigger_name;
7705
7706         trigger = lttng_triggers_get_at_index(triggers, i);
7707         trigger_status = lttng_trigger_get_name(trigger, &this_trigger_name);
7708         assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7709
7710         if (strcmp(this_trigger_name, trigger_name) == 0) {
7711             /* Trigger found: subscribe with its condition */
7712             enum lttng_notification_channel_status notification_channel_status;
7713
7714             notification_channel_status = lttng_notification_channel_subscribe(
7715                 notification_channel,
7716                 lttng_trigger_get_const_condition(trigger));
7717             assert(notification_channel_status ==
7718                    LTTNG_NOTIFICATION_CHANNEL_STATUS_OK);
7719             ret = true;
7720             break;
7721         }
7722     }
7723
7724     lttng_triggers_destroy(triggers);
7725     return ret;
7726 }
7727
7728 /*
7729  * Handles the evaluation `evaluation` of a single notification.
7730  */
7731 static void handle_evaluation(const struct lttng_evaluation *evaluation)
7732 {
7733     enum lttng_evaluation_status evaluation_status;
7734     const struct lttng_event_field_value *array_field_value;
7735     const struct lttng_event_field_value *string_field_value;
7736     enum lttng_event_field_value_status event_field_value_status;
7737     const char *string_field_string_value;
7738
7739     /* Get the value of the first captured (string) field */
7740     evaluation_status = lttng_evaluation_event_rule_matches_get_captured_values(
7741         evaluation, &array_field_value);
7742     assert(evaluation_status == LTTNG_EVALUATION_STATUS_OK);
7743     event_field_value_status =
7744         lttng_event_field_value_array_get_element_at_index(
7745             array_field_value, 0, &string_field_value);
7746     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7747     assert(lttng_event_field_value_get_type(string_field_value) ==
7748            LTTNG_EVENT_FIELD_VALUE_TYPE_STRING);
7749     event_field_value_status = lttng_event_field_value_string_get_value(
7750             string_field_value, &string_field_string_value);
7751     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7752
7753     /* Print the string value of the field */
7754     puts(string_field_string_value);
7755 }
7756
7757 int main(int argc, char *argv[])
7758 {
7759     int exit_status = EXIT_SUCCESS;
7760     struct lttng_notification_channel *notification_channel;
7761     enum lttng_notification_channel_status notification_channel_status;
7762     const struct lttng_condition *condition;
7763     const char *trigger_name;
7764     bool subscribe_res;
7765
7766     assert(argc >= 2);
7767     trigger_name = argv[1];
7768
7769     /*
7770      * Create a notification channel.
7771      *
7772      * A notification channel connects the user application to the LTTng
7773      * session daemon.
7774      *
7775      * You can use this notification channel to listen to various types
7776      * of notifications.
7777      */
7778     notification_channel = lttng_notification_channel_create(
7779         lttng_session_daemon_notification_endpoint);
7780     assert(notification_channel);
7781
7782     /*
7783      * Subscribe to notifications which match the condition of the
7784      * trigger named `trigger_name`.
7785      */
7786     if (!subscribe(notification_channel, trigger_name)) {
7787         fprintf(stderr,
7788                 "Error: Failed to subscribe to notifications (trigger `%s`).\n",
7789                 trigger_name);
7790         exit_status = EXIT_FAILURE;
7791         goto end;
7792     }
7793
7794     /*
7795      * Notification loop.
7796      *
7797      * Put this in a dedicated thread to avoid blocking the main thread.
7798      */
7799     while (true) {
7800         struct lttng_notification *notification;
7801         enum lttng_notification_channel_status status;
7802         const struct lttng_evaluation *notification_evaluation;
7803
7804         /* Receive the next notification */
7805         status = lttng_notification_channel_get_next_notification(
7806             notification_channel, &notification);
7807
7808         switch (status) {
7809         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7810             break;
7811         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7812             /*
7813              * The session daemon can drop notifications if a receiving
7814              * application doesn't consume the notifications fast
7815              * enough.
7816              */
7817             continue;
7818         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7819             /*
7820              * The session daemon closed the notification channel.
7821              *
7822              * This is typically caused by a session daemon shutting
7823              * down.
7824              */
7825             goto end;
7826         default:
7827             /* Unhandled conditions or errors */
7828             exit_status = EXIT_FAILURE;
7829             goto end;
7830         }
7831
7832         /*
7833          * Handle the condition evaluation.
7834          *
7835          * A notification provides, amongst other things:
7836          *
7837          * * The condition that caused LTTng to send this notification.
7838          *
7839          * * The condition evaluation, which provides more specific
7840          *   information on the evaluation of the condition.
7841          */
7842         handle_evaluation(lttng_notification_get_evaluation(notification));
7843
7844         /* Destroy the notification object */
7845         lttng_notification_destroy(notification);
7846     }
7847
7848 end:
7849     lttng_notification_channel_destroy(notification_channel);
7850     return exit_status;
7851 }
7852 ----
7853 --
7854 +
7855 This application prints the first captured string field value of the
7856 condition evaluation of each LTTng notification it receives.
7857
7858 . Build the `notif-app` application,
7859   using https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
7860   to provide the right compiler and linker flags:
7861 +
7862 --
7863 [role="term"]
7864 ----
7865 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
7866 ----
7867 --
7868
7869 Now, to add the trigger to the root session daemon:
7870
7871 [start=3]
7872 . **If there's no currently running LTTng root session daemon**, start
7873   one:
7874 +
7875 [role="term"]
7876 ----
7877 # lttng-sessiond --daemonize
7878 ----
7879
7880 . Add the trigger, naming it `sched-switch-notif`, to the root
7881   session daemon:
7882 +
7883 [role="term"]
7884 ----
7885 # lttng add-trigger --name=sched-switch-notif \
7886                     --condition=event-rule-matches \
7887                     --type=kernel --name=sched_switch \
7888                     --filter='next_comm == "bash"' --capture=prev_comm \
7889                     --action=notify
7890 ----
7891 +
7892 Confirm that the `sched-switch-notif` trigger exists with the
7893 man:lttng-list-triggers(1) command:
7894 +
7895 [role="term"]
7896 ----
7897 # lttng list-triggers
7898 ----
7899
7900 Run the cmd:notif-app application, passing the name of the trigger
7901 of which to watch the notifications:
7902
7903 [role="term"]
7904 ----
7905 # ./notif-app sched-switch-notif
7906 ----
7907
7908 Now, in an interactive Bash, type a few keys to fire the
7909 `sched-switch-notif` trigger. Watch the `notif-app` application print
7910 the previous process names.
7911 ====
7912
7913 [role="since-2.6"]
7914 [[mi]]
7915 === Use the machine interface
7916
7917 With any command of the man:lttng(1) command-line tool, set the
7918 opt:lttng(1):--mi option to `xml` (before the command name) to get an
7919 XML machine interface output, for example:
7920
7921 [role="term"]
7922 ----
7923 $ lttng --mi=xml list my-session
7924 ----
7925
7926 A schema definition (XSD) is
7927 https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/mi-lttng-4.0.xsd[available]
7928 to ease the integration with external tools as much as possible.
7929
7930
7931 [role="since-2.8"]
7932 [[metadata-regenerate]]
7933 === Regenerate the metadata of an LTTng trace
7934
7935 An LTTng trace, which is a https://diamon.org/ctf[CTF] trace, has both
7936 data stream files and a metadata stream file. This metadata file
7937 contains, amongst other things, information about the offset of the
7938 clock sources which LTTng uses to assign timestamps to <<event,event
7939 records>> when recording.
7940
7941 If, once a <<tracing-session,recording session>> is
7942 <<basic-tracing-session-control,started>>, a major
7943 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
7944 happens, the clock offset of the trace also needs to be updated. Use
7945 the `metadata` item of the man:lttng-regenerate(1) command to do so.
7946
7947 The main use case of this command is to allow a system to boot with
7948 an incorrect wall time and have LTTng trace it before its wall time
7949 is corrected. Once the system is known to be in a state where its
7950 wall time is correct, you can run `lttng regenerate metadata`.
7951
7952 To regenerate the metadata stream files of the
7953 <<cur-tracing-session,current recording session>>:
7954
7955 * Use the `metadata` item of the man:lttng-regenerate(1) command:
7956 +
7957 --
7958 [role="term"]
7959 ----
7960 $ lttng regenerate metadata
7961 ----
7962 --
7963
7964
7965 [role="since-2.9"]
7966 [[regenerate-statedump]]
7967 === Regenerate the state dump event records of a recording session
7968
7969 The LTTng kernel and user space tracers generate state dump
7970 <<event,event records>> when the application starts or when you
7971 <<basic-tracing-session-control,start a recording session>>.
7972
7973 An analysis can use the state dump event records to set an initial state
7974 before it builds the rest of the state from the subsequent event
7975 records. http://tracecompass.org/[Trace Compass] and
7976 https://github.com/lttng/lttng-analyses[LTTng analyses] are notable
7977 examples of applications which use the state dump of an LTTng trace.
7978
7979 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7980 state dump event records aren't included in the snapshot trace files
7981 because they were recorded to a <<channel,sub-buffer>> that has been
7982 consumed or <<overwrite-mode,overwritten>> already.
7983
7984 Use the `statedump` item of the man:lttng-regenerate(1) command to emit
7985 and record the state dump events again.
7986
7987 To regenerate the state dump of the <<cur-tracing-session,current
7988 recording session>>, provided you created it in <<snapshot-mode,snapshot
7989 mode>>, before you take a snapshot:
7990
7991 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7992 +
7993 --
7994 [role="term"]
7995 ----
7996 $ lttng regenerate statedump
7997 ----
7998 --
7999
8000 . <<basic-tracing-session-control,Stop the recording session>>:
8001 +
8002 --
8003 [role="term"]
8004 ----
8005 $ lttng stop
8006 ----
8007 --
8008
8009 . <<taking-a-snapshot,Take a snapshot>>:
8010 +
8011 --
8012 [role="term"]
8013 ----
8014 $ lttng snapshot record --name=my-snapshot
8015 ----
8016 --
8017
8018 Depending on the event throughput, you should run steps{nbsp}1
8019 and{nbsp}2 as closely as possible.
8020
8021 [NOTE]
8022 ====
8023 To record the state dump events, you need to
8024 <<enabling-disabling-events,create recording event rules>> which enable
8025 them:
8026
8027 * The names of LTTng-UST state dump tracepoints start with
8028   `lttng_ust_statedump:`.
8029
8030 * The names of LTTng-modules state dump tracepoints start with
8031   `lttng_statedump_`.
8032 ====
8033
8034
8035 [role="since-2.7"]
8036 [[persistent-memory-file-systems]]
8037 === Record trace data on persistent memory file systems
8038
8039 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile
8040 random-access memory] (NVRAM) is random-access memory that retains its
8041 information when power is turned off (non-volatile). Systems with such
8042 memory can store data structures in RAM and retrieve them after a
8043 reboot, without flushing to typical _storage_.
8044
8045 Linux supports NVRAM file systems thanks to either
8046 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
8047 (requires Linux{nbsp}4.1+) or http://pramfs.sourceforge.net/[PRAMFS] (requires Linux{nbsp}<{nbsp}4).
8048
8049 This section doesn't describe how to operate such file systems; we
8050 assume that you have a working persistent memory file system.
8051
8052 When you <<creating-destroying-tracing-sessions,create a recording
8053 session>>, you can specify the path of the shared memory holding the
8054 sub-buffers. If you specify a location on an NVRAM file system, then you
8055 can retrieve the latest recorded trace data when the system reboots
8056 after a crash.
8057
8058 To record trace data on a persistent memory file system and retrieve the
8059 trace data after a system crash:
8060
8061 . Create a recording session with a <<channel,sub-buffer>> shared memory
8062   path located on an NVRAM file system:
8063 +
8064 --
8065 [role="term"]
8066 ----
8067 $ lttng create my-session --shm-path=/path/to/shm/on/nvram
8068 ----
8069 --
8070
8071 . Configure the recording session as usual with the man:lttng(1)
8072   command-line tool, and <<basic-tracing-session-control,start
8073   recording>>.
8074
8075 . After a system crash, use the man:lttng-crash(1) command-line tool to
8076   read the trace data recorded on the NVRAM file system:
8077 +
8078 --
8079 [role="term"]
8080 ----
8081 $ lttng-crash /path/to/shm/on/nvram
8082 ----
8083 --
8084
8085 The binary layout of the ring buffer files isn't exactly the same as the
8086 trace files layout. This is why you need to use man:lttng-crash(1)
8087 instead of some standard LTTng trace reader.
8088
8089 To convert the ring buffer files to LTTng trace files:
8090
8091 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
8092 +
8093 --
8094 [role="term"]
8095 ----
8096 $ lttng-crash --extract=/path/to/trace /path/to/shm/on/nvram
8097 ----
8098 --
8099
8100
8101 [role="since-2.10"]
8102 [[notif-trigger-api]]
8103 === Get notified when the buffer usage of a channel is too high or too low
8104
8105 With the notification and <<trigger,trigger>> C{nbsp}API of
8106 <<liblttng-ctl-lttng,`liblttng-ctl`>>, LTTng can notify your user
8107 application when the buffer usage of one or more <<channel,channels>>
8108 becomes too low or too high.
8109
8110 Use this API and enable or disable <<event,recording event rules>> while
8111 a recording session <<basic-tracing-session-control,is active>> to avoid
8112 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>, for
8113 example.
8114
8115 .Send a notification to a user application when the buffer usage of an LTTng channel is too high.
8116 ====
8117 In this example, we create and build an application which gets notified
8118 when the buffer usage of a specific LTTng channel is higher than
8119 75{nbsp}%.
8120
8121 We only print that it's the case in this example, but we could as well
8122 use the `liblttng-ctl` C{nbsp}API to <<enabling-disabling-events,disable
8123 recording event rules>> when this happens, for example.
8124
8125 . Create the C{nbsp}source file of the application:
8126 +
8127 --
8128 [source,c]
8129 .path:{notif-app.c}
8130 ----
8131 #include <stdlib.h>
8132 #include <stdio.h>
8133 #include <assert.h>
8134 #include <lttng/lttng.h>
8135
8136 int main(int argc, char *argv[])
8137 {
8138     int exit_status = EXIT_SUCCESS;
8139     struct lttng_notification_channel *notification_channel;
8140     struct lttng_condition *condition;
8141     struct lttng_action *action;
8142     struct lttng_trigger *trigger;
8143     const char *recording_session_name;
8144     const char *channel_name;
8145
8146     assert(argc >= 3);
8147     recording_session_name = argv[1];
8148     channel_name = argv[2];
8149
8150     /*
8151      * Create a notification channel.
8152      *
8153      * A notification channel connects the user application to the LTTng
8154      * session daemon.
8155      *
8156      * You can use this notification channel to listen to various types
8157      * of notifications.
8158      */
8159     notification_channel = lttng_notification_channel_create(
8160         lttng_session_daemon_notification_endpoint);
8161
8162     /*
8163      * Create a "buffer usage becomes greater than" condition.
8164      *
8165      * In this case, the condition is satisfied when the buffer usage
8166      * becomes greater than or equal to 75 %.
8167      *
8168      * We create the condition for a specific recording session name,
8169      * channel name, and for the user space tracing domain.
8170      *
8171      * The following condition types also exist:
8172      *
8173      * * The buffer usage of a channel becomes less than a given value.
8174      *
8175      * * The consumed data size of a recording session becomes greater
8176      *   than a given value.
8177      *
8178      * * A recording session rotation becomes ongoing.
8179      *
8180      * * A recording session rotation becomes completed.
8181      *
8182      * * A given event rule matches an event.
8183      */
8184     condition = lttng_condition_buffer_usage_high_create();
8185     lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
8186     lttng_condition_buffer_usage_set_session_name(condition,
8187                                                   recording_session_name);
8188     lttng_condition_buffer_usage_set_channel_name(condition,
8189                                                   channel_name);
8190     lttng_condition_buffer_usage_set_domain_type(condition,
8191                                                  LTTNG_DOMAIN_UST);
8192
8193     /*
8194      * Create an action (receive a notification) to execute when the
8195      * condition created above is satisfied.
8196      */
8197     action = lttng_action_notify_create();
8198
8199     /*
8200      * Create a trigger.
8201      *
8202      * A trigger associates a condition to an action: LTTng executes
8203      * the action when the condition is satisfied.
8204      */
8205     trigger = lttng_trigger_create(condition, action);
8206
8207     /* Register the trigger to the LTTng session daemon. */
8208     lttng_register_trigger(trigger);
8209
8210     /*
8211      * Now that we have registered a trigger, LTTng will send a
8212      * notification every time its condition is met through a
8213      * notification channel.
8214      *
8215      * To receive this notification, we must subscribe to notifications
8216      * which match the same condition.
8217      */
8218     lttng_notification_channel_subscribe(notification_channel,
8219                                          condition);
8220
8221     /*
8222      * Notification loop.
8223      *
8224      * Put this in a dedicated thread to avoid blocking the main thread.
8225      */
8226     for (;;) {
8227         struct lttng_notification *notification;
8228         enum lttng_notification_channel_status status;
8229         const struct lttng_evaluation *notification_evaluation;
8230         const struct lttng_condition *notification_condition;
8231         double buffer_usage;
8232
8233         /* Receive the next notification. */
8234         status = lttng_notification_channel_get_next_notification(
8235             notification_channel, &notification);
8236
8237         switch (status) {
8238         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
8239             break;
8240         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
8241             /*
8242              * The session daemon can drop notifications if a monitoring
8243              * application isn't consuming the notifications fast
8244              * enough.
8245              */
8246             continue;
8247         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
8248             /*
8249              * The session daemon closed the notification channel.
8250              *
8251              * This is typically caused by a session daemon shutting
8252              * down.
8253              */
8254             goto end;
8255         default:
8256             /* Unhandled conditions or errors. */
8257             exit_status = EXIT_FAILURE;
8258             goto end;
8259         }
8260
8261         /*
8262          * A notification provides, amongst other things:
8263          *
8264          * * The condition that caused LTTng to send this notification.
8265          *
8266          * * The condition evaluation, which provides more specific
8267          *   information on the evaluation of the condition.
8268          *
8269          * The condition evaluation provides the buffer usage
8270          * value at the moment the condition was satisfied.
8271          */
8272         notification_condition = lttng_notification_get_condition(
8273             notification);
8274         notification_evaluation = lttng_notification_get_evaluation(
8275             notification);
8276
8277         /* We're subscribed to only one condition. */
8278         assert(lttng_condition_get_type(notification_condition) ==
8279             LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
8280
8281         /*
8282          * Get the exact sampled buffer usage from the condition
8283          * evaluation.
8284          */
8285         lttng_evaluation_buffer_usage_get_usage_ratio(
8286             notification_evaluation, &buffer_usage);
8287
8288         /*
8289          * At this point, instead of printing a message, we could do
8290          * something to reduce the buffer usage of the channel, like
8291          * disable specific events, for example.
8292          */
8293         printf("Buffer usage is %f %% in recording session \"%s\", "
8294                "user space channel \"%s\".\n", buffer_usage * 100,
8295                recording_session_name, channel_name);
8296
8297         /* Destroy the notification object. */
8298         lttng_notification_destroy(notification);
8299     }
8300
8301 end:
8302     lttng_action_destroy(action);
8303     lttng_condition_destroy(condition);
8304     lttng_trigger_destroy(trigger);
8305     lttng_notification_channel_destroy(notification_channel);
8306     return exit_status;
8307 }
8308 ----
8309 --
8310
8311 . Build the `notif-app` application, linking it with `liblttng-ctl`:
8312 +
8313 --
8314 [role="term"]
8315 ----
8316 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
8317 ----
8318 --
8319
8320 . <<creating-destroying-tracing-sessions,Create a recording session>>,
8321   <<enabling-disabling-events,create a recording event rule>> matching
8322   all the user space tracepoint events, and
8323   <<basic-tracing-session-control,start recording>>:
8324 +
8325 --
8326 [role="term"]
8327 ----
8328 $ lttng create my-session
8329 $ lttng enable-event --userspace --all
8330 $ lttng start
8331 ----
8332 --
8333 +
8334 If you create the channel manually with the man:lttng-enable-channel(1)
8335 command, you can set its <<channel-monitor-timer,monitor timer>> to
8336 control how frequently LTTng samples the current values of the channel
8337 properties to evaluate user conditions.
8338
8339 . Run the `notif-app` application.
8340 +
8341 This program accepts the <<tracing-session,recording session>> and
8342 user space channel names as its two first arguments. The channel
8343 which LTTng automatically creates with the man:lttng-enable-event(1)
8344 command above is named `channel0`:
8345 +
8346 --
8347 [role="term"]
8348 ----
8349 $ ./notif-app my-session channel0
8350 ----
8351 --
8352
8353 . In another terminal, run an application with a very high event
8354   throughput so that the 75{nbsp}% buffer usage condition is reached.
8355 +
8356 In the first terminal, the application should print lines like this:
8357 +
8358 ----
8359 Buffer usage is 81.45197 % in recording session "my-session", user space
8360 channel "channel0".
8361 ----
8362 +
8363 If you don't see anything, try to make the threshold of the condition in
8364 path:{notif-app.c} lower (0.1{nbsp}%, for example), and then rebuild the
8365 `notif-app` application (step{nbsp}2) and run it again (step{nbsp}4).
8366 ====
8367
8368
8369 [[reference]]
8370 == Reference
8371
8372 [[lttng-modules-ref]]
8373 === noch:{LTTng-modules}
8374
8375
8376 [role="since-2.9"]
8377 [[lttng-tracepoint-enum]]
8378 ==== `LTTNG_TRACEPOINT_ENUM()` usage
8379
8380 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
8381
8382 [source,c]
8383 ----
8384 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
8385 ----
8386
8387 Replace:
8388
8389 * `name` with the name of the enumeration (C identifier, unique
8390   amongst all the defined enumerations).
8391 * `entries` with a list of enumeration entries.
8392
8393 The available enumeration entry macros are:
8394
8395 +ctf_enum_value(__name__, __value__)+::
8396   Entry named +__name__+ mapped to the integral value +__value__+.
8397
8398 +ctf_enum_range(__name__, __begin__, __end__)+::
8399   Entry named +__name__+ mapped to the range of integral values between
8400   +__begin__+ (included) and +__end__+ (included).
8401
8402 +ctf_enum_auto(__name__)+::
8403   Entry named +__name__+ mapped to the integral value following the
8404   last mapping value.
8405 +
8406 The last value of a `ctf_enum_value()` entry is its +__value__+
8407 parameter.
8408 +
8409 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
8410 +
8411 If `ctf_enum_auto()` is the first entry in the list, its integral
8412 value is 0.
8413
8414 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
8415 to use a defined enumeration as a tracepoint field.
8416
8417 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
8418 ====
8419 [source,c]
8420 ----
8421 LTTNG_TRACEPOINT_ENUM(
8422     my_enum,
8423     TP_ENUM_VALUES(
8424         ctf_enum_auto("AUTO: EXPECT 0")
8425         ctf_enum_value("VALUE: 23", 23)
8426         ctf_enum_value("VALUE: 27", 27)
8427         ctf_enum_auto("AUTO: EXPECT 28")
8428         ctf_enum_range("RANGE: 101 TO 303", 101, 303)
8429         ctf_enum_auto("AUTO: EXPECT 304")
8430     )
8431 )
8432 ----
8433 ====
8434
8435
8436 [role="since-2.7"]
8437 [[lttng-modules-tp-fields]]
8438 ==== Tracepoint fields macros (for `TP_FIELDS()`)
8439
8440 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
8441 tracepoint fields, which must be listed within `TP_FIELDS()` in
8442 `LTTNG_TRACEPOINT_EVENT()`, are:
8443
8444 [role="func-desc growable",cols="asciidoc,asciidoc"]
8445 .Available macros to define LTTng-modules tracepoint fields
8446 |====
8447 |Macro |Description and parameters
8448
8449 |
8450 +ctf_integer(__t__, __n__, __e__)+
8451
8452 +ctf_integer_nowrite(__t__, __n__, __e__)+
8453
8454 +ctf_user_integer(__t__, __n__, __e__)+
8455
8456 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
8457 |
8458 Standard integer, displayed in base{nbsp}10.
8459
8460 +__t__+::
8461   Integer C type (`int`, `long`, `size_t`, ...).
8462
8463 +__n__+::
8464   Field name.
8465
8466 +__e__+::
8467   Argument expression.
8468
8469 |
8470 +ctf_integer_hex(__t__, __n__, __e__)+
8471
8472 +ctf_user_integer_hex(__t__, __n__, __e__)+
8473 |
8474 Standard integer, displayed in base{nbsp}16.
8475
8476 +__t__+::
8477   Integer C type.
8478
8479 +__n__+::
8480   Field name.
8481
8482 +__e__+::
8483   Argument expression.
8484
8485 |+ctf_integer_oct(__t__, __n__, __e__)+
8486 |
8487 Standard integer, displayed in base{nbsp}8.
8488
8489 +__t__+::
8490   Integer C type.
8491
8492 +__n__+::
8493   Field name.
8494
8495 +__e__+::
8496   Argument expression.
8497
8498 |
8499 +ctf_integer_network(__t__, __n__, __e__)+
8500
8501 +ctf_user_integer_network(__t__, __n__, __e__)+
8502 |
8503 Integer in network byte order (big-endian), displayed in base{nbsp}10.
8504
8505 +__t__+::
8506   Integer C type.
8507
8508 +__n__+::
8509   Field name.
8510
8511 +__e__+::
8512   Argument expression.
8513
8514 |
8515 +ctf_integer_network_hex(__t__, __n__, __e__)+
8516
8517 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
8518 |
8519 Integer in network byte order, displayed in base{nbsp}16.
8520
8521 +__t__+::
8522   Integer C type.
8523
8524 +__n__+::
8525   Field name.
8526
8527 +__e__+::
8528   Argument expression.
8529
8530 |
8531 +ctf_enum(__N__, __t__, __n__, __e__)+
8532
8533 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
8534
8535 +ctf_user_enum(__N__, __t__, __n__, __e__)+
8536
8537 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
8538 |
8539 Enumeration.
8540
8541 +__N__+::
8542   Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
8543
8544 +__t__+::
8545   Integer C type (`int`, `long`, `size_t`, ...).
8546
8547 +__n__+::
8548   Field name.
8549
8550 +__e__+::
8551   Argument expression.
8552
8553 |
8554 +ctf_string(__n__, __e__)+
8555
8556 +ctf_string_nowrite(__n__, __e__)+
8557
8558 +ctf_user_string(__n__, __e__)+
8559
8560 +ctf_user_string_nowrite(__n__, __e__)+
8561 |
8562 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
8563
8564 +__n__+::
8565   Field name.
8566
8567 +__e__+::
8568   Argument expression.
8569
8570 |
8571 +ctf_array(__t__, __n__, __e__, __s__)+
8572
8573 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
8574
8575 +ctf_user_array(__t__, __n__, __e__, __s__)+
8576
8577 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
8578 |
8579 Statically-sized array of integers.
8580
8581 +__t__+::
8582   Array element C type.
8583
8584 +__n__+::
8585   Field name.
8586
8587 +__e__+::
8588   Argument expression.
8589
8590 +__s__+::
8591   Number of elements.
8592
8593 |
8594 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
8595
8596 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8597
8598 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
8599
8600 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8601 |
8602 Statically-sized array of bits.
8603
8604 The type of +__e__+ must be an integer type. +__s__+ is the number
8605 of elements of such type in +__e__+, not the number of bits.
8606
8607 +__t__+::
8608   Array element C type.
8609
8610 +__n__+::
8611   Field name.
8612
8613 +__e__+::
8614   Argument expression.
8615
8616 +__s__+::
8617   Number of elements.
8618
8619 |
8620 +ctf_array_text(__t__, __n__, __e__, __s__)+
8621
8622 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
8623
8624 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
8625
8626 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
8627 |
8628 Statically-sized array, printed as text.
8629
8630 The string doesn't need to be null-terminated.
8631
8632 +__t__+::
8633   Array element C type (always `char`).
8634
8635 +__n__+::
8636   Field name.
8637
8638 +__e__+::
8639   Argument expression.
8640
8641 +__s__+::
8642   Number of elements.
8643
8644 |
8645 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
8646
8647 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8648
8649 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
8650
8651 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8652 |
8653 Dynamically-sized array of integers.
8654
8655 The type of +__E__+ must be unsigned.
8656
8657 +__t__+::
8658   Array element C type.
8659
8660 +__n__+::
8661   Field name.
8662
8663 +__e__+::
8664   Argument expression.
8665
8666 +__T__+::
8667   Length expression C type.
8668
8669 +__E__+::
8670   Length expression.
8671
8672 |
8673 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8674
8675 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8676 |
8677 Dynamically-sized array of integers, displayed in base{nbsp}16.
8678
8679 The type of +__E__+ must be unsigned.
8680
8681 +__t__+::
8682   Array element C type.
8683
8684 +__n__+::
8685   Field name.
8686
8687 +__e__+::
8688   Argument expression.
8689
8690 +__T__+::
8691   Length expression C type.
8692
8693 +__E__+::
8694   Length expression.
8695
8696 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
8697 |
8698 Dynamically-sized array of integers in network byte order (big-endian),
8699 displayed in base{nbsp}10.
8700
8701 The type of +__E__+ must be unsigned.
8702
8703 +__t__+::
8704   Array element C type.
8705
8706 +__n__+::
8707   Field name.
8708
8709 +__e__+::
8710   Argument expression.
8711
8712 +__T__+::
8713   Length expression C type.
8714
8715 +__E__+::
8716   Length expression.
8717
8718 |
8719 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8720
8721 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8722
8723 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8724
8725 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8726 |
8727 Dynamically-sized array of bits.
8728
8729 The type of +__e__+ must be an integer type. +__s__+ is the number
8730 of elements of such type in +__e__+, not the number of bits.
8731
8732 The type of +__E__+ must be unsigned.
8733
8734 +__t__+::
8735   Array element C type.
8736
8737 +__n__+::
8738   Field name.
8739
8740 +__e__+::
8741   Argument expression.
8742
8743 +__T__+::
8744   Length expression C type.
8745
8746 +__E__+::
8747   Length expression.
8748
8749 |
8750 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8751
8752 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8753
8754 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8755
8756 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8757 |
8758 Dynamically-sized array, displayed as text.
8759
8760 The string doesn't need to be null-terminated.
8761
8762 The type of +__E__+ must be unsigned.
8763
8764 The behaviour is undefined if +__e__+ is `NULL`.
8765
8766 +__t__+::
8767   Sequence element C type (always `char`).
8768
8769 +__n__+::
8770   Field name.
8771
8772 +__e__+::
8773   Argument expression.
8774
8775 +__T__+::
8776   Length expression C type.
8777
8778 +__E__+::
8779   Length expression.
8780 |====
8781
8782 Use the `_user` versions when the argument expression, `e`, is
8783 a user space address. In the cases of `ctf_user_integer*()` and
8784 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
8785 be addressable.
8786
8787 The `_nowrite` versions omit themselves from the trace data, but are
8788 otherwise identical. This means LTTng won't write the `_nowrite` fields
8789 to the recorded trace. Their primary purpose is to make some of the
8790 event context available to the <<enabling-disabling-events,recording
8791 event rule filters>> without having to commit the data to
8792 <<channel,sub-buffers>>.
8793
8794
8795 [[glossary]]
8796 == Glossary
8797
8798 Terms related to LTTng and to tracing in general:
8799
8800 [[def-action]]action::
8801   The part of a <<def-trigger,trigger>> which LTTng executes when the
8802   trigger <<def-condition,condition>> is satisfied.
8803
8804 Babeltrace::
8805   The https://diamon.org/babeltrace[Babeltrace] project, which includes:
8806 +
8807 * The
8808   https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
8809   command-line interface.
8810 * The libbabeltrace2 library which offers a
8811   https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
8812 * https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
8813 * Plugins.
8814
8815 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
8816   A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
8817
8818 [[def-channel]]<<channel,channel>>::
8819   An entity which is responsible for a set of
8820   <<def-ring-buffer,ring buffers>>.
8821 +
8822 <<def-recording-event-rule,Recording event rules>> are always attached
8823 to a specific channel.
8824
8825 clock::
8826   A source of time for a <<def-tracer,tracer>>.
8827
8828 [[def-condition]]condition::
8829   The part of a <<def-trigger,trigger>> which must be satisfied for
8830   LTTng to attempt to execute the trigger <<def-action,actions>>.
8831
8832 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
8833   A program which is responsible for consuming the full
8834   <<def-sub-buffer,sub-buffers>> and write them to a file system or
8835   send them over the network.
8836
8837 [[def-current-trace-chunk]]current trace chunk::
8838   A <<def-trace-chunk,trace chunk>> which includes the current content
8839   of all the <<def-sub-buffer,sub-buffers>> of the
8840   <<def-tracing-session,recording session>> and the stream files
8841   produced since the latest event amongst:
8842 +
8843 * The creation of the recording session.
8844 * The last <<def-tracing-session-rotation,recording session rotation>>, if
8845   any.
8846
8847 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
8848   The <<def-event-record-loss-mode,event record loss mode>> in which
8849   the <<def-tracer,tracer>> _discards_ new <<def-event-record,event
8850   records>> when there's no <<def-sub-buffer,sub-buffer>> space left to
8851   store them.
8852
8853 [[def-event]]event::
8854   The execution of an <<def-instrumentation-point,instrumentation
8855   point>>, like a <<def-tracepoint,tracepoint>> that you manually place
8856   in some source code, or a Linux kprobe.
8857 +
8858 When an instrumentation point is executed, LTTng creates an event.
8859 +
8860 When an <<def-event-rule,event rule>> matches the event,
8861 <<def-lttng,LTTng>> executes some action, for example:
8862 +
8863 * Record its payload to a <<def-sub-buffer,sub-buffer>> as an
8864   <<def-event-record,event record>>.
8865 * Attempt to execute the user-defined actions of a
8866   <<def-trigger,trigger>> with an
8867   <<add-event-rule-matches-trigger,``event rule matches''>> condition.
8868
8869 [[def-event-name]]event name::
8870   The name of an <<def-event,event>>, which is also the name of the
8871   <<def-event-record,event record>>.
8872 +
8873 This is also called the _instrumentation point name_.
8874
8875 [[def-event-record]]event record::
8876   A record (binary serialization), in a <<def-trace,trace>>, of the
8877   payload of an <<def-event,event>>.
8878 +
8879 The payload of an event record has zero or more _fields_.
8880
8881 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
8882   The mechanism by which event records of a given
8883   <<def-channel,channel>> are lost (not recorded) when there's no
8884   <<def-sub-buffer,sub-buffer>> space left to store them.
8885
8886 [[def-event-rule]]<<event-rule,event rule>>::
8887   Set of conditions which an <<def-event,event>> must satisfy
8888   for LTTng to execute some action.
8889 +
8890 An event rule is said to _match_ events, like a
8891 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
8892 matches strings.
8893 +
8894 A <<def-recording-event-rule,recording event rule>> is a specific type
8895 of event rule of which the action is to <<def-record,record>> the event
8896 to a <<def-sub-buffer,sub-buffer>>.
8897
8898 [[def-incl-set]]inclusion set::
8899   In the <<pid-tracking,process attribute inclusion set>> context: a
8900   set of <<def-proc-attr,process attributes>> of a given type.
8901
8902 <<instrumenting,instrumentation>>::
8903   The use of <<def-lttng,LTTng>> probes to make a kernel or
8904   <<def-user-application,user application>> traceable.
8905
8906 [[def-instrumentation-point]]instrumentation point::
8907   A point in the execution path of a kernel or
8908   <<def-user-application,user application>> which, when executed,
8909   create an <<def-event,event>>.
8910
8911 instrumentation point name::
8912   See _<<def-event-name,event name>>_.
8913
8914 `java.util.logging`::
8915   The
8916   https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
8917   of the Java platform.
8918
8919 log4j::
8920   A https://logging.apache.org/log4j/1.2/[logging library] for Java
8921   developed by the Apache Software Foundation.
8922
8923 log level::
8924   Level of severity of a log statement or user space
8925   <<def-instrumentation-point,instrumentation point>>.
8926
8927 [[def-lttng]]LTTng::
8928   The _Linux Trace Toolkit: next generation_ project.
8929
8930 <<lttng-cli,cmd:lttng>>::
8931   A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
8932   project which you can use to send and receive control messages to and
8933   from a <<def-session-daemon,session daemon>>.
8934
8935 LTTng analyses::
8936   The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
8937   which is a set of analyzing programs that you can use to obtain a
8938   higher level view of an <<def-lttng,LTTng>> <<def-trace,trace>>.
8939
8940 cmd:lttng-consumerd::
8941   The name of the <<def-consumer-daemon,consumer daemon>> program.
8942
8943 cmd:lttng-crash::
8944   A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8945   which can convert <<def-ring-buffer,ring buffer>> files (usually
8946   <<persistent-memory-file-systems,saved on a persistent memory file
8947   system>>) to <<def-trace,trace>> files.
8948 +
8949 See man:lttng-crash(1).
8950
8951 LTTng Documentation::
8952   This document.
8953
8954 <<lttng-live,LTTng live>>::
8955   A communication protocol between the <<lttng-relayd,relay daemon>> and
8956   live readers which makes it possible to show or analyze
8957   <<def-event-record,event records>> ``live'', as they're received by
8958   the <<def-relay-daemon,relay daemon>>.
8959
8960 <<lttng-modules,LTTng-modules>>::
8961   The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8962   which contains the Linux kernel modules to make the Linux kernel
8963   <<def-instrumentation-point,instrumentation points>> available for
8964   <<def-lttng,LTTng>> tracing.
8965
8966 cmd:lttng-relayd::
8967   The name of the <<def-relay-daemon,relay daemon>> program.
8968
8969 cmd:lttng-sessiond::
8970   The name of the <<def-session-daemon,session daemon>> program.
8971
8972 [[def-lttng-tools]]LTTng-tools::
8973   The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8974   contains the various programs and libraries used to
8975   <<controlling-tracing,control tracing>>.
8976
8977 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8978   The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8979   contains libraries to instrument
8980   <<def-user-application,user applications>>.
8981
8982 <<lttng-ust-agents,LTTng-UST Java agent>>::
8983   A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8984   allow the LTTng instrumentation of `java.util.logging` and Apache
8985   log4j{nbsp}1.2 logging statements.
8986
8987 <<lttng-ust-agents,LTTng-UST Python agent>>::
8988   A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8989   to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8990   statements.
8991
8992 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8993   The <<def-event-record-loss-mode,event record loss mode>> in which new
8994   <<def-event-record,event records>> _overwrite_ older event records
8995   when there's no <<def-sub-buffer,sub-buffer>> space left to store
8996   them.
8997
8998 <<channel-buffering-schemes,per-process buffering>>::
8999   A <<def-buffering-scheme,buffering scheme>> in which each instrumented
9000   process has its own <<def-sub-buffer,sub-buffers>> for a given user
9001   space <<def-channel,channel>>.
9002
9003 <<channel-buffering-schemes,per-user buffering>>::
9004   A <<def-buffering-scheme,buffering scheme>> in which all the processes
9005   of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
9006   given user space <<def-channel,channel>>.
9007
9008 [[def-proc-attr]]process attribute::
9009   In the <<pid-tracking,process attribute inclusion set>> context:
9010 +
9011 * A process ID.
9012 * A virtual process ID.
9013 * A Unix user ID.
9014 * A virtual Unix user ID.
9015 * A Unix group ID.
9016 * A virtual Unix group ID.
9017
9018 record (_noun_)::
9019   See <<def-event-record,_event record_>>.
9020
9021 [[def-record]]record (_verb_)::
9022   Serialize the binary payload of an <<def-event,event>> to a
9023   <<def-sub-buffer,sub-buffer>>.
9024
9025 [[def-recording-event-rule]]<<event,recording event rule>>::
9026   Specific type of <<def-event-rule,event rule>> of which the action is
9027   to <<def-record,record>> the matched event to a
9028   <<def-sub-buffer,sub-buffer>>.
9029
9030 [[def-tracing-session]][[def-recording-session]]<<tracing-session,recording session>>::
9031   A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
9032
9033 [[def-tracing-session-rotation]]<<session-rotation,recording session rotation>>::
9034   The action of archiving the
9035   <<def-current-trace-chunk,current trace chunk>> of a
9036   <<def-tracing-session,recording session>>.
9037
9038 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
9039   A process which is responsible for receiving the <<def-trace,trace>>
9040   data which a distant <<def-consumer-daemon,consumer daemon>> sends.
9041
9042 [[def-ring-buffer]]ring buffer::
9043   A set of <<def-sub-buffer,sub-buffers>>.
9044
9045 rotation::
9046   See _<<def-tracing-session-rotation,recording session rotation>>_.
9047
9048 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
9049   A process which receives control commands from you and orchestrates
9050   the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
9051
9052 <<taking-a-snapshot,snapshot>>::
9053   A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
9054   of a given <<def-tracing-session,recording session>>, saved as
9055   <<def-trace,trace>> files.
9056
9057 [[def-sub-buffer]]sub-buffer::
9058   One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
9059   which contains <<def-event-record,event records>>.
9060
9061 timestamp::
9062   The time information attached to an <<def-event,event>> when LTTng
9063   creates it.
9064
9065 [[def-trace]]trace (_noun_)::
9066   A set of:
9067 +
9068 * One https://diamon.org/ctf/[CTF] metadata stream file.
9069 * One or more CTF data stream files which are the concatenations of one
9070   or more flushed <<def-sub-buffer,sub-buffers>>.
9071
9072 [[def-trace-verb]]trace (_verb_)::
9073   From the perspective of a <<def-tracer,tracer>>: attempt to execute
9074   one or more actions when emitting an <<def-event,event>> in an
9075   application or in a system.
9076
9077 [[def-trace-chunk]]trace chunk::
9078   A self-contained <<def-trace,trace>> which is part of a
9079   <<def-tracing-session,recording session>>. Each
9080   <<def-tracing-session-rotation, recording session rotation>> produces a
9081   <<def-trace-chunk-archive,trace chunk archive>>.
9082
9083 [[def-trace-chunk-archive]]trace chunk archive::
9084   The result of a <<def-tracing-session-rotation, recording session
9085   rotation>>.
9086 +
9087 <<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
9088 containing <<def-tracing-session,recording session>> is still active: you
9089 are free to read it, modify it, move it, or remove it.
9090
9091 Trace Compass::
9092   The http://tracecompass.org[Trace Compass] project and application.
9093
9094 [[def-tracepoint]]tracepoint::
9095   An instrumentation point using the tracepoint mechanism of the Linux
9096   kernel or of <<def-lttng-ust,LTTng-UST>>.
9097
9098 tracepoint definition::
9099   The definition of a single <<def-tracepoint,tracepoint>>.
9100
9101 tracepoint name::
9102   The name of a <<def-tracepoint,tracepoint>>.
9103
9104 [[def-tracepoint-provider]]tracepoint provider::
9105   A set of functions providing <<def-tracepoint,tracepoints>> to an
9106   instrumented <<def-user-application,user application>>.
9107 +
9108 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
9109 provider package>>: many tracepoint providers can exist within a
9110 tracepoint provider package.
9111
9112 [[def-tracepoint-provider-package]]tracepoint provider package::
9113   One or more <<def-tracepoint-provider,tracepoint providers>> compiled
9114   as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
9115   link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
9116   library].
9117
9118 [[def-tracer]]tracer::
9119   A piece of software which executes some action when it emits
9120   an <<def-event,event>>, like <<def-record,record>> it to some
9121   buffer.
9122
9123 <<domain,tracing domain>>::
9124   A type of LTTng <<def-tracer,tracer>>.
9125
9126 <<tracing-group,tracing group>>::
9127   The Unix group which a Unix user can be part of to be allowed to
9128   control the Linux kernel LTTng <<def-tracer,tracer>>.
9129
9130 [[def-trigger]]<<trigger,trigger>>::
9131   A <<def-condition,condition>>-<<def-action,actions>> pair; when the
9132   condition of a trigger is satisfied, LTTng attempts to execute its
9133   actions.
9134
9135 [[def-user-application]]user application::
9136   An application (program or library) running in user space, as opposed
9137   to a Linux kernel module, for example.