2.13/lttng-docs-2.13.txt

   1 The LTTng Documentation
   2 =======================
   3 Philippe Proulx <pproulx@efficios.com>
   4 v2.13, 28 November 2023
   5
   6
   7 include::../common/copyright.txt[]
   8
   9
  10 include::../common/welcome.txt[]
  11
  12
  13 include::../common/audience.txt[]
  14
  15
  16 [[chapters]]
  17 === What's in this documentation?
  18
  19 The LTTng Documentation is divided into the following sections:
  20
  21 * ``**<<nuts-and-bolts,Nuts and bolts>>**'' explains the
  22   rudiments of software tracing and the rationale behind the
  23   LTTng project.
  24 +
  25 Skip this section if you’re familiar with software tracing and with the
  26 LTTng project.
  27
  28 * ``**<<installing-lttng,Installation>>**'' describes the steps to
  29   install the LTTng packages on common Linux distributions and from
  30   their sources.
  31 +
  32 Skip this section if you already properly installed LTTng on your target
  33 system.
  34
  35 * ``**<<getting-started,Quick start>>**'' is a concise guide to
  36   get started quickly with LTTng kernel and user space tracing.
  37 +
  38 We recommend this section if you're new to LTTng or to software tracing
  39 in general.
  40 +
  41 Skip this section if you're not new to LTTng.
  42
  43 * ``**<<core-concepts,Core concepts>>**'' explains the concepts at
  44   the heart of LTTng.
  45 +
  46 It's a good idea to become familiar with the core concepts
  47 before attempting to use the toolkit.
  48
  49 * ``**<<plumbing,Components of LTTng>>**'' describes the various
  50   components of the LTTng machinery, like the daemons, the libraries,
  51   and the command-line interface.
  52
  53 * ``**<<instrumenting,Instrumentation>>**'' shows different ways to
  54   instrument user applications and the Linux kernel for LTTng tracing.
  55 +
  56 Instrumenting source code is essential to provide a meaningful
  57 source of events.
  58 +
  59 Skip this section if you don't have a programming background.
  60
  61 * ``**<<controlling-tracing,Tracing control>>**'' is divided into topics
  62   which demonstrate how to use the vast array of features that
  63   LTTng{nbsp}{revision} offers.
  64
  65 * ``**<<reference,Reference>>**'' contains API reference tables.
  66
  67 * ``**<<glossary,Glossary>>**'' is a specialized dictionary of terms
  68   related to LTTng or to the field of software tracing.
  69
  70
  71 include::../common/convention.txt[]
  72
  73
  74 include::../common/acknowledgements.txt[]
  75
  76
  77 [[whats-new]]
  78 == What's new in LTTng{nbsp}{revision}?
  79
  80 LTTng{nbsp}{revision} bears the name _Nordicité_, the product of a
  81 collaboration between https://champlibre.co/[Champ Libre] and
  82 https://www.boreale.com/[Boréale]. This farmhouse IPA is brewed with
  83 https://en.wikipedia.org/wiki/Kveik[Kveik] yeast and Québec-grown
  84 barley, oats, and juniper branches. The result is a remarkable, fruity,
  85 hazy golden IPA that offers a balanced touch of resinous and woodsy
  86 bitterness.
  87
  88 New features and changes in LTTng{nbsp}{revision}:
  89
  90 General::
  91 +
  92 * The LTTng trigger API of <<liblttng-ctl-lttng,`liblttng-ctl`>> now
  93   offers the ``__event rule matches__'' condition (an <<event-rule,event
  94   rule>> matches an event) as well as the following new actions:
  95 +
  96 --
  97 * <<basic-tracing-session-control,Start or stop>> a recording session.
  98 * <<session-rotation,Archive the current trace chunk>> of a
  99   recording session (rotate).
 100 * <<taking-a-snapshot,Take a snapshot>> of a recording session.
 101 --
 102 +
 103 As a reminder, a <<trigger,trigger>> is a condition-actions pair. When
 104 the condition of a trigger is satisfied, LTTng attempts to execute its
 105 actions.
 106 +
 107 This feature is also available with the new man:lttng-add-trigger(1),
 108 man:lttng-remove-trigger(1), and man:lttng-list-triggers(1)
 109 <<lttng-cli,cmd:lttng>> commands.
 110 +
 111 Starting from LTTng{nbsp}{revision}, a trigger may have more than one
 112 action.
 113 +
 114 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
 115 trigger to a session daemon>>” to learn more.
 116
 117 * The LTTng <<lttng-ust,user space>> and <<lttng-modules,kernel>>
 118   tracers offer the new namespace context field `time_ns`, which is the
 119   inode number, in the proc file system, of the current clock namespace.
 120 +
 121 See man:lttng-add-context(1), man:lttng-ust(3), and
 122 man:time_namespaces(7).
 123
 124 * The link:/man[manual pages] of LTTng-tools now have a terminology and
 125   style which match the LTTng Documentation, many fixes, more internal
 126   and manual page links, clearer lists and procedures, superior
 127   consistency, and usage examples.
 128 +
 129 The new man:lttng-event-rule(7) manual page explains the new, common
 130 way to specify an event rule on the command line.
 131 +
 132 The new man:lttng-concepts(7) manual page explains the core concepts of
 133 LTTng. Its contents is essentially the ``<<core-concepts,Core
 134 concepts>>'' section of this documentation, but more adapted to the
 135 manual page style.
 136
 137 User space tracing::
 138 +
 139 [IMPORTANT]
 140 ====
 141 The major version part of the `liblttng-ust`
 142 https://en.wikipedia.org/wiki/Soname[soname] is bumped, which means you
 143 **must recompile** your instrumented applications/libraries and
 144 <<tracepoint-provider,tracepoint provider packages>> to use
 145 LTTng-UST{nbsp}{revision}.
 146
 147 This change became a necessity to clean up the library and for
 148 `liblttng-ust` to stop exporting private symbols.
 149
 150 Also, LTTng{nbsp}{revision} prepends the `lttng_ust_` and `LTTNG_UST_`
 151 prefix to all public macro/definition/function names to offer a
 152 consistent API namespace. The LTTng{nbsp}2.12 API is still available;
 153 see the ``Compatibility with previous APIs'' section of
 154 man:lttng-ust(3).
 155 ====
 156 +
 157 Other notable changes:
 158 +
 159 * The `liblttng-ust` C{nbsp}API offers the new man:lttng_ust_vtracef(3)
 160   and man:lttng_ust_vtracelog(3) macros which are to
 161   man:lttng_ust_tracef(3) and man:lttng_ust_tracelog(3) what
 162   man:vprintf(3) is to man:printf(3).
 163
 164 * LTTng-UST now only depends on https://liburcu.org/[`liburcu`] at build
 165   time, not at run time.
 166
 167 Kernel tracing::
 168 +
 169 * The preferred display base of event record integer fields which
 170   contain memory addresses is now hexadecimal instead of decimal.
 171
 172 * The `pid` field is removed from `lttng_statedump_file_descriptor`
 173   event records and the `file_table_address` field is added.
 174 +
 175 This new field is the address of the `files_struct` structure which
 176 contains the file descriptor.
 177 +
 178 See the
 179 ``https://github.com/lttng/lttng-modules/commit/e7a0ca7205fd4be7c829d171baa8823fe4784c90[statedump: introduce `file_table_address`]''
 180 patch to learn more.
 181
 182 * The `flags` field of `syscall_entry_clone` event records is now a
 183   structure containing two enumerations (exit signal and options).
 184 +
 185 This change makes the flag values more readable and meaningful.
 186 +
 187 See the
 188 ``https://github.com/lttng/lttng-modules/commit/d775625e2ba4825b73b5897e7701ad6e2bdba115[syscalls: Make `clone()`'s `flags` field a 2 enum struct]''
 189 patch to learn more.
 190
 191 * The memory footprint of the kernel tracer is improved: the latter only
 192   generates metadata for the specific system call recording event rules
 193   that you <<enabling-disabling-events,create>>.
 194
 195
 196 [[nuts-and-bolts]]
 197 == Nuts and bolts
 198
 199 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
 200 generation_ is a modern toolkit for tracing Linux systems and
 201 applications. So your first question might be:
 202 **what is tracing?**
 203
 204
 205 [[what-is-tracing]]
 206 === What is tracing?
 207
 208 As the history of software engineering progressed and led to what
 209 we now take for granted--complex, numerous and
 210 interdependent software applications running in parallel on
 211 sophisticated operating systems like Linux--the authors of such
 212 components, software developers, began feeling a natural
 213 urge to have tools that would ensure the robustness and good performance
 214 of their masterpieces.
 215
 216 One major achievement in this field is, inarguably, the
 217 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
 218 an essential tool for developers to find and fix bugs. But even the best
 219 debugger won't help make your software run faster, and nowadays, faster
 220 software means either more work done by the same hardware, or cheaper
 221 hardware for the same work.
 222
 223 A _profiler_ is often the tool of choice to identify performance
 224 bottlenecks. Profiling is suitable to identify _where_ performance is
 225 lost in a given piece of software. The profiler outputs a profile, a
 226 statistical summary of observed events, which you may use to discover
 227 which functions took the most time to execute. However, a profiler won't
 228 report _why_ some identified functions are the bottleneck. Bottlenecks
 229 might only occur when specific conditions are met, conditions that are
 230 sometimes impossible to capture by a statistical profiler, or impossible
 231 to reproduce with an application altered by the overhead of an
 232 event-based profiler. For a thorough investigation of software
 233 performance issues, a history of execution is essential, with the
 234 recorded values of variables and context fields you choose, and with as
 235 little influence as possible on the instrumented application. This is
 236 where tracing comes in handy.
 237
 238 _Tracing_ is a technique used to understand what goes on in a running
 239 software system. The piece of software used for tracing is called a
 240 _tracer_, which is conceptually similar to a tape recorder. When
 241 recording, specific instrumentation points placed in the software source
 242 code generate events that are saved on a giant tape: a _trace_ file. You
 243 can record user application and operating system events at the same
 244 time, opening the possibility of resolving a wide range of problems that
 245 would otherwise be extremely challenging.
 246
 247 Tracing is often compared to _logging_. However, tracers and loggers are
 248 two different tools, serving two different purposes. Tracers are
 249 designed to record much lower-level events that occur much more
 250 frequently than log messages, often in the range of thousands per
 251 second, with very little execution overhead. Logging is more appropriate
 252 for a very high-level analysis of less frequent events: user accesses,
 253 exceptional conditions (errors and warnings, for example), database
 254 transactions, instant messaging communications, and such. Simply put,
 255 logging is one of the many use cases that can be satisfied with tracing.
 256
 257 The list of recorded events inside a trace file can be read manually
 258 like a log file for the maximum level of detail, but it's generally
 259 much more interesting to perform application-specific analyses to
 260 produce reduced statistics and graphs that are useful to resolve a
 261 given problem. Trace viewers and analyzers are specialized tools
 262 designed to do this.
 263
 264 In the end, this is what LTTng is: a powerful, open source set of
 265 tools to trace the Linux kernel and user applications at the same time.
 266 LTTng is composed of several components actively maintained and
 267 developed by its link:/community/#where[community].
 268
 269
 270 [[lttng-alternatives]]
 271 === Alternatives to noch:{LTTng}
 272
 273 Excluding proprietary solutions, a few competing software tracers
 274 exist for Linux:
 275
 276 https://github.com/dtrace4linux/linux[dtrace4linux]::
 277     A port of Sun Microsystems' DTrace to Linux.
 278 +
 279 The cmd:dtrace tool interprets user scripts and is responsible for
 280 loading code into the Linux kernel for further execution and collecting
 281 the outputted data.
 282
 283 https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
 284     A subsystem in the Linux kernel in which a virtual machine can
 285     execute programs passed from the user space to the kernel.
 286 +
 287 You can attach such programs to tracepoints and kprobes thanks to a
 288 system call, and they can output data to the user space when executed
 289 thanks to different mechanisms (pipe, VM register values, and eBPF maps,
 290 to name a few).
 291
 292 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
 293     The de facto function tracer of the Linux kernel.
 294 +
 295 Its user interface is a set of special files in sysfs.
 296
 297 https://perf.wiki.kernel.org/[perf]::
 298     A performance analysis tool for Linux which supports hardware
 299     performance counters, tracepoints, as well as other counters and
 300     types of probes.
 301 +
 302 The controlling utility of perf is the cmd:perf command line/text UI
 303 tool.
 304
 305 https://linux.die.net/man/1/strace[strace]::
 306     A command-line utility which records system calls made by a
 307     user process, as well as signal deliveries and changes of process
 308     state.
 309 +
 310 strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
 311 fulfill its function.
 312
 313 https://www.sysdig.org/[sysdig]::
 314     Like SystemTap, uses scripts to analyze Linux kernel events.
 315 +
 316 You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
 317 sysdig executes them while it traces the system or afterwards. The
 318 interface of sysdig is the cmd:sysdig command-line tool as well as the
 319 text UI-based cmd:csysdig tool.
 320
 321 https://sourceware.org/systemtap/[SystemTap]::
 322     A Linux kernel and user space tracer which uses custom user scripts
 323     to produce plain text traces.
 324 +
 325 SystemTap converts the scripts to the C language, and then compiles them
 326 as Linux kernel modules which are loaded to produce trace data. The
 327 primary user interface of SystemTap is the cmd:stap command-line tool.
 328
 329 The main distinctive features of LTTng is that it produces correlated
 330 kernel and user space traces, as well as doing so with the lowest
 331 overhead amongst other solutions. It produces trace files in the
 332 https://diamon.org/ctf[CTF] format, a file format optimized
 333 for the production and analyses of multi-gigabyte data.
 334
 335 LTTng is the result of more than 10{nbsp}years of active open source
 336 development by a community of passionate developers. LTTng is currently
 337 available on major desktop and server Linux distributions.
 338
 339 The main interface for tracing control is a single command-line tool
 340 named cmd:lttng. The latter can create several recording sessions, enable
 341 and disable recording event rules on the fly, filter events efficiently
 342 with custom user expressions, start and stop tracing, and much more.
 343 LTTng can write the traces on the file system or send them over the
 344 network, and keep them totally or partially. You can make LTTng execute
 345 user-defined actions when LTTng emits an event. You can view the traces
 346 once tracing becomes inactive or as LTTng records events.
 347
 348 <<installing-lttng,Install LTTng now>> and
 349 <<getting-started,start tracing>>!
 350
 351
 352 [[installing-lttng]]
 353 == Installation
 354
 355 **LTTng** is a set of software <<plumbing,components>> which interact to
 356 <<instrumenting,instrument>> the Linux kernel and user applications, and
 357 to <<controlling-tracing,control tracing>> (start and stop
 358 recording, create recording event rules, and the rest). Those
 359 components are bundled into the following packages:
 360
 361 LTTng-tools::
 362     Libraries and command-line interface to control tracing.
 363
 364 LTTng-modules::
 365     Linux kernel modules to instrument and trace the kernel.
 366
 367 LTTng-UST::
 368     Libraries and Java/Python packages to instrument and trace user
 369     applications.
 370
 371 Most distributions mark the LTTng-modules and LTTng-UST packages as
 372 optional when installing LTTng-tools (which is always required). In the
 373 following sections, we always provide the steps to install all three,
 374 but note that:
 375
 376 * You only need to install LTTng-modules if you intend to use
 377   the Linux kernel LTTng tracer.
 378
 379 * You only need to install LTTng-UST if you intend to use the user
 380   space LTTng tracer.
 381
 382 [role="growable"]
 383 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 17{nbsp}October{nbsp}2023.
 384
 385 |===
 386 |Distribution |Available in releases
 387
 388 |https://www.ubuntu.com/[Ubuntu]
 389 |xref:ubuntu[Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_].
 390
 391 Ubuntu{nbsp}18.04 LTS _Bionic Beaver_ and Ubuntu{nbsp}20.04 LTS _Focal Fossa_:
 392 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 393
 394 |https://www.debian.org/[Debian]
 395 |<<debian,Debian{nbsp}12 _bookworm_>>.
 396
 397 |https://getfedora.org/[Fedora]
 398 |xref:fedora[Fedora{nbsp}37, Fedora{nbsp}38, and Fedora{nbsp}39].
 399
 400 |https://www.archlinux.org/[Arch Linux]
 401 |<<arch-linux,_extra_ repository and AUR>>.
 402
 403 |https://alpinelinux.org/[Alpine Linux]
 404 |xref:alpine-linux[Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, and Alpine Linux{nbsp}3.18].
 405
 406 |https://buildroot.org/[Buildroot]
 407 |xref:buildroot[Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 408 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 409 Buildroot{nbsp}2023.05, and Buildroot{nbsp}2023.08].
 410
 411 |https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
 412 https://www.yoctoproject.org/[Yocto]
 413 |xref:oe-yocto[Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 414 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 415 Yocto Project{nbsp}4.3 _Nanbield_].
 416
 417 |====
 418
 419 [NOTE]
 420 ====
 421 For https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
 422 packages, see https://packages.efficios.com/[EfficiOS Enterprise
 423 Packages].
 424
 425 For other distributions, <<building-from-source,build LTTng from
 426 source>>.
 427 ====
 428
 429 [[ubuntu]]
 430 === [[ubuntu-official-repository]]Ubuntu
 431
 432 LTTng{nbsp}{revision} is available on Ubuntu 22.04 LTS _Jammy Jellyfish_, Ubuntu 23.04 _Lunar Lobster_, and Ubuntu 23.10 _Mantic Minotaur_. For previous supported releases of Ubuntu, <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 433
 434 To install LTTng{nbsp}{revision} on Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_:
 435
 436 . Install the main LTTng{nbsp}{revision} packages:
 437 +
 438 --
 439 [role="term"]
 440 ----
 441 # apt-get install lttng-tools
 442 # apt-get install lttng-modules-dkms
 443 # apt-get install liblttng-ust-dev
 444 ----
 445 --
 446
 447 . **If you need to instrument and trace <<java-application,Java applications>>**,
 448 install the LTTng-UST Java agent:
 449 +
 450 --
 451 [role="term"]
 452 ----
 453 # apt-get install liblttng-ust-agent-java
 454 ----
 455 --
 456
 457 . **If you need to instrument and trace <<python-application,Python{nbsp}3
 458 applications>>**, install the LTTng-UST Python agent:
 459 +
 460 --
 461 [role="term"]
 462 ----
 463 # apt-get install python3-lttngust
 464 ----
 465 --
 466
 467 [[ubuntu-ppa]]
 468 === Ubuntu: noch:{LTTng} Stable {revision} PPA
 469
 470 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
 471 Stable{nbsp}{revision} PPA] offers the latest stable LTTng{nbsp}{revision}
 472 packages for Ubuntu{nbsp}18.04 LTS _Bionic Beaver_, Ubuntu{nbsp}20.04 LTS _Focal Fossa_,
 473 and Ubuntu{nbsp}22.04 LTS _Jammy Jellyfish_.
 474
 475 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
 476 PPA:
 477
 478 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
 479   list of packages:
 480 +
 481 --
 482 [role="term",subs="attributes"]
 483 ----
 484 # apt-add-repository ppa:lttng/stable-{revision}
 485 # apt-get update
 486 ----
 487 --
 488
 489 . Install the main LTTng{nbsp}{revision} packages:
 490 +
 491 --
 492 [role="term"]
 493 ----
 494 # apt-get install lttng-tools
 495 # apt-get install lttng-modules-dkms
 496 # apt-get install liblttng-ust-dev
 497 ----
 498 --
 499
 500 . **If you need to instrument and trace
 501   <<java-application,Java applications>>**, install the LTTng-UST
 502   Java agent:
 503 +
 504 --
 505 [role="term"]
 506 ----
 507 # apt-get install liblttng-ust-agent-java
 508 ----
 509 --
 510
 511 . **If you need to instrument and trace
 512   <<python-application,Python{nbsp}3 applications>>**, install the
 513   LTTng-UST Python agent:
 514 +
 515 --
 516 [role="term"]
 517 ----
 518 # apt-get install python3-lttngust
 519 ----
 520 --
 521
 522 [[debian]]
 523 === Debian
 524
 525 To install LTTng{nbsp}{revision} on Debian{nbsp}12 _bookworm_:
 526
 527 . Install the main LTTng{nbsp}{revision} packages:
 528 +
 529 --
 530 [role="term"]
 531 ----
 532 # apt install lttng-modules-dkms
 533 # apt install liblttng-ust-dev
 534 # apt install lttng-tools
 535 ----
 536 --
 537
 538 . **If you need to instrument and trace <<java-application,Java
 539   applications>>**, install the LTTng-UST Java agent:
 540 +
 541 --
 542 [role="term"]
 543 ----
 544 # apt install liblttng-ust-agent-java
 545 ----
 546 --
 547
 548 . **If you need to instrument and trace <<python-application,Python
 549   applications>>**, install the LTTng-UST Python agent:
 550 +
 551 --
 552 [role="term"]
 553 ----
 554 # apt install python3-lttngust
 555 ----
 556 --
 557
 558 [[fedora]]
 559 === Fedora
 560
 561 To install LTTng{nbsp}{revision} on Fedora{nbsp}37, Fedora{nbsp}38, or
 562 Fedora{nbsp}39:
 563
 564 . Install the LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision}
 565   packages:
 566 +
 567 --
 568 [role="term"]
 569 ----
 570 # yum install lttng-tools
 571 # yum install lttng-ust
 572 ----
 573 --
 574
 575 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 576 +
 577 --
 578 [role="term",subs="attributes,specialcharacters"]
 579 ----
 580 $ cd $(mktemp -d) &&
 581   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 582   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 583   cd lttng-modules-{revision}.* &&
 584   make &&
 585   sudo make modules_install &&
 586   sudo depmod -a
 587 ----
 588 --
 589
 590 [IMPORTANT]
 591 .Java and Python application instrumentation and tracing
 592 ====
 593 If you need to instrument and trace <<java-application,Java
 594 applications>> on Fedora, you need to build and install
 595 LTTng-UST{nbsp}{revision} <<building-from-source,from source>> and pass
 596 the `--enable-java-agent-jul`, `--enable-java-agent-log4j`, or
 597 `--enable-java-agent-all` options to the `configure` script, depending
 598 on which Java logging framework you use.
 599
 600 If you need to instrument and trace <<python-application,Python
 601 applications>> on Fedora, you need to build and install
 602 LTTng-UST{nbsp}{revision} from source and pass the
 603 `--enable-python-agent` option to the `configure` script.
 604 ====
 605
 606
 607 [[arch-linux]]
 608 === Arch Linux
 609
 610 LTTng-UST{nbsp}{revision} is available in the _extra_
 611 repository of Arch Linux, while LTTng-tools{nbsp}{revision} and
 612 LTTng-modules{nbsp}{revision} are available in the
 613 https://aur.archlinux.org/[AUR].
 614
 615 To install LTTng{nbsp}{revision} on Arch Linux, using
 616 https://github.com/Jguer/yay[yay] for the AUR packages:
 617
 618 . Install the main LTTng{nbsp}{revision} packages:
 619 +
 620 --
 621 [role="term"]
 622 ----
 623 # pacman -Sy lttng-ust
 624 $ yay -Sy lttng-tools
 625 $ yay -Sy lttng-modules
 626 ----
 627 --
 628
 629 . **If you need to instrument and trace <<python-application,Python
 630   applications>>**, install the LTTng-UST Python agent:
 631 +
 632 --
 633 [role="term"]
 634 ----
 635 # pacman -Sy python-lttngust
 636 ----
 637 --
 638
 639
 640 [[alpine-linux]]
 641 === Alpine Linux
 642
 643 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
 644 Alpine Linux{nbsp}3.16, Alpine Linux{nbsp}3.17, or Alpine Linux{nbsp}3.18:
 645
 646 . Add the LTTng packages:
 647 +
 648 --
 649 [role="term"]
 650 ----
 651 # apk add lttng-tools
 652 # apk add lttng-ust-dev
 653 ----
 654 --
 655
 656 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 657 +
 658 --
 659 [role="term",subs="attributes,specialcharacters"]
 660 ----
 661 $ cd $(mktemp -d) &&
 662   wget http://lttng.org/files/lttng-modules/lttng-modules-latest-{revision}.tar.bz2 &&
 663   tar -xf lttng-modules-latest-{revision}.tar.bz2 &&
 664   cd lttng-modules-{revision}.* &&
 665   make &&
 666   sudo make modules_install &&
 667   sudo depmod -a
 668 ----
 669 --
 670
 671
 672 [[buildroot]]
 673 === Buildroot
 674
 675 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2022.02, Buildroot{nbsp}2022.05,
 676 Buildroot{nbsp}2022.08, Buildroot{nbsp}2022.11, Buildroot{nbsp}2023.02,
 677 Buildroot{nbsp}2023.05, or Buildroot{nbsp}2023.08:
 678
 679 . Launch the Buildroot configuration tool:
 680 +
 681 --
 682 [role="term"]
 683 ----
 684 $ make menuconfig
 685 ----
 686 --
 687
 688 . In **Kernel**, check **Linux kernel**.
 689 . In **Toolchain**, check **Enable WCHAR support**.
 690 . In **Target packages**{nbsp}&#8594; **Debugging, profiling and benchmark**,
 691   check **lttng-modules** and **lttng-tools**.
 692 . In **Target packages**{nbsp}&#8594; **Libraries**{nbsp}&#8594;
 693   **Other**, check **lttng-libust**.
 694
 695
 696 [[oe-yocto]]
 697 === OpenEmbedded and Yocto
 698
 699 LTTng{nbsp}{revision} recipes are available in the
 700 https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
 701 layer for Yocto Project{nbsp}3.3 _Honister_, Yocto Project{nbsp}4.0 _Kirkstone_,
 702 Yocto Project{nbsp}4.1 _Langdale_, Yocto Project{nbsp}4.2 _Mickledore_, and
 703 Yocto Project{nbsp}4.3 _Nanbield_ under the following names:
 704
 705 * `lttng-tools`
 706 * `lttng-modules`
 707 * `lttng-ust`
 708
 709 With BitBake, the simplest way to include LTTng recipes in your target
 710 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
 711
 712 ----
 713 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
 714 ----
 715
 716 If you use Hob:
 717
 718 . Select a machine and an image recipe.
 719 . Click **Edit image recipe**.
 720 . Under the **All recipes** tab, search for **lttng**.
 721 . Check the desired LTTng recipes.
 722
 723
 724 [[building-from-source]]
 725 === Build from source
 726
 727 To build and install LTTng{nbsp}{revision} from source:
 728
 729 . Using the package manager of your distribution, or from source,
 730   install the following dependencies of LTTng-tools and LTTng-UST:
 731 +
 732 --
 733 * https://sourceforge.net/projects/libuuid/[libuuid]
 734 * https://directory.fsf.org/wiki/Popt[popt]
 735 * https://liburcu.org/[Userspace RCU]
 736 * http://www.xmlsoft.org/[libxml2]
 737 * **Optional**: https://github.com/numactl/numactl[numactl]
 738 --
 739
 740 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 741 +
 742 --
 743 [role="term"]
 744 ----
 745 $ cd $(mktemp -d) &&
 746   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
 747   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
 748   cd lttng-modules-2.13.* &&
 749   make &&
 750   sudo make modules_install &&
 751   sudo depmod -a
 752 ----
 753 --
 754
 755 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
 756 +
 757 --
 758 [role="term"]
 759 ----
 760 $ cd $(mktemp -d) &&
 761   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
 762   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
 763   cd lttng-ust-2.13.* &&
 764   ./configure &&
 765   make &&
 766   sudo make install &&
 767   sudo ldconfig
 768 ----
 769 --
 770 +
 771 Add `--disable-numa` to `./configure` if you don't have
 772 https://github.com/numactl/numactl[numactl].
 773 +
 774 --
 775 [IMPORTANT]
 776 .Java and Python application tracing
 777 ====
 778 If you need to instrument and have LTTng trace <<java-application,Java
 779 applications>>, pass the `--enable-java-agent-jul`,
 780 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
 781 `configure` script, depending on which Java logging framework you use.
 782
 783 If you need to instrument and have LTTng trace
 784 <<python-application,Python applications>>, pass the
 785 `--enable-python-agent` option to the `configure` script. You can set
 786 the env:PYTHON environment variable to the path to the Python interpreter
 787 for which to install the LTTng-UST Python agent package.
 788 ====
 789 --
 790 +
 791 --
 792 [NOTE]
 793 ====
 794 By default, LTTng-UST libraries are installed to
 795 dir:{/usr/local/lib}, which is the de facto directory in which to
 796 keep self-compiled and third-party libraries.
 797
 798 When <<building-tracepoint-providers-and-user-application,linking an
 799 instrumented user application with `liblttng-ust`>>:
 800
 801 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
 802   variable.
 803
 804 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
 805   man:gcc(1), man:g++(1), or man:clang(1).
 806 ====
 807 --
 808
 809 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
 810 +
 811 --
 812 [role="term"]
 813 ----
 814 $ cd $(mktemp -d) &&
 815   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
 816   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
 817   cd lttng-tools-2.13.* &&
 818   ./configure &&
 819   make &&
 820   sudo make install &&
 821   sudo ldconfig
 822 ----
 823 --
 824
 825 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
 826 previous steps automatically for a given version of LTTng and confine
 827 the installed files to a specific directory. This can be useful to try
 828 LTTng without installing it on your system.
 829
 830 [[linux-kernel-sig]]
 831 === Linux kernel module signature
 832
 833 Linux kernel modules require trusted signatures in order to be loaded
 834 when any of the following is true:
 835
 836 * The system boots with
 837   https://uefi.org/specs/UEFI/2.10/32_Secure_Boot_and_Driver_Signing.html#secure-boot-and-driver-signing[Secure Boot]
 838   enabled.
 839
 840 * The Linux kernel which boots is configured with
 841   `CONFIG_MODULE_SIG_FORCE`.
 842
 843 * The Linux kernel boots with a command line containing
 844   `module.sig_enforce=1`.
 845
 846 .`root` user running <<lttng-sessiond,`lttng-sessiond`>> which fails to load a required <<lttng-modules,kernel module>> due to the signature enforcement policies.
 847 ====
 848 [role="term"]
 849 ----
 850 # lttng-sessiond
 851 Warning: No tracing group detected
 852 modprobe: ERROR: could not insert 'lttng_ring_buffer_client_discard': Key was rejected by service
 853 Error: Unable to load required module lttng-ring-buffer-client-discard
 854 Warning: No kernel tracer available
 855 ----
 856 ====
 857
 858 There are several methods to enroll trusted keys for signing modules
 859 that are built from source. The precise details vary from one Linux
 860 version to another, and distributions may have their own mechanisms. For
 861 example, https://github.com/dell/dkms[DKMS] may autogenerate a key and
 862 sign modules, but the key isn't automatically enrolled.
 863
 864 See
 865 https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html[Kernel
 866 module signing facility] and the documentation of your distribution
 867 to learn more about signing Linux kernel modules.
 868
 869 [[getting-started]]
 870 == Quick start
 871
 872 This is a short guide to get started quickly with LTTng kernel and user
 873 space tracing.
 874
 875 Before you follow this guide, make sure to <<installing-lttng,install>>
 876 LTTng.
 877
 878 This tutorial walks you through the steps to:
 879
 880 . <<tracing-the-linux-kernel,Record Linux kernel events>>.
 881
 882 . <<tracing-your-own-user-application,Record the events of a user
 883   application>> written in C.
 884
 885 . <<viewing-and-analyzing-your-traces,View and analyze the
 886   recorded events>>.
 887
 888
 889 [[tracing-the-linux-kernel]]
 890 === Record Linux kernel events
 891
 892 NOTE: The following command lines start with the `#` prompt because you
 893 need root privileges to control the Linux kernel LTTng tracer. You can
 894 also control the kernel tracer as a regular user if your Unix user is a
 895 member of the <<tracing-group,tracing group>>.
 896
 897 . Create a <<tracing-session,recording session>> to write LTTng traces
 898   to dir:{/tmp/my-kernel-trace}:
 899 +
 900 --
 901 [role="term"]
 902 ----
 903 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
 904 ----
 905 --
 906
 907 . List the available kernel tracepoints and system calls:
 908 +
 909 --
 910 [role="term"]
 911 ----
 912 # lttng list --kernel
 913 # lttng list --kernel --syscall
 914 ----
 915 --
 916
 917 . Create <<event,recording event rules>> which match events having
 918   the desired names, for example the `sched_switch` and
 919   `sched_process_fork` tracepoints, and the man:open(2) and man:close(2)
 920   system calls:
 921 +
 922 --
 923 [role="term"]
 924 ----
 925 # lttng enable-event --kernel sched_switch,sched_process_fork
 926 # lttng enable-event --kernel --syscall open,close
 927 ----
 928 --
 929 +
 930 Create a recording event rule which matches _all_ the Linux kernel
 931 tracepoint events with the opt:lttng-enable-event(1):--all option
 932 (recording with such a recording event rule generates a lot of data):
 933 +
 934 --
 935 [role="term"]
 936 ----
 937 # lttng enable-event --kernel --all
 938 ----
 939 --
 940
 941 . <<basic-tracing-session-control,Start recording>>:
 942 +
 943 --
 944 [role="term"]
 945 ----
 946 # lttng start
 947 ----
 948 --
 949
 950 . Do some operation on your system for a few seconds. For example,
 951   load a website, or list the files of a directory.
 952
 953 . <<creating-destroying-tracing-sessions,Destroy>> the current
 954   recording session:
 955 +
 956 --
 957 [role="term"]
 958 ----
 959 # lttng destroy
 960 ----
 961 --
 962 +
 963 The man:lttng-destroy(1) command doesn't destroy the trace data; it
 964 only destroys the state of the recording session.
 965 +
 966 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
 967 implicitly (see ``<<basic-tracing-session-control,Start and stop a
 968 recording session>>''). You need to stop recording to make LTTng flush
 969 the remaining trace data and make the trace readable.
 970
 971 . For the sake of this example, make the recorded trace accessible to
 972   the non-root users:
 973 +
 974 --
 975 [role="term"]
 976 ----
 977 # chown -R $(whoami) /tmp/my-kernel-trace
 978 ----
 979 --
 980
 981 See ``<<viewing-and-analyzing-your-traces,View and analyze the
 982 recorded events>>'' to view the recorded events.
 983
 984
 985 [[tracing-your-own-user-application]]
 986 === Record user application events
 987
 988 This section walks you through a simple example to record the events of
 989 a _Hello world_ program written in{nbsp}C.
 990
 991 To create the traceable user application:
 992
 993 . Create the tracepoint provider header file, which defines the
 994   tracepoints and the events they can generate:
 995 +
 996 --
 997 [source,c]
 998 .path:{hello-tp.h}
 999 ----
1000 #undef LTTNG_UST_TRACEPOINT_PROVIDER
1001 #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
1002
1003 #undef LTTNG_UST_TRACEPOINT_INCLUDE
1004 #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
1005
1006 #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
1007 #define _HELLO_TP_H
1008
1009 #include <lttng/tracepoint.h>
1010
1011 LTTNG_UST_TRACEPOINT_EVENT(
1012     hello_world,
1013     my_first_tracepoint,
1014     LTTNG_UST_TP_ARGS(
1015         int, my_integer_arg,
1016         char *, my_string_arg
1017     ),
1018     LTTNG_UST_TP_FIELDS(
1019         lttng_ust_field_string(my_string_field, my_string_arg)
1020         lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
1021     )
1022 )
1023
1024 #endif /* _HELLO_TP_H */
1025
1026 #include <lttng/tracepoint-event.h>
1027 ----
1028 --
1029
1030 . Create the tracepoint provider package source file:
1031 +
1032 --
1033 [source,c]
1034 .path:{hello-tp.c}
1035 ----
1036 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
1037 #define LTTNG_UST_TRACEPOINT_DEFINE
1038
1039 #include "hello-tp.h"
1040 ----
1041 --
1042
1043 . Build the tracepoint provider package:
1044 +
1045 --
1046 [role="term"]
1047 ----
1048 $ gcc -c -I. hello-tp.c
1049 ----
1050 --
1051
1052 . Create the _Hello World_ application source file:
1053 +
1054 --
1055 [source,c]
1056 .path:{hello.c}
1057 ----
1058 #include <stdio.h>
1059 #include "hello-tp.h"
1060
1061 int main(int argc, char *argv[])
1062 {
1063     unsigned int i;
1064
1065     puts("Hello, World!\nPress Enter to continue...");
1066
1067     /*
1068      * The following getchar() call only exists for the purpose of this
1069      * demonstration, to pause the application in order for you to have
1070      * time to list its tracepoints. You don't need it otherwise.
1071      */
1072     getchar();
1073
1074     /*
1075      * An lttng_ust_tracepoint() call.
1076      *
1077      * Arguments, as defined in `hello-tp.h`:
1078      *
1079      * 1. Tracepoint provider name (required)
1080      * 2. Tracepoint name (required)
1081      * 3. `my_integer_arg` (first user-defined argument)
1082      * 4. `my_string_arg` (second user-defined argument)
1083      *
1084      * Notice the tracepoint provider and tracepoint names are
1085      * C identifiers, NOT strings: they're in fact parts of variables
1086      * that the macros in `hello-tp.h` create.
1087      */
1088     lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,
1089                          "hi there!");
1090
1091     for (i = 0; i < argc; i++) {
1092         lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1093                              i, argv[i]);
1094     }
1095
1096     puts("Quitting now!");
1097     lttng_ust_tracepoint(hello_world, my_first_tracepoint,
1098                          i * i, "i^2");
1099     return 0;
1100 }
1101 ----
1102 --
1103
1104 . Build the application:
1105 +
1106 --
1107 [role="term"]
1108 ----
1109 $ gcc -c hello.c
1110 ----
1111 --
1112
1113 . Link the application with the tracepoint provider package,
1114   `liblttng-ust` and `libdl`:
1115 +
1116 --
1117 [role="term"]
1118 ----
1119 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1120 ----
1121 --
1122
1123 Here's the whole build process:
1124
1125 [role="img-100"]
1126 .Build steps of the user space tracing tutorial.
1127 image::ust-flow.png[]
1128
1129 To record the events of the user application:
1130
1131 . Run the application with a few arguments:
1132 +
1133 --
1134 [role="term"]
1135 ----
1136 $ ./hello world and beyond
1137 ----
1138 --
1139 +
1140 You see:
1141 +
1142 --
1143 ----
1144 Hello, World!
1145 Press Enter to continue...
1146 ----
1147 --
1148
1149 . Start an LTTng <<lttng-sessiond,session daemon>>:
1150 +
1151 --
1152 [role="term"]
1153 ----
1154 $ lttng-sessiond --daemonize
1155 ----
1156 --
1157 +
1158 NOTE: A session daemon might already be running, for example as a
1159 service that the service manager of your distribution started.
1160
1161 . List the available user space tracepoints:
1162 +
1163 --
1164 [role="term"]
1165 ----
1166 $ lttng list --userspace
1167 ----
1168 --
1169 +
1170 You see the `hello_world:my_first_tracepoint` tracepoint listed
1171 under the `./hello` process.
1172
1173 . Create a <<tracing-session,recording session>>:
1174 +
1175 --
1176 [role="term"]
1177 ----
1178 $ lttng create my-user-space-session
1179 ----
1180 --
1181
1182 . Create a <<event,recording event rule>> which matches user space
1183   tracepoint events named `hello_world:my_first_tracepoint`:
1184 +
1185 --
1186 [role="term"]
1187 ----
1188 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1189 ----
1190 --
1191
1192 . <<basic-tracing-session-control,Start recording>>:
1193 +
1194 --
1195 [role="term"]
1196 ----
1197 $ lttng start
1198 ----
1199 --
1200
1201 . Go back to the running `hello` application and press **Enter**.
1202 +
1203 The program executes all `lttng_ust_tracepoint()` instrumentation
1204 points, emitting events as the event rule you created in step{nbsp}5
1205 matches them, and
1206 exits.
1207
1208 . <<creating-destroying-tracing-sessions,Destroy>> the current
1209   recording session:
1210 +
1211 --
1212 [role="term"]
1213 ----
1214 $ lttng destroy
1215 ----
1216 --
1217 +
1218 The man:lttng-destroy(1) command doesn't destroy the trace data; it
1219 only destroys the state of the recording session.
1220 +
1221 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1222 implicitly (see ``<<basic-tracing-session-control,Start and stop a
1223 recording session>>''). You need to stop recording to make LTTng flush
1224 the remaining trace data and make the trace readable.
1225
1226 By default, LTTng saves the traces to the
1227 +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+ directory, where
1228 +__NAME__+ is the recording session name. The env:LTTNG_HOME environment
1229 variable defaults to `$HOME` if not set.
1230
1231
1232 [[viewing-and-analyzing-your-traces]]
1233 === View and analyze the recorded events
1234
1235 Once you have completed the <<tracing-the-linux-kernel,Record Linux
1236 kernel events>> and <<tracing-your-own-user-application,Record user
1237 application events>> tutorials, you can inspect the recorded events.
1238
1239 There are tools you can use to read LTTng traces:
1240
1241 https://babeltrace.org/[Babeltrace{nbsp}2]::
1242     A rich, flexible trace manipulation toolkit which includes
1243     a versatile command-line interface
1244     (man:babeltrace2(1)),
1245     a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C{nbsp}library],
1246     and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1247     so that you can easily process or convert an LTTng trace with
1248     your own script.
1249 +
1250 The Babeltrace{nbsp}2 project ships with a plugin
1251 (man:babeltrace2-plugin-ctf(7)) which supports the format of the traces
1252 which LTTng produces, https://diamon.org/ctf/[CTF].
1253
1254 http://tracecompass.org/[Trace Compass]::
1255     A graphical user interface for viewing and analyzing any type of
1256     logs or traces, including those of LTTng.
1257
1258 NOTE: This section assumes that LTTng wrote the traces it recorded
1259 during the previous tutorials to their default location, in the
1260 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1261 environment variable defaults to `$HOME` if not set.
1262
1263
1264 [[viewing-and-analyzing-your-traces-bt]]
1265 ==== Use the cmd:babeltrace2 command-line tool
1266
1267 The simplest way to list all the recorded events of an LTTng trace is to
1268 pass its path to man:babeltrace2(1), without options:
1269
1270 [role="term"]
1271 ----
1272 $ babeltrace2 ~/lttng-traces/my-user-space-session*
1273 ----
1274
1275 The cmd:babeltrace2 command finds all traces recursively within the
1276 given path and prints all their events, sorting them chronologically.
1277
1278 Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1279 further filtering:
1280
1281 [role="term"]
1282 ----
1283 $ babeltrace2 /tmp/my-kernel-trace | grep _switch
1284 ----
1285
1286 Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1287 the recorded events:
1288
1289 [role="term"]
1290 ----
1291 $ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1292 ----
1293
1294
1295 [[viewing-and-analyzing-your-traces-bt-python]]
1296 ==== Use the Babeltrace{nbsp}2 Python bindings
1297
1298 The <<viewing-and-analyzing-your-traces-bt,text output of
1299 cmd:babeltrace2>> is useful to isolate event records by simple matching
1300 using man:grep(1) and similar utilities. However, more elaborate
1301 filters, such as keeping only event records with a field value falling
1302 within a specific range, are not trivial to write using a shell.
1303 Moreover, reductions and even the most basic computations involving
1304 multiple event records are virtually impossible to implement.
1305
1306 Fortunately, Babeltrace{nbsp}2 ships with
1307 https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1308 which make it easy to read the event records of an LTTng trace
1309 sequentially and compute the desired information.
1310
1311 The following script accepts an LTTng Linux kernel trace path as its
1312 first argument and prints the short names of the top five running
1313 processes on CPU{nbsp}0 during the whole trace:
1314
1315 [source,python]
1316 .path:{top5proc.py}
1317 ----
1318 import bt2
1319 import sys
1320 import collections
1321
1322
1323 def top5proc():
1324     # Get the trace path from the first command-line argument
1325     it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1326
1327     # This counter dictionary will hold execution times:
1328     #
1329     #     Task command name -> Total execution time (ns)
1330     exec_times = collections.Counter()
1331
1332     # This holds the last `sched_switch` timestamp
1333     last_ts = None
1334
1335     for msg in it:
1336         # We only care about event messages
1337         if type(msg) is not bt2._EventMessageConst:
1338             continue
1339
1340         # Event of the event message
1341         event = msg.event
1342
1343         # Keep only `sched_switch` events
1344         if event.cls.name != 'sched_switch':
1345             continue
1346
1347         # Keep only records of events which LTTng emitted from CPU 0
1348         if event.packet.context_field['cpu_id'] != 0:
1349             continue
1350
1351         # Event timestamp (ns)
1352         cur_ts = msg.default_clock_snapshot.ns_from_origin
1353
1354         if last_ts is None:
1355             # Start here
1356             last_ts = cur_ts
1357
1358         # (Short) name of the previous task command
1359         prev_comm = str(event.payload_field['prev_comm'])
1360
1361         # Initialize an entry in our dictionary if not done yet
1362         if prev_comm not in exec_times:
1363             exec_times[prev_comm] = 0
1364
1365         # Compute previous command execution time
1366         diff = cur_ts - last_ts
1367
1368         # Update execution time of this command
1369         exec_times[prev_comm] += diff
1370
1371         # Update last timestamp
1372         last_ts = cur_ts
1373
1374     # Print top 5
1375     for name, ns in exec_times.most_common(5):
1376         print('{:20}{} s'.format(name, ns / 1e9))
1377
1378
1379 if __name__ == '__main__':
1380     top5proc()
1381 ----
1382
1383 Run this script:
1384
1385 [role="term"]
1386 ----
1387 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1388 ----
1389
1390 Output example:
1391
1392 ----
1393 swapper/0           48.607245889 s
1394 chromium            7.192738188 s
1395 pavucontrol         0.709894415 s
1396 Compositor          0.660867933 s
1397 Xorg.bin            0.616753786 s
1398 ----
1399
1400 Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1401 since we weren't using the CPU that much when recording, its first
1402 position in the list makes sense.
1403
1404
1405 [[core-concepts]]
1406 == [[understanding-lttng]]Core concepts
1407
1408 From a user's perspective, the LTTng system is built on a few concepts,
1409 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1410 operates by sending commands to the <<lttng-sessiond,session daemon>>
1411 (through <<liblttng-ctl-lttng,`liblttng-ctl`>>).
1412
1413 Understanding how those objects relate to each other is key to master
1414 the toolkit.
1415
1416 The core concepts of LTTng are:
1417
1418 * <<"event-rule","Instrumentation point, event rule, and event">>
1419 * <<trigger,Trigger>>
1420 * <<tracing-session,Recording session>>
1421 * <<domain,Tracing domain>>
1422 * <<channel,Channel and ring buffer>>
1423 * <<event,Recording event rule and event record>>
1424
1425 NOTE: The man:lttng-concepts(7) manual page also documents the core
1426 concepts of LTTng, with more links to other LTTng-tools manual pages.
1427
1428
1429 [[event-rule]]
1430 === Instrumentation point, event rule, and event
1431
1432 An _instrumentation point_ is a point, within a piece of software,
1433 which, when executed, creates an LTTng _event_.
1434
1435 LTTng offers various <<instrumentation-point-types,types of
1436 instrumentation>>.
1437
1438 An _event rule_ is a set of conditions to match a set of events.
1439
1440 When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is
1441 said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies _all_ the
1442 conditions of{nbsp}__ER__. This concept is similar to a
1443 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
1444 which matches a set of strings.
1445
1446 When an event rule matches an event, LTTng _emits_ the event, therefore
1447 attempting to execute one or more actions.
1448
1449 [IMPORTANT]
1450 ====
1451 [[event-creation-emission-opti]]The event creation and emission
1452 processes are documentation concepts to help understand the journey from
1453 an instrumentation point to the execution of actions.
1454
1455 The actual creation of an event can be costly because LTTng needs to
1456 evaluate the arguments of the instrumentation point.
1457
1458 In practice, LTTng implements various optimizations for the Linux kernel
1459 and user space <<domain,tracing domains>> to avoid actually creating an
1460 event when the tracer knows, thanks to properties which are independent
1461 from the event payload and current context, that it would never emit
1462 such an event. Those properties are:
1463
1464 * The <<instrumentation-point-types,instrumentation point type>>.
1465
1466 * The instrumentation point name.
1467
1468 * The instrumentation point log level.
1469
1470 * For a <<event,recording event rule>>:
1471 ** The status of the rule itself.
1472 ** The status of the <<channel,channel>>.
1473 ** The activity of the <<tracing-session,recording session>>.
1474 ** Whether or not the process for which LTTng would create the event is
1475    <<pid-tracking,allowed to record events>>.
1476
1477 In other words: if, for a given instrumentation point{nbsp}__IP__, the
1478 LTTng tracer knows that it would never emit an event,
1479 executing{nbsp}__IP__ represents a simple boolean variable check and,
1480 for a Linux kernel recording event rule, a few process attribute checks.
1481 ====
1482
1483 As of LTTng{nbsp}{revision}, there are two places where you can find an
1484 event rule:
1485
1486 <<event,Recording event rule>>::
1487     A specific type of event rule of which the action is to record the
1488     matched event as an event record.
1489 +
1490 See ``<<enabling-disabling-events,Create and enable a recording event
1491 rule>>'' to learn more.
1492
1493 ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13)::
1494     When the event rule of the trigger condition matches an event, LTTng
1495     can execute user-defined actions such as sending an LTTng
1496     notification,
1497     <<basic-tracing-session-control,starting a recording session>>,
1498     and more.
1499 +
1500 See “<<add-event-rule-matches-trigger,Add an ``event rule matches''
1501 trigger to a session daemon>>” to learn more.
1502
1503 For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy _all_ the
1504 basic conditions of an event rule{nbsp}__ER__, that is:
1505
1506 * The instrumentation point from which LTTng
1507   creates{nbsp}__E__ has a specific
1508   <<instrumentation-point-types,type>>.
1509
1510 * A pattern matches the name of{nbsp}__E__ while another pattern
1511   doesn't.
1512
1513 * The log level of the instrumentation point from which LTTng
1514   creates{nbsp}__E__ is at least as severe as some value, or is exactly
1515   some value.
1516
1517 * The fields of the payload of{nbsp}__E__ and the current context fields
1518   satisfy a filter expression.
1519
1520 A <<event,recording event rule>> has additional, implicit conditions to
1521 satisfy.
1522
1523
1524 [[instrumentation-point-types]]
1525 ==== Instrumentation point types
1526
1527 As of LTTng{nbsp}{revision}, the available instrumentation point
1528 types are, depending on the <<domain,tracing domain>>:
1529
1530 Linux kernel::
1531     LTTng tracepoint:::
1532         A statically defined point in the source code of the kernel
1533         image or of a kernel module using the
1534         <<lttng-modules,LTTng-modules>> macros.
1535
1536     Linux kernel system call:::
1537         Entry, exit, or both of a Linux kernel system call.
1538
1539     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kprobe]:::
1540         A single probe dynamically placed in the compiled kernel code.
1541 +
1542 When you create such an instrumentation point, you set its memory
1543 address or symbol name.
1544
1545     Linux user space probe:::
1546         A single probe dynamically placed at the entry of a compiled
1547         user space application/library function through the kernel.
1548 +
1549 When you create such an instrumentation point, you set:
1550 +
1551 --
1552 With the ELF method::
1553     Its application/library path and its symbol name.
1554
1555 With the USDT method::
1556     Its application/library path, its provider name, and its probe name.
1557 +
1558 ``USDT'' stands for _SystemTap User-level Statically Defined Tracing_,
1559 a http://dtrace.org/blogs/about/[DTrace]-style marker.
1560 --
1561 +
1562 As of LTTng{nbsp}{revision}, LTTng only supports USDT probes which
1563 are _not_ reference-counted.
1564
1565     Linux https://www.kernel.org/doc/html/latest/trace/kprobes.html[kretprobe]:::
1566         Entry, exit, or both of a Linux kernel function.
1567 +
1568 When you create such an instrumentation point, you set the memory
1569 address or symbol name of its function.
1570
1571 User space::
1572     LTTng tracepoint:::
1573         A statically defined point in the source code of a C/$$C++$$
1574         application/library using the
1575         <<lttng-ust,LTTng-UST>> macros.
1576
1577 `java.util.logging`, Apache log4j, and Python::
1578     Java or Python logging statement:::
1579         A method call on a Java or Python logger attached to an
1580         LTTng-UST handler.
1581
1582 See ``<<list-instrumentation-points,List the available instrumentation
1583 points>>'' to learn how to list available Linux kernel, user space, and
1584 logging instrumentation points.
1585
1586
1587 [[trigger]]
1588 === Trigger
1589
1590 A _trigger_ associates a condition to one or more actions.
1591
1592 When the condition of a trigger is satisfied, LTTng attempts to execute
1593 its actions.
1594
1595 As of LTTng{nbsp}{revision}, the available trigger conditions and
1596 actions are:
1597
1598 Conditions::
1599 +
1600 * The consumed buffer size of a given <<tracing-session,recording
1601   session>> becomes greater than some value.
1602
1603 * The buffer usage of a given <<channel,channel>> becomes greater than
1604   some value.
1605
1606 * The buffer usage of a given channel becomes less than some value.
1607
1608 * There's an ongoing <<session-rotation,recording session rotation>>.
1609
1610 * A recording session rotation becomes completed.
1611
1612 * An <<add-event-rule-matches-trigger,event rule matches>> an event.
1613
1614 Actions::
1615 +
1616 * <<trigger-event-notif,Send a notification>> to a user application.
1617 * <<basic-tracing-session-control,Start>> a given recording session.
1618 * <<basic-tracing-session-control,Stop>> a given recording session.
1619 * <<session-rotation,Archive the current trace chunk>> of a given
1620   recording session (rotate).
1621 * <<taking-a-snapshot,Take a snapshot>> of a given recording session.
1622
1623 A trigger belongs to a <<lttng-sessiond,session daemon>>, not to a
1624 specific recording session. For a given session daemon, each Unix user has
1625 its own, private triggers. Note, however, that the `root` Unix user may,
1626 for the root session daemon:
1627
1628 * Add a trigger as another Unix user.
1629
1630 * List all the triggers, regardless of their owner.
1631
1632 * Remove a trigger which belongs to another Unix user.
1633
1634 For a given session daemon and Unix user, a trigger has a unique name.
1635
1636
1637 [[tracing-session]]
1638 === Recording session
1639
1640 A _recording session_ (named ``tracing session'' prior to
1641 LTTng{nbsp}2.13) is a stateful dialogue between you and a
1642 <<lttng-sessiond,session daemon>> for everything related to
1643 <<event,event recording>>.
1644
1645 Everything that you do when you control LTTng tracers to record events
1646 happens within a recording session. In particular, a recording session:
1647
1648 * Has its own name, unique for a given session daemon.
1649
1650 * Has its own set of trace files, if any.
1651
1652 * Has its own state of activity (started or stopped).
1653 +
1654 An active recording session is an implicit <<event,recording event rule>>
1655 condition.
1656
1657 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1658   snapshot, or live).
1659
1660 * Has its own <<channel,channels>> to which are attached their own
1661   recording event rules.
1662
1663 * Has its own <<pid-tracking,process attribute inclusion sets>>.
1664
1665 [role="img-100"]
1666 .A _recording session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,recording event rules>>.
1667 image::concepts.png[]
1668
1669 Those attributes and objects are completely isolated between different
1670 recording sessions.
1671
1672 A recording session is like an
1673 https://en.wikipedia.org/wiki/Automated_teller_machine[ATM] session: the
1674 operations you do on the banking system through the ATM don't alter the
1675 data of other users of the same system. In the case of the ATM, a
1676 session lasts as long as your bank card is inside. In the case of LTTng,
1677 a recording session lasts from the man:lttng-create(1) command to the
1678 man:lttng-destroy(1) command.
1679
1680 [role="img-100"]
1681 .Each Unix user has its own set of recording sessions.
1682 image::many-sessions.png[]
1683
1684 A recording session belongs to a <<lttng-sessiond,session daemon>>. For a
1685 given session daemon, each Unix user has its own, private recording
1686 sessions. Note, however, that the `root` Unix user may operate on or
1687 destroy another user's recording session.
1688
1689
1690 [[tracing-session-mode]]
1691 ==== Recording session mode
1692
1693 LTTng offers four recording session modes:
1694
1695 [[local-mode]]Local mode::
1696     Write the trace data to the local file system.
1697
1698 [[net-streaming-mode]]Network streaming mode::
1699     Send the trace data over the network to a listening
1700     <<lttng-relayd,relay daemon>>.
1701
1702 [[snapshot-mode]]Snapshot mode::
1703     Only write the trace data to the local file system or send it to a
1704     listening relay daemon when LTTng <<taking-a-snapshot,takes a
1705     snapshot>>.
1706 +
1707 LTTng forces all the <<channel,channels>>
1708 to be created to be configured to be snapshot-ready.
1709 +
1710 LTTng takes a snapshot of such a recording session when:
1711 +
1712 --
1713 * You run the man:lttng-snapshot(1) command.
1714
1715 * LTTng executes a `snapshot-session` <<trigger,trigger>> action.
1716 --
1717
1718 [[live-mode]]Live mode::
1719     Send the trace data over the network to a listening relay daemon
1720     for <<lttng-live,live reading>>.
1721 +
1722 An LTTng live reader (for example, man:babeltrace2(1)) can connect to
1723 the same relay daemon to receive trace data while the recording session is
1724 active.
1725
1726
1727 [[domain]]
1728 === Tracing domain
1729
1730 A _tracing domain_ identifies a type of LTTng tracer.
1731
1732 A tracing domain has its own properties and features.
1733
1734 There are currently five available tracing domains:
1735
1736 * Linux kernel
1737 * User space
1738 * `java.util.logging` (JUL)
1739 * log4j
1740 * Python
1741
1742 You must specify a tracing domain to target a type of LTTng tracer when
1743 using some <<lttng-cli,cmd:lttng>> commands to avoid ambiguity. For
1744 example, because the Linux kernel and user space tracing domains support
1745 named tracepoints as <<event-rule,instrumentation points>>, you need to
1746 specify a tracing domain when you <<enabling-disabling-events,create
1747 an event rule>> because both tracing domains could have tracepoints
1748 sharing the same name.
1749
1750 You can create <<channel,channels>> in the Linux kernel and user space
1751 tracing domains. The other tracing domains have a single, default
1752 channel.
1753
1754
1755 [[channel]]
1756 === Channel and ring buffer
1757
1758 A _channel_ is an object which is responsible for a set of
1759 _ring buffers_.
1760
1761 Each ring buffer is divided into multiple _sub-buffers_. When a
1762 <<event,recording event rule>>
1763 matches an event, LTTng can record it to one or more sub-buffers of one
1764 or more channels.
1765
1766 When you <<enabling-disabling-channels,create a channel>>, you set its
1767 final attributes, that is:
1768
1769 * Its <<channel-buffering-schemes,buffering scheme>>.
1770
1771 * What to do <<channel-overwrite-mode-vs-discard-mode,when there's no
1772   space left>> for a new event record because all sub-buffers are full.
1773
1774 * The <<channel-subbuf-size-vs-subbuf-count,size of each ring buffer and
1775   how many sub-buffers>> a ring buffer has.
1776
1777 * The <<tracefile-rotation,size of each trace file LTTng writes for this
1778   channel and the maximum count>> of trace files.
1779
1780 * The periods of its <<channel-read-timer,read>>,
1781   <<channel-switch-timer,switch>>, and <<channel-monitor-timer,monitor>>
1782   timers.
1783
1784 * For a Linux kernel channel: its output type.
1785 +
1786 See the opt:lttng-enable-channel(1):--output option of the
1787 man:lttng-enable-channel(1) command.
1788
1789 * For a user space channel: the value of its
1790   <<blocking-timeout-example,blocking timeout>>.
1791
1792 A channel is always associated to a <<domain,tracing domain>>. The
1793 `java.util.logging` (JUL), log4j, and Python tracing domains each have a
1794 default channel which you can't configure.
1795
1796 A channel owns <<event,recording event rules>>.
1797
1798
1799 [[channel-buffering-schemes]]
1800 ==== Buffering scheme
1801
1802 A channel has at least one ring buffer _per CPU_. LTTng always records
1803 an event to the ring buffer dedicated to the CPU which emits it.
1804
1805 The buffering scheme of a user space channel determines what has its own
1806 set of per-CPU ring buffers:
1807
1808 Per-user buffering::
1809   Allocate one set of ring buffers--one per CPU--shared by all the
1810   instrumented processes of:
1811     If your Unix user is `root`:::
1812       Each Unix user.
1813 +
1814 --
1815 [role="img-100"]
1816 .Per-user buffering scheme (recording session belongs to the `root` Unix user).
1817 image::per-user-buffering-root.png[]
1818 --
1819
1820     Otherwise:::
1821       Your Unix user.
1822 +
1823 --
1824 [role="img-100"]
1825 .Per-user buffering scheme (recording session belongs to the `Bob` Unix user).
1826 image::per-user-buffering.png[]
1827 --
1828
1829 Per-process buffering::
1830   Allocate one set of ring buffers--one per CPU--for each
1831   instrumented process of:
1832     If your Unix user is `root`:::
1833       All Unix users.
1834 +
1835 --
1836 [role="img-100"]
1837 .Per-process buffering scheme (recording session belongs to the `root` Unix user).
1838 image::per-process-buffering-root.png[]
1839 --
1840
1841     Otherwise:::
1842       Your Unix user.
1843 +
1844 --
1845 [role="img-100"]
1846 .Per-process buffering scheme (recording session belongs to the `Bob` Unix user).
1847 image::per-process-buffering.png[]
1848 --
1849
1850 The per-process buffering scheme tends to consume more memory than the
1851 per-user option because systems generally have more instrumented
1852 processes than Unix users running instrumented processes. However, the
1853 per-process buffering scheme ensures that one process having a high
1854 event throughput won't fill all the shared sub-buffers of the same Unix
1855 user, only its own.
1856
1857 The buffering scheme of a Linux kernel channel is always to allocate a
1858 single set of ring buffers for the whole system. This scheme is similar
1859 to the per-user option, but with a single, global user ``running'' the
1860 kernel.
1861
1862
1863 [[channel-overwrite-mode-vs-discard-mode]]
1864 ==== Event record loss mode
1865
1866 When LTTng emits an event, LTTng can record it to a specific, available
1867 sub-buffer within the ring buffers of specific channels. When there's no
1868 space left in a sub-buffer, the tracer marks it as consumable and
1869 another, available sub-buffer starts receiving the following event
1870 records. An LTTng <<lttng-consumerd,consumer daemon>> eventually
1871 consumes the marked sub-buffer, which returns to the available state.
1872
1873 [NOTE]
1874 [role="docsvg-channel-subbuf-anim"]
1875 ====
1876 {note-no-anim}
1877 ====
1878
1879 In an ideal world, sub-buffers are consumed faster than they're filled,
1880 as it's the case in the previous animation. In the real world,
1881 however, all sub-buffers can be full at some point, leaving no space to
1882 record the following events.
1883
1884 By default, <<lttng-modules,LTTng-modules>> and <<lttng-ust,LTTng-UST>>
1885 are _non-blocking_ tracers: when there's no available sub-buffer to
1886 record an event, it's acceptable to lose event records when the
1887 alternative would be to cause substantial delays in the execution of the
1888 instrumented application. LTTng privileges performance over integrity;
1889 it aims at perturbing the instrumented application as little as possible
1890 in order to make the detection of subtle race conditions and rare
1891 interrupt cascades possible.
1892
1893 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1894 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1895 example>> to learn how to use the blocking mode.
1896
1897 When it comes to losing event records because there's no available
1898 sub-buffer, or because the blocking timeout of
1899 the channel is reached, the _event record loss mode_ of the channel
1900 determines what to do. The available event record loss modes are:
1901
1902 [[discard-mode]]Discard mode::
1903     Drop the newest event records until a sub-buffer becomes available.
1904 +
1905 This is the only available mode when you specify a blocking timeout.
1906 +
1907 With this mode, LTTng increments a count of lost event records when an
1908 event record is lost and saves this count to the trace. A trace reader
1909 can use the saved discarded event record count of the trace to decide
1910 whether or not to perform some analysis even if trace data is known to
1911 be missing.
1912
1913 [[overwrite-mode]]Overwrite mode::
1914     Clear the sub-buffer containing the oldest event records and start
1915     writing the newest event records there.
1916 +
1917 This mode is sometimes called _flight recorder mode_ because it's
1918 similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight
1919 recorder]: always keep a fixed amount of the latest data. It's also
1920 similar to the roll mode of an oscilloscope.
1921 +
1922 Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer
1923 its sequence number within its data stream. With a <<local-mode,local>>,
1924 <<net-streaming-mode,network streaming>>, or <<live-mode,live>> recording
1925 session, a trace reader can use such sequence numbers to report lost
1926 packets. A trace reader can use the saved discarded sub-buffer (packet)
1927 count of the trace to decide whether or not to perform some analysis
1928 even if trace data is known to be missing.
1929 +
1930 With this mode, LTTng doesn't write to the trace the exact number of
1931 lost event records in the lost sub-buffers.
1932
1933 Which mechanism you should choose depends on your context: prioritize
1934 the newest or the oldest event records in the ring buffer?
1935
1936 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1937 as soon as a there's no space left for a new event record, whereas in
1938 discard mode, the tracer only discards the event record that doesn't
1939 fit.
1940
1941 There are a few ways to decrease your probability of losing event
1942 records. The ``<<channel-subbuf-size-vs-subbuf-count,Sub-buffer size and
1943 count>>'' section shows how to fine-tune the sub-buffer size and count
1944 of a channel to virtually stop losing event records, though at the cost
1945 of greater memory usage.
1946
1947
1948 [[channel-subbuf-size-vs-subbuf-count]]
1949 ==== Sub-buffer size and count
1950
1951 A channel has one or more ring buffer for each CPU of the target system.
1952
1953 See the ``<<channel-buffering-schemes,Buffering scheme>>'' section to
1954 learn how many ring buffers of a given channel are dedicated to each CPU
1955 depending on its buffering scheme.
1956
1957 Set the size of each sub-buffer the ring buffers of a channel contain
1958 and how many there are
1959 when you <<enabling-disabling-channels,create it>>.
1960
1961 Note that LTTng switching the current sub-buffer of a ring buffer
1962 (marking a full one as consumable and switching to an available one for
1963 LTTng to record the next events) introduces noticeable CPU overhead.
1964 Knowing this, the following list presents a few practical situations
1965 along with how to configure the sub-buffer size and count for them:
1966
1967 High event throughput::
1968     In general, prefer large sub-buffers to lower the risk of losing
1969     event records.
1970 +
1971 Having larger sub-buffers also ensures a lower sub-buffer switching
1972 frequency.
1973 +
1974 The sub-buffer count is only meaningful if you create the channel in
1975 <<overwrite-mode,overwrite mode>>: in this case, if LTTng overwrites a
1976 sub-buffer, then the other sub-buffers are left unaltered.
1977
1978 Low event throughput::
1979     In general, prefer smaller sub-buffers since the risk of losing
1980     event records is low.
1981 +
1982 Because LTTng emits events less frequently, the sub-buffer switching
1983 frequency should remain low and therefore the overhead of the tracer
1984 shouldn't be a problem.
1985
1986 Low memory system::
1987     If your target system has a low memory limit, prefer fewer first,
1988     then smaller sub-buffers.
1989 +
1990 Even if the system is limited in memory, you want to keep the
1991 sub-buffers as large as possible to avoid a high sub-buffer switching
1992 frequency.
1993
1994 Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format,
1995 which means event record data is very compact. For example, the average
1996 LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a
1997 sub-buffer size of 1{nbsp}MiB is considered large.
1998
1999 The previous scenarios highlight the major trade-off between a few large
2000 sub-buffers and more, smaller sub-buffers: sub-buffer switching
2001 frequency vs. how many event records are lost in overwrite mode.
2002 Assuming a constant event throughput and using the overwrite mode, the
2003 two following configurations have the same ring buffer total size:
2004
2005 [NOTE]
2006 [role="docsvg-channel-subbuf-size-vs-count-anim"]
2007 ====
2008 {note-no-anim}
2009 ====
2010
2011 Two sub-buffers of 4{nbsp}MiB each::
2012     Expect a very low sub-buffer switching frequency, but if LTTng
2013     ever needs to overwrite a sub-buffer, half of the event records so
2014     far (4{nbsp}MiB) are definitely lost.
2015
2016 Eight sub-buffers of 1{nbsp}MiB each::
2017     Expect four times the tracer overhead of the configuration above,
2018     but if LTTng needs to overwrite a sub-buffer, only the eighth of
2019     event records so far (1{nbsp}MiB) are definitely lost.
2020
2021 In <<discard-mode,discard mode>>, the sub-buffer count parameter is
2022 pointless: use two sub-buffers and set their size according to your
2023 requirements.
2024
2025
2026 [[tracefile-rotation]]
2027 ==== Maximum trace file size and count (trace file rotation)
2028
2029 By default, trace files can grow as large as needed.
2030
2031 Set the maximum size of each trace file that LTTng writes of a given
2032 channel when you <<enabling-disabling-channels,create it>>.
2033
2034 When the size of a trace file reaches the fixed maximum size of the
2035 channel, LTTng creates another file to contain the next event records.
2036 LTTng appends a file count to each trace file name in this case.
2037
2038 If you set the trace file size attribute when you create a channel, the
2039 maximum number of trace files that LTTng creates is _unlimited_ by
2040 default. To limit them, set a maximum number of trace files. When the
2041 number of trace files reaches the fixed maximum count of the channel,
2042 LTTng overwrites the oldest trace file. This mechanism is called _trace
2043 file rotation_.
2044
2045 [IMPORTANT]
2046 ====
2047 Even if you don't limit the trace file count, always assume that LTTng
2048 manages all the trace files of the recording session.
2049
2050 In other words, there's no safe way to know if LTTng still holds a given
2051 trace file open with the trace file rotation feature.
2052
2053 The only way to obtain an unmanaged, self-contained LTTng trace before
2054 you <<creating-destroying-tracing-sessions,destroy the recording session>>
2055 is with the <<session-rotation,recording session rotation>> feature, which
2056 is available since LTTng{nbsp}2.11.
2057 ====
2058
2059
2060 [[channel-timers]]
2061 ==== Timers
2062
2063 Each channel can have up to three optional timers:
2064
2065 [[channel-switch-timer]]Switch timer::
2066     When this timer expires, a sub-buffer switch happens: for each ring
2067     buffer of the channel, LTTng marks the current sub-buffer as
2068     consumable and _switches_ to an available one to record the next
2069     events.
2070 +
2071 [NOTE]
2072 [role="docsvg-channel-switch-timer"]
2073 ====
2074 {note-no-anim}
2075 ====
2076 +
2077 A switch timer is useful to ensure that LTTng consumes and commits trace
2078 data to trace files or to a distant <<lttng-relayd,relay daemon>>
2079 periodically in case of a low event throughput.
2080 +
2081 Such a timer is also convenient when you use large
2082 <<channel-subbuf-size-vs-subbuf-count,sub-buffers>> to cope with a
2083 sporadic high event throughput, even if the throughput is otherwise low.
2084 +
2085 Set the period of the switch timer of a channel when you
2086 <<enabling-disabling-channels,create it>> with
2087 the opt:lttng-enable-channel(1):--switch-timer option.
2088
2089 [[channel-read-timer]]Read timer::
2090     When this timer expires, LTTng checks for full, consumable
2091     sub-buffers.
2092 +
2093 By default, the LTTng tracers use an asynchronous message mechanism to
2094 signal a full sub-buffer so that a <<lttng-consumerd,consumer daemon>>
2095 can consume it.
2096 +
2097 When such messages must be avoided, for example in real-time
2098 applications, use this timer instead.
2099 +
2100 Set the period of the read timer of a channel when you
2101 <<enabling-disabling-channels,create it>> with the
2102 opt:lttng-enable-channel(1):--read-timer option.
2103
2104 [[channel-monitor-timer]]Monitor timer::
2105     When this timer expires, the consumer daemon samples some channel
2106     statistics to evaluate the following <<trigger,trigger>>
2107     conditions:
2108 +
2109 --
2110 . The consumed buffer size of a given <<tracing-session,recording
2111   session>> becomes greater than some value.
2112 . The buffer usage of a given channel becomes greater than some value.
2113 . The buffer usage of a given channel becomes less than some value.
2114 --
2115 +
2116 If you disable the monitor timer of a channel{nbsp}__C__:
2117 +
2118 --
2119 * The consumed buffer size value of the recording session of{nbsp}__C__
2120   could be wrong for trigger condition type{nbsp}1: the consumed buffer
2121   size of{nbsp}__C__ won't be part of the grand total.
2122
2123 * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3)
2124   for{nbsp}__C__ will never be satisfied.
2125 --
2126 +
2127 Set the period of the monitor timer of a channel when you
2128 <<enabling-disabling-channels,create it>> with the
2129 opt:lttng-enable-channel(1):--monitor-timer option.
2130
2131
2132 [[event]]
2133 === Recording event rule and event record
2134
2135 A _recording event rule_ is a specific type of <<event-rule,event rule>>
2136 of which the action is to serialize and record the matched event as an
2137 _event record_.
2138
2139 Set the explicit conditions of a recording event rule when you
2140 <<enabling-disabling-events,create it>>. A recording event rule also has
2141 the following implicit conditions:
2142
2143 * The recording event rule itself is enabled.
2144 +
2145 A recording event rule is enabled on creation.
2146
2147 * The <<channel,channel>> to which the recording event rule is attached
2148   is enabled.
2149 +
2150 A channel is enabled on creation.
2151
2152 * The <<tracing-session,recording session>> of the recording event rule is
2153   <<basic-tracing-session-control,active>> (started).
2154 +
2155 A recording session is inactive (stopped) on creation.
2156
2157 * The process for which LTTng creates an event to match is
2158   <<pid-tracking,allowed to record events>>.
2159 +
2160 All processes are allowed to record events on recording session
2161 creation.
2162
2163 You always attach a recording event rule to a channel, which belongs to
2164 a recording session, when you create it.
2165
2166 When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__,
2167 LTTng attempts to serialize and record{nbsp}__E__ to one of the
2168 available sub-buffers of the channel to which{nbsp}__E__ is attached.
2169
2170 When multiple matching recording event rules are attached to the same
2171 channel, LTTng attempts to serialize and record the matched event
2172 _once_. In the following example, the second recording event rule is
2173 redundant when both are enabled:
2174
2175 [role="term"]
2176 ----
2177 $ lttng enable-event --userspace hello:world
2178 $ lttng enable-event --userspace hello:world --loglevel=INFO
2179 ----
2180
2181 [role="img-100"]
2182 .Logical path from an instrumentation point to an event record.
2183 image::event-rule.png[]
2184
2185 As of LTTng{nbsp}{revision}, you cannot remove a recording event
2186 rule: it exists as long as its recording session exists.
2187
2188
2189 [[plumbing]]
2190 == Components of noch:{LTTng}
2191
2192 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
2193 to call LTTng a simple _tool_ since it's composed of multiple
2194 interacting components.
2195
2196 This section describes those components, explains their respective
2197 roles, and shows how they connect together to form the LTTng ecosystem.
2198
2199 The following diagram shows how the most important components of LTTng
2200 interact with user applications, the Linux kernel, and you:
2201
2202 [role="img-100"]
2203 .Control and trace data paths between LTTng components.
2204 image::plumbing.png[]
2205
2206 The LTTng project integrates:
2207
2208 LTTng-tools::
2209     Libraries and command-line interface to control recording sessions:
2210 +
2211 * <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
2212 * <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
2213 * <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
2214 * <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
2215 * <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
2216 * <<persistent-memory-file-systems,`lttng-crash` command-line tool>>
2217   (man:lttng-crash(1)).
2218
2219 LTTng-UST::
2220     Libraries and Java/Python packages to instrument and trace user
2221     applications:
2222 +
2223 * <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
2224   headers to instrument and trace any native user application.
2225 * <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
2226 ** `liblttng-ust-libc-wrapper`
2227 ** `liblttng-ust-pthread-wrapper`
2228 ** `liblttng-ust-cyg-profile`
2229 ** `liblttng-ust-cyg-profile-fast`
2230 ** `liblttng-ust-dl`
2231 * <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
2232   Java applications using `java.util.logging` or
2233   Apache log4j{nbsp}1.2 logging.
2234 * <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
2235   Python applications using the standard `logging` package.
2236
2237 LTTng-modules::
2238     <<lttng-modules,Linux kernel modules>> to instrument and trace the
2239     kernel:
2240 +
2241 * LTTng kernel tracer module.
2242 * Recording ring buffer kernel modules.
2243 * Probe kernel modules.
2244 * LTTng logger kernel module.
2245
2246
2247 [[lttng-cli]]
2248 === Tracing control command-line interface
2249
2250 The _man:lttng(1) command-line tool_ is the standard user interface to
2251 control LTTng <<tracing-session,recording sessions>>.
2252
2253 The cmd:lttng tool is part of LTTng-tools.
2254
2255 The cmd:lttng tool is linked with
2256 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
2257 one or more <<lttng-sessiond,session daemons>> behind the scenes.
2258
2259 The cmd:lttng tool has a Git-like interface:
2260
2261 [role="term"]
2262 ----
2263 $ lttng [GENERAL OPTIONS] <COMMAND> [COMMAND OPTIONS]
2264 ----
2265
2266 The ``<<controlling-tracing,Tracing control>>'' section explores the
2267 available features of LTTng through its cmd:lttng tool.
2268
2269
2270 [[liblttng-ctl-lttng]]
2271 === Tracing control library
2272
2273 [role="img-100"]
2274 .The tracing control library.
2275 image::plumbing-liblttng-ctl.png[]
2276
2277 The _LTTng control library_, `liblttng-ctl`, is used to communicate with
2278 a <<lttng-sessiond,session daemon>> using a C{nbsp}API that hides the
2279 underlying details of the protocol.
2280
2281 `liblttng-ctl` is part of LTTng-tools.
2282
2283 The <<lttng-cli,cmd:lttng command-line tool>> is linked with
2284 `liblttng-ctl`.
2285
2286 Use `liblttng-ctl` in C or $$C++$$ source code by including its
2287 ``master'' header:
2288
2289 [source,c]
2290 ----
2291 #include <lttng/lttng.h>
2292 ----
2293
2294 As of LTTng{nbsp}{revision}, the best available developer documentation
2295 for `liblttng-ctl` is its installed header files. Functions and
2296 structures are documented with header comments.
2297
2298
2299 [[lttng-ust]]
2300 === User space tracing library
2301
2302 [role="img-100"]
2303 .The user space tracing library.
2304 image::plumbing-liblttng-ust.png[]
2305
2306 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
2307 is the LTTng user space tracer.
2308
2309 `liblttng-ust` receives commands from a <<lttng-sessiond,session
2310 daemon>>, for example to allow specific instrumentation points to emit
2311 LTTng <<event-rule,events>>, and writes event records to <<channel,ring
2312 buffers>> shared with a <<lttng-consumerd,consumer daemon>>.
2313
2314 `liblttng-ust` is part of LTTng-UST.
2315
2316 `liblttng-ust` can also send asynchronous messages to the session daemon
2317 when it emits an event. This supports the ``event rule matches''
2318 <<trigger,trigger>> condition feature (see
2319 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2320 to a session daemon>>”).
2321
2322 Public C{nbsp}header files are installed beside `liblttng-ust` to
2323 instrument any <<c-application,C or $$C++$$ application>>.
2324
2325 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
2326 packages, use their own <<tracepoint-provider,tracepoint provider
2327 package>> which is linked with `liblttng-ust`.
2328
2329 An application or library doesn't have to initialize `liblttng-ust`
2330 manually: its constructor does the necessary tasks to register the
2331 application to a session daemon. The initialization phase also
2332 configures instrumentation points depending on the <<event-rule,event
2333 rules>> that you already created.
2334
2335
2336 [[lttng-ust-agents]]
2337 === User space tracing agents
2338
2339 [role="img-100"]
2340 .The user space tracing agents.
2341 image::plumbing-lttng-ust-agents.png[]
2342
2343 The _LTTng-UST Java and Python agents_ are regular Java and Python
2344 packages which add LTTng tracing capabilities to the
2345 native logging frameworks.
2346
2347 The LTTng-UST agents are part of LTTng-UST.
2348
2349 In the case of Java, the
2350 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
2351 core logging facilities] and
2352 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
2353 Note that Apache Log4j{nbsp}2 isn't supported.
2354
2355 In the case of Python, the standard
2356 https://docs.python.org/3/library/logging.html[`logging`] package
2357 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
2358 LTTng-UST Python agent package.
2359
2360 The applications using the LTTng-UST agents are in the
2361 `java.util.logging` (JUL), log4j, and Python <<domain,tracing domains>>.
2362
2363 Both agents use the same mechanism to convert log statements to LTTng
2364 events. When an agent initializes, it creates a log handler that
2365 attaches to the root logger. The agent also registers to a
2366 <<lttng-sessiond,session daemon>>. When the user application executes a
2367 log statement, the root logger passes it to the log handler of the
2368 agent. The custom log handler of the agent calls a native function in a
2369 tracepoint provider package shared library linked with
2370 <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
2371 other fields, like its logger name and its log level. This native
2372 function contains a user space instrumentation point, therefore tracing
2373 the log statement.
2374
2375 The log level condition of a <<event,recording event rule>> is
2376 considered when tracing a Java or a Python application, and it's
2377 compatible with the standard `java.util.logging`, log4j, and Python log
2378 levels.
2379
2380
2381 [[lttng-modules]]
2382 === LTTng kernel modules
2383
2384 [role="img-100"]
2385 .The LTTng kernel modules.
2386 image::plumbing-lttng-modules.png[]
2387
2388 The _LTTng kernel modules_ are a set of Linux kernel modules
2389 which implement the kernel tracer of the LTTng project.
2390
2391 The LTTng kernel modules are part of LTTng-modules.
2392
2393 The LTTng kernel modules include:
2394
2395 * A set of _probe_ modules.
2396 +
2397 Each module attaches to a specific subsystem
2398 of the Linux kernel using its tracepoint instrument points.
2399 +
2400 There are also modules to attach to the entry and return points of the
2401 Linux system call functions.
2402
2403 * _Ring buffer_ modules.
2404 +
2405 A ring buffer implementation is provided as kernel modules. The LTTng
2406 kernel tracer writes to ring buffers; a
2407 <<lttng-consumerd,consumer daemon>> reads from ring buffers.
2408
2409 * The _LTTng kernel tracer_ module.
2410 * The <<proc-lttng-logger-abi,_LTTng logger_>> module.
2411 +
2412 The LTTng logger module implements the special path:{/proc/lttng-logger}
2413 (and path:{/dev/lttng-logger}, since LTTng{nbsp}2.11) files so that any
2414 executable can generate LTTng events by opening those files and
2415 writing to them.
2416
2417 The LTTng kernel tracer can also send asynchronous messages to the
2418 <<lttng-sessiond,session daemon>> when it emits an event.
2419 This supports the ``event rule matches''
2420 <<trigger,trigger>> condition feature (see
2421 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2422 to a session daemon>>”).
2423
2424 Generally, you don't have to load the LTTng kernel modules manually
2425 (using man:modprobe(8), for example): a root session daemon loads the
2426 necessary modules when starting. If you have extra probe modules, you
2427 can specify to load them to the session daemon on the command line
2428 (see the opt:lttng-sessiond(8):--extra-kmod-probes option). See also
2429 <<linux-kernel-sig,Linux kernel module signature>>.
2430
2431 The LTTng kernel modules are installed in
2432 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
2433 the kernel release (output of `uname --kernel-release`).
2434
2435
2436 [[lttng-sessiond]]
2437 === Session daemon
2438
2439 [role="img-100"]
2440 .The session daemon.
2441 image::plumbing-sessiond.png[]
2442
2443 The _session daemon_, man:lttng-sessiond(8), is a
2444 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which:
2445
2446 * Manages <<tracing-session,recording sessions>>.
2447
2448 * Controls the various components (like tracers and
2449   <<lttng-consumerd,consumer daemons>>) of LTTng.
2450
2451 * Sends <<notif-trigger-api,asynchronous notifications>> to user
2452   applications.
2453
2454 The session daemon is part of LTTng-tools.
2455
2456 The session daemon sends control requests to and receives control
2457 responses from:
2458
2459 * The <<lttng-ust,user space tracing library>>.
2460 +
2461 Any instance of the user space tracing library first registers to
2462 a session daemon. Then, the session daemon can send requests to
2463 this instance, such as:
2464 +
2465 --
2466 ** Get the list of tracepoints.
2467 ** Share a <<event,recording event rule>> so that the user space tracing
2468    library can decide whether or not a given tracepoint can emit events.
2469    Amongst the possible conditions of a recording event rule is a filter
2470    expression which `liblttng-ust` evaluates before it emits an event.
2471 ** Share <<channel,channel>> attributes and ring buffer locations.
2472 --
2473 +
2474 The session daemon and the user space tracing library use a Unix
2475 domain socket to communicate.
2476
2477 * The <<lttng-ust-agents,user space tracing agents>>.
2478 +
2479 Any instance of a user space tracing agent first registers to
2480 a session daemon. Then, the session daemon can send requests to
2481 this instance, such as:
2482 +
2483 --
2484 ** Get the list of loggers.
2485 ** Enable or disable a specific logger.
2486 --
2487 +
2488 The session daemon and the user space tracing agent use a TCP connection
2489 to communicate.
2490
2491 * The <<lttng-modules,LTTng kernel tracer>>.
2492 * The <<lttng-consumerd,consumer daemon>>.
2493 +
2494 The session daemon sends requests to the consumer daemon to instruct
2495 it where to send the trace data streams, amongst other information.
2496
2497 * The <<lttng-relayd,relay daemon>>.
2498
2499 The session daemon receives commands from the
2500 <<liblttng-ctl-lttng,tracing control library>>.
2501
2502 The session daemon can receive asynchronous messages from the
2503 <<lttng-ust,user space>> and <<lttng-modules,kernel>> tracers
2504 when they emit events. This supports the ``event rule matches''
2505 <<trigger,trigger>> condition feature (see
2506 “<<add-event-rule-matches-trigger,Add an ``event rule matches'' trigger
2507 to a session daemon>>”).
2508
2509 The root session daemon loads the appropriate
2510 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2511 one or more <<lttng-consumerd,consumer daemons>> as soon as you create
2512 a <<event,recording event rule>>.
2513
2514 The session daemon doesn't send and receive trace data: this is the
2515 role of the <<lttng-consumerd,consumer daemon>> and
2516 <<lttng-relayd,relay daemon>>. It does, however, generate the
2517 https://diamon.org/ctf/[CTF] metadata stream.
2518
2519 Each Unix user can have its own session daemon instance. The
2520 recording sessions which different session daemons manage are completely
2521 independent.
2522
2523 The root user's session daemon is the only one which is
2524 allowed to control the LTTng kernel tracer, and its spawned consumer
2525 daemon is the only one which is allowed to consume trace data from the
2526 LTTng kernel tracer. Note, however, that any Unix user which is a member
2527 of the <<tracing-group,tracing group>> is allowed
2528 to create <<channel,channels>> in the
2529 Linux kernel <<domain,tracing domain>>, and therefore to use the Linux
2530 kernel LTTng tracer.
2531
2532 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2533 session daemon when using its `create` command if none is currently
2534 running. You can also start the session daemon manually.
2535
2536
2537 [[lttng-consumerd]]
2538 === Consumer daemon
2539
2540 [role="img-100"]
2541 .The consumer daemon.
2542 image::plumbing-consumerd.png[]
2543
2544 The _consumer daemon_, cmd:lttng-consumerd, is a
2545 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] which shares
2546 ring buffers with user applications or with the LTTng kernel modules to
2547 collect trace data and send it to some location (file system or to a
2548 <<lttng-relayd,relay daemon>> over the network).
2549
2550 The consumer daemon is part of LTTng-tools.
2551
2552 You don't start a consumer daemon manually: a consumer daemon is always
2553 spawned by a <<lttng-sessiond,session daemon>> as soon as you create a
2554 <<event,recording event rule>>, that is, before you start recording. When
2555 you kill its owner session daemon, the consumer daemon also exits
2556 because it's the child process of the session daemon. Command-line
2557 options of man:lttng-sessiond(8) target the consumer daemon process.
2558
2559 There are up to two running consumer daemons per Unix user, whereas only
2560 one session daemon can run per user. This is because each process can be
2561 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2562 and 64-bit processes, it's more efficient to have separate
2563 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2564 exception: it can have up to _three_ running consumer daemons: 32-bit
2565 and 64-bit instances for its user applications, and one more
2566 reserved for collecting kernel trace data.
2567
2568
2569 [[lttng-relayd]]
2570 === Relay daemon
2571
2572 [role="img-100"]
2573 .The relay daemon.
2574 image::plumbing-relayd.png[]
2575
2576 The _relay daemon_, man:lttng-relayd(8), is a
2577 https://en.wikipedia.org/wiki/Daemon_(computing)[daemon] acting as a bridge
2578 between remote session and consumer daemons, local trace files, and a
2579 remote live trace reader.
2580
2581 The relay daemon is part of LTTng-tools.
2582
2583 The main purpose of the relay daemon is to implement a receiver of
2584 <<sending-trace-data-over-the-network,trace data over the network>>.
2585 This is useful when the target system doesn't have much file system
2586 space to write trace files locally.
2587
2588 The relay daemon is also a server to which a
2589 <<lttng-live,live trace reader>> can
2590 connect. The live trace reader sends requests to the relay daemon to
2591 receive trace data as the target system records events. The
2592 communication protocol is named _LTTng live_; it's used over TCP
2593 connections.
2594
2595 Note that you can start the relay daemon on the target system directly.
2596 This is the setup of choice when the use case is to view/analyze events
2597 as the target system records them without the need of a remote system.
2598
2599
2600 [[instrumenting]]
2601 == [[using-lttng]]Instrumentation
2602
2603 There are many examples of tracing and monitoring in our everyday life:
2604
2605 * You have access to real-time and historical weather reports and
2606   forecasts thanks to weather stations installed around the country.
2607 * You know your heart is safe thanks to an electrocardiogram.
2608 * You make sure not to drive your car too fast and to have enough fuel
2609   to reach your destination thanks to gauges visible on your dashboard.
2610
2611 All the previous examples have something in common: they rely on
2612 **instruments**. Without the electrodes attached to the surface of your
2613 body skin, cardiac monitoring is futile.
2614
2615 LTTng, as a tracer, is no different from those real life examples. If
2616 you're about to trace a software system or, in other words, record its
2617 history of execution, you better have **instrumentation points** in the
2618 subject you're tracing, that is, the actual software system.
2619
2620 <<instrumentation-point-types,Various ways>> were developed to
2621 instrument a piece of software for LTTng tracing. The most
2622 straightforward one is to manually place static instrumentation points,
2623 called _tracepoints_, in the source code of the application. The Linux
2624 kernel <<domain,tracing domain>> also makes it possible to dynamically
2625 add instrumentation points.
2626
2627 If you're only interested in tracing the Linux kernel, your
2628 instrumentation needs are probably already covered by the built-in
2629 <<lttng-modules,Linux kernel instrumentation points>> of LTTng. You may
2630 also wish to have LTTng trace a user application which is already
2631 instrumented for LTTng tracing. In such cases, skip this whole section
2632 and read the topics of the ``<<controlling-tracing,Tracing control>>''
2633 section.
2634
2635 Many methods are available to instrument a piece of software for LTTng
2636 tracing:
2637
2638 * <<c-application,Instrument a C/$$C++$$ user application>>.
2639 * <<prebuilt-ust-helpers,Load a prebuilt user space tracing helper>>.
2640 * <<java-application,Instrument a Java application>>.
2641 * <<python-application,Instrument a Python application>>.
2642 * <<proc-lttng-logger-abi,Use the LTTng logger>>.
2643 * <<instrumenting-linux-kernel,Instrument a Linux kernel image or module>>.
2644
2645
2646 [[c-application]]
2647 === [[cxx-application]]Instrument a C/$$C++$$ user application
2648
2649 The high level procedure to instrument a C or $$C++$$ user application
2650 with the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`,
2651 is:
2652
2653 . <<tracepoint-provider,Create the source files of a tracepoint provider
2654   package>>.
2655
2656 . <<probing-the-application-source-code,Add tracepoints to
2657   the source code of the application>>.
2658
2659 . <<building-tracepoint-providers-and-user-application,Build and link
2660   a tracepoint provider package and the user application>>.
2661
2662 If you need quick, man:printf(3)-like instrumentation, skip those steps
2663 and use <<tracef,`lttng_ust_tracef()`>> or
2664 <<tracelog,`lttng_ust_tracelog()`>> instead.
2665
2666 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2667 instrument a user application with `liblttng-ust`.
2668
2669
2670 [[tracepoint-provider]]
2671 ==== Create the source files of a tracepoint provider package
2672
2673 A _tracepoint provider_ is a set of compiled functions which provide
2674 **tracepoints** to an application, the type of instrumentation point
2675 which LTTng-UST provides.
2676
2677 Those functions can make LTTng emit events with user-defined fields and
2678 serialize those events as event records to one or more LTTng-UST
2679 <<channel,channel>> sub-buffers. The `lttng_ust_tracepoint()` macro,
2680 which you <<probing-the-application-source-code,insert in the source
2681 code of a user application>>, calls those functions.
2682
2683 A _tracepoint provider package_ is an object file (`.o`) or a shared
2684 library (`.so`) which contains one or more tracepoint providers. Its
2685 source files are:
2686
2687 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2688 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2689
2690 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2691 the LTTng user space tracer, at run time.
2692
2693 [role="img-100"]
2694 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2695 image::ust-app.png[]
2696
2697 NOTE: If you need quick, man:printf(3)-like instrumentation, skip
2698 creating and using a tracepoint provider and use
2699 <<tracef,`lttng_ust_tracef()`>> or <<tracelog,`lttng_ust_tracelog()`>>
2700 instead.
2701
2702
2703 [[tpp-header]]
2704 ===== Create a tracepoint provider header file template
2705
2706 A _tracepoint provider header file_ contains the tracepoint definitions
2707 of a tracepoint provider.
2708
2709 To create a tracepoint provider header file:
2710
2711 . Start from this template:
2712 +
2713 --
2714 [source,c]
2715 .Tracepoint provider header file template (`.h` file extension).
2716 ----
2717 #undef LTTNG_UST_TRACEPOINT_PROVIDER
2718 #define LTTNG_UST_TRACEPOINT_PROVIDER provider_name
2719
2720 #undef LTTNG_UST_TRACEPOINT_INCLUDE
2721 #define LTTNG_UST_TRACEPOINT_INCLUDE "./tp.h"
2722
2723 #if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
2724 #define _TP_H
2725
2726 #include <lttng/tracepoint.h>
2727
2728 /*
2729  * Use LTTNG_UST_TRACEPOINT_EVENT(), LTTNG_UST_TRACEPOINT_EVENT_CLASS(),
2730  * LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(), and
2731  * LTTNG_UST_TRACEPOINT_LOGLEVEL() here.
2732  */
2733
2734 #endif /* _TP_H */
2735
2736 #include <lttng/tracepoint-event.h>
2737 ----
2738 --
2739 +
2740 Replace:
2741 +
2742 * +__provider_name__+ with the name of your tracepoint provider.
2743 * `"tp.h"` with the name of your tracepoint provider header file.
2744
2745 . Below the `#include <lttng/tracepoint.h>` line, put your
2746   <<defining-tracepoints,tracepoint definitions>>.
2747
2748 Your tracepoint provider name must be unique amongst all the possible
2749 tracepoint provider names used on the same target system. We suggest to
2750 include the name of your project or company in the name, for example,
2751 `org_lttng_my_project_tpp`.
2752
2753
2754 [[defining-tracepoints]]
2755 ===== Create a tracepoint definition
2756
2757 A _tracepoint definition_ defines, for a given tracepoint:
2758
2759 * Its **input arguments**.
2760 +
2761 They're the macro parameters that the `lttng_ust_tracepoint()` macro
2762 accepts for this particular tracepoint in the source code of the user
2763 application.
2764
2765 * Its **output event fields**.
2766 +
2767 They're the sources of event fields that form the payload of any event
2768 that the execution of the `lttng_ust_tracepoint()` macro emits for this
2769 particular tracepoint.
2770
2771 Create a tracepoint definition with the
2772 `LTTNG_UST_TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2773 line in the
2774 <<tpp-header,tracepoint provider header file template>>.
2775
2776 The syntax of the `LTTNG_UST_TRACEPOINT_EVENT()` macro is:
2777
2778 [source,c]
2779 .`LTTNG_UST_TRACEPOINT_EVENT()` macro syntax.
2780 ----
2781 LTTNG_UST_TRACEPOINT_EVENT(
2782     /* Tracepoint provider name */
2783     provider_name,
2784
2785     /* Tracepoint name */
2786     tracepoint_name,
2787
2788     /* Input arguments */
2789     LTTNG_UST_TP_ARGS(
2790         arguments
2791     ),
2792
2793     /* Output event fields */
2794     LTTNG_UST_TP_FIELDS(
2795         fields
2796     )
2797 )
2798 ----
2799
2800 Replace:
2801
2802 * +__provider_name__+ with your tracepoint provider name.
2803 * +__tracepoint_name__+ with your tracepoint name.
2804 * +__arguments__+ with the <<tpp-def-input-args,input arguments>>.
2805 * +__fields__+ with the <<tpp-def-output-fields,output event field>>
2806   definitions.
2807
2808 The full name of this tracepoint is `provider_name:tracepoint_name`.
2809
2810 [IMPORTANT]
2811 .Event name length limitation
2812 ====
2813 The concatenation of the tracepoint provider name and the tracepoint
2814 name must not exceed **254{nbsp}characters**. If it does, the
2815 instrumented application compiles and runs, but LTTng throws multiple
2816 warnings and you could experience serious issues.
2817 ====
2818
2819 [[tpp-def-input-args]]The syntax of the `LTTNG_UST_TP_ARGS()` macro is:
2820
2821 [source,c]
2822 .`LTTNG_UST_TP_ARGS()` macro syntax.
2823 ----
2824 LTTNG_UST_TP_ARGS(
2825     type, arg_name
2826 )
2827 ----
2828
2829 Replace:
2830
2831 * +__type__+ with the C{nbsp}type of the argument.
2832 * +__arg_name__+ with the argument name.
2833
2834 You can repeat +__type__+ and +__arg_name__+ up to 10{nbsp}times to have
2835 more than one argument.
2836
2837 .`LTTNG_UST_TP_ARGS()` usage with three arguments.
2838 ====
2839 [source,c]
2840 ----
2841 LTTNG_UST_TP_ARGS(
2842     int, count,
2843     float, ratio,
2844     const char*, query
2845 )
2846 ----
2847 ====
2848
2849 The `LTTNG_UST_TP_ARGS()` and `LTTNG_UST_TP_ARGS(void)` forms are valid
2850 to create a tracepoint definition with no input arguments.
2851
2852 [[tpp-def-output-fields]]The `LTTNG_UST_TP_FIELDS()` macro contains a
2853 list of `lttng_ust_field_*()` macros. Each `lttng_ust_field_*()` macro
2854 defines one event field. See man:lttng-ust(3) for a complete description
2855 of the available `lttng_ust_field_*()` macros. A `lttng_ust_field_*()`
2856 macro specifies the type, size, and byte order of one event field.
2857
2858 Each `lttng_ust_field_*()` macro takes an _argument expression_
2859 parameter. This is a C{nbsp}expression that the tracer evaluates at the
2860 `lttng_ust_tracepoint()` macro site in the source code of the
2861 application. This expression provides the source of data of a field. The
2862 argument expression can include input argument names listed in the
2863 `LTTNG_UST_TP_ARGS()` macro.
2864
2865 Each `lttng_ust_field_*()` macro also takes a _field name_ parameter.
2866 Field names must be unique within a given tracepoint definition.
2867
2868 Here's a complete tracepoint definition example:
2869
2870 .Tracepoint definition.
2871 ====
2872 The following tracepoint definition defines a tracepoint which takes
2873 three input arguments and has four output event fields.
2874
2875 [source,c]
2876 ----
2877 #include "my-custom-structure.h"
2878
2879 LTTNG_UST_TRACEPOINT_EVENT(
2880     my_provider,
2881     my_tracepoint,
2882     LTTNG_UST_TP_ARGS(
2883         const struct my_custom_structure *, my_custom_structure,
2884         float, ratio,
2885         const char *, query
2886     ),
2887     LTTNG_UST_TP_FIELDS(
2888         lttng_ust_field_string(query_field, query)
2889         lttng_ust_field_float(double, ratio_field, ratio)
2890         lttng_ust_field_integer(int, recv_size,
2891                                 my_custom_structure->recv_size)
2892         lttng_ust_field_integer(int, send_size,
2893                                 my_custom_structure->send_size)
2894     )
2895 )
2896 ----
2897
2898 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
2899 macro in the source code of your application like this:
2900
2901 [source,c]
2902 ----
2903 lttng_ust_tracepoint(my_provider, my_tracepoint,
2904                      my_structure, some_ratio, the_query);
2905 ----
2906 ====
2907
2908 NOTE: The LTTng-UST tracer only evaluates the arguments of a tracepoint
2909 at run time when such a tracepoint _could_ emit an event. See
2910 <<event-creation-emission-opti,this note>> to learn more.
2911
2912
2913 [[using-tracepoint-classes]]
2914 ===== Use a tracepoint class
2915
2916 A _tracepoint class_ is a class of tracepoints which share the same
2917 output event field definitions. A _tracepoint instance_ is one
2918 instance of such a defined tracepoint class, with its own tracepoint
2919 name.
2920
2921 The <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()` macro>> is
2922 actually a shorthand which defines both a tracepoint class and a
2923 tracepoint instance at the same time.
2924
2925 When you build a tracepoint provider package, the C or $$C++$$ compiler
2926 creates one serialization function for each **tracepoint class**. A
2927 serialization function is responsible for serializing the event fields
2928 of a tracepoint to a sub-buffer when recording.
2929
2930 For various performance reasons, when your situation requires multiple
2931 tracepoint definitions with different names, but with the same event
2932 fields, we recommend that you manually create a tracepoint class and
2933 instantiate as many tracepoint instances as needed. One positive effect
2934 of such a design, amongst other advantages, is that all tracepoint
2935 instances of the same tracepoint class reuse the same serialization
2936 function, thus reducing
2937 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2938
2939 .Use a tracepoint class and tracepoint instances.
2940 ====
2941 Consider the following three tracepoint definitions:
2942
2943 [source,c]
2944 ----
2945 LTTNG_UST_TRACEPOINT_EVENT(
2946     my_app,
2947     get_account,
2948     LTTNG_UST_TP_ARGS(
2949         int, userid,
2950         size_t, len
2951     ),
2952     LTTNG_UST_TP_FIELDS(
2953         lttng_ust_field_integer(int, userid, userid)
2954         lttng_ust_field_integer(size_t, len, len)
2955     )
2956 )
2957
2958 LTTNG_UST_TRACEPOINT_EVENT(
2959     my_app,
2960     get_settings,
2961     LTTNG_UST_TP_ARGS(
2962         int, userid,
2963         size_t, len
2964     ),
2965     LTTNG_UST_TP_FIELDS(
2966         lttng_ust_field_integer(int, userid, userid)
2967         lttng_ust_field_integer(size_t, len, len)
2968     )
2969 )
2970
2971 LTTNG_UST_TRACEPOINT_EVENT(
2972     my_app,
2973     get_transaction,
2974     LTTNG_UST_TP_ARGS(
2975         int, userid,
2976         size_t, len
2977     ),
2978     LTTNG_UST_TP_FIELDS(
2979         lttng_ust_field_integer(int, userid, userid)
2980         lttng_ust_field_integer(size_t, len, len)
2981     )
2982 )
2983 ----
2984
2985 In this case, we create three tracepoint classes, with one implicit
2986 tracepoint instance for each of them: `get_account`, `get_settings`, and
2987 `get_transaction`. However, they all share the same event field names
2988 and types. Hence three identical, yet independent serialization
2989 functions are created when you build the tracepoint provider package.
2990
2991 A better design choice is to define a single tracepoint class and three
2992 tracepoint instances:
2993
2994 [source,c]
2995 ----
2996 /* The tracepoint class */
2997 LTTNG_UST_TRACEPOINT_EVENT_CLASS(
2998     /* Tracepoint class provider name */
2999     my_app,
3000
3001     /* Tracepoint class name */
3002     my_class,
3003
3004     /* Input arguments */
3005     LTTNG_UST_TP_ARGS(
3006         int, userid,
3007         size_t, len
3008     ),
3009
3010     /* Output event fields */
3011     LTTNG_UST_TP_FIELDS(
3012         lttng_ust_field_integer(int, userid, userid)
3013         lttng_ust_field_integer(size_t, len, len)
3014     )
3015 )
3016
3017 /* The tracepoint instances */
3018 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3019     /* Tracepoint class provider name */
3020     my_app,
3021
3022     /* Tracepoint class name */
3023     my_class,
3024
3025     /* Instance provider name */
3026     my_app,
3027
3028     /* Tracepoint name */
3029     get_account,
3030
3031     /* Input arguments */
3032     LTTNG_UST_TP_ARGS(
3033         int, userid,
3034         size_t, len
3035     )
3036 )
3037 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3038     my_app,
3039     my_class,
3040     get_settings,
3041     LTTNG_UST_TP_ARGS(
3042         int, userid,
3043         size_t, len
3044     )
3045 )
3046 LTTNG_UST_TRACEPOINT_EVENT_INSTANCE(
3047     my_app,
3048     my_class,
3049     get_transaction,
3050     LTTNG_UST_TP_ARGS(
3051         int, userid,
3052         size_t, len
3053     )
3054 )
3055 ----
3056 ====
3057
3058 The tracepoint class and instance provider names must be the same if the
3059 `LTTNG_UST_TRACEPOINT_EVENT_CLASS()` and
3060 `LTTNG_UST_TRACEPOINT_EVENT_INSTANCE()` expansions are part of the same
3061 translation unit. See man:lttng-ust(3) to learn more.
3062
3063
3064 [[assigning-log-levels]]
3065 ===== Assign a log level to a tracepoint definition
3066
3067 Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
3068 with the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro.
3069
3070 Assigning different levels of severity to tracepoint definitions can be
3071 useful: when you <<enabling-disabling-events,create a recording event
3072 rule>>, you can target tracepoints having a log level at least as severe
3073 as a specific value.
3074
3075 The concept of LTTng-UST log levels is similar to the levels found
3076 in typical logging frameworks:
3077
3078 * In a logging framework, the log level is given by the function
3079   or method name you use at the log statement site: `debug()`,
3080   `info()`, `warn()`, `error()`, and so on.
3081
3082 * In LTTng-UST, you statically assign the log level to a tracepoint
3083   definition; any `lttng_ust_tracepoint()` macro invocation which refers
3084   to this definition has this log level.
3085
3086 You must use `LTTNG_UST_TRACEPOINT_LOGLEVEL()` _after_ the
3087 <<defining-tracepoints,`LTTNG_UST_TRACEPOINT_EVENT()`>> or
3088 <<using-tracepoint-classes,`LTTNG_UST_TRACEPOINT_INSTANCE()`>> macro for
3089 a given tracepoint.
3090
3091 The syntax of the `LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro is:
3092
3093 [source,c]
3094 .`LTTNG_UST_TRACEPOINT_LOGLEVEL()` macro syntax.
3095 ----
3096 LTTNG_UST_TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
3097 ----
3098
3099 Replace:
3100
3101 * +__provider_name__+ with the tracepoint provider name.
3102 * +__tracepoint_name__+ with the tracepoint name.
3103 * +__log_level__+ with the log level to assign to the tracepoint
3104   definition named +__tracepoint_name__+ in the +__provider_name__+
3105   tracepoint provider.
3106 +
3107 See man:lttng-ust(3) for a list of available log level names.
3108
3109 .Assign the `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT` log level to a tracepoint definition.
3110 ====
3111 [source,c]
3112 ----
3113 /* Tracepoint definition */
3114 LTTNG_UST_TRACEPOINT_EVENT(
3115     my_app,
3116     get_transaction,
3117     LTTNG_UST_TP_ARGS(
3118         int, userid,
3119         size_t, len
3120     ),
3121     LTTNG_UST_TP_FIELDS(
3122         lttng_ust_field_integer(int, userid, userid)
3123         lttng_ust_field_integer(size_t, len, len)
3124     )
3125 )
3126
3127 /* Log level assignment */
3128 LTTNG_UST_TRACEPOINT_LOGLEVEL(my_app, get_transaction,
3129                               LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT)
3130 ----
3131 ====
3132
3133
3134 [[tpp-source]]
3135 ===== Create a tracepoint provider package source file
3136
3137 A _tracepoint provider package source file_ is a C source file which
3138 includes a <<tpp-header,tracepoint provider header file>> to expand its
3139 macros into event serialization and other functions.
3140
3141 Use the following tracepoint provider package source file template:
3142
3143 [source,c]
3144 .Tracepoint provider package source file template.
3145 ----
3146 #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
3147
3148 #include "tp.h"
3149 ----
3150
3151 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
3152 header file>> name. You may also include more than one tracepoint
3153 provider header file here to create a tracepoint provider package
3154 holding more than one tracepoint providers.
3155
3156
3157 [[probing-the-application-source-code]]
3158 ==== Add tracepoints to the source code of an application
3159
3160 Once you <<tpp-header,create a tracepoint provider header file>>, use
3161 the `lttng_ust_tracepoint()` macro in the source code of your
3162 application to insert the tracepoints that this header
3163 <<defining-tracepoints,defines>>.
3164
3165 The `lttng_ust_tracepoint()` macro takes at least two parameters: the
3166 tracepoint provider name and the tracepoint name. The corresponding
3167 tracepoint definition defines the other parameters.
3168
3169 .`lttng_ust_tracepoint()` usage.
3170 ====
3171 The following <<defining-tracepoints,tracepoint definition>> defines a
3172 tracepoint which takes two input arguments and has two output event
3173 fields.
3174
3175 [source,c]
3176 .Tracepoint provider header file.
3177 ----
3178 #include "my-custom-structure.h"
3179
3180 LTTNG_UST_TRACEPOINT_EVENT(
3181     my_provider,
3182     my_tracepoint,
3183     LTTNG_UST_TP_ARGS(
3184         int, argc,
3185         const char *, cmd_name
3186     ),
3187     LTTNG_UST_TP_FIELDS(
3188         lttng_ust_field_string(cmd_name, cmd_name)
3189         lttng_ust_field_integer(int, number_of_args, argc)
3190     )
3191 )
3192 ----
3193
3194 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3195 macro in the source code of your application like this:
3196
3197 [source,c]
3198 .Application source file.
3199 ----
3200 #include "tp.h"
3201
3202 int main(int argc, char* argv[])
3203 {
3204     lttng_ust_tracepoint(my_provider, my_tracepoint, argc, argv[0]);
3205     return 0;
3206 }
3207 ----
3208
3209 Note how the source code of the application includes
3210 the tracepoint provider header file containing the tracepoint
3211 definitions to use, path:{tp.h}.
3212 ====
3213
3214 .`lttng_ust_tracepoint()` usage with a complex tracepoint definition.
3215 ====
3216 Consider this complex tracepoint definition, where multiple event
3217 fields refer to the same input arguments in their argument expression
3218 parameter:
3219
3220 [source,c]
3221 .Tracepoint provider header file.
3222 ----
3223 /* For `struct stat` */
3224 #include <sys/types.h>
3225 #include <sys/stat.h>
3226 #include <unistd.h>
3227
3228 LTTNG_UST_TRACEPOINT_EVENT(
3229     my_provider,
3230     my_tracepoint,
3231     LTTNG_UST_TP_ARGS(
3232         int, my_int_arg,
3233         char *, my_str_arg,
3234         struct stat *, st
3235     ),
3236     LTTNG_UST_TP_FIELDS(
3237         lttng_ust_field_integer(int, my_constant_field, 23 + 17)
3238         lttng_ust_field_integer(int, my_int_arg_field, my_int_arg)
3239         lttng_ust_field_integer(int, my_int_arg_field2,
3240                                 my_int_arg * my_int_arg)
3241         lttng_ust_field_integer(int, sum4_field,
3242                                 my_str_arg[0] + my_str_arg[1] +
3243                                 my_str_arg[2] + my_str_arg[3])
3244         lttng_ust_field_string(my_str_arg_field, my_str_arg)
3245         lttng_ust_field_integer_hex(off_t, size_field, st->st_size)
3246         lttng_ust_field_float(double, size_dbl_field, (double) st->st_size)
3247         lttng_ust_field_sequence_text(char, half_my_str_arg_field,
3248                                       my_str_arg, size_t,
3249                                       strlen(my_str_arg) / 2)
3250     )
3251 )
3252 ----
3253
3254 Refer to this tracepoint definition with the `lttng_ust_tracepoint()`
3255 macro in the source code of your application like this:
3256
3257 [source,c]
3258 .Application source file.
3259 ----
3260 #define LTTNG_UST_TRACEPOINT_DEFINE
3261 #include "tp.h"
3262
3263 int main(void)
3264 {
3265     struct stat s;
3266
3267     stat("/etc/fstab", &s);
3268     lttng_ust_tracepoint(my_provider, my_tracepoint, 23,
3269                          "Hello, World!", &s);
3270
3271     return 0;
3272 }
3273 ----
3274
3275 If you look at the event record that LTTng writes when recording this
3276 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
3277 it should look like this:
3278
3279 .Event record fields
3280 |====
3281 |Field name              |Field value
3282 |`my_constant_field`     |40
3283 |`my_int_arg_field`      |23
3284 |`my_int_arg_field2`     |529
3285 |`sum4_field`            |389
3286 |`my_str_arg_field`      |`Hello, World!`
3287 |`size_field`            |0x12d
3288 |`size_dbl_field`        |301.0
3289 |`half_my_str_arg_field` |`Hello,`
3290 |====
3291 ====
3292
3293 Sometimes, the arguments you pass to `lttng_ust_tracepoint()` are
3294 expensive to evaluate--they use the call stack, for example. To avoid
3295 this computation when LTTng wouldn't emit any event anyway, use the
3296 `lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros.
3297
3298 The syntax of the `lttng_ust_tracepoint_enabled()` and
3299 `lttng_ust_do_tracepoint()` macros is:
3300
3301 [source,c]
3302 .`lttng_ust_tracepoint_enabled()` and `lttng_ust_do_tracepoint()` macros syntax.
3303 ----
3304 lttng_ust_tracepoint_enabled(provider_name, tracepoint_name)
3305
3306 lttng_ust_do_tracepoint(provider_name, tracepoint_name, ...)
3307 ----
3308
3309 Replace:
3310
3311 * +__provider_name__+ with the tracepoint provider name.
3312 * +__tracepoint_name__+ with the tracepoint name.
3313
3314 `lttng_ust_tracepoint_enabled()` returns a non-zero value if executing
3315 the tracepoint named `tracepoint_name` from the provider named
3316 `provider_name` _could_ make LTTng emit an event, depending on the
3317 payload of said event.
3318
3319 `lttng_ust_do_tracepoint()` is like `lttng_ust_tracepoint()`, except
3320 that it doesn't check what `lttng_ust_tracepoint_enabled()` checks.
3321 Using `lttng_ust_tracepoint()` with `lttng_ust_tracepoint_enabled()` is
3322 dangerous because `lttng_ust_tracepoint()` also contains the
3323 `lttng_ust_tracepoint_enabled()` check; therefore, a race condition is
3324 possible in this situation:
3325
3326 [source,c]
3327 .Possible race condition when using `lttng_ust_tracepoint_enabled()` with `lttng_ust_tracepoint()`.
3328 ----
3329 if (lttng_ust_tracepoint_enabled(my_provider, my_tracepoint)) {
3330     stuff = prepare_stuff();
3331 }
3332
3333 lttng_ust_tracepoint(my_provider, my_tracepoint, stuff);
3334 ----
3335
3336 If `lttng_ust_tracepoint_enabled()` is false, but would be true after
3337 the conditional block, then `stuff` isn't prepared: the emitted event
3338 will either contain wrong data, or the whole application could crash
3339 (with a segmentation fault, for example).
3340
3341 NOTE: Neither `lttng_ust_tracepoint_enabled()` nor
3342 `lttng_ust_do_tracepoint()` have an `STAP_PROBEV()` call. If you need
3343 it, you must emit this call yourself.
3344
3345
3346 [[building-tracepoint-providers-and-user-application]]
3347 ==== Build and link a tracepoint provider package and an application
3348
3349 Once you have one or more <<tpp-header,tracepoint provider header
3350 files>> and a <<tpp-source,tracepoint provider package source file>>,
3351 create the tracepoint provider package by compiling its source
3352 file. From here, multiple build and run scenarios are possible. The
3353 following table shows common application and library configurations
3354 along with the required command lines to achieve them.
3355
3356 In the following diagrams, we use the following file names:
3357
3358 `app`::
3359   Executable application.
3360
3361 `app.o`::
3362   Application object file.
3363
3364 `tpp.o`::
3365   Tracepoint provider package object file.
3366
3367 `tpp.a`::
3368   Tracepoint provider package archive file.
3369
3370 `libtpp.so`::
3371   Tracepoint provider package shared object file.
3372
3373 `emon.o`::
3374   User library object file.
3375
3376 `libemon.so`::
3377   User library shared object file.
3378
3379 We use the following symbols in the diagrams of table below:
3380
3381 [role="img-100"]
3382 .Symbols used in the build scenario diagrams.
3383 image::ust-sit-symbols.png[]
3384
3385 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
3386 variable in the following instructions.
3387
3388 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
3389 .Common tracepoint provider package scenarios.
3390 |====
3391 |Scenario |Instructions
3392
3393 |
3394 The instrumented application is statically linked with
3395 the tracepoint provider package object.
3396
3397 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
3398
3399 |
3400 include::../common/ust-sit-step-tp-o.txt[]
3401
3402 To build the instrumented application:
3403
3404 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3405 +
3406 --
3407 [source,c]
3408 ----
3409 #define LTTNG_UST_TRACEPOINT_DEFINE
3410 ----
3411 --
3412
3413 . Compile the application source file:
3414 +
3415 --
3416 [role="term"]
3417 ----
3418 $ gcc -c app.c
3419 ----
3420 --
3421
3422 . Build the application:
3423 +
3424 --
3425 [role="term"]
3426 ----
3427 $ gcc -o app app.o tpp.o -llttng-ust -ldl
3428 ----
3429 --
3430
3431 To run the instrumented application:
3432
3433 * Start the application:
3434 +
3435 --
3436 [role="term"]
3437 ----
3438 $ ./app
3439 ----
3440 --
3441
3442 |
3443 The instrumented application is statically linked with the
3444 tracepoint provider package archive file.
3445
3446 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
3447
3448 |
3449 To create the tracepoint provider package archive file:
3450
3451 . Compile the <<tpp-source,tracepoint provider package source file>>:
3452 +
3453 --
3454 [role="term"]
3455 ----
3456 $ gcc -I. -c tpp.c
3457 ----
3458 --
3459
3460 . Create the tracepoint provider package archive file:
3461 +
3462 --
3463 [role="term"]
3464 ----
3465 $ ar rcs tpp.a tpp.o
3466 ----
3467 --
3468
3469 To build the instrumented application:
3470
3471 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3472 +
3473 --
3474 [source,c]
3475 ----
3476 #define LTTNG_UST_TRACEPOINT_DEFINE
3477 ----
3478 --
3479
3480 . Compile the application source file:
3481 +
3482 --
3483 [role="term"]
3484 ----
3485 $ gcc -c app.c
3486 ----
3487 --
3488
3489 . Build the application:
3490 +
3491 --
3492 [role="term"]
3493 ----
3494 $ gcc -o app app.o tpp.a -llttng-ust -ldl
3495 ----
3496 --
3497
3498 To run the instrumented application:
3499
3500 * Start the application:
3501 +
3502 --
3503 [role="term"]
3504 ----
3505 $ ./app
3506 ----
3507 --
3508
3509 |
3510 The instrumented application is linked with the tracepoint provider
3511 package shared object.
3512
3513 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
3514
3515 |
3516 include::../common/ust-sit-step-tp-so.txt[]
3517
3518 To build the instrumented application:
3519
3520 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3521 +
3522 --
3523 [source,c]
3524 ----
3525 #define LTTNG_UST_TRACEPOINT_DEFINE
3526 ----
3527 --
3528
3529 . Compile the application source file:
3530 +
3531 --
3532 [role="term"]
3533 ----
3534 $ gcc -c app.c
3535 ----
3536 --
3537
3538 . Build the application:
3539 +
3540 --
3541 [role="term"]
3542 ----
3543 $ gcc -o app app.o -ldl -L. -ltpp
3544 ----
3545 --
3546
3547 To run the instrumented application:
3548
3549 * Start the application:
3550 +
3551 --
3552 [role="term"]
3553 ----
3554 $ ./app
3555 ----
3556 --
3557
3558 |
3559 The tracepoint provider package shared object is preloaded before the
3560 instrumented application starts.
3561
3562 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3563
3564 |
3565 include::../common/ust-sit-step-tp-so.txt[]
3566
3567 To build the instrumented application:
3568
3569 . In path:{app.c}, before including path:{tpp.h}, add the
3570   following lines:
3571 +
3572 --
3573 [source,c]
3574 ----
3575 #define LTTNG_UST_TRACEPOINT_DEFINE
3576 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3577 ----
3578 --
3579
3580 . Compile the application source file:
3581 +
3582 --
3583 [role="term"]
3584 ----
3585 $ gcc -c app.c
3586 ----
3587 --
3588
3589 . Build the application:
3590 +
3591 --
3592 [role="term"]
3593 ----
3594 $ gcc -o app app.o -ldl
3595 ----
3596 --
3597
3598 To run the instrumented application with tracing support:
3599
3600 * Preload the tracepoint provider package shared object and
3601   start the application:
3602 +
3603 --
3604 [role="term"]
3605 ----
3606 $ LD_PRELOAD=./libtpp.so ./app
3607 ----
3608 --
3609
3610 To run the instrumented application without tracing support:
3611
3612 * Start the application:
3613 +
3614 --
3615 [role="term"]
3616 ----
3617 $ ./app
3618 ----
3619 --
3620
3621 |
3622 The instrumented application dynamically loads the tracepoint provider
3623 package shared object.
3624
3625 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3626
3627 |
3628 include::../common/ust-sit-step-tp-so.txt[]
3629
3630 To build the instrumented application:
3631
3632 . In path:{app.c}, before including path:{tpp.h}, add the
3633   following lines:
3634 +
3635 --
3636 [source,c]
3637 ----
3638 #define LTTNG_UST_TRACEPOINT_DEFINE
3639 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3640 ----
3641 --
3642
3643 . Compile the application source file:
3644 +
3645 --
3646 [role="term"]
3647 ----
3648 $ gcc -c app.c
3649 ----
3650 --
3651
3652 . Build the application:
3653 +
3654 --
3655 [role="term"]
3656 ----
3657 $ gcc -o app app.o -ldl
3658 ----
3659 --
3660
3661 To run the instrumented application:
3662
3663 * Start the application:
3664 +
3665 --
3666 [role="term"]
3667 ----
3668 $ ./app
3669 ----
3670 --
3671
3672 |
3673 The application is linked with the instrumented user library.
3674
3675 The instrumented user library is statically linked with the tracepoint
3676 provider package object file.
3677
3678 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3679
3680 |
3681 include::../common/ust-sit-step-tp-o-fpic.txt[]
3682
3683 To build the instrumented user library:
3684
3685 . In path:{emon.c}, before including path:{tpp.h}, add the
3686   following line:
3687 +
3688 --
3689 [source,c]
3690 ----
3691 #define LTTNG_UST_TRACEPOINT_DEFINE
3692 ----
3693 --
3694
3695 . Compile the user library source file:
3696 +
3697 --
3698 [role="term"]
3699 ----
3700 $ gcc -I. -fpic -c emon.c
3701 ----
3702 --
3703
3704 . Build the user library shared object:
3705 +
3706 --
3707 [role="term"]
3708 ----
3709 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3710 ----
3711 --
3712
3713 To build the application:
3714
3715 . Compile the application source file:
3716 +
3717 --
3718 [role="term"]
3719 ----
3720 $ gcc -c app.c
3721 ----
3722 --
3723
3724 . Build the application:
3725 +
3726 --
3727 [role="term"]
3728 ----
3729 $ gcc -o app app.o -L. -lemon
3730 ----
3731 --
3732
3733 To run the application:
3734
3735 * Start the application:
3736 +
3737 --
3738 [role="term"]
3739 ----
3740 $ ./app
3741 ----
3742 --
3743
3744 |
3745 The application is linked with the instrumented user library.
3746
3747 The instrumented user library is linked with the tracepoint provider
3748 package shared object.
3749
3750 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3751
3752 |
3753 include::../common/ust-sit-step-tp-so.txt[]
3754
3755 To build the instrumented user library:
3756
3757 . In path:{emon.c}, before including path:{tpp.h}, add the
3758   following line:
3759 +
3760 --
3761 [source,c]
3762 ----
3763 #define LTTNG_UST_TRACEPOINT_DEFINE
3764 ----
3765 --
3766
3767 . Compile the user library source file:
3768 +
3769 --
3770 [role="term"]
3771 ----
3772 $ gcc -I. -fpic -c emon.c
3773 ----
3774 --
3775
3776 . Build the user library shared object:
3777 +
3778 --
3779 [role="term"]
3780 ----
3781 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3782 ----
3783 --
3784
3785 To build the application:
3786
3787 . Compile the application source file:
3788 +
3789 --
3790 [role="term"]
3791 ----
3792 $ gcc -c app.c
3793 ----
3794 --
3795
3796 . Build the application:
3797 +
3798 --
3799 [role="term"]
3800 ----
3801 $ gcc -o app app.o -L. -lemon
3802 ----
3803 --
3804
3805 To run the application:
3806
3807 * Start the application:
3808 +
3809 --
3810 [role="term"]
3811 ----
3812 $ ./app
3813 ----
3814 --
3815
3816 |
3817 The tracepoint provider package shared object is preloaded before the
3818 application starts.
3819
3820 The application is linked with the instrumented user library.
3821
3822 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3823
3824 |
3825 include::../common/ust-sit-step-tp-so.txt[]
3826
3827 To build the instrumented user library:
3828
3829 . In path:{emon.c}, before including path:{tpp.h}, add the
3830   following lines:
3831 +
3832 --
3833 [source,c]
3834 ----
3835 #define LTTNG_UST_TRACEPOINT_DEFINE
3836 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3837 ----
3838 --
3839
3840 . Compile the user library source file:
3841 +
3842 --
3843 [role="term"]
3844 ----
3845 $ gcc -I. -fpic -c emon.c
3846 ----
3847 --
3848
3849 . Build the user library shared object:
3850 +
3851 --
3852 [role="term"]
3853 ----
3854 $ gcc -shared -o libemon.so emon.o -ldl
3855 ----
3856 --
3857
3858 To build the application:
3859
3860 . Compile the application source file:
3861 +
3862 --
3863 [role="term"]
3864 ----
3865 $ gcc -c app.c
3866 ----
3867 --
3868
3869 . Build the application:
3870 +
3871 --
3872 [role="term"]
3873 ----
3874 $ gcc -o app app.o -L. -lemon
3875 ----
3876 --
3877
3878 To run the application with tracing support:
3879
3880 * Preload the tracepoint provider package shared object and
3881   start the application:
3882 +
3883 --
3884 [role="term"]
3885 ----
3886 $ LD_PRELOAD=./libtpp.so ./app
3887 ----
3888 --
3889
3890 To run the application without tracing support:
3891
3892 * Start the application:
3893 +
3894 --
3895 [role="term"]
3896 ----
3897 $ ./app
3898 ----
3899 --
3900
3901 |
3902 The application is linked with the instrumented user library.
3903
3904 The instrumented user library dynamically loads the tracepoint provider
3905 package shared object.
3906
3907 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3908
3909 |
3910 include::../common/ust-sit-step-tp-so.txt[]
3911
3912 To build the instrumented user library:
3913
3914 . In path:{emon.c}, before including path:{tpp.h}, add the
3915   following lines:
3916 +
3917 --
3918 [source,c]
3919 ----
3920 #define LTTNG_UST_TRACEPOINT_DEFINE
3921 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3922 ----
3923 --
3924
3925 . Compile the user library source file:
3926 +
3927 --
3928 [role="term"]
3929 ----
3930 $ gcc -I. -fpic -c emon.c
3931 ----
3932 --
3933
3934 . Build the user library shared object:
3935 +
3936 --
3937 [role="term"]
3938 ----
3939 $ gcc -shared -o libemon.so emon.o -ldl
3940 ----
3941 --
3942
3943 To build the application:
3944
3945 . Compile the application source file:
3946 +
3947 --
3948 [role="term"]
3949 ----
3950 $ gcc -c app.c
3951 ----
3952 --
3953
3954 . Build the application:
3955 +
3956 --
3957 [role="term"]
3958 ----
3959 $ gcc -o app app.o -L. -lemon
3960 ----
3961 --
3962
3963 To run the application:
3964
3965 * Start the application:
3966 +
3967 --
3968 [role="term"]
3969 ----
3970 $ ./app
3971 ----
3972 --
3973
3974 |
3975 The application dynamically loads the instrumented user library.
3976
3977 The instrumented user library is linked with the tracepoint provider
3978 package shared object.
3979
3980 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3981
3982 |
3983 include::../common/ust-sit-step-tp-so.txt[]
3984
3985 To build the instrumented user library:
3986
3987 . In path:{emon.c}, before including path:{tpp.h}, add the
3988   following line:
3989 +
3990 --
3991 [source,c]
3992 ----
3993 #define LTTNG_UST_TRACEPOINT_DEFINE
3994 ----
3995 --
3996
3997 . Compile the user library source file:
3998 +
3999 --
4000 [role="term"]
4001 ----
4002 $ gcc -I. -fpic -c emon.c
4003 ----
4004 --
4005
4006 . Build the user library shared object:
4007 +
4008 --
4009 [role="term"]
4010 ----
4011 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
4012 ----
4013 --
4014
4015 To build the application:
4016
4017 . Compile the application source file:
4018 +
4019 --
4020 [role="term"]
4021 ----
4022 $ gcc -c app.c
4023 ----
4024 --
4025
4026 . Build the application:
4027 +
4028 --
4029 [role="term"]
4030 ----
4031 $ gcc -o app app.o -ldl -L. -lemon
4032 ----
4033 --
4034
4035 To run the application:
4036
4037 * Start the application:
4038 +
4039 --
4040 [role="term"]
4041 ----
4042 $ ./app
4043 ----
4044 --
4045
4046 |
4047 The application dynamically loads the instrumented user library.
4048
4049 The instrumented user library dynamically loads the tracepoint provider
4050 package shared object.
4051
4052 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
4053
4054 |
4055 include::../common/ust-sit-step-tp-so.txt[]
4056
4057 To build the instrumented user library:
4058
4059 . In path:{emon.c}, before including path:{tpp.h}, add the
4060   following lines:
4061 +
4062 --
4063 [source,c]
4064 ----
4065 #define LTTNG_UST_TRACEPOINT_DEFINE
4066 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4067 ----
4068 --
4069
4070 . Compile the user library source file:
4071 +
4072 --
4073 [role="term"]
4074 ----
4075 $ gcc -I. -fpic -c emon.c
4076 ----
4077 --
4078
4079 . Build the user library shared object:
4080 +
4081 --
4082 [role="term"]
4083 ----
4084 $ gcc -shared -o libemon.so emon.o -ldl
4085 ----
4086 --
4087
4088 To build the application:
4089
4090 . Compile the application source file:
4091 +
4092 --
4093 [role="term"]
4094 ----
4095 $ gcc -c app.c
4096 ----
4097 --
4098
4099 . Build the application:
4100 +
4101 --
4102 [role="term"]
4103 ----
4104 $ gcc -o app app.o -ldl -L. -lemon
4105 ----
4106 --
4107
4108 To run the application:
4109
4110 * Start the application:
4111 +
4112 --
4113 [role="term"]
4114 ----
4115 $ ./app
4116 ----
4117 --
4118
4119 |
4120 The tracepoint provider package shared object is preloaded before the
4121 application starts.
4122
4123 The application dynamically loads the instrumented user library.
4124
4125 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
4126
4127 |
4128 include::../common/ust-sit-step-tp-so.txt[]
4129
4130 To build the instrumented user library:
4131
4132 . In path:{emon.c}, before including path:{tpp.h}, add the
4133   following lines:
4134 +
4135 --
4136 [source,c]
4137 ----
4138 #define LTTNG_UST_TRACEPOINT_DEFINE
4139 #define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
4140 ----
4141 --
4142
4143 . Compile the user library source file:
4144 +
4145 --
4146 [role="term"]
4147 ----
4148 $ gcc -I. -fpic -c emon.c
4149 ----
4150 --
4151
4152 . Build the user library shared object:
4153 +
4154 --
4155 [role="term"]
4156 ----
4157 $ gcc -shared -o libemon.so emon.o -ldl
4158 ----
4159 --
4160
4161 To build the application:
4162
4163 . Compile the application source file:
4164 +
4165 --
4166 [role="term"]
4167 ----
4168 $ gcc -c app.c
4169 ----
4170 --
4171
4172 . Build the application:
4173 +
4174 --
4175 [role="term"]
4176 ----
4177 $ gcc -o app app.o -L. -lemon
4178 ----
4179 --
4180
4181 To run the application with tracing support:
4182
4183 * Preload the tracepoint provider package shared object and
4184   start the application:
4185 +
4186 --
4187 [role="term"]
4188 ----
4189 $ LD_PRELOAD=./libtpp.so ./app
4190 ----
4191 --
4192
4193 To run the application without tracing support:
4194
4195 * Start the application:
4196 +
4197 --
4198 [role="term"]
4199 ----
4200 $ ./app
4201 ----
4202 --
4203
4204 |
4205 The application is statically linked with the tracepoint provider
4206 package object file.
4207
4208 The application is linked with the instrumented user library.
4209
4210 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
4211
4212 |
4213 include::../common/ust-sit-step-tp-o.txt[]
4214
4215 To build the instrumented user library:
4216
4217 . In path:{emon.c}, before including path:{tpp.h}, add the
4218   following line:
4219 +
4220 --
4221 [source,c]
4222 ----
4223 #define LTTNG_UST_TRACEPOINT_DEFINE
4224 ----
4225 --
4226
4227 . Compile the user library source file:
4228 +
4229 --
4230 [role="term"]
4231 ----
4232 $ gcc -I. -fpic -c emon.c
4233 ----
4234 --
4235
4236 . Build the user library shared object:
4237 +
4238 --
4239 [role="term"]
4240 ----
4241 $ gcc -shared -o libemon.so emon.o
4242 ----
4243 --
4244
4245 To build the application:
4246
4247 . Compile the application source file:
4248 +
4249 --
4250 [role="term"]
4251 ----
4252 $ gcc -c app.c
4253 ----
4254 --
4255
4256 . Build the application:
4257 +
4258 --
4259 [role="term"]
4260 ----
4261 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
4262 ----
4263 --
4264
4265 To run the instrumented application:
4266
4267 * Start the application:
4268 +
4269 --
4270 [role="term"]
4271 ----
4272 $ ./app
4273 ----
4274 --
4275
4276 |
4277 The application is statically linked with the tracepoint provider
4278 package object file.
4279
4280 The application dynamically loads the instrumented user library.
4281
4282 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
4283
4284 |
4285 include::../common/ust-sit-step-tp-o.txt[]
4286
4287 To build the application:
4288
4289 . In path:{app.c}, before including path:{tpp.h}, add the following line:
4290 +
4291 --
4292 [source,c]
4293 ----
4294 #define LTTNG_UST_TRACEPOINT_DEFINE
4295 ----
4296 --
4297
4298 . Compile the application source file:
4299 +
4300 --
4301 [role="term"]
4302 ----
4303 $ gcc -c app.c
4304 ----
4305 --
4306
4307 . Build the application:
4308 +
4309 --
4310 [role="term"]
4311 ----
4312 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
4313       -llttng-ust -ldl
4314 ----
4315 --
4316 +
4317 The `--export-dynamic` option passed to the linker is necessary for the
4318 dynamically loaded library to ``see'' the tracepoint symbols defined in
4319 the application.
4320
4321 To build the instrumented user library:
4322
4323 . Compile the user library source file:
4324 +
4325 --
4326 [role="term"]
4327 ----
4328 $ gcc -I. -fpic -c emon.c
4329 ----
4330 --
4331
4332 . Build the user library shared object:
4333 +
4334 --
4335 [role="term"]
4336 ----
4337 $ gcc -shared -o libemon.so emon.o
4338 ----
4339 --
4340
4341 To run the application:
4342
4343 * Start the application:
4344 +
4345 --
4346 [role="term"]
4347 ----
4348 $ ./app
4349 ----
4350 --
4351 |====
4352
4353
4354 [[using-lttng-ust-with-daemons]]
4355 ===== Use noch:{LTTng-UST} with daemons
4356
4357 If your instrumented application calls man:fork(2), man:clone(2),
4358 or BSD's man:rfork(2), without a following man:exec(3)-family
4359 system call, you must preload the path:{liblttng-ust-fork.so} shared
4360 object when you start the application.
4361
4362 [role="term"]
4363 ----
4364 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
4365 ----
4366
4367 If your tracepoint provider package is
4368 a shared library which you also preload, you must put both
4369 shared objects in env:LD_PRELOAD:
4370
4371 [role="term"]
4372 ----
4373 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
4374 ----
4375
4376
4377 [role="since-2.9"]
4378 [[liblttng-ust-fd]]
4379 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
4380
4381 If your instrumented application closes one or more file descriptors
4382 which it did not open itself, you must preload the
4383 path:{liblttng-ust-fd.so} shared object when you start the application:
4384
4385 [role="term"]
4386 ----
4387 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
4388 ----
4389
4390 Typical use cases include closing all the file descriptors after
4391 man:fork(2) or man:rfork(2) and buggy applications doing
4392 ``double closes''.
4393
4394
4395 [[lttng-ust-pkg-config]]
4396 ===== Use noch:{pkg-config}
4397
4398 On some distributions, LTTng-UST ships with a
4399 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
4400 metadata file. If this is your case, then use cmd:pkg-config to
4401 build an application on the command line:
4402
4403 [role="term"]
4404 ----
4405 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
4406 ----
4407
4408
4409 [[instrumenting-32-bit-app-on-64-bit-system]]
4410 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
4411
4412 In order to trace a 32-bit application running on a 64-bit system,
4413 LTTng must use a dedicated 32-bit
4414 <<lttng-consumerd,consumer daemon>>.
4415
4416 The following steps show how to build and install a 32-bit consumer
4417 daemon, which is _not_ part of the default 64-bit LTTng build, how to
4418 build and install the 32-bit LTTng-UST libraries, and how to build and
4419 link an instrumented 32-bit application in that context.
4420
4421 To build a 32-bit instrumented application for a 64-bit target system,
4422 assuming you have a fresh target system with no installed Userspace RCU
4423 or LTTng packages:
4424
4425 . Download, build, and install a 32-bit version of Userspace RCU:
4426 +
4427 --
4428 [role="term"]
4429 ----
4430 $ cd $(mktemp -d) &&
4431   wget https://lttng.org/files/urcu/userspace-rcu-latest-0.13.tar.bz2 &&
4432   tar -xf userspace-rcu-latest-0.13.tar.bz2 &&
4433   cd userspace-rcu-0.13.* &&
4434   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
4435   make &&
4436   sudo make install &&
4437   sudo ldconfig
4438 ----
4439 --
4440
4441 . Using the package manager of your distribution, or from source,
4442   install the 32-bit versions of the following dependencies of
4443   LTTng-tools and LTTng-UST:
4444 +
4445 --
4446 * https://sourceforge.net/projects/libuuid/[libuuid]
4447 * https://directory.fsf.org/wiki/Popt[popt]
4448 * https://www.xmlsoft.org/[libxml2]
4449 * **Optional**: https://github.com/numactl/numactl[numactl]
4450 --
4451
4452 . Download, build, and install a 32-bit version of the latest
4453   LTTng-UST{nbsp}{revision}:
4454 +
4455 --
4456 [role="term"]
4457 ----
4458 $ cd $(mktemp -d) &&
4459   wget https://lttng.org/files/lttng-ust/lttng-ust-latest-2.13.tar.bz2 &&
4460   tar -xf lttng-ust-latest-2.13.tar.bz2 &&
4461   cd lttng-ust-2.13.* &&
4462   ./configure --libdir=/usr/local/lib32 \
4463               CFLAGS=-m32 CXXFLAGS=-m32 \
4464               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
4465   make &&
4466   sudo make install &&
4467   sudo ldconfig
4468 ----
4469 --
4470 +
4471 Add `--disable-numa` to `./configure` if you don't have
4472 https://github.com/numactl/numactl[numactl].
4473 +
4474 [NOTE]
4475 ====
4476 Depending on your distribution, 32-bit libraries could be installed at a
4477 different location than `/usr/lib32`. For example, Debian is known to
4478 install some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
4479
4480 In this case, make sure to set `LDFLAGS` to all the
4481 relevant 32-bit library paths, for example:
4482
4483 [role="term"]
4484 ----
4485 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
4486 ----
4487 ====
4488
4489 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
4490   the 32-bit consumer daemon:
4491 +
4492 --
4493 [role="term"]
4494 ----
4495 $ cd $(mktemp -d) &&
4496   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4497   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4498   cd lttng-tools-2.13.* &&
4499   ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
4500               LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
4501               --disable-bin-lttng --disable-bin-lttng-crash \
4502               --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
4503   make &&
4504   cd src/bin/lttng-consumerd &&
4505   sudo make install &&
4506   sudo ldconfig
4507 ----
4508 --
4509
4510 . From your distribution or from source, <<installing-lttng,install>>
4511   the 64-bit versions of LTTng-UST and Userspace RCU.
4512
4513 . Download, build, and install the 64-bit version of the
4514   latest LTTng-tools{nbsp}{revision}:
4515 +
4516 --
4517 [role="term"]
4518 ----
4519 $ cd $(mktemp -d) &&
4520   wget https://lttng.org/files/lttng-tools/lttng-tools-latest-2.13.tar.bz2 &&
4521   tar -xf lttng-tools-latest-2.13.tar.bz2 &&
4522   cd lttng-tools-2.13.* &&
4523   ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4524               --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4525   make &&
4526   sudo make install &&
4527   sudo ldconfig
4528 ----
4529 --
4530
4531 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4532   when linking your 32-bit application:
4533 +
4534 ----
4535 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4536 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4537 ----
4538 +
4539 For example, let's rebuild the quick start example in
4540 ``<<tracing-your-own-user-application,Record user application events>>''
4541 as an instrumented 32-bit application:
4542 +
4543 --
4544 [role="term"]
4545 ----
4546 $ gcc -m32 -c -I. hello-tp.c
4547 $ gcc -m32 -c hello.c
4548 $ gcc -m32 -o hello hello.o hello-tp.o \
4549       -L/usr/lib32 -L/usr/local/lib32 \
4550       -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4551       -llttng-ust -ldl
4552 ----
4553 --
4554
4555 No special action is required to execute the 32-bit application and
4556 for LTTng to trace it: use the command-line man:lttng(1) tool as usual.
4557
4558
4559 [role="since-2.5"]
4560 [[tracef]]
4561 ==== Use `lttng_ust_tracef()`
4562
4563 man:lttng_ust_tracef(3) is a small LTTng-UST API designed for quick,
4564 man:printf(3)-like instrumentation without the burden of
4565 <<tracepoint-provider,creating>> and
4566 <<building-tracepoint-providers-and-user-application,building>>
4567 a tracepoint provider package.
4568
4569 To use `lttng_ust_tracef()` in your application:
4570
4571 . In the C or $$C++$$ source files where you need to use
4572   `lttng_ust_tracef()`, include `<lttng/tracef.h>`:
4573 +
4574 --
4575 [source,c]
4576 ----
4577 #include <lttng/tracef.h>
4578 ----
4579 --
4580
4581 . In the source code of the application, use `lttng_ust_tracef()` like
4582   you would use man:printf(3):
4583 +
4584 --
4585 [source,c]
4586 ----
4587     /* ... */
4588
4589     lttng_ust_tracef("my message: %d (%s)", my_integer, my_string);
4590
4591     /* ... */
4592 ----
4593 --
4594
4595 . Link your application with `liblttng-ust`:
4596 +
4597 --
4598 [role="term"]
4599 ----
4600 $ gcc -o app app.c -llttng-ust
4601 ----
4602 --
4603
4604 To record the events that `lttng_ust_tracef()` calls emit:
4605
4606 * <<enabling-disabling-events,Create a recording event rule>> which
4607   matches user space events named `lttng_ust_tracef:*`:
4608 +
4609 --
4610 [role="term"]
4611 ----
4612 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4613 ----
4614 --
4615
4616 [IMPORTANT]
4617 .Limitations of `lttng_ust_tracef()`
4618 ====
4619 The `lttng_ust_tracef()` utility function was developed to make user
4620 space tracing super simple, albeit with notable disadvantages compared
4621 to <<defining-tracepoints,user-defined tracepoints>>:
4622
4623 * All the created events have the same tracepoint provider and
4624   tracepoint names, respectively `lttng_ust_tracef` and `event`.
4625 * There's no static type checking.
4626 * The only event record field you actually get, named `msg`, is a string
4627   potentially containing the values you passed to `lttng_ust_tracef()`
4628   using your own format string. This also means that you can't filter
4629   events with a custom expression at run time because there are no
4630   isolated fields.
4631 * Since `lttng_ust_tracef()` uses the man:vasprintf(3) function of the
4632   C{nbsp}standard library behind the scenes to format the strings at run
4633   time, its expected performance is lower than with user-defined
4634   tracepoints, which don't require a conversion to a string.
4635
4636 Taking this into consideration, `lttng_ust_tracef()` is useful for some
4637 quick prototyping and debugging, but you shouldn't consider it for any
4638 permanent and serious applicative instrumentation.
4639 ====
4640
4641
4642 [role="since-2.7"]
4643 [[tracelog]]
4644 ==== Use `lttng_ust_tracelog()`
4645
4646 The man:tracelog(3) API is very similar to
4647 <<tracef,`lttng_ust_tracef()`>>, with the difference that it accepts an
4648 additional log level parameter.
4649
4650 The goal of `lttng_ust_tracelog()` is to ease the migration from logging
4651 to tracing.
4652
4653 To use `lttng_ust_tracelog()` in your application:
4654
4655 . In the C or $$C++$$ source files where you need to use `tracelog()`,
4656   include `<lttng/tracelog.h>`:
4657 +
4658 --
4659 [source,c]
4660 ----
4661 #include <lttng/tracelog.h>
4662 ----
4663 --
4664
4665 . In the source code of the application, use `lttng_ust_tracelog()` like
4666   you would use man:printf(3), except for the first parameter which is
4667   the log level:
4668 +
4669 --
4670 [source,c]
4671 ----
4672     /* ... */
4673
4674     tracelog(LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING,
4675             "my message: %d (%s)", my_integer, my_string);
4676
4677     /* ... */
4678 ----
4679 --
4680 +
4681 See man:lttng-ust(3) for a list of available log level names.
4682
4683 . Link your application with `liblttng-ust`:
4684 +
4685 --
4686 [role="term"]
4687 ----
4688 $ gcc -o app app.c -llttng-ust
4689 ----
4690 --
4691
4692 To record the events that `lttng_ust_tracelog()` calls emit with a log
4693 level _at least as severe as_ a specific log level:
4694
4695 * <<enabling-disabling-events,Create a recording event rule>> which
4696   matches user space tracepoint events named `lttng_ust_tracelog:*` and
4697   with some minimum level of severity:
4698 +
4699 --
4700 [role="term"]
4701 ----
4702 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4703                      --loglevel=WARNING
4704 ----
4705 --
4706
4707 To record the events that `lttng_ust_tracelog()` calls emit with a
4708 _specific log level_:
4709
4710 * Create a recording event rule which matches tracepoint events named
4711   `lttng_ust_tracelog:*` and with a specific log level:
4712 +
4713 --
4714 [role="term"]
4715 ----
4716 $ lttng enable-event --userspace 'lttng_ust_tracelog:*' \
4717                      --loglevel-only=INFO
4718 ----
4719 --
4720
4721
4722 [[prebuilt-ust-helpers]]
4723 === Load a prebuilt user space tracing helper
4724
4725 The LTTng-UST package provides a few helpers in the form of preloadable
4726 shared objects which automatically instrument system functions and
4727 calls.
4728
4729 The helper shared objects are normally found in dir:{/usr/lib}. If you
4730 built LTTng-UST <<building-from-source,from source>>, they're probably
4731 located in dir:{/usr/local/lib}.
4732
4733 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4734 are:
4735
4736 path:{liblttng-ust-libc-wrapper.so}::
4737 path:{liblttng-ust-pthread-wrapper.so}::
4738   <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4739   memory and POSIX threads function tracing>>.
4740
4741 path:{liblttng-ust-cyg-profile.so}::
4742 path:{liblttng-ust-cyg-profile-fast.so}::
4743   <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4744
4745 path:{liblttng-ust-dl.so}::
4746   <<liblttng-ust-dl,Dynamic linker tracing>>.
4747
4748 To use a user space tracing helper with any user application:
4749
4750 * Preload the helper shared object when you start the application:
4751 +
4752 --
4753 [role="term"]
4754 ----
4755 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4756 ----
4757 --
4758 +
4759 You can preload more than one helper:
4760 +
4761 --
4762 [role="term"]
4763 ----
4764 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4765 ----
4766 --
4767
4768
4769 [role="since-2.3"]
4770 [[liblttng-ust-libc-pthread-wrapper]]
4771 ==== Instrument C standard library memory and POSIX threads functions
4772
4773 The path:{liblttng-ust-libc-wrapper.so} and
4774 path:{liblttng-ust-pthread-wrapper.so} helpers
4775 add instrumentation to some C standard library and POSIX
4776 threads functions.
4777
4778 [role="growable"]
4779 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4780 |====
4781 |TP provider name |TP name |Instrumented function
4782
4783 .6+|`lttng_ust_libc` |`malloc`         |man:malloc(3)
4784                      |`calloc`         |man:calloc(3)
4785                      |`realloc`        |man:realloc(3)
4786                      |`free`           |man:free(3)
4787                      |`memalign`       |man:memalign(3)
4788                      |`posix_memalign` |man:posix_memalign(3)
4789 |====
4790
4791 [role="growable"]
4792 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4793 |====
4794 |TP provider name |TP name |Instrumented function
4795
4796 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4797                         |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4798                         |`pthread_mutex_trylock`  |man:pthread_mutex_trylock(3p)
4799                         |`pthread_mutex_unlock`   |man:pthread_mutex_unlock(3p)
4800 |====
4801
4802 When you preload the shared object, it replaces the functions listed
4803 in the previous tables by wrappers which contain tracepoints and call
4804 the replaced functions.
4805
4806
4807 [[liblttng-ust-cyg-profile]]
4808 ==== Instrument function entry and exit
4809
4810 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4811 to the entry and exit points of functions.
4812
4813 man:gcc(1) and man:clang(1) have an option named
4814 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4815 which generates instrumentation calls for entry and exit to functions.
4816 The LTTng-UST function tracing helpers,
4817 path:{liblttng-ust-cyg-profile.so} and
4818 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4819 to add tracepoints to the two generated functions (which contain
4820 `cyg_profile` in their names, hence the name of the helper).
4821
4822 To use the LTTng-UST function tracing helper, the source files to
4823 instrument must be built using the `-finstrument-functions` compiler
4824 flag.
4825
4826 There are two versions of the LTTng-UST function tracing helper:
4827
4828 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4829   that you should only use when it can be _guaranteed_ that the
4830   complete event stream is recorded without any lost event record.
4831   Any kind of duplicate information is left out.
4832 +
4833 Assuming no event record is lost, having only the function addresses on
4834 entry is enough to create a call graph, since an event record always
4835 contains the ID of the CPU that generated it.
4836 +
4837 Use a tool like man:addr2line(1) to convert function addresses back to
4838 source file names and line numbers.
4839
4840 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4841 which also works in use cases where event records might get discarded or
4842 not recorded from application startup.
4843 In these cases, the trace analyzer needs more information to be
4844 able to reconstruct the program flow.
4845
4846 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4847 points of this helper.
4848
4849 All the tracepoints that this helper provides have the log level
4850 `LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4851
4852 TIP: It's sometimes a good idea to limit the number of source files that
4853 you compile with the `-finstrument-functions` option to prevent LTTng
4854 from writing an excessive amount of trace data at run time. When using
4855 man:gcc(1), use the
4856 `-finstrument-functions-exclude-function-list` option to avoid
4857 instrument entries and exits of specific function names.
4858
4859
4860 [role="since-2.4"]
4861 [[liblttng-ust-dl]]
4862 ==== Instrument the dynamic linker
4863
4864 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4865 man:dlopen(3) and man:dlclose(3) function calls.
4866
4867 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4868 of this helper.
4869
4870
4871 [role="since-2.4"]
4872 [[java-application]]
4873 === Instrument a Java application
4874
4875 You can instrument any Java application which uses one of the following
4876 logging frameworks:
4877
4878 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4879   (JUL) core logging facilities.
4880
4881 * https://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4882   LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4883
4884 [role="img-100"]
4885 .LTTng-UST Java agent imported by a Java application.
4886 image::java-app.png[]
4887
4888 Note that the methods described below are new in LTTng{nbsp}2.8.
4889 Previous LTTng versions use another technique.
4890
4891 NOTE: We use https://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4892 and https://ci.lttng.org/[continuous integration], thus this version is
4893 directly supported. However, the LTTng-UST Java agent is also tested
4894 with OpenJDK{nbsp}7.
4895
4896
4897 [role="since-2.8"]
4898 [[jul]]
4899 ==== Use the LTTng-UST Java agent for `java.util.logging`
4900
4901 To use the LTTng-UST Java agent in a Java application which uses
4902 `java.util.logging` (JUL):
4903
4904 . In the source code of the Java application, import the LTTng-UST log
4905   handler package for `java.util.logging`:
4906 +
4907 --
4908 [source,java]
4909 ----
4910 import org.lttng.ust.agent.jul.LttngLogHandler;
4911 ----
4912 --
4913
4914 . Create an LTTng-UST `java.util.logging` log handler:
4915 +
4916 --
4917 [source,java]
4918 ----
4919 Handler lttngUstLogHandler = new LttngLogHandler();
4920 ----
4921 --
4922
4923 . Add this handler to the `java.util.logging` loggers which should emit
4924   LTTng events:
4925 +
4926 --
4927 [source,java]
4928 ----
4929 Logger myLogger = Logger.getLogger("some-logger");
4930
4931 myLogger.addHandler(lttngUstLogHandler);
4932 ----
4933 --
4934
4935 . Use `java.util.logging` log statements and configuration as usual.
4936   The loggers with an attached LTTng-UST log handler can emit
4937   LTTng events.
4938
4939 . Before exiting the application, remove the LTTng-UST log handler from
4940   the loggers attached to it and call its `close()` method:
4941 +
4942 --
4943 [source,java]
4944 ----
4945 myLogger.removeHandler(lttngUstLogHandler);
4946 lttngUstLogHandler.close();
4947 ----
4948 --
4949 +
4950 This isn't strictly necessary, but it's recommended for a clean
4951 disposal of the resources of the handler.
4952
4953 . Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
4954   path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4955   in the
4956   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4957   path] when you build the Java application.
4958 +
4959 The JAR files are typically located in dir:{/usr/share/java}.
4960 +
4961 IMPORTANT: The LTTng-UST Java agent must be
4962 <<installing-lttng,installed>> for the logging framework your
4963 application uses.
4964
4965 .Use the LTTng-UST Java agent for `java.util.logging`.
4966 ====
4967 [source,java]
4968 .path:{Test.java}
4969 ----
4970 import java.io.IOException;
4971 import java.util.logging.Handler;
4972 import java.util.logging.Logger;
4973 import org.lttng.ust.agent.jul.LttngLogHandler;
4974
4975 public class Test
4976 {
4977     private static final int answer = 42;
4978
4979     public static void main(String[] argv) throws Exception
4980     {
4981         // Create a logger
4982         Logger logger = Logger.getLogger("jello");
4983
4984         // Create an LTTng-UST log handler
4985         Handler lttngUstLogHandler = new LttngLogHandler();
4986
4987         // Add the LTTng-UST log handler to our logger
4988         logger.addHandler(lttngUstLogHandler);
4989
4990         // Log at will!
4991         logger.info("some info");
4992         logger.warning("some warning");
4993         Thread.sleep(500);
4994         logger.finer("finer information; the answer is " + answer);
4995         Thread.sleep(123);
4996         logger.severe("error!");
4997
4998         // Not mandatory, but cleaner
4999         logger.removeHandler(lttngUstLogHandler);
5000         lttngUstLogHandler.close();
5001     }
5002 }
5003 ----
5004
5005 Build this example:
5006
5007 [role="term"]
5008 ----
5009 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5010 ----
5011
5012 <<creating-destroying-tracing-sessions,Create a recording session>>,
5013 <<enabling-disabling-events,create a recording event rule>> matching JUL
5014 events named `jello`, and <<basic-tracing-session-control,start
5015 recording>>:
5016
5017 [role="term"]
5018 ----
5019 $ lttng create
5020 $ lttng enable-event --jul jello
5021 $ lttng start
5022 ----
5023
5024 Run the compiled class:
5025
5026 [role="term"]
5027 ----
5028 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5029 ----
5030
5031 <<basic-tracing-session-control,Stop recording>> and inspect the
5032 recorded events:
5033
5034 [role="term"]
5035 ----
5036 $ lttng stop
5037 $ lttng view
5038 ----
5039 ====
5040
5041 In the resulting trace, an <<event,event record>> which a Java
5042 application using `java.util.logging` generated is named
5043 `lttng_jul:event` and has the following fields:
5044
5045 `msg`::
5046   Log record message.
5047
5048 `logger_name`::
5049   Logger name.
5050
5051 `class_name`::
5052   Name of the class in which the log statement was executed.
5053
5054 `method_name`::
5055   Name of the method in which the log statement was executed.
5056
5057 `long_millis`::
5058   Logging time (timestamp in milliseconds).
5059
5060 `int_loglevel`::
5061   Log level integer value.
5062
5063 `int_threadid`::
5064   ID of the thread in which the log statement was executed.
5065
5066 Use the opt:lttng-enable-event(1):--loglevel or
5067 opt:lttng-enable-event(1):--loglevel-only option of the
5068 man:lttng-enable-event(1) command to target a range of
5069 `java.util.logging` log levels or a specific `java.util.logging` log
5070 level.
5071
5072
5073 [role="since-2.8"]
5074 [[log4j]]
5075 ==== Use the LTTng-UST Java agent for Apache log4j
5076
5077 To use the LTTng-UST Java agent in a Java application which uses
5078 Apache log4j{nbsp}1.2:
5079
5080 . In the source code of the Java application, import the LTTng-UST log
5081   appender package for Apache log4j:
5082 +
5083 --
5084 [source,java]
5085 ----
5086 import org.lttng.ust.agent.log4j.LttngLogAppender;
5087 ----
5088 --
5089
5090 . Create an LTTng-UST log4j log appender:
5091 +
5092 --
5093 [source,java]
5094 ----
5095 Appender lttngUstLogAppender = new LttngLogAppender();
5096 ----
5097 --
5098
5099 . Add this appender to the log4j loggers which should emit LTTng events:
5100 +
5101 --
5102 [source,java]
5103 ----
5104 Logger myLogger = Logger.getLogger("some-logger");
5105
5106 myLogger.addAppender(lttngUstLogAppender);
5107 ----
5108 --
5109
5110 . Use Apache log4j log statements and configuration as usual. The
5111   loggers with an attached LTTng-UST log appender can emit LTTng events.
5112
5113 . Before exiting the application, remove the LTTng-UST log appender from
5114   the loggers attached to it and call its `close()` method:
5115 +
5116 --
5117 [source,java]
5118 ----
5119 myLogger.removeAppender(lttngUstLogAppender);
5120 lttngUstLogAppender.close();
5121 ----
5122 --
5123 +
5124 This isn't strictly necessary, but it's recommended for a clean
5125 disposal of the resources of the appender.
5126
5127 . Include the common and log4j-specific JAR
5128   files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
5129   path:{lttng-ust-agent-log4j.jar}, in the
5130   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
5131   path] when you build the Java application.
5132 +
5133 The JAR files are typically located in dir:{/usr/share/java}.
5134 +
5135 IMPORTANT: The LTTng-UST Java agent must be
5136 <<installing-lttng,installed>> for the logging framework your
5137 application uses.
5138
5139 .Use the LTTng-UST Java agent for Apache log4j.
5140 ====
5141 [source,java]
5142 .path:{Test.java}
5143 ----
5144 import org.apache.log4j.Appender;
5145 import org.apache.log4j.Logger;
5146 import org.lttng.ust.agent.log4j.LttngLogAppender;
5147
5148 public class Test
5149 {
5150     private static final int answer = 42;
5151
5152     public static void main(String[] argv) throws Exception
5153     {
5154         // Create a logger
5155         Logger logger = Logger.getLogger("jello");
5156
5157         // Create an LTTng-UST log appender
5158         Appender lttngUstLogAppender = new LttngLogAppender();
5159
5160         // Add the LTTng-UST log appender to our logger
5161         logger.addAppender(lttngUstLogAppender);
5162
5163         // Log at will!
5164         logger.info("some info");
5165         logger.warn("some warning");
5166         Thread.sleep(500);
5167         logger.debug("debug information; the answer is " + answer);
5168         Thread.sleep(123);
5169         logger.fatal("error!");
5170
5171         // Not mandatory, but cleaner
5172         logger.removeAppender(lttngUstLogAppender);
5173         lttngUstLogAppender.close();
5174     }
5175 }
5176
5177 ----
5178
5179 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
5180 file):
5181
5182 [role="term"]
5183 ----
5184 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
5185 ----
5186
5187 <<creating-destroying-tracing-sessions,Create a recording session>>,
5188 <<enabling-disabling-events,create a recording event rule>> matching
5189 log4j events named `jello`, and <<basic-tracing-session-control,start
5190 recording>>:
5191
5192 [role="term"]
5193 ----
5194 $ lttng create
5195 $ lttng enable-event --log4j jello
5196 $ lttng start
5197 ----
5198
5199 Run the compiled class:
5200
5201 [role="term"]
5202 ----
5203 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
5204 ----
5205
5206 <<basic-tracing-session-control,Stop recording>> and inspect the
5207 recorded events:
5208
5209 [role="term"]
5210 ----
5211 $ lttng stop
5212 $ lttng view
5213 ----
5214 ====
5215
5216 In the resulting trace, an <<event,event record>> which a Java
5217 application using log4j generated is named `lttng_log4j:event` and
5218 has the following fields:
5219
5220 `msg`::
5221   Log record message.
5222
5223 `logger_name`::
5224   Logger name.
5225
5226 `class_name`::
5227   Name of the class in which the log statement was executed.
5228
5229 `method_name`::
5230   Name of the method in which the log statement was executed.
5231
5232 `filename`::
5233   Name of the file in which the executed log statement is located.
5234
5235 `line_number`::
5236   Line number at which the log statement was executed.
5237
5238 `timestamp`::
5239   Logging timestamp.
5240
5241 `int_loglevel`::
5242   Log level integer value.
5243
5244 `thread_name`::
5245   Name of the Java thread in which the log statement was executed.
5246
5247 Use the opt:lttng-enable-event(1):--loglevel or
5248 opt:lttng-enable-event(1):--loglevel-only option of the
5249 man:lttng-enable-event(1) command to target a range of Apache log4j
5250 log levels or a specific log4j log level.
5251
5252
5253 [role="since-2.8"]
5254 [[java-application-context]]
5255 ==== Provide application-specific context fields in a Java application
5256
5257 A Java application-specific context field is a piece of state which
5258 the Java application provides. You can <<adding-context,add>> such
5259 a context field to be recorded, using the
5260 man:lttng-add-context(1) command, to each <<event,event record>>
5261 which the log statements of this application produce.
5262
5263 For example, a given object might have a current request ID variable.
5264 You can create a context information retriever for this object and
5265 assign a name to this current request ID. You can then, using the
5266 man:lttng-add-context(1) command, add this context field by name so that
5267 LTTng writes it to the event records of a given `java.util.logging` or
5268 log4j <<channel,channel>>.
5269
5270 To provide application-specific context fields in a Java application:
5271
5272 . In the source code of the Java application, import the LTTng-UST
5273   Java agent context classes and interfaces:
5274 +
5275 --
5276 [source,java]
5277 ----
5278 import org.lttng.ust.agent.context.ContextInfoManager;
5279 import org.lttng.ust.agent.context.IContextInfoRetriever;
5280 ----
5281 --
5282
5283 . Create a context information retriever class, that is, a class which
5284   implements the `IContextInfoRetriever` interface:
5285 +
5286 --
5287 [source,java]
5288 ----
5289 class MyContextInfoRetriever implements IContextInfoRetriever
5290 {
5291     @Override
5292     public Object retrieveContextInfo(String key)
5293     {
5294         if (key.equals("intCtx")) {
5295             return (short) 17;
5296         } else if (key.equals("strContext")) {
5297             return "context value!";
5298         } else {
5299             return null;
5300         }
5301     }
5302 }
5303 ----
5304 --
5305 +
5306 This `retrieveContextInfo()` method is the only member of the
5307 `IContextInfoRetriever` interface. Its role is to return the current
5308 value of a state by name to create a context field. The names of the
5309 context fields and which state variables they return depends on your
5310 specific scenario.
5311 +
5312 All primitive types and objects are supported as context fields.
5313 When `retrieveContextInfo()` returns an object, the context field
5314 serializer calls its `toString()` method to add a string field to
5315 event records. The method can also return `null`, which means that
5316 no context field is available for the required name.
5317
5318 . Register an instance of your context information retriever class to
5319   the context information manager singleton:
5320 +
5321 --
5322 [source,java]
5323 ----
5324 IContextInfoRetriever cir = new MyContextInfoRetriever();
5325 ContextInfoManager cim = ContextInfoManager.getInstance();
5326 cim.registerContextInfoRetriever("retrieverName", cir);
5327 ----
5328 --
5329
5330 . Before exiting the application, remove your context information
5331   retriever from the context information manager singleton:
5332 +
5333 --
5334 [source,java]
5335 ----
5336 ContextInfoManager cim = ContextInfoManager.getInstance();
5337 cim.unregisterContextInfoRetriever("retrieverName");
5338 ----
5339 --
5340 +
5341 This isn't strictly necessary, but it's recommended for a clean
5342 disposal of some resources of the manager.
5343
5344 . Build your Java application with LTTng-UST Java agent support as
5345   usual, following the procedure for either the
5346   <<jul,`java.util.logging`>> or <<log4j,Apache log4j>> framework.
5347
5348 .Provide application-specific context fields in a Java application.
5349 ====
5350 [source,java]
5351 .path:{Test.java}
5352 ----
5353 import java.util.logging.Handler;
5354 import java.util.logging.Logger;
5355 import org.lttng.ust.agent.jul.LttngLogHandler;
5356 import org.lttng.ust.agent.context.ContextInfoManager;
5357 import org.lttng.ust.agent.context.IContextInfoRetriever;
5358
5359 public class Test
5360 {
5361     // Our context information retriever class
5362     private static class MyContextInfoRetriever
5363     implements IContextInfoRetriever
5364     {
5365         @Override
5366         public Object retrieveContextInfo(String key) {
5367             if (key.equals("intCtx")) {
5368                 return (short) 17;
5369             } else if (key.equals("strContext")) {
5370                 return "context value!";
5371             } else {
5372                 return null;
5373             }
5374         }
5375     }
5376
5377     private static final int answer = 42;
5378
5379     public static void main(String args[]) throws Exception
5380     {
5381         // Get the context information manager instance
5382         ContextInfoManager cim = ContextInfoManager.getInstance();
5383
5384         // Create and register our context information retriever
5385         IContextInfoRetriever cir = new MyContextInfoRetriever();
5386         cim.registerContextInfoRetriever("myRetriever", cir);
5387
5388         // Create a logger
5389         Logger logger = Logger.getLogger("jello");
5390
5391         // Create an LTTng-UST log handler
5392         Handler lttngUstLogHandler = new LttngLogHandler();
5393
5394         // Add the LTTng-UST log handler to our logger
5395         logger.addHandler(lttngUstLogHandler);
5396
5397         // Log at will!
5398         logger.info("some info");
5399         logger.warning("some warning");
5400         Thread.sleep(500);
5401         logger.finer("finer information; the answer is " + answer);
5402         Thread.sleep(123);
5403         logger.severe("error!");
5404
5405         // Not mandatory, but cleaner
5406         logger.removeHandler(lttngUstLogHandler);
5407         lttngUstLogHandler.close();
5408         cim.unregisterContextInfoRetriever("myRetriever");
5409     }
5410 }
5411 ----
5412
5413 Build this example:
5414
5415 [role="term"]
5416 ----
5417 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
5418 ----
5419
5420 <<creating-destroying-tracing-sessions,Create a recording session>> and
5421 <<enabling-disabling-events,create a recording event rule>> matching
5422 `java.util.logging` events named `jello`:
5423
5424 [role="term"]
5425 ----
5426 $ lttng create
5427 $ lttng enable-event --jul jello
5428 ----
5429
5430 <<adding-context,Add the application-specific context fields>> to be
5431 recorded to the event records of the `java.util.logging` channel:
5432
5433 [role="term"]
5434 ----
5435 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
5436 $ lttng add-context --jul --type='$app.myRetriever:strContext'
5437 ----
5438
5439 <<basic-tracing-session-control,Start recording>>:
5440
5441 [role="term"]
5442 ----
5443 $ lttng start
5444 ----
5445
5446 Run the compiled class:
5447
5448 [role="term"]
5449 ----
5450 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
5451 ----
5452
5453 <<basic-tracing-session-control,Stop recording>> and inspect the
5454 recorded events:
5455
5456 [role="term"]
5457 ----
5458 $ lttng stop
5459 $ lttng view
5460 ----
5461 ====
5462
5463
5464 [role="since-2.7"]
5465 [[python-application]]
5466 === Instrument a Python application
5467
5468 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
5469 uses the standard
5470 https://docs.python.org/3/library/logging.html[`logging`] package.
5471
5472 Each log statement creates an LTTng event once the application module
5473 imports the <<lttng-ust-agents,LTTng-UST Python agent>> package.
5474
5475 [role="img-100"]
5476 .A Python application importing the LTTng-UST Python agent.
5477 image::python-app.png[]
5478
5479 To use the LTTng-UST Python agent:
5480
5481 . In the source code of the Python application, import the LTTng-UST
5482   Python agent:
5483 +
5484 --
5485 [source,python]
5486 ----
5487 import lttngust
5488 ----
5489 --
5490 +
5491 The LTTng-UST Python agent automatically adds its logging handler to the
5492 root logger at import time.
5493 +
5494 A log statement that the application executes before this import doesn't
5495 create an LTTng event.
5496 +
5497 IMPORTANT: The LTTng-UST Python agent must be
5498 <<installing-lttng,installed>>.
5499
5500 . Use log statements and logging configuration as usual.
5501   Since the LTTng-UST Python agent adds a handler to the _root_
5502   logger, any log statement from any logger can emit an LTTng event.
5503
5504 .Use the LTTng-UST Python agent.
5505 ====
5506 [source,python]
5507 .path:{test.py}
5508 ----
5509 import lttngust
5510 import logging
5511 import time
5512
5513
5514 def example():
5515     logging.basicConfig()
5516     logger = logging.getLogger('my-logger')
5517
5518     while True:
5519         logger.debug('debug message')
5520         logger.info('info message')
5521         logger.warn('warn message')
5522         logger.error('error message')
5523         logger.critical('critical message')
5524         time.sleep(1)
5525
5526
5527 if __name__ == '__main__':
5528     example()
5529 ----
5530
5531 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5532 logging handler which prints to the standard error stream, isn't
5533 strictly required for LTTng-UST tracing to work, but in versions of
5534 Python preceding{nbsp}3.2, you could see a warning message which
5535 indicates that no handler exists for the logger `my-logger`.
5536
5537 <<creating-destroying-tracing-sessions,Create a recording session>>,
5538 <<enabling-disabling-events,create a recording event rule>> matching
5539 Python logging events named `my-logger`, and
5540 <<basic-tracing-session-control,start recording>>:
5541
5542 [role="term"]
5543 ----
5544 $ lttng create
5545 $ lttng enable-event --python my-logger
5546 $ lttng start
5547 ----
5548
5549 Run the Python script:
5550
5551 [role="term"]
5552 ----
5553 $ python test.py
5554 ----
5555
5556 <<basic-tracing-session-control,Stop recording>> and inspect the
5557 recorded events:
5558
5559 [role="term"]
5560 ----
5561 $ lttng stop
5562 $ lttng view
5563 ----
5564 ====
5565
5566 In the resulting trace, an <<event,event record>> which a Python
5567 application generated is named `lttng_python:event` and has the
5568 following fields:
5569
5570 `asctime`::
5571   Logging time (string).
5572
5573 `msg`::
5574   Log record message.
5575
5576 `logger_name`::
5577   Logger name.
5578
5579 `funcName`::
5580   Name of the function in which the log statement was executed.
5581
5582 `lineno`::
5583   Line number at which the log statement was executed.
5584
5585 `int_loglevel`::
5586   Log level integer value.
5587
5588 `thread`::
5589   ID of the Python thread in which the log statement was executed.
5590
5591 `threadName`::
5592   Name of the Python thread in which the log statement was executed.
5593
5594 Use the opt:lttng-enable-event(1):--loglevel or
5595 opt:lttng-enable-event(1):--loglevel-only option of the
5596 man:lttng-enable-event(1) command to target a range of Python log levels
5597 or a specific Python log level.
5598
5599 When an application imports the LTTng-UST Python agent, the agent tries
5600 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5601 <<start-sessiond,start the session daemon>> _before_ you run the Python
5602 application. If a session daemon is found, the agent tries to register
5603 to it during five seconds, after which the application continues
5604 without LTTng tracing support. Override this timeout value with
5605 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5606 (milliseconds).
5607
5608 If the session daemon stops while a Python application with an imported
5609 LTTng-UST Python agent runs, the agent retries to connect and to
5610 register to a session daemon every three seconds. Override this
5611 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5612 variable.
5613
5614
5615 [role="since-2.5"]
5616 [[proc-lttng-logger-abi]]
5617 === Use the LTTng logger
5618
5619 The `lttng-tracer` Linux kernel module, part of
5620 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5621 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5622 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5623 to any of those files to create one or more LTTng events.
5624
5625 [role="img-100"]
5626 .An application writes to the LTTng logger file to create one or more LTTng events.
5627 image::lttng-logger.png[]
5628
5629 The LTTng logger is the quickest method--not the most efficient,
5630 however--to add instrumentation to an application. It's designed
5631 mostly to instrument shell scripts:
5632
5633 [role="term"]
5634 ----
5635 $ echo "Some message, some $variable" > /dev/lttng-logger
5636 ----
5637
5638 Any event that the LTTng logger creates is named `lttng_logger` and
5639 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5640 other instrumentation points in the kernel tracing domain, **any Unix
5641 user** can <<enabling-disabling-events,create a recording event rule>>
5642 which matches events named `lttng_logger`, not only the root user or
5643 users in the <<tracing-group,tracing group>>.
5644
5645 To use the LTTng logger:
5646
5647 * From any application, write text data to the path:{/dev/lttng-logger}
5648   file.
5649
5650 The `msg` field of `lttng_logger` event records contains the
5651 recorded message.
5652
5653 NOTE: The maximum message length of an LTTng logger event is
5654 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5655 than one event to contain the remaining data.
5656
5657 You shouldn't use the LTTng logger to trace a user application which you
5658 can instrument in a more efficient way, namely:
5659
5660 * <<c-application,C and $$C++$$ applications>>.
5661 * <<java-application,Java applications>>.
5662 * <<python-application,Python applications>>.
5663
5664 .Use the LTTng logger.
5665 ====
5666 [source,bash]
5667 .path:{test.bash}
5668 ----
5669 echo 'Hello, World!' > /dev/lttng-logger
5670 sleep 2
5671 df --human-readable --print-type / > /dev/lttng-logger
5672 ----
5673
5674 <<creating-destroying-tracing-sessions,Create a recording session>>,
5675 <<enabling-disabling-events,create a recording event rule>> matching
5676 Linux kernel tracepoint events named `lttng_logger`, and
5677 <<basic-tracing-session-control,start recording>>:
5678
5679 [role="term"]
5680 ----
5681 $ lttng create
5682 $ lttng enable-event --kernel lttng_logger
5683 $ lttng start
5684 ----
5685
5686 Run the Bash script:
5687
5688 [role="term"]
5689 ----
5690 $ bash test.bash
5691 ----
5692
5693 <<basic-tracing-session-control,Stop recording>> and inspect the recorded
5694 events:
5695
5696 [role="term"]
5697 ----
5698 $ lttng stop
5699 $ lttng view
5700 ----
5701 ====
5702
5703
5704 [[instrumenting-linux-kernel]]
5705 === Instrument a Linux kernel image or module
5706
5707 NOTE: This section shows how to _add_ instrumentation points to the
5708 Linux kernel. The subsystems of the kernel are already thoroughly
5709 instrumented at strategic points for LTTng when you
5710 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5711 package.
5712
5713
5714 [[linux-add-lttng-layer]]
5715 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5716
5717 This section shows how to add an LTTng layer to existing ftrace
5718 instrumentation using the `TRACE_EVENT()` API.
5719
5720 This section doesn't document the `TRACE_EVENT()` macro. Read the
5721 following articles to learn more about this API:
5722
5723 * https://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5724 * https://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5725 * https://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5726
5727 The following procedure assumes that your ftrace tracepoints are
5728 correctly defined in their own header and that they're created in
5729 one source file using the `CREATE_TRACE_POINTS` definition.
5730
5731 To add an LTTng layer over an existing ftrace tracepoint:
5732
5733 . Make sure the following kernel configuration options are
5734   enabled:
5735 +
5736 --
5737 * `CONFIG_MODULES`
5738 * `CONFIG_KALLSYMS`
5739 * `CONFIG_HIGH_RES_TIMERS`
5740 * `CONFIG_TRACEPOINTS`
5741 --
5742
5743 . Build the Linux source tree with your custom ftrace tracepoints.
5744 . Boot the resulting Linux image on your target system.
5745 +
5746 Confirm that the tracepoints exist by looking for their names in the
5747 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5748 is your subsystem name.
5749
5750 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5751 +
5752 --
5753 [role="term"]
5754 ----
5755 $ cd $(mktemp -d) &&
5756   wget https://lttng.org/files/lttng-modules/lttng-modules-latest-2.13.tar.bz2 &&
5757   tar -xf lttng-modules-latest-2.13.tar.bz2 &&
5758   cd lttng-modules-2.13.*
5759 ----
5760 --
5761
5762 . In dir:{instrumentation/events/lttng-module}, relative to the root
5763   of the LTTng-modules source tree, create a header file named
5764   +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5765   LTTng-modules tracepoint definitions using the LTTng-modules
5766   macros in it.
5767 +
5768 Start with this template:
5769 +
5770 --
5771 [source,c]
5772 .path:{instrumentation/events/lttng-module/my_subsys.h}
5773 ----
5774 #undef TRACE_SYSTEM
5775 #define TRACE_SYSTEM my_subsys
5776
5777 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5778 #define _LTTNG_MY_SUBSYS_H
5779
5780 #include "../../../probes/lttng-tracepoint-event.h"
5781 #include <linux/tracepoint.h>
5782
5783 LTTNG_TRACEPOINT_EVENT(
5784     /*
5785      * Format is identical to the TRACE_EVENT() version for the three
5786      * following macro parameters:
5787      */
5788     my_subsys_my_event,
5789     TP_PROTO(int my_int, const char *my_string),
5790     TP_ARGS(my_int, my_string),
5791
5792     /* LTTng-modules specific macros */
5793     TP_FIELDS(
5794         ctf_integer(int, my_int_field, my_int)
5795         ctf_string(my_bar_field, my_bar)
5796     )
5797 )
5798
5799 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5800
5801 #include "../../../probes/define_trace.h"
5802 ----
5803 --
5804 +
5805 The entries in the `TP_FIELDS()` section are the list of fields for the
5806 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5807 the `TRACE_EVENT()` ftrace macro.
5808 +
5809 See ``<<lttng-modules-tp-fields,Tracepoint fields macros>>'' for a
5810 complete description of the available `ctf_*()` macros.
5811
5812 . Create the kernel module C{nbsp}source file of the LTTng-modules
5813   probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5814   subsystem name:
5815 +
5816 --
5817 [source,c]
5818 .path:{probes/lttng-probe-my-subsys.c}
5819 ----
5820 #include <linux/module.h>
5821 #include "../lttng-tracer.h"
5822
5823 /*
5824  * Build-time verification of mismatch between mainline
5825  * TRACE_EVENT() arguments and the LTTng-modules adaptation
5826  * layer LTTNG_TRACEPOINT_EVENT() arguments.
5827  */
5828 #include <trace/events/my_subsys.h>
5829
5830 /* Create LTTng tracepoint probes */
5831 #define LTTNG_PACKAGE_BUILD
5832 #define CREATE_TRACE_POINTS
5833 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5834
5835 #include "../instrumentation/events/lttng-module/my_subsys.h"
5836
5837 MODULE_LICENSE("GPL and additional rights");
5838 MODULE_AUTHOR("Your name <your-email>");
5839 MODULE_DESCRIPTION("LTTng my_subsys probes");
5840 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5841     __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5842     __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5843     LTTNG_MODULES_EXTRAVERSION);
5844 ----
5845 --
5846
5847 . Edit path:{probes/KBuild} and add your new kernel module object
5848   next to the existing ones:
5849 +
5850 --
5851 [source,make]
5852 .path:{probes/KBuild}
5853 ----
5854 # ...
5855
5856 obj-m += lttng-probe-module.o
5857 obj-m += lttng-probe-power.o
5858
5859 obj-m += lttng-probe-my-subsys.o
5860
5861 # ...
5862 ----
5863 --
5864
5865 . Build and install the LTTng kernel modules:
5866 +
5867 --
5868 [role="term"]
5869 ----
5870 $ make KERNELDIR=/path/to/linux
5871 # make modules_install && depmod -a
5872 ----
5873 --
5874 +
5875 Replace `/path/to/linux` with the path to the Linux source tree where
5876 you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
5877
5878 Note that you can also use the
5879 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5880 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5881 C{nbsp}code that need to be executed before LTTng records the event
5882 fields.
5883
5884 The best way to learn how to use the previous LTTng-modules macros is to
5885 inspect the existing LTTng-modules tracepoint definitions in the
5886 dir:{instrumentation/events/lttng-module} header files. Compare them
5887 with the Linux kernel mainline versions in the
5888 dir:{include/trace/events} directory of the Linux source tree.
5889
5890
5891 [role="since-2.7"]
5892 [[lttng-tracepoint-event-code]]
5893 ===== Use custom C code to access the data for tracepoint fields
5894
5895 Although we recommended to always use the
5896 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5897 the arguments and fields of an LTTng-modules tracepoint when possible,
5898 sometimes you need a more complex process to access the data that the
5899 tracer records as event record fields. In other words, you need local
5900 variables and multiple C{nbsp}statements instead of simple
5901 argument-based expressions that you pass to the
5902 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5903
5904 Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5905 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5906 a block of C{nbsp}code to be executed before LTTng records the fields.
5907 The structure of this macro is:
5908
5909 [source,c]
5910 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5911 ----
5912 LTTNG_TRACEPOINT_EVENT_CODE(
5913     /*
5914      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5915      * version for the following three macro parameters:
5916      */
5917     my_subsys_my_event,
5918     TP_PROTO(int my_int, const char *my_string),
5919     TP_ARGS(my_int, my_string),
5920
5921     /* Declarations of custom local variables */
5922     TP_locvar(
5923         int a = 0;
5924         unsigned long b = 0;
5925         const char *name = "(undefined)";
5926         struct my_struct *my_struct;
5927     ),
5928
5929     /*
5930      * Custom code which uses both tracepoint arguments
5931      * (in TP_ARGS()) and local variables (in TP_locvar()).
5932      *
5933      * Local variables are actually members of a structure pointed
5934      * to by the special variable tp_locvar.
5935      */
5936     TP_code(
5937         if (my_int) {
5938             tp_locvar->a = my_int + 17;
5939             tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5940             tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5941             tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5942             put_my_struct(tp_locvar->my_struct);
5943
5944             if (tp_locvar->b) {
5945                 tp_locvar->a = 1;
5946             }
5947         }
5948     ),
5949
5950     /*
5951      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5952      * version for this, except that tp_locvar members can be
5953      * used in the argument expression parameters of
5954      * the ctf_*() macros.
5955      */
5956     TP_FIELDS(
5957         ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5958         ctf_integer(int, my_struct_a, tp_locvar->a)
5959         ctf_string(my_string_field, my_string)
5960         ctf_string(my_struct_name, tp_locvar->name)
5961     )
5962 )
5963 ----
5964
5965 IMPORTANT: The C code defined in `TP_code()` must not have any side
5966 effects when executed. In particular, the code must not allocate
5967 memory or get resources without deallocating this memory or putting
5968 those resources afterwards.
5969
5970
5971 [[instrumenting-linux-kernel-tracing]]
5972 ==== Load and unload a custom probe kernel module
5973
5974 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5975 kernel module>> in the kernel before it can emit LTTng events.
5976
5977 To load the default probe kernel modules and a custom probe kernel
5978 module:
5979
5980 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5981   probe modules to load when starting a root <<lttng-sessiond,session
5982   daemon>>:
5983 +
5984 --
5985 .Load the `my_subsys`, `usb`, and the default probe modules.
5986 ====
5987 [role="term"]
5988 ----
5989 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5990 ----
5991 ====
5992 --
5993 +
5994 You only need to pass the subsystem name, not the whole kernel module
5995 name.
5996
5997 To load _only_ a given custom probe kernel module:
5998
5999 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
6000   modules to load when starting a root session daemon:
6001 +
6002 --
6003 .Load only the `my_subsys` and `usb` probe modules.
6004 ====
6005 [role="term"]
6006 ----
6007 # lttng-sessiond --kmod-probes=my_subsys,usb
6008 ----
6009 ====
6010 --
6011
6012 To confirm that a probe module is loaded:
6013
6014 * Use man:lsmod(8):
6015 +
6016 --
6017 [role="term"]
6018 ----
6019 $ lsmod | grep lttng_probe_usb
6020 ----
6021 --
6022
6023 To unload the loaded probe modules:
6024
6025 * Kill the session daemon with `SIGTERM`:
6026 +
6027 --
6028 [role="term"]
6029 ----
6030 # pkill lttng-sessiond
6031 ----
6032 --
6033 +
6034 You can also use the `--remove` option of man:modprobe(8) if the session
6035 daemon terminates abnormally.
6036
6037
6038 [[controlling-tracing]]
6039 == Tracing control
6040
6041 Once an application or a Linux kernel is <<instrumenting,instrumented>>
6042 for LTTng tracing, you can _trace_ it.
6043
6044 In the LTTng context, _tracing_ means making sure that LTTng attempts to
6045 execute some action(s) when a CPU executes an instrumentation point.
6046
6047 This section is divided in topics on how to use the various
6048 <<plumbing,components of LTTng>>, in particular the
6049 <<lttng-cli,cmd:lttng command-line tool>>, to _control_ the LTTng
6050 daemons and tracers.
6051
6052 NOTE: In the following subsections, we refer to an man:lttng(1) command
6053 using its man page name. For example, instead of ``Run the `create`
6054 command to'', we write ``Run the man:lttng-create(1) command to''.
6055
6056
6057 [[start-sessiond]]
6058 === Start a session daemon
6059
6060 In some situations, you need to run a <<lttng-sessiond,session daemon>>
6061 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
6062 command-line tool.
6063
6064 You will see the following error when you run a command while no session
6065 daemon is running:
6066
6067 ----
6068 Error: No session daemon is available
6069 ----
6070
6071 The only command that automatically runs a session daemon is
6072 man:lttng-create(1), which you use to
6073 <<creating-destroying-tracing-sessions,create a recording session>>. While
6074 this could be your most used first operation, sometimes it's not. Some
6075 examples are:
6076
6077 * <<list-instrumentation-points,List the available instrumentation points>>.
6078 * <<saving-loading-tracing-session,Load a recording session configuration>>.
6079 * <<add-event-rule-matches-trigger,Add a trigger>>.
6080
6081 All the examples above don't require a recording session to operate on.
6082
6083 [[tracing-group]] Each Unix user can have its own running session daemon
6084 to use the user space LTTng tracer. The session daemon that the `root`
6085 user starts is the only one allowed to control the LTTng kernel tracer.
6086 Members of the Unix _tracing group_ may connect to and control the root
6087 session daemon, even for user space tracing. See the ``Session daemon
6088 connection'' section of man:lttng(1) to learn more about the Unix
6089 tracing group.
6090
6091 To start a user session daemon:
6092
6093 * Run man:lttng-sessiond(8):
6094 +
6095 --
6096 [role="term"]
6097 ----
6098 $ lttng-sessiond --daemonize
6099 ----
6100 --
6101
6102 To start the root session daemon:
6103
6104 * Run man:lttng-sessiond(8) as the `root` user:
6105 +
6106 --
6107 [role="term"]
6108 ----
6109 # lttng-sessiond --daemonize
6110 ----
6111 --
6112
6113 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
6114 start the session daemon in foreground.
6115
6116 To stop a session daemon, kill its process (see man:kill(1)) with the
6117 standard `TERM` signal.
6118
6119 Note that some Linux distributions could manage the LTTng session daemon
6120 as a service. In this case, we suggest that you use the service manager
6121 to start, restart, and stop session daemons.
6122
6123
6124 [[creating-destroying-tracing-sessions]]
6125 === Create and destroy a recording session
6126
6127 Many LTTng control operations happen in the scope of a
6128 <<tracing-session,recording session>>, which is the dialogue between the
6129 <<lttng-sessiond,session daemon>> and you for everything related to
6130 <<event,event recording>>.
6131
6132 To create a recording session with a generated name:
6133
6134 * Use the man:lttng-create(1) command:
6135 +
6136 --
6137 [role="term"]
6138 ----
6139 $ lttng create
6140 ----
6141 --
6142
6143 The name of the created recording session is `auto` followed by the
6144 creation date.
6145
6146 To create a recording session with a specific name:
6147
6148 * Use the optional argument of the man:lttng-create(1) command:
6149 +
6150 --
6151 [role="term"]
6152 ----
6153 $ lttng create SESSION
6154 ----
6155 --
6156 +
6157 Replace +__SESSION__+ with your specific recording session name.
6158
6159 In <<local-mode,local mode>>, LTTng writes the traces of a recording
6160 session to the +$LTTNG_HOME/lttng-traces/__NAME__-__DATE__-__TIME__+
6161 directory by default, where +__NAME__+ is the name of the recording
6162 session. Note that the env:LTTNG_HOME environment variable defaults to
6163 `$HOME` if not set.
6164
6165 To output LTTng traces to a non-default location:
6166
6167 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1)
6168   command:
6169 +
6170 --
6171 [role="term"]
6172 ----
6173 $ lttng create my-session --output=/tmp/some-directory
6174 ----
6175 --
6176
6177 You may create as many recording sessions as you wish.
6178
6179 To list all the existing recording sessions for your Unix user, or for
6180 all users if your Unix user is `root`:
6181
6182 * Use the man:lttng-list(1) command:
6183 +
6184 --
6185 [role="term"]
6186 ----
6187 $ lttng list
6188 ----
6189 --
6190
6191 [[cur-tracing-session]]When you create a recording session, the
6192 man:lttng-create(1) command sets it as the _current recording session_.
6193 The following man:lttng(1) commands operate on the current recording
6194 session when you don't specify one:
6195
6196 [role="list-3-cols"]
6197 * man:lttng-add-context(1)
6198 * man:lttng-clear(1)
6199 * man:lttng-destroy(1)
6200 * man:lttng-disable-channel(1)
6201 * man:lttng-disable-event(1)
6202 * man:lttng-disable-rotation(1)
6203 * man:lttng-enable-channel(1)
6204 * man:lttng-enable-event(1)
6205 * man:lttng-enable-rotation(1)
6206 * man:lttng-load(1)
6207 * man:lttng-regenerate(1)
6208 * man:lttng-rotate(1)
6209 * man:lttng-save(1)
6210 * man:lttng-snapshot(1)
6211 * man:lttng-start(1)
6212 * man:lttng-status(1)
6213 * man:lttng-stop(1)
6214 * man:lttng-track(1)
6215 * man:lttng-untrack(1)
6216 * man:lttng-view(1)
6217
6218 To change the current recording session:
6219
6220 * Use the man:lttng-set-session(1) command:
6221 +
6222 --
6223 [role="term"]
6224 ----
6225 $ lttng set-session SESSION
6226 ----
6227 --
6228 +
6229 Replace +__SESSION__+ with the name of the new current recording session.
6230
6231 When you're done recording in a given recording session, destroy it.
6232 This operation frees the resources taken by the recording session to
6233 destroy; it doesn't destroy the trace data that LTTng wrote for this
6234 recording session (see ``<<clear,Clear a recording session>>'' for one
6235 way to do this).
6236
6237 To destroy the current recording session:
6238
6239 * Use the man:lttng-destroy(1) command:
6240 +
6241 --
6242 [role="term"]
6243 ----
6244 $ lttng destroy
6245 ----
6246 --
6247
6248 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
6249 implicitly (see ``<<basic-tracing-session-control,Start and stop a
6250 recording session>>''). You need to stop recording to make LTTng flush the
6251 remaining trace data and make the trace readable.
6252
6253
6254 [[list-instrumentation-points]]
6255 === List the available instrumentation points
6256
6257 The <<lttng-sessiond,session daemon>> can query the running instrumented
6258 user applications and the Linux kernel to get a list of available
6259 instrumentation points:
6260
6261 * LTTng tracepoints and system calls for the Linux kernel
6262   <<domain,tracing domain>>.
6263
6264 * LTTng tracepoints for the user space tracing domain.
6265
6266 To list the available instrumentation points:
6267
6268 . <<start-sessiond,Make sure>> there's a running
6269   <<lttng-sessiond,session daemon>> to which your Unix user can
6270   connect.
6271
6272 . Use the man:lttng-list(1) command with the option of the requested
6273   tracing domain amongst:
6274 +
6275 --
6276 opt:lttng-list(1):--kernel::
6277     Linux kernel tracepoints.
6278 +
6279 Your Unix user must be `root`, or it must be a member of the Unix
6280 <<tracing-group,tracing group>>.
6281
6282 opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
6283     Linux kernel system calls.
6284 +
6285 Your Unix user must be `root`, or it must be a member of the Unix
6286 <<tracing-group,tracing group>>.
6287
6288 opt:lttng-list(1):--userspace::
6289     User space tracepoints.
6290
6291 opt:lttng-list(1):--jul::
6292    `java.util.logging` loggers.
6293
6294 opt:lttng-list(1):--log4j::
6295     Apache log4j loggers.
6296
6297 opt:lttng-list(1):--python::
6298     Python loggers.
6299 --
6300
6301 .List the available user space tracepoints.
6302 ====
6303 [role="term"]
6304 ----
6305 $ lttng list --userspace
6306 ----
6307 ====
6308
6309 .List the available Linux kernel system calls.
6310 ====
6311 [role="term"]
6312 ----
6313 $ lttng list --kernel --syscall
6314 ----
6315 ====
6316
6317
6318 [[enabling-disabling-events]]
6319 === Create and enable a recording event rule
6320
6321 Once you <<creating-destroying-tracing-sessions,create a recording
6322 session>>, you can create <<event,recording event rules>> with the
6323 man:lttng-enable-event(1) command.
6324
6325 The man:lttng-enable-event(1) command always attaches an event rule to a
6326 <<channel,channel>> on creation. The command can create a _default
6327 channel_, named `channel0`, for you. The man:lttng-enable-event(1)
6328 command reuses the default channel each time you run it for the same
6329 tracing domain and session.
6330
6331 A recording event rule is always enabled at creation time.
6332
6333 The following examples show how to combine the command-line arguments of
6334 the man:lttng-enable-event(1) command to create simple to more complex
6335 recording event rules within the <<cur-tracing-session,current recording
6336 session>>.
6337
6338 .Create a recording event rule matching specific Linux kernel tracepoint events (default channel).
6339 ====
6340 [role="term"]
6341 ----
6342 # lttng enable-event --kernel sched_switch
6343 ----
6344 ====
6345
6346 .Create a recording event rule matching Linux kernel system call events with four specific names (default channel).
6347 ====
6348 [role="term"]
6349 ----
6350 # lttng enable-event --kernel --syscall open,write,read,close
6351 ----
6352 ====
6353
6354 .Create recording event rules matching tracepoint events which satisfy a filter expressions (default channel).
6355 ====
6356 [role="term"]
6357 ----
6358 # lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
6359 ----
6360
6361 [role="term"]
6362 ----
6363 # lttng enable-event --kernel --all \
6364                      --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
6365 ----
6366
6367 [role="term"]
6368 ----
6369 $ lttng enable-event --jul my_logger \
6370                      --filter='$app.retriever:cur_msg_id > 3'
6371 ----
6372
6373 IMPORTANT: Make sure to always single-quote the filter string when you
6374 run man:lttng(1) from a shell.
6375
6376 See also ``<<pid-tracking,Allow specific processes to record events>>''
6377 which offers another, more efficient filtering mechanism for process ID,
6378 user ID, and group ID attributes.
6379 ====
6380
6381 .Create a recording event rule matching any user space event from the `my_app` tracepoint provider and with a log level range (default channel).
6382 ====
6383 [role="term"]
6384 ----
6385 $ lttng enable-event --userspace my_app:'*' --loglevel=INFO
6386 ----
6387
6388 IMPORTANT: Make sure to always single-quote the wildcard character when
6389 you run man:lttng(1) from a shell.
6390 ====
6391
6392 .Create a recording event rule matching user space events named specifically, but with name exclusions (default channel).
6393 ====
6394 [role="term"]
6395 ----
6396 $ lttng enable-event --userspace my_app:'*' \
6397                      --exclude=my_app:set_user,my_app:handle_sig
6398 ----
6399 ====
6400
6401 .Create a recording event rule matching any Apache log4j event with a specific log level (default channel).
6402 ====
6403 [role="term"]
6404 ----
6405 $ lttng enable-event --log4j --all --loglevel-only=WARN
6406 ----
6407 ====
6408
6409 .Create a recording event rule, attached to a specific channel, and matching user space tracepoint events named `my_app:my_tracepoint`.
6410 ====
6411 [role="term"]
6412 ----
6413 $ lttng enable-event --userspace my_app:my_tracepoint \
6414                      --channel=my-channel
6415 ----
6416 ====
6417
6418 .Create a recording event rule matching user space probe events for the `malloc` function entry in path:{/usr/lib/libc.so.6}:
6419 ====
6420 [role="term"]
6421 ----
6422 # lttng enable-event --kernel \
6423                      --userspace-probe=/usr/lib/libc.so.6:malloc \
6424                      libc_malloc
6425 ----
6426 ====
6427
6428 .Create a recording event rule matching user space probe events for the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
6429 ====
6430 [role="term"]
6431 ----
6432 # lttng enable-event --kernel \
6433                      --userspace-probe=sdt:serv:server:accept_request \
6434                      server_accept_request
6435 ----
6436 ====
6437
6438 The recording event rules of a given channel form a whitelist: as soon
6439 as an event rule matches an event, LTTng emits it _once_ and therefore
6440 <<channel-overwrite-mode-vs-discard-mode,can>> record it. For example,
6441 the following rules both match user space tracepoint events named
6442 `my_app:my_tracepoint` with an `INFO` log level:
6443
6444 [role="term"]
6445 ----
6446 $ lttng enable-event --userspace my_app:my_tracepoint
6447 $ lttng enable-event --userspace my_app:my_tracepoint \
6448                      --loglevel=INFO
6449 ----
6450
6451 The second recording event rule is redundant: the first one includes the
6452 second one.
6453
6454
6455 [[disable-event-rule]]
6456 === Disable a recording event rule
6457
6458 To disable a <<event,recording event rule>> that you
6459 <<enabling-disabling-events,created>> previously, use the
6460 man:lttng-disable-event(1) command.
6461
6462 man:lttng-disable-event(1) can only find recording event rules to
6463 disable by their <<instrumentation-point-types,instrumentation point
6464 type>> and event name conditions. Therefore, you cannot disable
6465 recording event rules having a specific instrumentation point log level
6466 condition, for example.
6467
6468 LTTng doesn't emit (and, therefore, won't record) an event which only
6469 _disabled_ recording event rules match.
6470
6471 .Disable event rules matching Python logging events from the `my-logger` logger (default <<channel,channel>>, <<cur-tracing-session,current recording session>>).
6472 ====
6473 [role="term"]
6474 ----
6475 $ lttng disable-event --python my-logger
6476 ----
6477 ====
6478
6479 .Disable event rules matching all `java.util.logging` events (default channel, recording session `my-session`).
6480 ====
6481 [role="term"]
6482 ----
6483 $ lttng disable-event --jul --session=my-session '*'
6484 ----
6485 ====
6486
6487 .Disable _all_ the Linux kernel recording event rules (channel `my-chan`, current recording session).
6488 ====
6489 The opt:lttng-disable-event(1):--all-events option isn't, like the
6490 opt:lttng-enable-event(1):--all option of the man:lttng-enable-event(1)
6491 command, an alias for the event name globbing pattern `*`: it disables
6492 _all_ the recording event rules of a given channel.
6493
6494 [role="term"]
6495 ----
6496 # lttng disable-event --kernel --channel=my-chan --all-events
6497 ----
6498 ====
6499
6500 NOTE: You can't _remove_ a recording event rule once you create it.
6501
6502
6503 [[status]]
6504 === Get the status of a recording session
6505
6506 To get the status of the <<cur-tracing-session,current recording
6507 session>>, that is, its parameters, its channels, recording event rules,
6508 and their attributes:
6509
6510 * Use the man:lttng-status(1) command:
6511 +
6512 --
6513 [role="term"]
6514 ----
6515 $ lttng status
6516 ----
6517 --
6518
6519 To get the status of any recording session:
6520
6521 * Use the man:lttng-list(1) command with the name of the recording
6522   session:
6523 +
6524 --
6525 [role="term"]
6526 ----
6527 $ lttng list SESSION
6528 ----
6529 --
6530 +
6531 Replace +__SESSION__+ with the recording session name.
6532
6533
6534 [[basic-tracing-session-control]]
6535 === Start and stop a recording session
6536
6537 Once you <<creating-destroying-tracing-sessions,create a recording
6538 session>> and <<enabling-disabling-events,create one or more recording
6539 event rules>>, you can start and stop the tracers for this recording
6540 session.
6541
6542 To start the <<cur-tracing-session,current recording session>>:
6543
6544 * Use the man:lttng-start(1) command:
6545 +
6546 --
6547 [role="term"]
6548 ----
6549 $ lttng start
6550 ----
6551 --
6552
6553 LTTng is flexible: you can launch user applications before or after you
6554 start the tracers. An LTTng tracer only <<event,records an event>> if a
6555 recording event rule matches it, which means the tracer is active.
6556
6557 The `start-session` <<trigger,trigger>> action can also start a recording
6558 session.
6559
6560 To stop the current recording session:
6561
6562 * Use the man:lttng-stop(1) command:
6563 +
6564 --
6565 [role="term"]
6566 ----
6567 $ lttng stop
6568 ----
6569 --
6570 +
6571 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6572 records>> or lost sub-buffers since the last time you ran
6573 man:lttng-start(1), the man:lttng-stop(1) command prints corresponding
6574 warnings.
6575
6576 IMPORTANT: You need to stop recording to make LTTng flush the remaining
6577 trace data and make the trace readable. Note that the
6578 man:lttng-destroy(1) command (see
6579 ``<<creating-destroying-tracing-sessions,Create and destroy a recording
6580 session>>'') also runs the man:lttng-stop(1) command implicitly.
6581
6582 The `stop-session` <<trigger,trigger>> action can also stop a recording
6583 session.
6584
6585 [role="since-2.12"]
6586 [[clear]]
6587 === Clear a recording session
6588
6589 You might need to remove all the current tracing data of one or more
6590 <<tracing-session,recording sessions>> between multiple attempts to
6591 reproduce a problem without interrupting the LTTng recording activity.
6592
6593 To clear the tracing data of the
6594 <<cur-tracing-session,current recording session>>:
6595
6596 * Use the man:lttng-clear(1) command:
6597 +
6598 --
6599 [role="term"]
6600 ----
6601 $ lttng clear
6602 ----
6603 --
6604
6605 To clear the tracing data of all the recording sessions:
6606
6607 * Use the `lttng clear` command with its opt:lttng-clear(1):--all
6608   option:
6609 +
6610 --
6611 [role="term"]
6612 ----
6613 $ lttng clear --all
6614 ----
6615 --
6616
6617
6618 [[enabling-disabling-channels]]
6619 === Create a channel
6620
6621 Once you <<creating-destroying-tracing-sessions,create a recording
6622 session>>, you can create a <<channel,channel>> with the
6623 man:lttng-enable-channel(1) command.
6624
6625 Note that LTTng can automatically create a default channel when you
6626 <<enabling-disabling-events,create a recording event rule>>.
6627 Therefore, you only need to create a channel when you need non-default
6628 attributes.
6629
6630 Specify each non-default channel attribute with a command-line
6631 option when you run the man:lttng-enable-channel(1) command.
6632
6633 You can only create a custom channel in the Linux kernel and user space
6634 <<domain,tracing domains>>: the Java/Python logging tracing domains have
6635 their own default channel which LTTng automatically creates when you
6636 <<enabling-disabling-events,create a recording event rule>>.
6637
6638 [IMPORTANT]
6639 ====
6640 As of LTTng{nbsp}{revision}, you may _not_ perform the
6641 following operations with the man:lttng-enable-channel(1) command:
6642
6643 * Change an attribute of an existing channel.
6644
6645 * Enable a disabled channel once its recording session has been
6646   <<basic-tracing-session-control,active>> at least once.
6647
6648 * Create a channel once its recording session has been active at
6649   least once.
6650
6651 * Create a user space channel with a given
6652   <<channel-buffering-schemes,buffering scheme>> and create a second
6653   user space channel with a different buffering scheme in the same
6654   recording session.
6655 ====
6656
6657 The following examples show how to combine the command-line options of
6658 the man:lttng-enable-channel(1) command to create simple to more complex
6659 channels within the <<cur-tracing-session,current recording session>>.
6660
6661 .Create a Linux kernel channel with default attributes.
6662 ====
6663 [role="term"]
6664 ----
6665 # lttng enable-channel --kernel my-channel
6666 ----
6667 ====
6668
6669 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6670 ====
6671 [role="term"]
6672 ----
6673 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6674                        --buffers-pid my-channel
6675 ----
6676 ====
6677
6678 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6679 ====
6680 <<creating-destroying-tracing-sessions,Create a recording session>>,
6681 create the channel, <<enabling-disabling-events,create a recording event
6682 rule>>, and <<basic-tracing-session-control,start recording>>:
6683
6684 [role="term"]
6685 ----
6686 $ lttng create
6687 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-chan
6688 $ lttng enable-event --userspace --channel=blocking-chan --all
6689 $ lttng start
6690 ----
6691
6692 Run an application instrumented with LTTng-UST tracepoints and allow it
6693 to block:
6694
6695 [role="term"]
6696 ----
6697 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6698 ----
6699 ====
6700
6701 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream.
6702 ====
6703 [role="term"]
6704 ----
6705 # lttng enable-channel --kernel --tracefile-count=8 \
6706                        --tracefile-size=4194304 my-channel
6707 ----
6708 ====
6709
6710 .Create a user space channel in <<overwrite-mode,overwrite>> (or ``flight recorder'') mode.
6711 ====
6712 [role="term"]
6713 ----
6714 $ lttng enable-channel --userspace --overwrite my-channel
6715 ----
6716 ====
6717
6718 .<<enabling-disabling-events,Create>> the same <<event,recording event rule>> attached to two different channels.
6719 ====
6720 [role="term"]
6721 ----
6722 $ lttng enable-event --userspace --channel=my-channel app:tp
6723 $ lttng enable-event --userspace --channel=other-channel app:tp
6724 ----
6725
6726 When a CPU executes the `app:tp` <<c-application,user space
6727 tracepoint>>, the two recording event rules above match the created
6728 event, making LTTng emit the event. Because the recording event rules
6729 are not attached to the same channel, LTTng records the event twice.
6730 ====
6731
6732
6733 [[disable-channel]]
6734 === Disable a channel
6735
6736 To disable a specific channel that you
6737 <<enabling-disabling-channels,created>> previously, use the
6738 man:lttng-disable-channel(1) command.
6739
6740 .Disable a specific Linux kernel channel (<<cur-tracing-session,current recording session>>).
6741 ====
6742 [role="term"]
6743 ----
6744 # lttng disable-channel --kernel my-channel
6745 ----
6746 ====
6747
6748 An enabled channel is an implicit <<event,recording event rule>>
6749 condition.
6750
6751 NOTE: As of LTTng{nbsp}{revision}, you may _not_ enable a disabled
6752 channel once its recording session has been
6753 <<basic-tracing-session-control,started>> at least once.
6754
6755
6756 [[adding-context]]
6757 === Add context fields to be recorded to the event records of a channel
6758
6759 <<event,Event record>> fields in trace files provide important
6760 information about previously emitted events, but sometimes some external
6761 context may help you solve a problem faster.
6762
6763 Examples of context fields are:
6764
6765 * The **process ID**, **thread ID**, **process name**, and
6766   **process priority** of the thread from which LTTng emits the event.
6767
6768 * The **hostname** of the system on which LTTng emits the event.
6769
6770 * The Linux kernel and user call stacks (since LTTng{nbsp}2.11).
6771
6772 * The current values of many possible **performance counters** using
6773   perf, for example:
6774
6775 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6776 ** Cache misses.
6777 ** Branch instructions, misses, and loads.
6778 ** CPU faults.
6779
6780 * Any state defined at the application level (supported for the
6781   `java.util.logging` and Apache log4j <<domain,tracing domains>>).
6782
6783 To get the full list of available context fields:
6784
6785 * Use the opt:lttng-add-context(1):--list option of the
6786   man:lttng-add-context(1) command:
6787 +
6788 [role="term"]
6789 ----
6790 $ lttng add-context --list
6791 ----
6792
6793 .Add context fields to be recorded to the event records of all the <<channel,channels>> of the <<cur-tracing-session,current recording session>>.
6794 ====
6795 The following command line adds the virtual process identifier and the
6796 per-thread CPU cycles count fields to all the user space channels of the
6797 current recording session.
6798
6799 [role="term"]
6800 ----
6801 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6802 ----
6803 ====
6804
6805 .Add performance counter context fields by raw ID
6806 ====
6807 See man:lttng-add-context(1) for the exact format of the context field
6808 type, which is partly compatible with the format used in
6809 man:perf-record(1).
6810
6811 [role="term"]
6812 ----
6813 # lttng add-context --userspace --type=perf:thread:raw:r0110:test
6814 # lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6815 ----
6816 ====
6817
6818 .Add context fields to be recorded to the event records of a specific channel.
6819 ====
6820 The following command line adds the thread identifier and user call
6821 stack context fields to the Linux kernel channel named `my-channel` of
6822 the <<cur-tracing-session,current recording session>>.
6823
6824 [role="term"]
6825 ----
6826 # lttng add-context --kernel --channel=my-channel \
6827                     --type=tid --type=callstack-user
6828 ----
6829 ====
6830
6831 .Add an <<java-application-context,application-specific context field>> to be recorded to the event records of a specific channel.
6832 ====
6833 The following command line makes sure LTTng writes the `cur_msg_id`
6834 context field of the `retriever` context retriever to all the Java
6835 logging <<event,event records>> of the channel named `my-channel`:
6836
6837 [role="term"]
6838 ----
6839 # lttng add-context --kernel --channel=my-channel \
6840                     --type='$app:retriever:cur_msg_id'
6841 ----
6842
6843 IMPORTANT: Make sure to always single-quote the `$` character when you
6844 run man:lttng-add-context(1) from a shell.
6845 ====
6846
6847 NOTE: You can't undo what the man:lttng-add-context(1) command does.
6848
6849
6850 [role="since-2.7"]
6851 [[pid-tracking]]
6852 === Allow specific processes to record events
6853
6854 It's often useful to only allow processes with specific attributes to
6855 record events. For example, you may wish to record all the system calls
6856 which a given process makes (à la man:strace(1)).
6857
6858 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6859 purpose. Both commands operate on _inclusion sets_ of process
6860 attributes. The available process attribute types are:
6861
6862 Linux kernel <<domain,tracing domain>>::
6863 +
6864 * Process ID (PID).
6865
6866 * Virtual process ID (VPID).
6867 +
6868 This is the PID as seen by the application.
6869
6870 * Unix user ID (UID).
6871
6872 * Virtual Unix user ID (VUID).
6873 +
6874 This is the UID as seen by the application.
6875
6876 * Unix group ID (GID).
6877
6878 * Virtual Unix group ID (VGID).
6879 +
6880 This is the GID as seen by the application.
6881
6882 User space tracing domain::
6883 +
6884 * VPID
6885 * VUID
6886 * VGID
6887
6888 A <<tracing-session,recording session>> has nine process
6889 attribute inclusion sets: six for the Linux kernel <<domain,tracing domain>>
6890 and three for the user space tracing domain.
6891
6892 For a given recording session, a process{nbsp}__P__ is allowed to record
6893 LTTng events for a given <<domain,tracing domain>>{nbsp}__D__ if _all_
6894 the attributes of{nbsp}__P__ are part of the inclusion sets
6895 of{nbsp}__D__.
6896
6897 Whether a process is allowed or not to record LTTng events is an
6898 implicit condition of all <<event,recording event rules>>. Therefore, if
6899 LTTng creates an event{nbsp}__E__ for a given process, but this process
6900 may not record events, then no recording event rule matches{nbsp}__E__,
6901 which means LTTng won't emit and record{nbsp}__E__.
6902
6903 When you <<creating-destroying-tracing-sessions,create a recording
6904 session>>, all its process attribute inclusion sets contain all the
6905 possible values. In other words, all processes are allowed to record
6906 events.
6907
6908 Add values to an inclusion set with the man:lttng-track(1) command and
6909 remove values with the man:lttng-untrack(1) command.
6910
6911 [NOTE]
6912 ====
6913 The process attribute values are _numeric_.
6914
6915 Should a process with a given ID (part of an inclusion set), for
6916 example, exit, and then a new process be given this same ID, then the
6917 latter would also be allowed to record events.
6918
6919 With the man:lttng-track(1) command, you can add Unix user and group
6920 _names_ to the user and group inclusion sets: the
6921 <<lttng-sessiond,session daemon>> finds the corresponding UID, VUID,
6922 GID, or VGID once on _addition_ to the inclusion set. This means that if
6923 you rename the user or group after you run the man:lttng-track(1)
6924 command, its user/group ID remains part of the inclusion sets.
6925 ====
6926
6927 .Allow processes to record events based on their virtual process ID (VPID).
6928 ====
6929 For the sake of the following example, assume the target system has
6930 16{nbsp}possible VPIDs.
6931
6932 When you
6933 <<creating-destroying-tracing-sessions,create a recording session>>,
6934 the user space VPID inclusion set contains _all_ the possible VPIDs:
6935
6936 [role="img-100"]
6937 .The VPID inclusion set is full.
6938 image::track-all.png[]
6939
6940 When the inclusion set is full and you run the man:lttng-track(1)
6941 command to specify some VPIDs, LTTng:
6942
6943 . Clears the inclusion set.
6944 . Adds the specific VPIDs to the inclusion set.
6945
6946 After:
6947
6948 [role="term"]
6949 ----
6950 $ lttng track --userspace --vpid=3,4,7,10,13
6951 ----
6952
6953 the VPID inclusion set is:
6954
6955 [role="img-100"]
6956 .The VPID inclusion set contains the VPIDs 3, 4, 7, 10, and 13.
6957 image::track-3-4-7-10-13.png[]
6958
6959 Add more VPIDs to the inclusion set afterwards:
6960
6961 [role="term"]
6962 ----
6963 $ lttng track --userspace --vpid=1,15,16
6964 ----
6965
6966 The result is:
6967
6968 [role="img-100"]
6969 .VPIDs 1, 15, and 16 are added to the inclusion set.
6970 image::track-1-3-4-7-10-13-15-16.png[]
6971
6972 The man:lttng-untrack(1) command removes entries from process attribute
6973 inclusion sets. Given the previous example, the following command:
6974
6975 [role="term"]
6976 ----
6977 $ lttng untrack --userspace --vpid=3,7,10,13
6978 ----
6979
6980 leads to this VPID inclusion set:
6981
6982 [role="img-100"]
6983 .VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6984 image::track-1-4-15-16.png[]
6985
6986 You can make the VPID inclusion set full again with the
6987 opt:lttng-track(1):--all option:
6988
6989 [role="term"]
6990 ----
6991 $ lttng track --userspace --vpid --all
6992 ----
6993
6994 The result is, again:
6995
6996 [role="img-100"]
6997 .The VPID inclusion set is full.
6998 image::track-all.png[]
6999 ====
7000
7001 .Allow specific processes to record events based on their user ID (UID).
7002 ====
7003 A typical use case with process attribute inclusion sets is to start
7004 with an empty inclusion set, then <<basic-tracing-session-control,start
7005 the tracers>>, and finally add values manually while the tracers are
7006 active.
7007
7008 Use the opt:lttng-untrack(1):--all option of the
7009 man:lttng-untrack(1) command to clear the inclusion set after you
7010 <<creating-destroying-tracing-sessions,create a recording session>>, for
7011 example (with UIDs):
7012
7013 [role="term"]
7014 ----
7015 # lttng untrack --kernel --uid --all
7016 ----
7017
7018 gives:
7019
7020 [role="img-100"]
7021 .The UID inclusion set is empty.
7022 image::untrack-all.png[]
7023
7024 If the LTTng tracer runs with this inclusion set configuration, it
7025 records no events within the <<cur-tracing-session,current recording
7026 session>> because no processes is allowed to do so. Use the
7027 man:lttng-track(1) command as usual to add specific values to the UID
7028 inclusion set when you need to, for example:
7029
7030 [role="term"]
7031 ----
7032 # lttng track --kernel --uid=http,11
7033 ----
7034
7035 Result:
7036
7037 [role="img-100"]
7038 .UIDs 6 (`http`) and 11 are part of the UID inclusion set.
7039 image::track-6-11.png[]
7040 ====
7041
7042
7043 [role="since-2.5"]
7044 [[saving-loading-tracing-session]]
7045 === Save and load recording session configurations
7046
7047 Configuring a <<tracing-session,recording session>> can be long. Some of
7048 the tasks involved are:
7049
7050 * <<enabling-disabling-channels,Create channels>> with
7051   specific attributes.
7052
7053 * <<adding-context,Add context fields>> to be recorded to the
7054   <<event,event records>> of specific channels.
7055
7056 * <<enabling-disabling-events,Create recording event rules>> with
7057   specific log level, filter, and other conditions.
7058
7059 If you use LTTng to solve real world problems, chances are you have to
7060 record events using the same recording session setup over and over,
7061 modifying a few variables each time in your instrumented program or
7062 environment.
7063
7064 To avoid constant recording session reconfiguration, the man:lttng(1)
7065 command-line tool can save and load recording session configurations
7066 to/from XML files.
7067
7068 To save a given recording session configuration:
7069
7070 * Use the man:lttng-save(1) command:
7071 +
7072 --
7073 [role="term"]
7074 ----
7075 $ lttng save SESSION
7076 ----
7077 --
7078 +
7079 Replace +__SESSION__+ with the name of the recording session to save.
7080
7081 LTTng saves recording session configurations to
7082 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
7083 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. See
7084 man:lttng-save(1) to learn more about the recording session configuration
7085 output path.
7086
7087 LTTng saves all configuration parameters, for example:
7088
7089 * The recording session name.
7090 * The trace data output path.
7091 * The <<channel,channels>>, with their state and all their attributes.
7092 * The context fields you added to channels.
7093 * The <<event,recording event rules>> with their state and conditions.
7094
7095 To load a recording session:
7096
7097 * Use the man:lttng-load(1) command:
7098 +
7099 --
7100 [role="term"]
7101 ----
7102 $ lttng load SESSION
7103 ----
7104 --
7105 +
7106 Replace +__SESSION__+ with the name of the recording session to load.
7107
7108 When LTTng loads a configuration, it restores your saved recording session
7109 as if you just configured it manually.
7110
7111 You can also save and load many sessions at a time; see
7112 man:lttng-save(1) and man:lttng-load(1) to learn more.
7113
7114
7115 [[sending-trace-data-over-the-network]]
7116 === Send trace data over the network
7117
7118 LTTng can send the recorded trace data of a <<tracing-session,recording
7119 session>> to a remote system over the network instead of writing it to
7120 the local file system.
7121
7122 To send the trace data over the network:
7123
7124 . On the _remote_ system (which can also be the target system),
7125   start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
7126 +
7127 --
7128 [role="term"]
7129 ----
7130 $ lttng-relayd
7131 ----
7132 --
7133
7134 . On the _target_ system, create a recording session
7135   <<net-streaming-mode,configured>> to send trace data over the network:
7136 +
7137 --
7138 [role="term"]
7139 ----
7140 $ lttng create my-session --set-url=net://remote-system
7141 ----
7142 --
7143 +
7144 Replace +__remote-system__+ with the host name or IP address of the
7145 remote system. See man:lttng-create(1) for the exact URL format.
7146
7147 . On the target system, use the man:lttng(1) command-line tool as usual.
7148 +
7149 When recording is <<basic-tracing-session-control,active>>, the
7150 <<lttng-consumerd,consumer daemon>> of the target sends the contents of
7151 <<channel,sub-buffers>> to the remote relay daemon instead of flushing
7152 them to the local file system. The relay daemon writes the received
7153 packets to its local file system.
7154
7155 See the ``Output directory'' section of man:lttng-relayd(8) to learn
7156 where a relay daemon writes its received trace data.
7157
7158
7159 [role="since-2.4"]
7160 [[lttng-live]]
7161 === View events as LTTng records them (noch:{LTTng} live)
7162
7163 _LTTng live_ is a network protocol implemented by the
7164 <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)) to allow compatible
7165 trace readers to display or analyze <<event,event records>> as LTTng
7166 records events on the target system while recording is
7167 <<basic-tracing-session-control,active>>.
7168
7169 The relay daemon creates a _tee_: it forwards the trace data to both the
7170 local file system and to connected live readers:
7171
7172 [role="img-90"]
7173 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live reader.
7174 image::live.png[]
7175
7176 To use LTTng live:
7177
7178 . On the _target system_, create a <<tracing-session,recording session>>
7179   in _live mode_:
7180 +
7181 --
7182 [role="term"]
7183 ----
7184 $ lttng create my-session --live
7185 ----
7186 --
7187 +
7188 This operation spawns a local relay daemon.
7189
7190 . Start the live reader and configure it to connect to the relay daemon.
7191 +
7192 For example, with man:babeltrace2(1):
7193 +
7194 --
7195 [role="term"]
7196 ----
7197 $ babeltrace2 net://localhost/host/HOSTNAME/my-session
7198 ----
7199 --
7200 +
7201 Replace +__HOSTNAME__+ with the host name of the target system.
7202
7203 . Configure the recording session as usual with the man:lttng(1)
7204   command-line tool, and <<basic-tracing-session-control,start recording>>.
7205
7206 List the available live recording sessions with man:babeltrace2(1):
7207
7208 [role="term"]
7209 ----
7210 $ babeltrace2 net://localhost
7211 ----
7212
7213 You can start the relay daemon on another system. In this case, you need
7214 to specify the URL of the relay daemon when you
7215 <<creating-destroying-tracing-sessions,create the recording session>> with
7216 the opt:lttng-create(1):--set-url option of the man:lttng-create(1)
7217 command. You also need to replace +__localhost__+ in the procedure above
7218 with the host name of the system on which the relay daemon runs.
7219
7220
7221 [role="since-2.3"]
7222 [[taking-a-snapshot]]
7223 === Take a snapshot of the current sub-buffers of a recording session
7224
7225 The normal behavior of LTTng is to append full sub-buffers to growing
7226 trace data files. This is ideal to keep a full history of the events
7227 which the target system emitted, but it can represent too much data in
7228 some situations.
7229
7230 For example, you may wish to have LTTng record your application
7231 continuously until some critical situation happens, in which case you
7232 only need the latest few recorded events to perform the desired
7233 analysis, not multi-gigabyte trace files.
7234
7235 With the man:lttng-snapshot(1) command, you can take a _snapshot_ of the
7236 current <<channel,sub-buffers>> of a given <<tracing-session,recording
7237 session>>. LTTng can write the snapshot to the local file system or send
7238 it over the network.
7239
7240 [role="img-100"]
7241 .A snapshot is a copy of the current sub-buffers, which LTTng does _not_ clear after the operation.
7242 image::snapshot.png[]
7243
7244 The snapshot feature of LTTng is similar to how a
7245 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder] or the
7246 ``roll'' mode of an oscilloscope work.
7247
7248 TIP: If you wish to create unmanaged, self-contained, non-overlapping
7249 trace chunk archives instead of a simple copy of the current
7250 sub-buffers, see the <<session-rotation,recording session rotation>>
7251 feature (available since LTTng{nbsp}2.11).
7252
7253 To take a snapshot of the <<cur-tracing-session,current recording
7254 session>>:
7255
7256 . Create a recording session in <<snapshot-mode,snapshot mode>>:
7257 +
7258 --
7259 [role="term"]
7260 ----
7261 $ lttng create my-session --snapshot
7262 ----
7263 --
7264 +
7265 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
7266 <<channel,channels>> created in this mode is automatically set to
7267 <<overwrite-mode,_overwrite_>>.
7268
7269 . Configure the recording session as usual with the man:lttng(1)
7270   command-line tool, and <<basic-tracing-session-control,start
7271   recording>>.
7272
7273 . **Optional**: When you need to take a snapshot,
7274   <<basic-tracing-session-control,stop recording>>.
7275 +
7276 You can take a snapshot when the tracers are active, but if you stop
7277 them first, you're guaranteed that the trace data in the sub-buffers
7278 doesn't change before you actually take the snapshot.
7279
7280 . Take a snapshot:
7281 +
7282 --
7283 [role="term"]
7284 ----
7285 $ lttng snapshot record --name=my-first-snapshot
7286 ----
7287 --
7288 +
7289 LTTng writes the current sub-buffers of all the channels of the
7290 <<cur-tracing-session,current recording session>> to
7291 trace files on the local file system. Those trace files have
7292 `my-first-snapshot` in their name.
7293
7294 There's no difference between the format of a normal trace file and the
7295 format of a snapshot: LTTng trace readers also support LTTng snapshots.
7296
7297 By default, LTTng writes snapshot files to the path shown by
7298
7299 [role="term"]
7300 ----
7301 $ lttng snapshot list-output
7302 ----
7303
7304 You can change this path or decide to send snapshots over the network
7305 using either:
7306
7307 . An output path or URL that you specify when you
7308   <<creating-destroying-tracing-sessions,create the recording session>>.
7309
7310 . A snapshot output path or URL that you add using the
7311   `add-output` action of the man:lttng-snapshot(1) command.
7312
7313 . An output path or URL that you provide directly to the
7314   `record` action of the man:lttng-snapshot(1) command.
7315
7316 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
7317 you specify a URL, a <<lttng-relayd,relay daemon>> must listen on a
7318 remote system (see ``<<sending-trace-data-over-the-network,Send trace
7319 data over the network>>'').
7320
7321 The `snapshot-session` <<trigger,trigger>> action can also take
7322 a recording session snapshot.
7323
7324
7325 [role="since-2.11"]
7326 [[session-rotation]]
7327 === Archive the current trace chunk (rotate a recording session)
7328
7329 The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
7330 current sub-buffers of a recording session to the file system or send them
7331 over the network. When you take a snapshot, LTTng doesn't clear the ring
7332 buffers of the recording session: if you take another snapshot immediately
7333 after, both snapshots could contain overlapping trace data.
7334
7335 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
7336 _recording session rotation_ is a feature which appends the content of the
7337 ring buffers to what's already on the file system or sent over the
7338 network since the creation of the recording session or since the last
7339 rotation, and then clears those ring buffers to avoid trace data
7340 overlaps.
7341
7342 What LTTng is about to write when performing a recording session rotation
7343 is called the _current trace chunk_. When LTTng writes or sends over the
7344 network this current trace chunk, it becomes a _trace chunk archive_.
7345 Therefore, a recording session rotation operation _archives_ the current
7346 trace chunk.
7347
7348 [role="img-100"]
7349 .A recording session rotation operation _archives_ the current trace chunk.
7350 image::rotation.png[]
7351
7352 A trace chunk archive is a self-contained LTTng trace which LTTng
7353 doesn't manage anymore: you can read it, modify it, move it, or remove
7354 it.
7355
7356 As of LTTng{nbsp}{revision}, there are three methods to perform a
7357 recording session rotation:
7358
7359 * <<immediate-rotation,Immediately>>.
7360
7361 * With a <<rotation-schedule,rotation schedule>>.
7362
7363 * Through the execution of a `rotate-session` <<trigger,trigger>>
7364   action.
7365
7366 [[immediate-rotation]]To perform an immediate rotation of the
7367 <<cur-tracing-session,current recording session>>:
7368
7369 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7370   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7371   mode>> (only those two recording session modes support recording session
7372   rotation):
7373 +
7374 --
7375 [role="term"]
7376 ----
7377 # lttng create my-session
7378 ----
7379 --
7380
7381 . <<enabling-disabling-events,Create one or more recording event rules>>
7382   and <<basic-tracing-session-control,start recording>>:
7383 +
7384 --
7385 [role="term"]
7386 ----
7387 # lttng enable-event --kernel sched_'*'
7388 # lttng start
7389 ----
7390 --
7391
7392 . When needed, immediately rotate the current recording session:
7393 +
7394 --
7395 [role="term"]
7396 ----
7397 # lttng rotate
7398 ----
7399 --
7400 +
7401 The man:lttng-rotate(1) command prints the path to the created trace
7402 chunk archive. See its manual page to learn about the format of trace
7403 chunk archive directory names.
7404 +
7405 Perform other immediate rotations while the recording session is active.
7406 It's guaranteed that all the trace chunk archives don't contain
7407 overlapping trace data. You can also perform an immediate rotation once
7408 you have <<basic-tracing-session-control,stopped>> the recording session.
7409
7410 . When you're done recording,
7411   <<creating-destroying-tracing-sessions,destroy the current recording
7412   session>>:
7413 +
7414 --
7415 [role="term"]
7416 ----
7417 # lttng destroy
7418 ----
7419 --
7420 +
7421 The recording session destruction operation creates one last trace chunk
7422 archive from the current trace chunk.
7423
7424 [[rotation-schedule]]A recording session rotation schedule is a planned
7425 rotation which LTTng performs automatically based on one of the
7426 following conditions:
7427
7428 * A timer with a configured period expires.
7429
7430 * The total size of the _flushed_ part of the current trace chunk
7431   becomes greater than or equal to a configured value.
7432
7433 To schedule a rotation of the <<cur-tracing-session,current recording
7434 session>>, set a _rotation schedule_:
7435
7436 . <<creating-destroying-tracing-sessions,Create a recording session>> in
7437   <<local-mode,local mode>> or <<net-streaming-mode,network streaming
7438   mode>> (only those two creation modes support recording session
7439   rotation):
7440 +
7441 --
7442 [role="term"]
7443 ----
7444 # lttng create my-session
7445 ----
7446 --
7447
7448 . <<enabling-disabling-events,Create one or more recording event rules>>:
7449 +
7450 --
7451 [role="term"]
7452 ----
7453 # lttng enable-event --kernel sched_'*'
7454 ----
7455 --
7456
7457 . Set a recording session rotation schedule:
7458 +
7459 --
7460 [role="term"]
7461 ----
7462 # lttng enable-rotation --timer=10s
7463 ----
7464 --
7465 +
7466 In this example, we set a rotation schedule so that LTTng performs a
7467 recording session rotation every ten seconds.
7468 +
7469 See man:lttng-enable-rotation(1) to learn more about other ways to set a
7470 rotation schedule.
7471
7472 . <<basic-tracing-session-control,Start recording>>:
7473 +
7474 --
7475 [role="term"]
7476 ----
7477 # lttng start
7478 ----
7479 --
7480 +
7481 LTTng performs recording session rotations automatically while the
7482 recording session is active thanks to the rotation schedule.
7483
7484 . When you're done recording,
7485   <<creating-destroying-tracing-sessions,destroy the current recording
7486   session>>:
7487 +
7488 --
7489 [role="term"]
7490 ----
7491 # lttng destroy
7492 ----
7493 --
7494 +
7495 The recording session destruction operation creates one last trace chunk
7496 archive from the current trace chunk.
7497
7498 Unset a recording session rotation schedule with the
7499 man:lttng-disable-rotation(1) command.
7500
7501
7502 [role="since-2.13"]
7503 [[add-event-rule-matches-trigger]]
7504 === Add an ``event rule matches'' trigger to a session daemon
7505
7506 With the man:lttng-add-trigger(1) command, you can add a
7507 <<trigger,trigger>> to a <<lttng-sessiond,session daemon>>.
7508
7509 A trigger associates an LTTng tracing condition to one or more actions:
7510 when the condition is satisfied, LTTng attempts to execute the actions.
7511
7512 A trigger doesn't need any <<tracing-session,recording session>> to exist:
7513 it belongs to a session daemon.
7514
7515 As of LTTng{nbsp}{revision}, many condition types are available through
7516 the <<liblttng-ctl-lttng,`liblttng-ctl`>> C{nbsp}API, but the
7517 man:lttng-add-trigger(1) command only accepts the ``event rule matches''
7518 condition.
7519
7520 An ``event rule matches'' condition is satisfied when its event rule
7521 matches an event.
7522
7523 Unlike a <<event,recording event rule>>, the event rule of an
7524 ``event rule matches'' trigger condition has no implicit conditions,
7525 that is:
7526
7527 * It has no enabled/disabled state.
7528 * It has no attached <<channel,channel>>.
7529 * It doesn't belong to a <<tracing-session,recording session>>.
7530
7531 Both the man:lttng-add-trigger(1) and man:lttng-enable-event(1) commands
7532 accept command-line arguments to specify an <<event-rule,event rule>>.
7533 That being said, the former is a more recent command and therefore
7534 follows the common event rule specification format (see
7535 man:lttng-event-rule(7)).
7536
7537 .Start a <<tracing-session,recording session>> when an event rule matches.
7538 ====
7539 This example shows how to add the following trigger to the root
7540 <<lttng-sessiond,session daemon>>:
7541
7542 Condition::
7543     An event rule matches a Linux kernel system call event of which the
7544     name starts with `exec` and `*/ls` matches the `filename` payload
7545     field.
7546 +
7547 With such an event rule, LTTng emits an event when the cmd:ls program
7548 starts.
7549
7550 Action::
7551     <<basic-tracing-session-control,Start the recording session>>
7552     named `pitou`.
7553
7554 To add such a trigger to the root session daemon:
7555
7556 . **If there's no currently running LTTng root session daemon**, start
7557   one:
7558 +
7559 [role="term"]
7560 ----
7561 # lttng-sessiond --daemonize
7562 ----
7563
7564 . <<creating-destroying-tracing-sessions,Create a recording session>>
7565   named `pitou` and
7566   <<enabling-disabling-events,create a recording event rule>> matching
7567   all the system call events:
7568 +
7569 [role="term"]
7570 ----
7571 # lttng create pitou
7572 # lttng enable-event --kernel --syscall --all
7573 ----
7574
7575 . Add the trigger to the root session daemon:
7576 +
7577 [role="term"]
7578 ----
7579 # lttng add-trigger --condition=event-rule-matches \
7580                     --type=syscall --name='exec*' \
7581                     --filter='filename == "*/ls"' \
7582                     --action=start-session pitou
7583 ----
7584 +
7585 Confirm that the trigger exists with the man:lttng-list-triggers(1)
7586 command:
7587 +
7588 [role="term"]
7589 ----
7590 # lttng list-triggers
7591 ----
7592
7593 . Make sure the `pitou` recording session is still inactive (stopped):
7594 +
7595 [role="term"]
7596 ----
7597 # lttng list pitou
7598 ----
7599 +
7600 The first line should be something like:
7601 +
7602 ----
7603 Recording session pitou: [inactive]
7604 ----
7605
7606 Run the cmd:ls program to fire the LTTng trigger above:
7607
7608 [role="term"]
7609 ----
7610 $ ls ~
7611 ----
7612
7613 At this point, the `pitou` recording session should be active
7614 (started). Confirm this with the man:lttng-list(1) command again:
7615
7616 [role="term"]
7617 ----
7618 # lttng list pitou
7619 ----
7620
7621 The first line should now look like:
7622
7623 ----
7624 Recording session pitou: [active]
7625 ----
7626
7627 This line confirms that the LTTng trigger you added fired, therefore
7628 starting the `pitou` recording session.
7629 ====
7630
7631 .[[trigger-event-notif]]Send a notification to a user application when an event rule matches.
7632 ====
7633 This example shows how to add the following trigger to the root
7634 <<lttng-sessiond,session daemon>>:
7635
7636 Condition::
7637     An event rule matches a Linux kernel tracepoint event named
7638     `sched_switch` and of which the value of the `next_comm` payload
7639     field is `bash`.
7640 +
7641 With such an event rule, LTTng emits an event when Linux gives access to
7642 the processor to a process named `bash`.
7643
7644 Action::
7645     Send an LTTng notification to a user application.
7646
7647 Moreover, we'll specify a _capture descriptor_ with the
7648 `event-rule-matches` trigger condition so that the user application can
7649 get the value of a specific `sched_switch` event payload field.
7650
7651 First, write and build the user application:
7652
7653 . Create the C{nbsp}source file of the application:
7654 +
7655 --
7656 [source,c]
7657 .path:{notif-app.c}
7658 ----
7659 #include <stdlib.h>
7660 #include <stdio.h>
7661 #include <stdbool.h>
7662 #include <assert.h>
7663 #include <string.h>
7664 #include <lttng/lttng.h>
7665
7666 /*
7667  * Subscribes to notifications, through the notification channel
7668  * `notification_channel`, which match the condition of the trigger
7669  * named `trigger_name`.
7670  *
7671  * Returns `true` on success.
7672  */
7673 static bool subscribe(struct lttng_notification_channel *notification_channel,
7674                       const char *trigger_name)
7675 {
7676     const struct lttng_condition *condition = NULL;
7677     struct lttng_triggers *triggers = NULL;
7678     unsigned int trigger_count;
7679     unsigned int i;
7680     enum lttng_error_code error_code;
7681     enum lttng_trigger_status trigger_status;
7682     bool ret = false;
7683
7684     /* Get all LTTng triggers */
7685     error_code = lttng_list_triggers(&triggers);
7686     assert(error_code == LTTNG_OK);
7687
7688     /* Get the number of triggers */
7689     trigger_status = lttng_triggers_get_count(triggers, &trigger_count);
7690     assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7691
7692     /* Find the trigger named `trigger_name` */
7693     for (i = 0; i < trigger_count; i++) {
7694         const struct lttng_trigger *trigger;
7695         const char *this_trigger_name;
7696
7697         trigger = lttng_triggers_get_at_index(triggers, i);
7698         trigger_status = lttng_trigger_get_name(trigger, &this_trigger_name);
7699         assert(trigger_status == LTTNG_TRIGGER_STATUS_OK);
7700
7701         if (strcmp(this_trigger_name, trigger_name) == 0) {
7702             /* Trigger found: subscribe with its condition */
7703             enum lttng_notification_channel_status notification_channel_status;
7704
7705             notification_channel_status = lttng_notification_channel_subscribe(
7706                 notification_channel,
7707                 lttng_trigger_get_const_condition(trigger));
7708             assert(notification_channel_status ==
7709                    LTTNG_NOTIFICATION_CHANNEL_STATUS_OK);
7710             ret = true;
7711             break;
7712         }
7713     }
7714
7715     lttng_triggers_destroy(triggers);
7716     return ret;
7717 }
7718
7719 /*
7720  * Handles the evaluation `evaluation` of a single notification.
7721  */
7722 static void handle_evaluation(const struct lttng_evaluation *evaluation)
7723 {
7724     enum lttng_evaluation_status evaluation_status;
7725     const struct lttng_event_field_value *array_field_value;
7726     const struct lttng_event_field_value *string_field_value;
7727     enum lttng_event_field_value_status event_field_value_status;
7728     const char *string_field_string_value;
7729
7730     /* Get the value of the first captured (string) field */
7731     evaluation_status = lttng_evaluation_event_rule_matches_get_captured_values(
7732         evaluation, &array_field_value);
7733     assert(evaluation_status == LTTNG_EVALUATION_STATUS_OK);
7734     event_field_value_status =
7735         lttng_event_field_value_array_get_element_at_index(
7736             array_field_value, 0, &string_field_value);
7737     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7738     assert(lttng_event_field_value_get_type(string_field_value) ==
7739            LTTNG_EVENT_FIELD_VALUE_TYPE_STRING);
7740     event_field_value_status = lttng_event_field_value_string_get_value(
7741             string_field_value, &string_field_string_value);
7742     assert(event_field_value_status == LTTNG_EVENT_FIELD_VALUE_STATUS_OK);
7743
7744     /* Print the string value of the field */
7745     puts(string_field_string_value);
7746 }
7747
7748 int main(int argc, char *argv[])
7749 {
7750     int exit_status = EXIT_SUCCESS;
7751     struct lttng_notification_channel *notification_channel;
7752     enum lttng_notification_channel_status notification_channel_status;
7753     const struct lttng_condition *condition;
7754     const char *trigger_name;
7755     bool subscribe_res;
7756
7757     assert(argc >= 2);
7758     trigger_name = argv[1];
7759
7760     /*
7761      * Create a notification channel.
7762      *
7763      * A notification channel connects the user application to the LTTng
7764      * session daemon.
7765      *
7766      * You can use this notification channel to listen to various types
7767      * of notifications.
7768      */
7769     notification_channel = lttng_notification_channel_create(
7770         lttng_session_daemon_notification_endpoint);
7771     assert(notification_channel);
7772
7773     /*
7774      * Subscribe to notifications which match the condition of the
7775      * trigger named `trigger_name`.
7776      */
7777     if (!subscribe(notification_channel, trigger_name)) {
7778         fprintf(stderr,
7779                 "Error: Failed to subscribe to notifications (trigger `%s`).\n",
7780                 trigger_name);
7781         exit_status = EXIT_FAILURE;
7782         goto end;
7783     }
7784
7785     /*
7786      * Notification loop.
7787      *
7788      * Put this in a dedicated thread to avoid blocking the main thread.
7789      */
7790     while (true) {
7791         struct lttng_notification *notification;
7792         enum lttng_notification_channel_status status;
7793         const struct lttng_evaluation *notification_evaluation;
7794
7795         /* Receive the next notification */
7796         status = lttng_notification_channel_get_next_notification(
7797             notification_channel, &notification);
7798
7799         switch (status) {
7800         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7801             break;
7802         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7803             /*
7804              * The session daemon can drop notifications if a receiving
7805              * application doesn't consume the notifications fast
7806              * enough.
7807              */
7808             continue;
7809         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7810             /*
7811              * The session daemon closed the notification channel.
7812              *
7813              * This is typically caused by a session daemon shutting
7814              * down.
7815              */
7816             goto end;
7817         default:
7818             /* Unhandled conditions or errors */
7819             exit_status = EXIT_FAILURE;
7820             goto end;
7821         }
7822
7823         /*
7824          * Handle the condition evaluation.
7825          *
7826          * A notification provides, amongst other things:
7827          *
7828          * * The condition that caused LTTng to send this notification.
7829          *
7830          * * The condition evaluation, which provides more specific
7831          *   information on the evaluation of the condition.
7832          */
7833         handle_evaluation(lttng_notification_get_evaluation(notification));
7834
7835         /* Destroy the notification object */
7836         lttng_notification_destroy(notification);
7837     }
7838
7839 end:
7840     lttng_notification_channel_destroy(notification_channel);
7841     return exit_status;
7842 }
7843 ----
7844 --
7845 +
7846 This application prints the first captured string field value of the
7847 condition evaluation of each LTTng notification it receives.
7848
7849 . Build the `notif-app` application,
7850   using https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
7851   to provide the right compiler and linker flags:
7852 +
7853 --
7854 [role="term"]
7855 ----
7856 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
7857 ----
7858 --
7859
7860 Now, to add the trigger to the root session daemon:
7861
7862 [start=3]
7863 . **If there's no currently running LTTng root session daemon**, start
7864   one:
7865 +
7866 [role="term"]
7867 ----
7868 # lttng-sessiond --daemonize
7869 ----
7870
7871 . Add the trigger, naming it `sched-switch-notif`, to the root
7872   session daemon:
7873 +
7874 [role="term"]
7875 ----
7876 # lttng add-trigger --name=sched-switch-notif \
7877                     --condition=event-rule-matches \
7878                     --type=kernel --name=sched_switch \
7879                     --filter='next_comm == "bash"' --capture=prev_comm \
7880                     --action=notify
7881 ----
7882 +
7883 Confirm that the `sched-switch-notif` trigger exists with the
7884 man:lttng-list-triggers(1) command:
7885 +
7886 [role="term"]
7887 ----
7888 # lttng list-triggers
7889 ----
7890
7891 Run the cmd:notif-app application, passing the name of the trigger
7892 of which to watch the notifications:
7893
7894 [role="term"]
7895 ----
7896 # ./notif-app sched-switch-notif
7897 ----
7898
7899 Now, in an interactive Bash, type a few keys to fire the
7900 `sched-switch-notif` trigger. Watch the `notif-app` application print
7901 the previous process names.
7902 ====
7903
7904 [role="since-2.6"]
7905 [[mi]]
7906 === Use the machine interface
7907
7908 With any command of the man:lttng(1) command-line tool, set the
7909 opt:lttng(1):--mi option to `xml` (before the command name) to get an
7910 XML machine interface output, for example:
7911
7912 [role="term"]
7913 ----
7914 $ lttng --mi=xml list my-session
7915 ----
7916
7917 A schema definition (XSD) is
7918 https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/mi-lttng-4.0.xsd[available]
7919 to ease the integration with external tools as much as possible.
7920
7921
7922 [role="since-2.8"]
7923 [[metadata-regenerate]]
7924 === Regenerate the metadata of an LTTng trace
7925
7926 An LTTng trace, which is a https://diamon.org/ctf[CTF] trace, has both
7927 data stream files and a metadata stream file. This metadata file
7928 contains, amongst other things, information about the offset of the
7929 clock sources which LTTng uses to assign timestamps to <<event,event
7930 records>> when recording.
7931
7932 If, once a <<tracing-session,recording session>> is
7933 <<basic-tracing-session-control,started>>, a major
7934 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
7935 happens, the clock offset of the trace also needs to be updated. Use
7936 the `metadata` item of the man:lttng-regenerate(1) command to do so.
7937
7938 The main use case of this command is to allow a system to boot with
7939 an incorrect wall time and have LTTng trace it before its wall time
7940 is corrected. Once the system is known to be in a state where its
7941 wall time is correct, you can run `lttng regenerate metadata`.
7942
7943 To regenerate the metadata stream files of the
7944 <<cur-tracing-session,current recording session>>:
7945
7946 * Use the `metadata` item of the man:lttng-regenerate(1) command:
7947 +
7948 --
7949 [role="term"]
7950 ----
7951 $ lttng regenerate metadata
7952 ----
7953 --
7954
7955
7956 [role="since-2.9"]
7957 [[regenerate-statedump]]
7958 === Regenerate the state dump event records of a recording session
7959
7960 The LTTng kernel and user space tracers generate state dump
7961 <<event,event records>> when the application starts or when you
7962 <<basic-tracing-session-control,start a recording session>>.
7963
7964 An analysis can use the state dump event records to set an initial state
7965 before it builds the rest of the state from the subsequent event
7966 records. http://tracecompass.org/[Trace Compass] is a notable
7967 example of an application which uses the state dump of an LTTng trace.
7968
7969 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7970 state dump event records aren't included in the snapshot trace files
7971 because they were recorded to a <<channel,sub-buffer>> that has been
7972 consumed or <<overwrite-mode,overwritten>> already.
7973
7974 Use the `statedump` item of the man:lttng-regenerate(1) command to emit
7975 and record the state dump events again.
7976
7977 To regenerate the state dump of the <<cur-tracing-session,current
7978 recording session>>, provided you created it in <<snapshot-mode,snapshot
7979 mode>>, before you take a snapshot:
7980
7981 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7982 +
7983 --
7984 [role="term"]
7985 ----
7986 $ lttng regenerate statedump
7987 ----
7988 --
7989
7990 . <<basic-tracing-session-control,Stop the recording session>>:
7991 +
7992 --
7993 [role="term"]
7994 ----
7995 $ lttng stop
7996 ----
7997 --
7998
7999 . <<taking-a-snapshot,Take a snapshot>>:
8000 +
8001 --
8002 [role="term"]
8003 ----
8004 $ lttng snapshot record --name=my-snapshot
8005 ----
8006 --
8007
8008 Depending on the event throughput, you should run steps{nbsp}1
8009 and{nbsp}2 as closely as possible.
8010
8011 [NOTE]
8012 ====
8013 To record the state dump events, you need to
8014 <<enabling-disabling-events,create recording event rules>> which enable
8015 them:
8016
8017 * The names of LTTng-UST state dump tracepoints start with
8018   `lttng_ust_statedump:`.
8019
8020 * The names of LTTng-modules state dump tracepoints start with
8021   `lttng_statedump_`.
8022 ====
8023
8024
8025 [role="since-2.7"]
8026 [[persistent-memory-file-systems]]
8027 === Record trace data on persistent memory file systems
8028
8029 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile
8030 random-access memory] (NVRAM) is random-access memory that retains its
8031 information when power is turned off (non-volatile). Systems with such
8032 memory can store data structures in RAM and retrieve them after a
8033 reboot, without flushing to typical _storage_.
8034
8035 Linux supports NVRAM file systems thanks to either
8036 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
8037 (requires Linux{nbsp}4.1+) or http://pramfs.sourceforge.net/[PRAMFS] (requires Linux{nbsp}<{nbsp}4).
8038
8039 This section doesn't describe how to operate such file systems; we
8040 assume that you have a working persistent memory file system.
8041
8042 When you <<creating-destroying-tracing-sessions,create a recording
8043 session>>, you can specify the path of the shared memory holding the
8044 sub-buffers. If you specify a location on an NVRAM file system, then you
8045 can retrieve the latest recorded trace data when the system reboots
8046 after a crash.
8047
8048 To record trace data on a persistent memory file system and retrieve the
8049 trace data after a system crash:
8050
8051 . Create a recording session with a <<channel,sub-buffer>> shared memory
8052   path located on an NVRAM file system:
8053 +
8054 --
8055 [role="term"]
8056 ----
8057 $ lttng create my-session --shm-path=/path/to/shm/on/nvram
8058 ----
8059 --
8060
8061 . Configure the recording session as usual with the man:lttng(1)
8062   command-line tool, and <<basic-tracing-session-control,start
8063   recording>>.
8064
8065 . After a system crash, use the man:lttng-crash(1) command-line tool to
8066   read the trace data recorded on the NVRAM file system:
8067 +
8068 --
8069 [role="term"]
8070 ----
8071 $ lttng-crash /path/to/shm/on/nvram
8072 ----
8073 --
8074
8075 The binary layout of the ring buffer files isn't exactly the same as the
8076 trace files layout. This is why you need to use man:lttng-crash(1)
8077 instead of some standard LTTng trace reader.
8078
8079 To convert the ring buffer files to LTTng trace files:
8080
8081 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
8082 +
8083 --
8084 [role="term"]
8085 ----
8086 $ lttng-crash --extract=/path/to/trace /path/to/shm/on/nvram
8087 ----
8088 --
8089
8090
8091 [role="since-2.10"]
8092 [[notif-trigger-api]]
8093 === Get notified when the buffer usage of a channel is too high or too low
8094
8095 With the notification and <<trigger,trigger>> C{nbsp}API of
8096 <<liblttng-ctl-lttng,`liblttng-ctl`>>, LTTng can notify your user
8097 application when the buffer usage of one or more <<channel,channels>>
8098 becomes too low or too high.
8099
8100 Use this API and enable or disable <<event,recording event rules>> while
8101 a recording session <<basic-tracing-session-control,is active>> to avoid
8102 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>, for
8103 example.
8104
8105 .Send a notification to a user application when the buffer usage of an LTTng channel is too high.
8106 ====
8107 In this example, we create and build an application which gets notified
8108 when the buffer usage of a specific LTTng channel is higher than
8109 75{nbsp}%.
8110
8111 We only print that it's the case in this example, but we could as well
8112 use the `liblttng-ctl` C{nbsp}API to <<enabling-disabling-events,disable
8113 recording event rules>> when this happens, for example.
8114
8115 . Create the C{nbsp}source file of the application:
8116 +
8117 --
8118 [source,c]
8119 .path:{notif-app.c}
8120 ----
8121 #include <stdlib.h>
8122 #include <stdio.h>
8123 #include <assert.h>
8124 #include <lttng/lttng.h>
8125
8126 int main(int argc, char *argv[])
8127 {
8128     int exit_status = EXIT_SUCCESS;
8129     struct lttng_notification_channel *notification_channel;
8130     struct lttng_condition *condition;
8131     struct lttng_action *action;
8132     struct lttng_trigger *trigger;
8133     const char *recording_session_name;
8134     const char *channel_name;
8135
8136     assert(argc >= 3);
8137     recording_session_name = argv[1];
8138     channel_name = argv[2];
8139
8140     /*
8141      * Create a notification channel.
8142      *
8143      * A notification channel connects the user application to the LTTng
8144      * session daemon.
8145      *
8146      * You can use this notification channel to listen to various types
8147      * of notifications.
8148      */
8149     notification_channel = lttng_notification_channel_create(
8150         lttng_session_daemon_notification_endpoint);
8151
8152     /*
8153      * Create a "buffer usage becomes greater than" condition.
8154      *
8155      * In this case, the condition is satisfied when the buffer usage
8156      * becomes greater than or equal to 75 %.
8157      *
8158      * We create the condition for a specific recording session name,
8159      * channel name, and for the user space tracing domain.
8160      *
8161      * The following condition types also exist:
8162      *
8163      * * The buffer usage of a channel becomes less than a given value.
8164      *
8165      * * The consumed data size of a recording session becomes greater
8166      *   than a given value.
8167      *
8168      * * A recording session rotation becomes ongoing.
8169      *
8170      * * A recording session rotation becomes completed.
8171      *
8172      * * A given event rule matches an event.
8173      */
8174     condition = lttng_condition_buffer_usage_high_create();
8175     lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
8176     lttng_condition_buffer_usage_set_session_name(condition,
8177                                                   recording_session_name);
8178     lttng_condition_buffer_usage_set_channel_name(condition,
8179                                                   channel_name);
8180     lttng_condition_buffer_usage_set_domain_type(condition,
8181                                                  LTTNG_DOMAIN_UST);
8182
8183     /*
8184      * Create an action (receive a notification) to execute when the
8185      * condition created above is satisfied.
8186      */
8187     action = lttng_action_notify_create();
8188
8189     /*
8190      * Create a trigger.
8191      *
8192      * A trigger associates a condition to an action: LTTng executes
8193      * the action when the condition is satisfied.
8194      */
8195     trigger = lttng_trigger_create(condition, action);
8196
8197     /* Register the trigger to the LTTng session daemon. */
8198     lttng_register_trigger(trigger);
8199
8200     /*
8201      * Now that we have registered a trigger, LTTng will send a
8202      * notification every time its condition is met through a
8203      * notification channel.
8204      *
8205      * To receive this notification, we must subscribe to notifications
8206      * which match the same condition.
8207      */
8208     lttng_notification_channel_subscribe(notification_channel,
8209                                          condition);
8210
8211     /*
8212      * Notification loop.
8213      *
8214      * Put this in a dedicated thread to avoid blocking the main thread.
8215      */
8216     for (;;) {
8217         struct lttng_notification *notification;
8218         enum lttng_notification_channel_status status;
8219         const struct lttng_evaluation *notification_evaluation;
8220         const struct lttng_condition *notification_condition;
8221         double buffer_usage;
8222
8223         /* Receive the next notification. */
8224         status = lttng_notification_channel_get_next_notification(
8225             notification_channel, &notification);
8226
8227         switch (status) {
8228         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
8229             break;
8230         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
8231             /*
8232              * The session daemon can drop notifications if a monitoring
8233              * application isn't consuming the notifications fast
8234              * enough.
8235              */
8236             continue;
8237         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
8238             /*
8239              * The session daemon closed the notification channel.
8240              *
8241              * This is typically caused by a session daemon shutting
8242              * down.
8243              */
8244             goto end;
8245         default:
8246             /* Unhandled conditions or errors. */
8247             exit_status = EXIT_FAILURE;
8248             goto end;
8249         }
8250
8251         /*
8252          * A notification provides, amongst other things:
8253          *
8254          * * The condition that caused LTTng to send this notification.
8255          *
8256          * * The condition evaluation, which provides more specific
8257          *   information on the evaluation of the condition.
8258          *
8259          * The condition evaluation provides the buffer usage
8260          * value at the moment the condition was satisfied.
8261          */
8262         notification_condition = lttng_notification_get_condition(
8263             notification);
8264         notification_evaluation = lttng_notification_get_evaluation(
8265             notification);
8266
8267         /* We're subscribed to only one condition. */
8268         assert(lttng_condition_get_type(notification_condition) ==
8269             LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
8270
8271         /*
8272          * Get the exact sampled buffer usage from the condition
8273          * evaluation.
8274          */
8275         lttng_evaluation_buffer_usage_get_usage_ratio(
8276             notification_evaluation, &buffer_usage);
8277
8278         /*
8279          * At this point, instead of printing a message, we could do
8280          * something to reduce the buffer usage of the channel, like
8281          * disable specific events, for example.
8282          */
8283         printf("Buffer usage is %f %% in recording session \"%s\", "
8284                "user space channel \"%s\".\n", buffer_usage * 100,
8285                recording_session_name, channel_name);
8286
8287         /* Destroy the notification object. */
8288         lttng_notification_destroy(notification);
8289     }
8290
8291 end:
8292     lttng_action_destroy(action);
8293     lttng_condition_destroy(condition);
8294     lttng_trigger_destroy(trigger);
8295     lttng_notification_channel_destroy(notification_channel);
8296     return exit_status;
8297 }
8298 ----
8299 --
8300
8301 . Build the `notif-app` application, linking it with `liblttng-ctl`:
8302 +
8303 --
8304 [role="term"]
8305 ----
8306 $ gcc -o notif-app notif-app.c $(pkg-config --cflags --libs lttng-ctl)
8307 ----
8308 --
8309
8310 . <<creating-destroying-tracing-sessions,Create a recording session>>,
8311   <<enabling-disabling-events,create a recording event rule>> matching
8312   all the user space tracepoint events, and
8313   <<basic-tracing-session-control,start recording>>:
8314 +
8315 --
8316 [role="term"]
8317 ----
8318 $ lttng create my-session
8319 $ lttng enable-event --userspace --all
8320 $ lttng start
8321 ----
8322 --
8323 +
8324 If you create the channel manually with the man:lttng-enable-channel(1)
8325 command, you can set its <<channel-monitor-timer,monitor timer>> to
8326 control how frequently LTTng samples the current values of the channel
8327 properties to evaluate user conditions.
8328
8329 . Run the `notif-app` application.
8330 +
8331 This program accepts the <<tracing-session,recording session>> and
8332 user space channel names as its two first arguments. The channel
8333 which LTTng automatically creates with the man:lttng-enable-event(1)
8334 command above is named `channel0`:
8335 +
8336 --
8337 [role="term"]
8338 ----
8339 $ ./notif-app my-session channel0
8340 ----
8341 --
8342
8343 . In another terminal, run an application with a very high event
8344   throughput so that the 75{nbsp}% buffer usage condition is reached.
8345 +
8346 In the first terminal, the application should print lines like this:
8347 +
8348 ----
8349 Buffer usage is 81.45197 % in recording session "my-session", user space
8350 channel "channel0".
8351 ----
8352 +
8353 If you don't see anything, try to make the threshold of the condition in
8354 path:{notif-app.c} lower (0.1{nbsp}%, for example), and then rebuild the
8355 `notif-app` application (step{nbsp}2) and run it again (step{nbsp}4).
8356 ====
8357
8358
8359 [[reference]]
8360 == Reference
8361
8362 [[lttng-modules-ref]]
8363 === noch:{LTTng-modules}
8364
8365
8366 [role="since-2.9"]
8367 [[lttng-tracepoint-enum]]
8368 ==== `LTTNG_TRACEPOINT_ENUM()` usage
8369
8370 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
8371
8372 [source,c]
8373 ----
8374 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
8375 ----
8376
8377 Replace:
8378
8379 * `name` with the name of the enumeration (C identifier, unique
8380   amongst all the defined enumerations).
8381 * `entries` with a list of enumeration entries.
8382
8383 The available enumeration entry macros are:
8384
8385 +ctf_enum_value(__name__, __value__)+::
8386   Entry named +__name__+ mapped to the integral value +__value__+.
8387
8388 +ctf_enum_range(__name__, __begin__, __end__)+::
8389   Entry named +__name__+ mapped to the range of integral values between
8390   +__begin__+ (included) and +__end__+ (included).
8391
8392 +ctf_enum_auto(__name__)+::
8393   Entry named +__name__+ mapped to the integral value following the
8394   last mapping value.
8395 +
8396 The last value of a `ctf_enum_value()` entry is its +__value__+
8397 parameter.
8398 +
8399 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
8400 +
8401 If `ctf_enum_auto()` is the first entry in the list, its integral
8402 value is 0.
8403
8404 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
8405 to use a defined enumeration as a tracepoint field.
8406
8407 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
8408 ====
8409 [source,c]
8410 ----
8411 LTTNG_TRACEPOINT_ENUM(
8412     my_enum,
8413     TP_ENUM_VALUES(
8414         ctf_enum_auto("AUTO: EXPECT 0")
8415         ctf_enum_value("VALUE: 23", 23)
8416         ctf_enum_value("VALUE: 27", 27)
8417         ctf_enum_auto("AUTO: EXPECT 28")
8418         ctf_enum_range("RANGE: 101 TO 303", 101, 303)
8419         ctf_enum_auto("AUTO: EXPECT 304")
8420     )
8421 )
8422 ----
8423 ====
8424
8425
8426 [role="since-2.7"]
8427 [[lttng-modules-tp-fields]]
8428 ==== Tracepoint fields macros (for `TP_FIELDS()`)
8429
8430 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
8431 tracepoint fields, which must be listed within `TP_FIELDS()` in
8432 `LTTNG_TRACEPOINT_EVENT()`, are:
8433
8434 [role="func-desc growable",cols="asciidoc,asciidoc"]
8435 .Available macros to define LTTng-modules tracepoint fields
8436 |====
8437 |Macro |Description and parameters
8438
8439 |
8440 +ctf_integer(__t__, __n__, __e__)+
8441
8442 +ctf_integer_nowrite(__t__, __n__, __e__)+
8443
8444 +ctf_user_integer(__t__, __n__, __e__)+
8445
8446 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
8447 |
8448 Standard integer, displayed in base{nbsp}10.
8449
8450 +__t__+::
8451   Integer C type (`int`, `long`, `size_t`, ...).
8452
8453 +__n__+::
8454   Field name.
8455
8456 +__e__+::
8457   Argument expression.
8458
8459 |
8460 +ctf_integer_hex(__t__, __n__, __e__)+
8461
8462 +ctf_user_integer_hex(__t__, __n__, __e__)+
8463 |
8464 Standard integer, displayed in base{nbsp}16.
8465
8466 +__t__+::
8467   Integer C type.
8468
8469 +__n__+::
8470   Field name.
8471
8472 +__e__+::
8473   Argument expression.
8474
8475 |+ctf_integer_oct(__t__, __n__, __e__)+
8476 |
8477 Standard integer, displayed in base{nbsp}8.
8478
8479 +__t__+::
8480   Integer C type.
8481
8482 +__n__+::
8483   Field name.
8484
8485 +__e__+::
8486   Argument expression.
8487
8488 |
8489 +ctf_integer_network(__t__, __n__, __e__)+
8490
8491 +ctf_user_integer_network(__t__, __n__, __e__)+
8492 |
8493 Integer in network byte order (big-endian), displayed in base{nbsp}10.
8494
8495 +__t__+::
8496   Integer C type.
8497
8498 +__n__+::
8499   Field name.
8500
8501 +__e__+::
8502   Argument expression.
8503
8504 |
8505 +ctf_integer_network_hex(__t__, __n__, __e__)+
8506
8507 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
8508 |
8509 Integer in network byte order, displayed in base{nbsp}16.
8510
8511 +__t__+::
8512   Integer C type.
8513
8514 +__n__+::
8515   Field name.
8516
8517 +__e__+::
8518   Argument expression.
8519
8520 |
8521 +ctf_enum(__N__, __t__, __n__, __e__)+
8522
8523 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
8524
8525 +ctf_user_enum(__N__, __t__, __n__, __e__)+
8526
8527 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
8528 |
8529 Enumeration.
8530
8531 +__N__+::
8532   Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
8533
8534 +__t__+::
8535   Integer C type (`int`, `long`, `size_t`, ...).
8536
8537 +__n__+::
8538   Field name.
8539
8540 +__e__+::
8541   Argument expression.
8542
8543 |
8544 +ctf_string(__n__, __e__)+
8545
8546 +ctf_string_nowrite(__n__, __e__)+
8547
8548 +ctf_user_string(__n__, __e__)+
8549
8550 +ctf_user_string_nowrite(__n__, __e__)+
8551 |
8552 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
8553
8554 +__n__+::
8555   Field name.
8556
8557 +__e__+::
8558   Argument expression.
8559
8560 |
8561 +ctf_array(__t__, __n__, __e__, __s__)+
8562
8563 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
8564
8565 +ctf_user_array(__t__, __n__, __e__, __s__)+
8566
8567 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
8568 |
8569 Statically-sized array of integers.
8570
8571 +__t__+::
8572   Array element C type.
8573
8574 +__n__+::
8575   Field name.
8576
8577 +__e__+::
8578   Argument expression.
8579
8580 +__s__+::
8581   Number of elements.
8582
8583 |
8584 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
8585
8586 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8587
8588 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
8589
8590 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
8591 |
8592 Statically-sized array of bits.
8593
8594 The type of +__e__+ must be an integer type. +__s__+ is the number
8595 of elements of such type in +__e__+, not the number of bits.
8596
8597 +__t__+::
8598   Array element C type.
8599
8600 +__n__+::
8601   Field name.
8602
8603 +__e__+::
8604   Argument expression.
8605
8606 +__s__+::
8607   Number of elements.
8608
8609 |
8610 +ctf_array_text(__t__, __n__, __e__, __s__)+
8611
8612 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
8613
8614 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
8615
8616 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
8617 |
8618 Statically-sized array, printed as text.
8619
8620 The string doesn't need to be null-terminated.
8621
8622 +__t__+::
8623   Array element C type (always `char`).
8624
8625 +__n__+::
8626   Field name.
8627
8628 +__e__+::
8629   Argument expression.
8630
8631 +__s__+::
8632   Number of elements.
8633
8634 |
8635 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
8636
8637 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8638
8639 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
8640
8641 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
8642 |
8643 Dynamically-sized array of integers.
8644
8645 The type of +__E__+ must be unsigned.
8646
8647 +__t__+::
8648   Array element C type.
8649
8650 +__n__+::
8651   Field name.
8652
8653 +__e__+::
8654   Argument expression.
8655
8656 +__T__+::
8657   Length expression C type.
8658
8659 +__E__+::
8660   Length expression.
8661
8662 |
8663 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8664
8665 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
8666 |
8667 Dynamically-sized array of integers, displayed in base{nbsp}16.
8668
8669 The type of +__E__+ must be unsigned.
8670
8671 +__t__+::
8672   Array element C type.
8673
8674 +__n__+::
8675   Field name.
8676
8677 +__e__+::
8678   Argument expression.
8679
8680 +__T__+::
8681   Length expression C type.
8682
8683 +__E__+::
8684   Length expression.
8685
8686 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
8687 |
8688 Dynamically-sized array of integers in network byte order (big-endian),
8689 displayed in base{nbsp}10.
8690
8691 The type of +__E__+ must be unsigned.
8692
8693 +__t__+::
8694   Array element C type.
8695
8696 +__n__+::
8697   Field name.
8698
8699 +__e__+::
8700   Argument expression.
8701
8702 +__T__+::
8703   Length expression C type.
8704
8705 +__E__+::
8706   Length expression.
8707
8708 |
8709 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8710
8711 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8712
8713 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
8714
8715 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
8716 |
8717 Dynamically-sized array of bits.
8718
8719 The type of +__e__+ must be an integer type. +__s__+ is the number
8720 of elements of such type in +__e__+, not the number of bits.
8721
8722 The type of +__E__+ must be unsigned.
8723
8724 +__t__+::
8725   Array element C type.
8726
8727 +__n__+::
8728   Field name.
8729
8730 +__e__+::
8731   Argument expression.
8732
8733 +__T__+::
8734   Length expression C type.
8735
8736 +__E__+::
8737   Length expression.
8738
8739 |
8740 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8741
8742 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8743
8744 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
8745
8746 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
8747 |
8748 Dynamically-sized array, displayed as text.
8749
8750 The string doesn't need to be null-terminated.
8751
8752 The type of +__E__+ must be unsigned.
8753
8754 The behaviour is undefined if +__e__+ is `NULL`.
8755
8756 +__t__+::
8757   Sequence element C type (always `char`).
8758
8759 +__n__+::
8760   Field name.
8761
8762 +__e__+::
8763   Argument expression.
8764
8765 +__T__+::
8766   Length expression C type.
8767
8768 +__E__+::
8769   Length expression.
8770 |====
8771
8772 Use the `_user` versions when the argument expression, `e`, is
8773 a user space address. In the cases of `ctf_user_integer*()` and
8774 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
8775 be addressable.
8776
8777 The `_nowrite` versions omit themselves from the trace data, but are
8778 otherwise identical. This means LTTng won't write the `_nowrite` fields
8779 to the recorded trace. Their primary purpose is to make some of the
8780 event context available to the <<enabling-disabling-events,recording
8781 event rule filters>> without having to commit the data to
8782 <<channel,sub-buffers>>.
8783
8784
8785 [[glossary]]
8786 == Glossary
8787
8788 Terms related to LTTng and to tracing in general:
8789
8790 [[def-action]]action::
8791   The part of a <<def-trigger,trigger>> which LTTng executes when the
8792   trigger <<def-condition,condition>> is satisfied.
8793
8794 Babeltrace::
8795   The https://diamon.org/babeltrace[Babeltrace] project, which includes:
8796 +
8797 * The
8798   https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
8799   command-line interface.
8800 * The libbabeltrace2 library which offers a
8801   https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
8802 * https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
8803 * Plugins.
8804
8805 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
8806   A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
8807
8808 [[def-channel]]<<channel,channel>>::
8809   An entity which is responsible for a set of
8810   <<def-ring-buffer,ring buffers>>.
8811 +
8812 <<def-recording-event-rule,Recording event rules>> are always attached
8813 to a specific channel.
8814
8815 clock::
8816   A source of time for a <<def-tracer,tracer>>.
8817
8818 [[def-condition]]condition::
8819   The part of a <<def-trigger,trigger>> which must be satisfied for
8820   LTTng to attempt to execute the trigger <<def-action,actions>>.
8821
8822 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
8823   A program which is responsible for consuming the full
8824   <<def-sub-buffer,sub-buffers>> and write them to a file system or
8825   send them over the network.
8826
8827 [[def-current-trace-chunk]]current trace chunk::
8828   A <<def-trace-chunk,trace chunk>> which includes the current content
8829   of all the <<def-sub-buffer,sub-buffers>> of the
8830   <<def-tracing-session,recording session>> and the stream files
8831   produced since the latest event amongst:
8832 +
8833 * The creation of the recording session.
8834 * The last <<def-tracing-session-rotation,recording session rotation>>, if
8835   any.
8836
8837 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
8838   The <<def-event-record-loss-mode,event record loss mode>> in which
8839   the <<def-tracer,tracer>> _discards_ new <<def-event-record,event
8840   records>> when there's no <<def-sub-buffer,sub-buffer>> space left to
8841   store them.
8842
8843 [[def-event]]event::
8844   The execution of an <<def-instrumentation-point,instrumentation
8845   point>>, like a <<def-tracepoint,tracepoint>> that you manually place
8846   in some source code, or a Linux kprobe.
8847 +
8848 When an instrumentation point is executed, LTTng creates an event.
8849 +
8850 When an <<def-event-rule,event rule>> matches the event,
8851 <<def-lttng,LTTng>> executes some action, for example:
8852 +
8853 * Record its payload to a <<def-sub-buffer,sub-buffer>> as an
8854   <<def-event-record,event record>>.
8855 * Attempt to execute the user-defined actions of a
8856   <<def-trigger,trigger>> with an
8857   <<add-event-rule-matches-trigger,``event rule matches''>> condition.
8858
8859 [[def-event-name]]event name::
8860   The name of an <<def-event,event>>, which is also the name of the
8861   <<def-event-record,event record>>.
8862 +
8863 This is also called the _instrumentation point name_.
8864
8865 [[def-event-record]]event record::
8866   A record (binary serialization), in a <<def-trace,trace>>, of the
8867   payload of an <<def-event,event>>.
8868 +
8869 The payload of an event record has zero or more _fields_.
8870
8871 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
8872   The mechanism by which event records of a given
8873   <<def-channel,channel>> are lost (not recorded) when there's no
8874   <<def-sub-buffer,sub-buffer>> space left to store them.
8875
8876 [[def-event-rule]]<<event-rule,event rule>>::
8877   Set of conditions which an <<def-event,event>> must satisfy
8878   for LTTng to execute some action.
8879 +
8880 An event rule is said to _match_ events, like a
8881 https://en.wikipedia.org/wiki/Regular_expression[regular expression]
8882 matches strings.
8883 +
8884 A <<def-recording-event-rule,recording event rule>> is a specific type
8885 of event rule of which the action is to <<def-record,record>> the event
8886 to a <<def-sub-buffer,sub-buffer>>.
8887
8888 [[def-incl-set]]inclusion set::
8889   In the <<pid-tracking,process attribute inclusion set>> context: a
8890   set of <<def-proc-attr,process attributes>> of a given type.
8891
8892 <<instrumenting,instrumentation>>::
8893   The use of <<def-lttng,LTTng>> probes to make a kernel or
8894   <<def-user-application,user application>> traceable.
8895
8896 [[def-instrumentation-point]]instrumentation point::
8897   A point in the execution path of a kernel or
8898   <<def-user-application,user application>> which, when executed,
8899   create an <<def-event,event>>.
8900
8901 instrumentation point name::
8902   See _<<def-event-name,event name>>_.
8903
8904 `java.util.logging`::
8905   The
8906   https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
8907   of the Java platform.
8908
8909 log4j::
8910   A https://logging.apache.org/log4j/1.2/[logging library] for Java
8911   developed by the Apache Software Foundation.
8912
8913 log level::
8914   Level of severity of a log statement or user space
8915   <<def-instrumentation-point,instrumentation point>>.
8916
8917 [[def-lttng]]LTTng::
8918   The _Linux Trace Toolkit: next generation_ project.
8919
8920 <<lttng-cli,cmd:lttng>>::
8921   A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
8922   project which you can use to send and receive control messages to and
8923   from a <<def-session-daemon,session daemon>>.
8924
8925 cmd:lttng-consumerd::
8926   The name of the <<def-consumer-daemon,consumer daemon>> program.
8927
8928 cmd:lttng-crash::
8929   A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8930   which can convert <<def-ring-buffer,ring buffer>> files (usually
8931   <<persistent-memory-file-systems,saved on a persistent memory file
8932   system>>) to <<def-trace,trace>> files.
8933 +
8934 See man:lttng-crash(1).
8935
8936 LTTng Documentation::
8937   This document.
8938
8939 <<lttng-live,LTTng live>>::
8940   A communication protocol between the <<lttng-relayd,relay daemon>> and
8941   live readers which makes it possible to show or analyze
8942   <<def-event-record,event records>> ``live'', as they're received by
8943   the <<def-relay-daemon,relay daemon>>.
8944
8945 <<lttng-modules,LTTng-modules>>::
8946   The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8947   which contains the Linux kernel modules to make the Linux kernel
8948   <<def-instrumentation-point,instrumentation points>> available for
8949   <<def-lttng,LTTng>> tracing.
8950
8951 cmd:lttng-relayd::
8952   The name of the <<def-relay-daemon,relay daemon>> program.
8953
8954 cmd:lttng-sessiond::
8955   The name of the <<def-session-daemon,session daemon>> program.
8956
8957 [[def-lttng-tools]]LTTng-tools::
8958   The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8959   contains the various programs and libraries used to
8960   <<controlling-tracing,control tracing>>.
8961
8962 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8963   The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8964   contains libraries to instrument
8965   <<def-user-application,user applications>>.
8966
8967 <<lttng-ust-agents,LTTng-UST Java agent>>::
8968   A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8969   allow the LTTng instrumentation of `java.util.logging` and Apache
8970   log4j{nbsp}1.2 logging statements.
8971
8972 <<lttng-ust-agents,LTTng-UST Python agent>>::
8973   A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8974   to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8975   statements.
8976
8977 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8978   The <<def-event-record-loss-mode,event record loss mode>> in which new
8979   <<def-event-record,event records>> _overwrite_ older event records
8980   when there's no <<def-sub-buffer,sub-buffer>> space left to store
8981   them.
8982
8983 <<channel-buffering-schemes,per-process buffering>>::
8984   A <<def-buffering-scheme,buffering scheme>> in which each instrumented
8985   process has its own <<def-sub-buffer,sub-buffers>> for a given user
8986   space <<def-channel,channel>>.
8987
8988 <<channel-buffering-schemes,per-user buffering>>::
8989   A <<def-buffering-scheme,buffering scheme>> in which all the processes
8990   of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
8991   given user space <<def-channel,channel>>.
8992
8993 [[def-proc-attr]]process attribute::
8994   In the <<pid-tracking,process attribute inclusion set>> context:
8995 +
8996 * A process ID.
8997 * A virtual process ID.
8998 * A Unix user ID.
8999 * A virtual Unix user ID.
9000 * A Unix group ID.
9001 * A virtual Unix group ID.
9002
9003 record (_noun_)::
9004   See <<def-event-record,_event record_>>.
9005
9006 [[def-record]]record (_verb_)::
9007   Serialize the binary payload of an <<def-event,event>> to a
9008   <<def-sub-buffer,sub-buffer>>.
9009
9010 [[def-recording-event-rule]]<<event,recording event rule>>::
9011   Specific type of <<def-event-rule,event rule>> of which the action is
9012   to <<def-record,record>> the matched event to a
9013   <<def-sub-buffer,sub-buffer>>.
9014
9015 [[def-tracing-session]][[def-recording-session]]<<tracing-session,recording session>>::
9016   A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
9017
9018 [[def-tracing-session-rotation]]<<session-rotation,recording session rotation>>::
9019   The action of archiving the
9020   <<def-current-trace-chunk,current trace chunk>> of a
9021   <<def-tracing-session,recording session>>.
9022
9023 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
9024   A process which is responsible for receiving the <<def-trace,trace>>
9025   data which a distant <<def-consumer-daemon,consumer daemon>> sends.
9026
9027 [[def-ring-buffer]]ring buffer::
9028   A set of <<def-sub-buffer,sub-buffers>>.
9029
9030 rotation::
9031   See _<<def-tracing-session-rotation,recording session rotation>>_.
9032
9033 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
9034   A process which receives control commands from you and orchestrates
9035   the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
9036
9037 <<taking-a-snapshot,snapshot>>::
9038   A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
9039   of a given <<def-tracing-session,recording session>>, saved as
9040   <<def-trace,trace>> files.
9041
9042 [[def-sub-buffer]]sub-buffer::
9043   One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
9044   which contains <<def-event-record,event records>>.
9045
9046 timestamp::
9047   The time information attached to an <<def-event,event>> when LTTng
9048   creates it.
9049
9050 [[def-trace]]trace (_noun_)::
9051   A set of:
9052 +
9053 * One https://diamon.org/ctf/[CTF] metadata stream file.
9054 * One or more CTF data stream files which are the concatenations of one
9055   or more flushed <<def-sub-buffer,sub-buffers>>.
9056
9057 [[def-trace-verb]]trace (_verb_)::
9058   From the perspective of a <<def-tracer,tracer>>: attempt to execute
9059   one or more actions when emitting an <<def-event,event>> in an
9060   application or in a system.
9061
9062 [[def-trace-chunk]]trace chunk::
9063   A self-contained <<def-trace,trace>> which is part of a
9064   <<def-tracing-session,recording session>>. Each
9065   <<def-tracing-session-rotation, recording session rotation>> produces a
9066   <<def-trace-chunk-archive,trace chunk archive>>.
9067
9068 [[def-trace-chunk-archive]]trace chunk archive::
9069   The result of a <<def-tracing-session-rotation, recording session
9070   rotation>>.
9071 +
9072 <<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
9073 containing <<def-tracing-session,recording session>> is still active: you
9074 are free to read it, modify it, move it, or remove it.
9075
9076 Trace Compass::
9077   The http://tracecompass.org[Trace Compass] project and application.
9078
9079 [[def-tracepoint]]tracepoint::
9080   An instrumentation point using the tracepoint mechanism of the Linux
9081   kernel or of <<def-lttng-ust,LTTng-UST>>.
9082
9083 tracepoint definition::
9084   The definition of a single <<def-tracepoint,tracepoint>>.
9085
9086 tracepoint name::
9087   The name of a <<def-tracepoint,tracepoint>>.
9088
9089 [[def-tracepoint-provider]]tracepoint provider::
9090   A set of functions providing <<def-tracepoint,tracepoints>> to an
9091   instrumented <<def-user-application,user application>>.
9092 +
9093 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
9094 provider package>>: many tracepoint providers can exist within a
9095 tracepoint provider package.
9096
9097 [[def-tracepoint-provider-package]]tracepoint provider package::
9098   One or more <<def-tracepoint-provider,tracepoint providers>> compiled
9099   as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
9100   link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
9101   library].
9102
9103 [[def-tracer]]tracer::
9104   A piece of software which executes some action when it emits
9105   an <<def-event,event>>, like <<def-record,record>> it to some
9106   buffer.
9107
9108 <<domain,tracing domain>>::
9109   A type of LTTng <<def-tracer,tracer>>.
9110
9111 <<tracing-group,tracing group>>::
9112   The Unix group which a Unix user can be part of to be allowed to
9113   control the Linux kernel LTTng <<def-tracer,tracer>>.
9114
9115 [[def-trigger]]<<trigger,trigger>>::
9116   A <<def-condition,condition>>-<<def-action,actions>> pair; when the
9117   condition of a trigger is satisfied, LTTng attempts to execute its
9118   actions.
9119
9120 [[def-user-application]]user application::
9121   An application (program or library) running in user space, as opposed
9122   to a Linux kernel module, for example.