2.12/lttng-docs-2.12.txt

   1 The LTTng Documentation
   2 =======================
   3 Philippe Proulx <pproulx@efficios.com>
   4 v2.12, 28 November 2023
   5
   6
   7 include::../common/copyright.txt[]
   8
   9
  10 include::../common/welcome.txt[]
  11
  12
  13 include::../common/audience.txt[]
  14
  15
  16 [[chapters]]
  17 === What's in this documentation?
  18
  19 The LTTng Documentation is divided into the following sections:
  20
  21 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
  22   rudiments of software tracing and the rationale behind the
  23   LTTng project.
  24 +
  25 Skip this section if you’re familiar with software tracing and with the
  26 LTTng project.
  27
  28 * **<<installing-lttng,Installation>>** describes the steps to
  29   install the LTTng packages on common Linux distributions and from
  30   their sources.
  31 +
  32 Skip this section if you already properly installed LTTng on your target
  33 system.
  34
  35 * **<<getting-started,Quick start>>** is a concise guide to
  36   getting started quickly with LTTng kernel and user space tracing.
  37 +
  38 We recommend this section if you're new to LTTng or to software tracing
  39 in general.
  40 +
  41 Skip this section if you're not new to LTTng.
  42
  43 * **<<core-concepts,Core concepts>>** explains the concepts at
  44   the heart of LTTng.
  45 +
  46 It's a good idea to become familiar with the core concepts
  47 before attempting to use the toolkit.
  48
  49 * **<<plumbing,Components of LTTng>>** describes the various components
  50   of the LTTng machinery, like the daemons, the libraries, and the
  51   command-line interface.
  52 * **<<instrumenting,Instrumentation>>** shows different ways to
  53   instrument user applications and the Linux kernel.
  54 +
  55 Instrumenting source code is essential to provide a meaningful
  56 source of events.
  57 +
  58 Skip this section if you don't have a programming background.
  59
  60 * **<<controlling-tracing,Tracing control>>** is divided into topics
  61   which demonstrate how to use the vast array of features that
  62   LTTng{nbsp}{revision} offers.
  63 * **<<reference,Reference>>** contains reference tables.
  64 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
  65   to LTTng or to the field of software tracing.
  66
  67
  68 include::../common/convention.txt[]
  69
  70
  71 include::../common/acknowledgements.txt[]
  72
  73
  74 [[whats-new]]
  75 == What's new in LTTng{nbsp}{revision}?
  76
  77 LTTng{nbsp}{revision} bears the name _Ta Meilleure_, a Northeast IPA
  78 beer brewed by https://lagabiere.com/[Lagabière]. Translating to ``Your
  79 best one'', this beer gives out strong aromas of passion fruit, lemon,
  80 and peaches. Tastewise, expect a lot of fruit, a creamy texture, and a
  81 smooth lingering hop bitterness.
  82
  83 New features and changes in LTTng{nbsp}{revision}:
  84
  85 Tracing control::
  86 +
  87 * Clear the contents of one or more <<tracing-session,tracing sessions>>
  88   without having to destroy and reconfigure them
  89   with the new man:lttng-clear(1) command.
  90 +
  91 This is especially useful to clear the tracing data of a tracing session
  92 between attempts to reproduce a problem.
  93 +
  94 See <<clear,Clear a tracing session>>.
  95
  96 * Before LTTng{nbsp}{revision}, the man:lttng-track(1) and
  97   man:lttng-untrack(1) commands used to add and remove process IDs
  98   (PIDs) to a whitelist so that LTTng would only trace processes with
  99   specific PIDs.
 100 +
 101 LTTng{nbsp}{revision} adds Unix user IDs (UIDs) and Unix group IDs
 102 (GIDs) to the available <<pid-tracking,process attributes to track>>.
 103 You can specify numeric user/group IDs and user/group names to track,
 104 for example:
 105 +
 106 [role="term"]
 107 ----
 108 $ lttng track --userspace --vuid=http,999 --vgid=mysql,9
 109 ----
 110 +
 111 While you can also track UIDs and GIDs with the
 112 opt:lttng-enable-event(1):--filter option of the `enable-event` command,
 113 this dedicated process attribute tracking approach reduces tracing
 114 overhead and prevents the creation of <<def-sub-buffer,sub-buffers>> for
 115 the users and groups which LTTng doesn't track.
 116 +
 117 In the command manual pages, the term ``whitelist'' is renamed to
 118 ``inclusion set'' to clarify the concept.
 119
 120 * The <<lttng-relayd,relay daemon>> can now maintain many files
 121   virtually opened without using as many file descriptors (FD). It does
 122   so by closing and reopening FDs as needed.
 123 +
 124 This feature is meant as a workaround for users who can't bump the
 125 system limit because of permission restrictions.
 126 +
 127 The new opt:lttng-relayd(8):--fd-pool-size relay daemon option
 128 sets the maximum number of simultaneously opened file descriptors
 129 (using the soft `RLIMIT_NOFILE` resource limit of the process by
 130 default; see man:getrlimit(2)).
 131
 132 * By default, the relay daemon writes its traces under a predefined
 133   directory hierarchy,
 134   +$LTTNG_HOME/lttng-traces/__host__/__session__/__domain__+, with:
 135 +
 136 --
 137 +__host__+::
 138     Remote hostname.
 139
 140 +__session__+::
 141     <<tracing-session,Tracing session>> name.
 142
 143 +__domain__+::
 144     <<domain,Tracing domain>> name (`ust` or `kernel`).
 145 --
 146 +
 147 Change this hierarchy to group traces by tracing session name rather
 148 than by hostname
 149 (+$LTTNG_HOME/lttng-traces/__session__/__host__/__domain__+) with the
 150 new opt:lttng-relayd(8):--group-output-by-session option of the
 151 relay daemon.
 152 +
 153 This feature is especially useful if you're tracing two or more hosts,
 154 having different hostnames, which share the same tracing session name as
 155 part of their configuration. In this scenario, you can use a descriptive
 156 tracing session name (for example, `connection-hang`) across a fleet of
 157 machines streaming to a single relay daemon.
 158
 159 * The relay daemon has a new opt:lttng-relayd(8):--working-directory
 160   option to override its working directory.
 161
 162 Linux kernel tracing::
 163 +
 164 * New instrumentation hooks to trace the entry and exit tracepoints of
 165   the network reception code paths of the Linux kernel.
 166 +
 167 Use the resulting event records to identify the bounds of a network
 168 reception and link the events that occur in the interim (for example,
 169 wake-ups) to a specific network reception instance. You can also
 170 analyze the latency of the network stack thanks to those event records.
 171
 172 * The `thread` field of the `irqaction` structure, which specifies the
 173   process to wake up when a threaded interrupt request (IRQ) occurs, is
 174   now part of the `lttng_statedump_interrupt` event record.
 175 +
 176 Use this information to discover which processes handle the various
 177 IRQs. You can also associate the events occurring in the context of
 178 those processes with their respective IRQ.
 179
 180 * New `lttng_statedump_cpu_topology` tracepoint to record the active
 181   CPU/NUMA topology.
 182 +
 183 Use this information to discover which CPUs are SMT siblings or part of
 184 the same socket. As of LTTng{nbsp}{revision}, only the x86 architecture
 185 is supported since all architectures describe their topologies
 186 differently.
 187 +
 188 The `architecture` field of the tracepoint is statically defined and
 189 exists for all architecture implementations. Analysis tools can
 190 therefore anticipate the layout of the event record.
 191 +
 192 Event record example:
 193 +
 194 [source,yaml]
 195 ----
 196 lttng_statedump_cpu_topology:
 197   architecture: x86
 198   cpu_id: 0
 199   vendor: GenuineIntel
 200   family: 6
 201   model: 142
 202   model_name: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
 203   physical_id: 0
 204   core_id: 0
 205   cores: 2
 206 ----
 207
 208 * New product UUID metadata environment field, `product_uuid`,
 209   which LTTng copies from the
 210   https://en.wikipedia.org/wiki/Desktop_Management_Interface[Desktop
 211   Management Interface] (DMI).
 212 +
 213 Use this environment field to uniquely identify a machine (virtual or
 214 physical) in order to correlate traces from multiple virtual machines.
 215
 216
 217 [[nuts-and-bolts]]
 218 == Nuts and bolts
 219
 220 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
 221 generation_ is a modern toolkit for tracing Linux systems and
 222 applications. So your first question might be:
 223 **what is tracing?**
 224
 225
 226 [[what-is-tracing]]
 227 === What is tracing?
 228
 229 As the history of software engineering progressed and led to what
 230 we now take for granted--complex, numerous and
 231 interdependent software applications running in parallel on
 232 sophisticated operating systems like Linux--the authors of such
 233 components, software developers, began feeling a natural
 234 urge to have tools that would ensure the robustness and good performance
 235 of their masterpieces.
 236
 237 One major achievement in this field is, inarguably, the
 238 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
 239 an essential tool for developers to find and fix bugs. But even the best
 240 debugger won't help make your software run faster, and nowadays, faster
 241 software means either more work done by the same hardware, or cheaper
 242 hardware for the same work.
 243
 244 A _profiler_ is often the tool of choice to identify performance
 245 bottlenecks. Profiling is suitable to identify _where_ performance is
 246 lost in a given software. The profiler outputs a profile, a statistical
 247 summary of observed events, which you may use to discover which
 248 functions took the most time to execute. However, a profiler won't
 249 report _why_ some identified functions are the bottleneck. Bottlenecks
 250 might only occur when specific conditions are met, conditions that are
 251 sometimes impossible to capture by a statistical profiler, or impossible
 252 to reproduce with an application altered by the overhead of an
 253 event-based profiler. For a thorough investigation of software
 254 performance issues, a history of execution is essential, with the
 255 recorded values of variables and context fields you choose, and
 256 with as little influence as possible on the instrumented software. This
 257 is where tracing comes in handy.
 258
 259 _Tracing_ is a technique used to understand what goes on in a running
 260 software system. The software used for tracing is called a _tracer_,
 261 which is conceptually similar to a tape recorder. When recording,
 262 specific instrumentation points placed in the software source code
 263 generate events that are saved on a giant tape: a _trace_ file. You
 264 can trace user applications and the operating system at the same time,
 265 opening the possibility of resolving a wide range of problems that would
 266 otherwise be extremely challenging.
 267
 268 Tracing is often compared to _logging_. However, tracers and loggers are
 269 two different tools, serving two different purposes. Tracers are
 270 designed to record much lower-level events that occur much more
 271 frequently than log messages, often in the range of thousands per
 272 second, with very little execution overhead. Logging is more appropriate
 273 for a very high-level analysis of less frequent events: user accesses,
 274 exceptional conditions (errors and warnings, for example), database
 275 transactions, instant messaging communications, and such. Simply put,
 276 logging is one of the many use cases that can be satisfied with tracing.
 277
 278 The list of recorded events inside a trace file can be read manually
 279 like a log file for the maximum level of detail, but it is generally
 280 much more interesting to perform application-specific analyses to
 281 produce reduced statistics and graphs that are useful to resolve a
 282 given problem. Trace viewers and analyzers are specialized tools
 283 designed to do this.
 284
 285 In the end, this is what LTTng is: a powerful, open source set of
 286 tools to trace the Linux kernel and user applications at the same time.
 287 LTTng is composed of several components actively maintained and
 288 developed by its link:/community/#where[community].
 289
 290
 291 [[lttng-alternatives]]
 292 === Alternatives to noch:{LTTng}
 293
 294 Excluding proprietary solutions, a few competing software tracers
 295 exist for Linux:
 296
 297 https://github.com/dtrace4linux/linux[dtrace4linux]::
 298     A port of Sun Microsystems' DTrace to Linux.
 299 +
 300 The cmd:dtrace tool interprets user scripts and is responsible for
 301 loading code into the Linux kernel for further execution and collecting
 302 the outputted data.
 303
 304 https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF]::
 305     A subsystem in the Linux kernel in which a virtual machine can
 306     execute programs passed from the user space to the kernel.
 307 +
 308 You can attach such programs to tracepoints and kprobes thanks to a
 309 system call, and they can output data to the user space when executed
 310 thanks to different mechanisms (pipe, VM register values, and eBPF maps,
 311 to name a few).
 312
 313 https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]::
 314     The de facto function tracer of the Linux kernel.
 315 +
 316 Its user interface is a set of special files in sysfs.
 317
 318 https://perf.wiki.kernel.org/[perf]::
 319     A performance analysis tool for Linux which supports hardware
 320     performance counters, tracepoints, as well as other counters and
 321     types of probes.
 322 +
 323 The controlling utility of perf is the cmd:perf command line/text UI
 324 tool.
 325
 326 http://linux.die.net/man/1/strace[strace]::
 327     A command-line utility which records system calls made by a
 328     user process, as well as signal deliveries and changes of process
 329     state.
 330 +
 331 strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace] to
 332 fulfill its function.
 333
 334 http://www.sysdig.org/[sysdig]::
 335     Like SystemTap, uses scripts to analyze Linux kernel events.
 336 +
 337 You write scripts, or _chisels_ in the jargon of sysdig, in Lua and
 338 sysdig executes them while it traces the system or afterwards. The
 339 interface of sysdig is the cmd:sysdig command-line tool as well as the
 340 text UI-based cmd:csysdig tool.
 341
 342 https://sourceware.org/systemtap/[SystemTap]::
 343     A Linux kernel and user space tracer which uses custom user scripts
 344     to produce plain text traces.
 345 +
 346 SystemTap converts the scripts to the C language, and then compiles them
 347 as Linux kernel modules which are loaded to produce trace data. The
 348 primary user interface of SystemTap is the cmd:stap command-line tool.
 349
 350 The main distinctive features of LTTng is that it produces correlated
 351 kernel and user space traces, as well as doing so with the lowest
 352 overhead amongst other solutions. It produces trace files in the
 353 http://diamon.org/ctf[CTF] format, a file format optimized
 354 for the production and analyses of multi-gigabyte data.
 355
 356 LTTng is the result of more than 10{nbsp}years of active open source
 357 development by a community of passionate developers. LTTng is currently
 358 available on major desktop and server Linux distributions.
 359
 360 The main interface for tracing control is a single command-line tool
 361 named cmd:lttng. The latter can create several tracing sessions, enable
 362 and disable events on the fly, filter events efficiently with custom
 363 user expressions, start and stop tracing, and much more. LTTng can
 364 record the traces on the file system or send them over the network, and
 365 keep them totally or partially. You can view the traces once tracing
 366 becomes inactive or in real-time.
 367
 368 <<installing-lttng,Install LTTng now>> and
 369 <<getting-started,start tracing>>!
 370
 371
 372 [[installing-lttng]]
 373 == Installation
 374
 375 **LTTng** is a set of software <<plumbing,components>> which interact to
 376 <<instrumenting,instrument>> the Linux kernel and user applications, and
 377 to <<controlling-tracing,control tracing>> (start and stop
 378 tracing, enable and disable event rules, and the rest). Those
 379 components are bundled into the following packages:
 380
 381 LTTng-tools::
 382     Libraries and command-line interface to control tracing.
 383
 384 LTTng-modules::
 385     Linux kernel modules to instrument and trace the kernel.
 386
 387 LTTng-UST::
 388     Libraries and Java/Python packages to instrument and trace user
 389     applications.
 390
 391 Most distributions mark the LTTng-modules and LTTng-UST packages as
 392 optional when installing LTTng-tools (which is always required). In the
 393 following sections, we always provide the steps to install all three,
 394 but note that:
 395
 396 * You only need to install LTTng-modules if you intend to trace the
 397   Linux kernel.
 398 * You only need to install LTTng-UST if you intend to trace user
 399   applications.
 400
 401 [role="growable"]
 402 .Availability of LTTng{nbsp}{revision} for major Linux distributions as of 3{nbsp}November{nbsp}2023.
 403 |====
 404 |Distribution |Available in releases
 405
 406 |https://www.ubuntu.com/[Ubuntu]
 407 |Ubuntu{nbsp}16.04 _Xenial Xerus_, Ubuntu{nbsp}18.04 _Bionic Beaver_,
 408 Ubuntu{nbsp}20.04 _Focal Fossa_, and Ubuntu{nbsp}22.04 _Jammy Jellyfish_:
 409 <<ubuntu-ppa,use the LTTng Stable{nbsp}{revision} PPA>>.
 410
 411 |https://www.debian.org/[Debian]
 412 |<<debian,Debian{nbsp}11 _bullseye_>>.
 413
 414 |https://www.redhat.com/[RHEL] and https://www.suse.com/[SLES]
 415 |See http://packages.efficios.com/[EfficiOS Enterprise Packages].
 416
 417 |https://alpinelinux.org/[Alpine Linux]
 418 |xref:alpine-linux[Alpine Linux{nbsp}3.12, Alpine Linux{nbsp}3.13,
 419 Alpine Linux{nbsp}3.14, and Alpine Linux{nbsp}3.15].
 420
 421 |https://buildroot.org/[Buildroot]
 422 |xref:buildroot[Buildroot{nbsp}2020.08, Buildroot{nbsp}2020.11,
 423 Builroot{nbsp}2021.02, Buildroot{nbsp}2021.05, Buildroot{nbsp}2021.08,
 424 and Builroot{nbsp}2021.11].
 425
 426 |https://www.openembedded.org/wiki/Main_Page[OpenEmbedded] and
 427 https://www.yoctoproject.org/[Yocto]
 428 |xref:oe-yocto[Yocto Project{nbsp}3.2 _Gatesgarth_ and Yocto Project{nbsp}3.3 _Hardknott_].
 429 |====
 430
 431
 432 [[ubuntu-ppa]]
 433 === Ubuntu: noch:{LTTng} Stable {revision} PPA
 434
 435 The https://launchpad.net/~lttng/+archive/ubuntu/stable-{revision}[LTTng
 436 Stable{nbsp}{revision} PPA] offers the latest stable
 437 LTTng{nbsp}{revision} packages for Ubuntu{nbsp}16.04 _Xenial Xerus_,
 438 Ubuntu{nbsp}18.04 _Bionic Beaver_, Ubuntu{nbsp}20.04 _Focal Fossa_,
 439 and Ubuntu{nbsp}22.04 _Jammy Jellyfish_.
 440
 441 To install LTTng{nbsp}{revision} from the LTTng Stable{nbsp}{revision}
 442 PPA:
 443
 444 . Add the LTTng Stable{nbsp}{revision} PPA repository and update the
 445   list of packages:
 446 +
 447 --
 448 [role="term"]
 449 ----
 450 # apt-add-repository ppa:lttng/stable-2.12
 451 # apt-get update
 452 ----
 453 --
 454
 455 . Install the main LTTng{nbsp}{revision} packages:
 456 +
 457 --
 458 [role="term"]
 459 ----
 460 # apt-get install lttng-tools
 461 # apt-get install lttng-modules-dkms
 462 # apt-get install liblttng-ust-dev
 463 ----
 464 --
 465
 466 . **If you need to instrument and trace
 467   <<java-application,Java applications>>**, install the LTTng-UST
 468   Java agent:
 469 +
 470 --
 471 [role="term"]
 472 ----
 473 # apt-get install liblttng-ust-agent-java
 474 ----
 475 --
 476
 477 . **If you need to instrument and trace
 478   <<python-application,Python{nbsp}3 applications>>**, install the
 479   LTTng-UST Python agent:
 480 +
 481 --
 482 [role="term"]
 483 ----
 484 # apt-get install python3-lttngust
 485 ----
 486 --
 487
 488
 489 [[debian]]
 490 === Debian
 491
 492 To install LTTng{nbsp}{revision} on Debian{nbsp}11 _bullseye_:
 493
 494 . Install the main LTTng{nbsp}{revision} packages:
 495 +
 496 --
 497 [role="term"]
 498 ----
 499 # apt-get install lttng-modules-dkms
 500 # apt-get install liblttng-ust-dev
 501 # apt-get install lttng-tools
 502 ----
 503 --
 504
 505 . **If you need to instrument and trace <<java-application,Java
 506   applications>>**, install the LTTng-UST Java agent:
 507 +
 508 --
 509 [role="term"]
 510 ----
 511 # apt-get install liblttng-ust-agent-java
 512 ----
 513 --
 514
 515 . **If you need to instrument and trace <<python-application,Python
 516   applications>>**, install the LTTng-UST Python agent:
 517 +
 518 --
 519 [role="term"]
 520 ----
 521 # apt-get install python3-lttngust
 522 ----
 523 --
 524
 525 [[alpine-linux]]
 526 === Alpine Linux
 527
 528 To install LTTng-tools{nbsp}{revision} and LTTng-UST{nbsp}{revision} on
 529 Alpine Linux{nbsp}3.12, Alpine Linux{nbsp}3.13, Alpine Linux{nbsp}3.14,
 530 or Alpine Linux{nbsp}3.15:
 531
 532 . Add the LTTng packages:
 533 +
 534 --
 535 [role="term"]
 536 ----
 537 # apk add lttng-tools
 538 # apk add lttng-ust-dev
 539 ----
 540 --
 541
 542 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 543 +
 544 --
 545 [role="term"]
 546 ----
 547 $ cd $(mktemp -d) &&
 548 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
 549 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
 550 cd lttng-modules-2.12.* &&
 551 make &&
 552 sudo make modules_install &&
 553 sudo depmod -a
 554 ----
 555 --
 556
 557
 558 [[buildroot]]
 559 === Buildroot
 560
 561 To install LTTng{nbsp}{revision} on Buildroot{nbsp}2020.08, Buildroot{nbsp}2020.11,
 562 Buildroot{nbsp}2021.02, Buildroot{nbsp}2021.05, Buildroot{nbsp}2021.08, or
 563 Buildroot{nbsp}2021.11:
 564
 565 . Launch the Buildroot configuration tool:
 566 +
 567 --
 568 [role="term"]
 569 ----
 570 $ make menuconfig
 571 ----
 572 --
 573
 574 . In **Kernel**, check **Linux kernel**.
 575 . In **Toolchain**, check **Enable WCHAR support**.
 576 . In **Target packages**{nbsp}&#8594; **Debugging, profiling and benchmark**,
 577   check **lttng-modules** and **lttng-tools**.
 578 . In **Target packages**{nbsp}&#8594; **Libraries**{nbsp}&#8594;
 579   **Other**, check **lttng-libust**.
 580
 581
 582 [[oe-yocto]]
 583 === OpenEmbedded and Yocto
 584
 585 LTTng{nbsp}{revision} recipes are available in the
 586 https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/[`openembedded-core`]
 587 layer for Yocto Project{nbsp}3.2 _Gatesgarth_ and Yocto Project{nbsp}3.3 _Hardknott_
 588 under the following names:
 589
 590 * `lttng-tools`
 591 * `lttng-modules`
 592 * `lttng-ust`
 593
 594 With BitBake, the simplest way to include LTTng recipes in your target
 595 image is to add them to `IMAGE_INSTALL_append` in path:{conf/local.conf}:
 596
 597 ----
 598 IMAGE_INSTALL_append = " lttng-tools lttng-modules lttng-ust"
 599 ----
 600
 601 If you use Hob:
 602
 603 . Select a machine and an image recipe.
 604 . Click **Edit image recipe**.
 605 . Under the **All recipes** tab, search for **lttng**.
 606 . Check the desired LTTng recipes.
 607
 608
 609 [[building-from-source]]
 610 === Build from source
 611
 612 To build and install LTTng{nbsp}{revision} from source:
 613
 614 . Using the package manager of your distribution, or from source,
 615   install the following dependencies of LTTng-tools and LTTng-UST:
 616 +
 617 --
 618 * https://sourceforge.net/projects/libuuid/[libuuid]
 619 * http://directory.fsf.org/wiki/Popt[popt]
 620 * http://liburcu.org/[Userspace RCU]
 621 * http://www.xmlsoft.org/[libxml2]
 622 * **Optional**: https://github.com/numactl/numactl[numactl]
 623 --
 624
 625 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 626 +
 627 --
 628 [role="term"]
 629 ----
 630 $ cd $(mktemp -d) &&
 631 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
 632 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
 633 cd lttng-modules-2.12.* &&
 634 make &&
 635 sudo make modules_install &&
 636 sudo depmod -a
 637 ----
 638 --
 639
 640 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
 641 +
 642 --
 643 [role="term"]
 644 ----
 645 $ cd $(mktemp -d) &&
 646 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
 647 tar -xf lttng-ust-latest-2.12.tar.bz2 &&
 648 cd lttng-ust-2.12.* &&
 649 ./configure &&
 650 make &&
 651 sudo make install &&
 652 sudo ldconfig
 653 ----
 654 --
 655 +
 656 Add `--disable-numa` to `./configure` if you don't have
 657 https://github.com/numactl/numactl[numactl].
 658 +
 659 --
 660 [IMPORTANT]
 661 .Java and Python application tracing
 662 ====
 663 If you need to instrument and trace <<java-application,Java
 664 applications>>, pass the `--enable-java-agent-jul`,
 665 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
 666 `configure` script, depending on which Java logging framework you use.
 667
 668 If you need to instrument and trace <<python-application,Python
 669 applications>>, pass the `--enable-python-agent` option to the
 670 `configure` script. You can set the `PYTHON` environment variable to the
 671 path to the Python interpreter for which to install the LTTng-UST Python
 672 agent package.
 673 ====
 674 --
 675 +
 676 --
 677 [NOTE]
 678 ====
 679 By default, LTTng-UST libraries are installed to
 680 dir:{/usr/local/lib}, which is the de facto directory in which to
 681 keep self-compiled and third-party libraries.
 682
 683 When <<building-tracepoint-providers-and-user-application,linking an
 684 instrumented user application with `liblttng-ust`>>:
 685
 686 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
 687   variable.
 688 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
 689   man:gcc(1), man:g++(1), or man:clang(1).
 690 ====
 691 --
 692
 693 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
 694 +
 695 --
 696 [role="term"]
 697 ----
 698 $ cd $(mktemp -d) &&
 699 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
 700 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
 701 cd lttng-tools-2.12.* &&
 702 ./configure &&
 703 make &&
 704 sudo make install &&
 705 sudo ldconfig
 706 ----
 707 --
 708
 709 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
 710 previous steps automatically for a given version of LTTng and confine
 711 the installed files in a specific directory. This can be useful to test
 712 LTTng without installing it on your system.
 713
 714
 715 [[linux-kernel-sig]]
 716 === Linux kernel module signature
 717
 718 Linux kernel modules require trusted signatures in order to be loaded
 719 when any of the following is true:
 720
 721 * The system boots with
 722   https://uefi.org/specs/UEFI/2.10/32_Secure_Boot_and_Driver_Signing.html#secure-boot-and-driver-signing[Secure Boot]
 723   enabled.
 724
 725 * The Linux kernel which boots is configured with
 726   `CONFIG_MODULE_SIG_FORCE`.
 727
 728 * The Linux kernel boots with a command line containing
 729   `module.sig_enforce=1`.
 730
 731 .`root` user running <<lttng-sessiond,`lttng-sessiond`>> which fails to load a required <<lttng-modules,kernel module>> due to the signature enforcement policies.
 732 ====
 733 [role="term"]
 734 ----
 735 # lttng-sessiond
 736 Warning: No tracing group detected
 737 modprobe: ERROR: could not insert 'lttng_ring_buffer_client_discard': Key was rejected by service
 738 Error: Unable to load required module lttng-ring-buffer-client-discard
 739 Warning: No kernel tracer available
 740 ----
 741 ====
 742
 743 There are several methods to enroll trusted keys for signing modules
 744 that are built from source. The precise details vary from one Linux
 745 version to another, and distributions may have their own mechanisms. For
 746 example, https://github.com/dell/dkms[DKMS] may autogenerate a key and
 747 sign modules, but the key isn't automatically enrolled.
 748
 749 See
 750 https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html[Kernel
 751 module signing facility] and the documentation of your distribution
 752 to learn more about signing Linux kernel modules.
 753
 754 [[getting-started]]
 755 == Quick start
 756
 757 This is a short guide to get started quickly with LTTng kernel and user
 758 space tracing.
 759
 760 Before you follow this guide, make sure to <<installing-lttng,install>>
 761 LTTng.
 762
 763 This tutorial walks you through the steps to:
 764
 765 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
 766 . <<tracing-your-own-user-application,Trace a user application>> written
 767   in C.
 768 . <<viewing-and-analyzing-your-traces,View and analyze the
 769   recorded events>>.
 770
 771
 772 [[tracing-the-linux-kernel]]
 773 === Trace the Linux kernel
 774
 775 The following command lines start with the `#` prompt because you need
 776 root privileges to trace the Linux kernel. You can also trace the kernel
 777 as a regular user if your Unix user is a member of the
 778 <<tracing-group,tracing group>>.
 779
 780 . Create a <<tracing-session,tracing session>> which writes its traces
 781   to dir:{/tmp/my-kernel-trace}:
 782 +
 783 --
 784 [role="term"]
 785 ----
 786 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
 787 ----
 788 --
 789
 790 . List the available kernel tracepoints and system calls:
 791 +
 792 --
 793 [role="term"]
 794 ----
 795 # lttng list --kernel
 796 # lttng list --kernel --syscall
 797 ----
 798 --
 799
 800 . Create <<event,event rules>> which match the desired instrumentation
 801   point names, for example the `sched_switch` and `sched_process_fork`
 802   tracepoints, and the man:open(2) and man:close(2) system calls:
 803 +
 804 --
 805 [role="term"]
 806 ----
 807 # lttng enable-event --kernel sched_switch,sched_process_fork
 808 # lttng enable-event --kernel --syscall open,close
 809 ----
 810 --
 811 +
 812 Create an event rule which matches _all_ the Linux kernel
 813 tracepoints with the opt:lttng-enable-event(1):--all option
 814 (this will generate a lot of data when tracing):
 815 +
 816 --
 817 [role="term"]
 818 ----
 819 # lttng enable-event --kernel --all
 820 ----
 821 --
 822
 823 . <<basic-tracing-session-control,Start tracing>>:
 824 +
 825 --
 826 [role="term"]
 827 ----
 828 # lttng start
 829 ----
 830 --
 831
 832 . Do some operation on your system for a few seconds. For example,
 833   load a website, or list the files of a directory.
 834 . <<creating-destroying-tracing-sessions,Destroy>> the current
 835   tracing session:
 836 +
 837 --
 838 [role="term"]
 839 ----
 840 # lttng destroy
 841 ----
 842 --
 843 +
 844 The man:lttng-destroy(1) command doesn't destroy the trace data; it
 845 only destroys the state of the tracing session.
 846 +
 847 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
 848 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
 849 session>>). You need to stop tracing to make LTTng flush the remaining
 850 trace data and make the trace readable.
 851
 852 . For the sake of this example, make the recorded trace accessible to
 853   the non-root users:
 854 +
 855 --
 856 [role="term"]
 857 ----
 858 # chown -R $(whoami) /tmp/my-kernel-trace
 859 ----
 860 --
 861
 862 See <<viewing-and-analyzing-your-traces,View and analyze the
 863 recorded events>> to view the recorded events.
 864
 865
 866 [[tracing-your-own-user-application]]
 867 === Trace a user application
 868
 869 This section steps you through a simple example to trace a
 870 _Hello world_ program written in C.
 871
 872 To create the traceable user application:
 873
 874 . Create the tracepoint provider header file, which defines the
 875   tracepoints and the events they can generate:
 876 +
 877 --
 878 [source,c]
 879 .path:{hello-tp.h}
 880 ----
 881 #undef TRACEPOINT_PROVIDER
 882 #define TRACEPOINT_PROVIDER hello_world
 883
 884 #undef TRACEPOINT_INCLUDE
 885 #define TRACEPOINT_INCLUDE "./hello-tp.h"
 886
 887 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
 888 #define _HELLO_TP_H
 889
 890 #include <lttng/tracepoint.h>
 891
 892 TRACEPOINT_EVENT(
 893     hello_world,
 894     my_first_tracepoint,
 895     TP_ARGS(
 896         int, my_integer_arg,
 897         char*, my_string_arg
 898     ),
 899     TP_FIELDS(
 900         ctf_string(my_string_field, my_string_arg)
 901         ctf_integer(int, my_integer_field, my_integer_arg)
 902     )
 903 )
 904
 905 #endif /* _HELLO_TP_H */
 906
 907 #include <lttng/tracepoint-event.h>
 908 ----
 909 --
 910
 911 . Create the tracepoint provider package source file:
 912 +
 913 --
 914 [source,c]
 915 .path:{hello-tp.c}
 916 ----
 917 #define TRACEPOINT_CREATE_PROBES
 918 #define TRACEPOINT_DEFINE
 919
 920 #include "hello-tp.h"
 921 ----
 922 --
 923
 924 . Build the tracepoint provider package:
 925 +
 926 --
 927 [role="term"]
 928 ----
 929 $ gcc -c -I. hello-tp.c
 930 ----
 931 --
 932
 933 . Create the _Hello World_ application source file:
 934 +
 935 --
 936 [source,c]
 937 .path:{hello.c}
 938 ----
 939 #include <stdio.h>
 940 #include "hello-tp.h"
 941
 942 int main(int argc, char *argv[])
 943 {
 944     int x;
 945
 946     puts("Hello, World!\nPress Enter to continue...");
 947
 948     /*
 949      * The following getchar() call is only placed here for the purpose
 950      * of this demonstration, to pause the application in order for
 951      * you to have time to list its tracepoints. It's not needed
 952      * otherwise.
 953      */
 954     getchar();
 955
 956     /*
 957      * A tracepoint() call.
 958      *
 959      * Arguments, as defined in hello-tp.h:
 960      *
 961      * 1. Tracepoint provider name   (required)
 962      * 2. Tracepoint name            (required)
 963      * 3. my_integer_arg             (first user-defined argument)
 964      * 4. my_string_arg              (second user-defined argument)
 965      *
 966      * Notice the tracepoint provider and tracepoint names are
 967      * NOT strings: they are in fact parts of variables that the
 968      * macros in hello-tp.h create.
 969      */
 970     tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
 971
 972     for (x = 0; x < argc; ++x) {
 973         tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
 974     }
 975
 976     puts("Quitting now!");
 977     tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
 978
 979     return 0;
 980 }
 981 ----
 982 --
 983
 984 . Build the application:
 985 +
 986 --
 987 [role="term"]
 988 ----
 989 $ gcc -c hello.c
 990 ----
 991 --
 992
 993 . Link the application with the tracepoint provider package,
 994   `liblttng-ust`, and `libdl`:
 995 +
 996 --
 997 [role="term"]
 998 ----
 999 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
1000 ----
1001 --
1002
1003 Here's the whole build process:
1004
1005 [role="img-100"]
1006 .Build steps of the user space tracing tutorial.
1007 image::ust-flow.png[]
1008
1009 To trace the user application:
1010
1011 . Run the application with a few arguments:
1012 +
1013 --
1014 [role="term"]
1015 ----
1016 $ ./hello world and beyond
1017 ----
1018 --
1019 +
1020 You see:
1021 +
1022 --
1023 ----
1024 Hello, World!
1025 Press Enter to continue...
1026 ----
1027 --
1028
1029 . Start an LTTng <<lttng-sessiond,session daemon>>:
1030 +
1031 --
1032 [role="term"]
1033 ----
1034 $ lttng-sessiond --daemonize
1035 ----
1036 --
1037 +
1038 Note that a session daemon might already be running, for example as a
1039 service that the service manager of the distribution started.
1040
1041 . List the available user space tracepoints:
1042 +
1043 --
1044 [role="term"]
1045 ----
1046 $ lttng list --userspace
1047 ----
1048 --
1049 +
1050 You see the `hello_world:my_first_tracepoint` tracepoint listed
1051 under the `./hello` process.
1052
1053 . Create a <<tracing-session,tracing session>>:
1054 +
1055 --
1056 [role="term"]
1057 ----
1058 $ lttng create my-user-space-session
1059 ----
1060 --
1061
1062 . Create an <<event,event rule>> which matches the
1063   `hello_world:my_first_tracepoint` event name:
1064 +
1065 --
1066 [role="term"]
1067 ----
1068 $ lttng enable-event --userspace hello_world:my_first_tracepoint
1069 ----
1070 --
1071
1072 . <<basic-tracing-session-control,Start tracing>>:
1073 +
1074 --
1075 [role="term"]
1076 ----
1077 $ lttng start
1078 ----
1079 --
1080
1081 . Go back to the running `hello` application and press Enter. The
1082   program executes all `tracepoint()` instrumentation points and exits.
1083 . <<creating-destroying-tracing-sessions,Destroy>> the current
1084   tracing session:
1085 +
1086 --
1087 [role="term"]
1088 ----
1089 $ lttng destroy
1090 ----
1091 --
1092 +
1093 The man:lttng-destroy(1) command doesn't destroy the trace data; it
1094 only destroys the state of the tracing session.
1095 +
1096 The man:lttng-destroy(1) command also runs the man:lttng-stop(1) command
1097 implicitly (see <<basic-tracing-session-control,Start and stop a tracing
1098 session>>). You need to stop tracing to make LTTng flush the remaining
1099 trace data and make the trace readable.
1100
1101 By default, LTTng saves the traces in
1102 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
1103 where +__name__+ is the tracing session name. The
1104 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
1105
1106 See <<viewing-and-analyzing-your-traces,View and analyze the
1107 recorded events>> to view the recorded events.
1108
1109
1110 [[viewing-and-analyzing-your-traces]]
1111 === View and analyze the recorded events
1112
1113 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
1114 kernel>> and <<tracing-your-own-user-application,Trace a user
1115 application>> tutorials, you can inspect the recorded events.
1116
1117 There are tools you can use to read LTTng traces:
1118
1119 https://babeltrace.org/[Babeltrace{nbsp}2]::
1120     A rich, flexible trace manipulation toolkit which includes
1121     a versatile command-line interface
1122     (https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]),
1123     a https://babeltrace.org/docs/v2.0/libbabeltrace2/[C library],
1124     and https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1125     so that you can easily process or convert an LTTng trace with
1126     your own script.
1127 +
1128 The Babeltrace{nbsp}2 project ships with a
1129 https://babeltrace.org/docs/v2.0/man7/babeltrace2-plugin-ctf.7/[plugin]
1130 which supports the format of the traces which LTTng produces,
1131 https://diamon.org/ctf/[CTF].
1132
1133 http://tracecompass.org/[Trace Compass]::
1134     A graphical user interface for viewing and analyzing any type of
1135     logs or traces, including those of LTTng.
1136
1137 NOTE: This section assumes that LTTng saved the traces it recorded
1138 during the previous tutorials to their default location, in the
1139 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
1140 environment variable defaults to `$HOME` if not set.
1141
1142
1143 [[viewing-and-analyzing-your-traces-bt]]
1144 ==== Use the cmd:babeltrace2 command-line tool
1145
1146 The simplest way to list all the recorded events of an LTTng trace is to
1147 pass its path to
1148 https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
1149 without options:
1150
1151 [role="term"]
1152 ----
1153 $ babeltrace2 ~/lttng-traces/my-user-space-session*
1154 ----
1155
1156 cmd:babeltrace2 finds all traces recursively within the given path and
1157 prints all their events, sorting them chronologically.
1158
1159 Pipe the output of cmd:babeltrace2 into a tool like man:grep(1) for
1160 further filtering:
1161
1162 [role="term"]
1163 ----
1164 $ babeltrace2 /tmp/my-kernel-trace | grep _switch
1165 ----
1166
1167 Pipe the output of cmd:babeltrace2 into a tool like man:wc(1) to count
1168 the recorded events:
1169
1170 [role="term"]
1171 ----
1172 $ babeltrace2 /tmp/my-kernel-trace | grep _open | wc --lines
1173 ----
1174
1175
1176 [[viewing-and-analyzing-your-traces-bt-python]]
1177 ==== Use the Babeltrace{nbsp}2 Python bindings
1178
1179 The <<viewing-and-analyzing-your-traces-bt,text output of
1180 cmd:babeltrace2>> is useful to isolate events by simple matching using
1181 man:grep(1) and similar utilities. However, more elaborate filters, such
1182 as keeping only event records with a field value falling within a
1183 specific range, are not trivial to write using a shell. Moreover,
1184 reductions and even the most basic computations involving multiple event
1185 records are virtually impossible to implement.
1186
1187 Fortunately, Babeltrace{nbsp}2 ships with
1188 https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings]
1189 which make it easy to read the event records of an LTTng trace
1190 sequentially and compute the desired information.
1191
1192 The following script accepts an LTTng Linux kernel trace path as its
1193 first argument and prints the short names of the top five running
1194 processes on CPU{nbsp}0 during the whole trace:
1195
1196 [source,python]
1197 .path:{top5proc.py}
1198 ----
1199 import bt2
1200 import sys
1201 import collections
1202
1203 def top5proc():
1204     # Get the trace path from the first command-line argument.
1205     it = bt2.TraceCollectionMessageIterator(sys.argv[1])
1206
1207     # This counter dictionary will hold execution times:
1208     #
1209     #     Task command name -> Total execution time (ns)
1210     exec_times = collections.Counter()
1211
1212     # This holds the last `sched_switch` timestamp.
1213     last_ts = None
1214
1215     for msg in it:
1216         # We only care about event messages.
1217         if type(msg) is not bt2._EventMessageConst:
1218             continue
1219
1220         # Event of the event message.
1221         event = msg.event
1222
1223         # Keep only `sched_switch` events.
1224         if event.cls.name != 'sched_switch':
1225             continue
1226
1227         # Keep only events which occurred on CPU 0.
1228         if event.packet.context_field['cpu_id'] != 0:
1229             continue
1230
1231         # Event timestamp (ns).
1232         cur_ts = msg.default_clock_snapshot.ns_from_origin
1233
1234         if last_ts is None:
1235             # We start here.
1236             last_ts = cur_ts
1237
1238         # (Short) name of the previous task command.
1239         prev_comm = str(event.payload_field['prev_comm'])
1240
1241         # Initialize an entry in our dictionary if not yet done.
1242         if prev_comm not in exec_times:
1243             exec_times[prev_comm] = 0
1244
1245         # Compute previous command execution time.
1246         diff = cur_ts - last_ts
1247
1248         # Update execution time of this command.
1249         exec_times[prev_comm] += diff
1250
1251         # Update last timestamp.
1252         last_ts = cur_ts
1253
1254     # Print top 5.
1255     for name, ns in exec_times.most_common(5):
1256         print('{:20}{} s'.format(name, ns / 1e9))
1257
1258
1259 if __name__ == '__main__':
1260     top5proc()
1261 ----
1262
1263 Run this script:
1264
1265 [role="term"]
1266 ----
1267 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1268 ----
1269
1270 Output example:
1271
1272 ----
1273 swapper/0           48.607245889 s
1274 chromium            7.192738188 s
1275 pavucontrol         0.709894415 s
1276 Compositor          0.660867933 s
1277 Xorg.bin            0.616753786 s
1278 ----
1279
1280 Note that `swapper/0` is the ``idle'' process of CPU{nbsp}0 on Linux;
1281 since we weren't using the CPU that much when tracing, its first
1282 position in the list makes sense.
1283
1284
1285 [[core-concepts]]
1286 == [[understanding-lttng]]Core concepts
1287
1288 From a user's perspective, the LTTng system is built on a few concepts,
1289 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1290 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1291 Understanding how those objects relate to eachother is key in mastering
1292 the toolkit.
1293
1294 The core concepts are:
1295
1296 * <<tracing-session,Tracing session>>
1297 * <<domain,Tracing domain>>
1298 * <<channel,Channel and ring buffer>>
1299 * <<"event","Instrumentation point, event rule, event, and event record">>
1300
1301
1302 [[tracing-session]]
1303 === Tracing session
1304
1305 A _tracing session_ is a stateful dialogue between you and
1306 a <<lttng-sessiond,session daemon>>. You can
1307 <<creating-destroying-tracing-sessions,create a new tracing
1308 session>> with the `lttng create` command.
1309
1310 Most of what you do when you control LTTng tracers happens within a
1311 tracing session. In particular, a tracing session:
1312
1313 * Has its own name.
1314 * Has its own set of trace files.
1315 * Has its own state of activity (started or stopped).
1316 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1317   snapshot, or live).
1318 * Has its own <<channel,channels>> to which are associated their own
1319   <<event,event rules>>.
1320 * Has its own <<pid-tracking,process attribute tracking>> inclusion
1321   sets.
1322
1323 [role="img-100"]
1324 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1325 image::concepts.png[]
1326
1327 Those attributes and objects are completely isolated between different
1328 tracing sessions.
1329
1330 A tracing session is analogous to a cash machine session:
1331 the operations you do on the banking system through the cash machine do
1332 not alter the data of other users of the same system. In the case of
1333 the cash machine, a session lasts as long as your bank card is inside.
1334 In the case of LTTng, a tracing session lasts from the `lttng create`
1335 command to the `lttng destroy` command.
1336
1337 [role="img-100"]
1338 .Each Unix user has its own set of tracing sessions.
1339 image::many-sessions.png[]
1340
1341
1342 [[tracing-session-mode]]
1343 ==== Tracing session mode
1344
1345 LTTng can send the generated trace data to different locations. The
1346 _tracing session mode_ dictates where to send it. The following modes
1347 are available in LTTng{nbsp}{revision}:
1348
1349 [[local-mode]]Local mode::
1350   LTTng writes the traces to the file system of the machine it traces
1351   (target system).
1352
1353 [[net-streaming-mode]]Network streaming mode::
1354   LTTng sends the traces over the network to a
1355   <<lttng-relayd,relay daemon>> running on a remote system.
1356
1357 Snapshot mode::
1358   LTTng doesn't write the traces by default.
1359 +
1360 Instead, you can request LTTng to <<taking-a-snapshot,take a snapshot>>,
1361 that is, a copy of the current sub-buffers of the tracing session, and
1362 to write it to the file system of the target or to send it over the
1363 network to a <<lttng-relayd,relay daemon>> running on a remote system.
1364
1365 [[live-mode]]Live mode::
1366   This mode is similar to the network streaming mode, but a live
1367   trace viewer can connect to the distant relay daemon to
1368   <<lttng-live,view event records as LTTng generates them>>.
1369
1370
1371 [[domain]]
1372 === Tracing domain
1373
1374 A _tracing domain_ is a namespace for event sources. A tracing domain
1375 has its own properties and features.
1376
1377 There are currently five available tracing domains:
1378
1379 * Linux kernel
1380 * User space
1381 * `java.util.logging` (JUL)
1382 * log4j
1383 * Python
1384
1385 You must specify a tracing domain when using some commands to avoid
1386 ambiguity. For example, since all the domains support named tracepoints
1387 as event sources (instrumentation points that you manually insert in the
1388 source code), you need to specify a tracing domain when
1389 <<enabling-disabling-events,creating an event rule>> because all the
1390 tracing domains could have tracepoints with the same names.
1391
1392 You can create <<channel,channels>> in the Linux kernel and user space
1393 tracing domains. The other tracing domains have a single default
1394 channel.
1395
1396
1397 [[channel]]
1398 === Channel and ring buffer
1399
1400 A _channel_ is an object which is responsible for a set of ring buffers.
1401 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1402 tracer emits an event, it can record it to one or more
1403 sub-buffers. The attributes of a channel determine what to do when
1404 there's no space left for a new event record because all sub-buffers
1405 are full, where to send a full sub-buffer, and other behaviours.
1406
1407 A channel is always associated to a <<domain,tracing domain>>. The
1408 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1409 a default channel which you can't configure.
1410
1411 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1412 an event, it records it to the sub-buffers of all
1413 the enabled channels with a satisfied event rule, as long as those
1414 channels are part of active <<tracing-session,tracing sessions>>.
1415
1416
1417 [[channel-buffering-schemes]]
1418 ==== Per-user vs. per-process buffering schemes
1419
1420 A channel has at least one ring buffer _per CPU_. LTTng always
1421 records an event to the ring buffer associated to the CPU on which it
1422 occurs.
1423
1424 Two _buffering schemes_ are available when you
1425 <<enabling-disabling-channels,create a channel>> in the
1426 user space <<domain,tracing domain>>:
1427
1428 Per-user buffering::
1429   Allocate one set of ring buffers--one per CPU--shared by all the
1430   instrumented processes of each Unix user.
1431 +
1432 --
1433 [role="img-100"]
1434 .Per-user buffering scheme.
1435 image::per-user-buffering.png[]
1436 --
1437
1438 Per-process buffering::
1439   Allocate one set of ring buffers--one per CPU--for each
1440   instrumented process.
1441 +
1442 --
1443 [role="img-100"]
1444 .Per-process buffering scheme.
1445 image::per-process-buffering.png[]
1446 --
1447 +
1448 The per-process buffering scheme tends to consume more memory than the
1449 per-user option because systems generally have more instrumented
1450 processes than Unix users running instrumented processes. However, the
1451 per-process buffering scheme ensures that one process having a high
1452 event throughput won't fill all the shared sub-buffers of the same
1453 user, only its own.
1454
1455 The Linux kernel tracing domain has only one available buffering scheme
1456 which is to allocate a single set of ring buffers for the whole system.
1457 This scheme is similar to the per-user option, but with a single, global
1458 user ``running'' the kernel.
1459
1460
1461 [[channel-overwrite-mode-vs-discard-mode]]
1462 ==== Overwrite vs. discard event record loss modes
1463
1464 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1465 arc in the following animations) of the ring buffer of a specific
1466 channel. When there's no space left in a sub-buffer, the tracer marks it
1467 as consumable (red) and another, empty sub-buffer starts receiving the
1468 following event records. A <<lttng-consumerd,consumer daemon>>
1469 eventually consumes the marked sub-buffer (returns to white).
1470
1471 [NOTE]
1472 [role="docsvg-channel-subbuf-anim"]
1473 ====
1474 {note-no-anim}
1475 ====
1476
1477 In an ideal world, sub-buffers are consumed faster than they are filled,
1478 as it is the case in the previous animation. In the real world,
1479 however, all sub-buffers can be full at some point, leaving no space to
1480 record the following events.
1481
1482 By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when
1483 no empty sub-buffer is available, it is acceptable to lose event records
1484 when the alternative would be to cause substantial delays in the
1485 execution of the instrumented application. LTTng privileges performance
1486 over integrity; it aims at perturbing the target system as little as
1487 possible in order to make tracing of subtle race conditions and rare
1488 interrupt cascades possible.
1489
1490 Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports
1491 a _blocking mode_. See the <<blocking-timeout-example,blocking timeout
1492 example>> to learn how to use the blocking mode.
1493
1494 When it comes to losing event records because no empty sub-buffer is
1495 available, or because the <<opt-blocking-timeout,blocking timeout>> is
1496 reached, the _event record loss mode_ of the channel determines what to
1497 do. The available event record loss modes are:
1498
1499 Discard mode::
1500   Drop the newest event records until the tracer releases a sub-buffer.
1501 +
1502 This is the only available mode when you specify a
1503 <<opt-blocking-timeout,blocking timeout>>.
1504
1505 Overwrite mode::
1506   Clear the sub-buffer containing the oldest event records and start
1507   writing the newest event records there.
1508 +
1509 This mode is sometimes called _flight recorder mode_ because it's
1510 similar to a
1511 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1512 always keep a fixed amount of the latest data.
1513
1514 Which mechanism you should choose depends on your context: prioritize
1515 the newest or the oldest event records in the ring buffer?
1516
1517 Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_
1518 as soon as a there's no space left for a new event record, whereas in
1519 discard mode, the tracer only discards the event record that doesn't
1520 fit.
1521
1522 In discard mode, LTTng increments a count of lost event records when an
1523 event record is lost and saves this count to the trace. Since
1524 LTTng{nbsp}2.8, in overwrite mode, LTTng writes to a given sub-buffer
1525 its sequence number within its data stream. With a <<local-mode,local>>,
1526 <<net-streaming-mode,network streaming>>, or <<live-mode,live>>
1527 <<tracing-session,tracing session>>, a trace reader can use such
1528 sequence numbers to report lost packets. In overwrite mode, LTTng
1529 doesn't write to the trace the exact number of lost event records in
1530 those lost sub-buffers.
1531
1532 Trace analyses can use saved discarded event record and sub-buffer
1533 (packet) counts of the trace to decide whether or not to perform the
1534 analyses even if trace data is known to be missing.
1535
1536 There are a few ways to decrease your probability of losing event
1537 records.
1538 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1539 how to fine-tune the sub-buffer count and size of a channel to virtually
1540 stop losing event records, though at the cost of greater memory usage.
1541
1542
1543 [[channel-subbuf-size-vs-subbuf-count]]
1544 ==== Sub-buffer count and size
1545
1546 When you <<enabling-disabling-channels,create a channel>>, you can
1547 set its number of sub-buffers and their size.
1548
1549 Note that there is noticeable CPU overhead introduced when
1550 switching sub-buffers (marking a full one as consumable and switching
1551 to an empty one for the following events to be recorded). Knowing this,
1552 the following list presents a few practical situations along with how
1553 to configure the sub-buffer count and size for them:
1554
1555 * **High event throughput**: In general, prefer bigger sub-buffers to
1556   lower the risk of losing event records.
1557 +
1558 Having bigger sub-buffers also ensures a lower
1559 <<channel-switch-timer,sub-buffer switching frequency>>.
1560 +
1561 The number of sub-buffers is only meaningful if you create the channel
1562 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1563 other sub-buffers are left unaltered.
1564
1565 * **Low event throughput**: In general, prefer smaller sub-buffers
1566   since the risk of losing event records is low.
1567 +
1568 Because events occur less frequently, the sub-buffer switching frequency
1569 should remain low and thus the overhead of the tracer shouldn't be a
1570 problem.
1571
1572 * **Low memory system**: If your target system has a low memory
1573   limit, prefer fewer first, then smaller sub-buffers.
1574 +
1575 Even if the system is limited in memory, you want to keep the
1576 sub-buffers as big as possible to avoid a high sub-buffer switching
1577 frequency.
1578
1579 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1580 which means event data is very compact. For example, the average
1581 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1582 sub-buffer size of 1{nbsp}MiB is considered big.
1583
1584 The previous situations highlight the major trade-off between a few big
1585 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1586 frequency vs. how much data is lost in overwrite mode. Assuming a
1587 constant event throughput and using the overwrite mode, the two
1588 following configurations have the same ring buffer total size:
1589
1590 [NOTE]
1591 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1592 ====
1593 {note-no-anim}
1594 ====
1595
1596 * **Two sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1597   switching frequency, but if a sub-buffer overwrite happens, half of
1598   the event records so far (4{nbsp}MiB) are definitely lost.
1599 * **Eight sub-buffers of 1{nbsp}MiB each**: Expect four times the
1600   overhead of the tracer as the previous configuration, but if a
1601   sub-buffer overwrite happens, only the eighth of event records so far
1602   are definitely lost.
1603
1604 In discard mode, the sub-buffers count parameter is pointless: use two
1605 sub-buffers and set their size according to the requirements of your
1606 situation.
1607
1608
1609 [[channel-switch-timer]]
1610 ==== Switch timer period
1611
1612 The _switch timer period_ is an important configurable attribute of
1613 a channel to ensure periodic sub-buffer flushing.
1614
1615 When the _switch timer_ expires, a sub-buffer switch happens. Set
1616 the switch timer period attribute when you
1617 <<enabling-disabling-channels,create a channel>> to ensure that LTTng
1618 consumes and commits trace data to trace files or to a distant relay
1619 daemon periodically in case of a low event throughput.
1620
1621 [NOTE]
1622 [role="docsvg-channel-switch-timer"]
1623 ====
1624 {note-no-anim}
1625 ====
1626
1627 This attribute is also convenient when you use big sub-buffers to cope
1628 with a sporadic high event throughput, even if the throughput is
1629 normally low.
1630
1631
1632 [[channel-read-timer]]
1633 ==== Read timer period
1634
1635 By default, the LTTng tracers use a notification mechanism to signal a
1636 full sub-buffer so that a consumer daemon can consume it. When such
1637 notifications must be avoided, for example in real-time applications,
1638 use the _read timer_ of the channel instead. When the read timer fires,
1639 the <<lttng-consumerd,consumer daemon>> checks for full, consumable
1640 sub-buffers.
1641
1642
1643 [[tracefile-rotation]]
1644 ==== Trace file count and size
1645
1646 By default, trace files can grow as large as needed. Set the maximum
1647 size of each trace file that a channel writes when you
1648 <<enabling-disabling-channels,create a channel>>. When the size of a
1649 trace file reaches the fixed maximum size of the channel, LTTng creates
1650 another file to contain the next event records. LTTng appends a file
1651 count to each trace file name in this case.
1652
1653 If you set the trace file size attribute when you create a channel, the
1654 maximum number of trace files that LTTng creates is _unlimited_ by
1655 default. To limit them, set a maximum number of trace files. When the
1656 number of trace files reaches the fixed maximum count of the channel,
1657 the oldest trace file is overwritten. This mechanism is called _trace
1658 file rotation_.
1659
1660 [IMPORTANT]
1661 ====
1662 Even if you don't limit the trace file count, you can't assume that
1663 LTTng doesn't manage any trace file.
1664
1665 In other words, there is no safe way to know if LTTng still holds a
1666 given trace file open with the trace file rotation feature.
1667
1668 The only way to obtain an unmanaged, self-contained LTTng trace before
1669 you <<creating-destroying-tracing-sessions,destroy>> the tracing session
1670 is with the <<session-rotation,tracing session rotation>> feature
1671 (available since LTTng{nbsp}2.11).
1672 ====
1673
1674
1675 [[event]]
1676 === Instrumentation point, event rule, event, and event record
1677
1678 An _event rule_ is a set of conditions which must be **all** satisfied
1679 for LTTng to record an occuring event.
1680
1681 You set the conditions when you <<enabling-disabling-events,create
1682 an event rule>>.
1683
1684 You always attach an event rule to a <<channel,channel>> when you create
1685 it.
1686
1687 When an event passes the conditions of an event rule, LTTng records it
1688 in one of the sub-buffers of the attached channel.
1689
1690 The available conditions, as of LTTng{nbsp}{revision}, are:
1691
1692 * The event rule _is enabled_.
1693 * The type of the instrumentation point _is{nbsp}T_.
1694 * The name of the instrumentation point (sometimes called _event name_)
1695   _matches{nbsp}N_, but _isn't{nbsp}E_.
1696 * The log level of the instrumentation point _is as severe as{nbsp}L_, or
1697   _is exactly{nbsp}L_.
1698 * The fields of the payload of the event _satisfy_ a filter
1699   expression{nbsp}__F__.
1700
1701 As you can see, all the conditions but the dynamic filter are related to
1702 the status of the event rule or to the instrumentation point, not to the
1703 occurring events. This is why, without a filter, checking if an event
1704 passes an event rule isn't a dynamic task: when you create or modify an
1705 event rule, all the tracers of its tracing domain enable or disable the
1706 instrumentation points themselves once. This is possible because the
1707 attributes of an instrumentation point (type, name, and log level) are
1708 defined statically. In other words, without a dynamic filter, the tracer
1709 _doesn't evaluate_ the arguments of an instrumentation point unless it
1710 matches an enabled event rule.
1711
1712 Note that, for LTTng to record an event, the <<channel,channel>> to
1713 which a matching event rule is attached must also be enabled, and the
1714 <<tracing-session,tracing session>> owning this channel must be active
1715 (started).
1716
1717 [role="img-100"]
1718 .Logical path from an instrumentation point to an event record.
1719 image::event-rule.png[]
1720
1721 .Event, event record, or event rule?
1722 ****
1723 With so many similar terms, it's easy to get confused.
1724
1725 An **event** is the consequence of the execution of an _instrumentation
1726 point_, like a tracepoint that you manually place in some source code,
1727 or a Linux kernel kprobe. An event is said to _occur_ at a specific
1728 time. Different actions can be taken upon the occurrence of an event,
1729 like record the payload of the event to a buffer.
1730
1731 An **event record** is the representation of an event in a sub-buffer. A
1732 tracer is responsible for capturing the payload of an event, current
1733 context variables, the ID of the event, and its timestamp. LTTng
1734 can append this sub-buffer to a trace file.
1735
1736 An **event rule** is a set of conditions which must _all_ be satisfied
1737 for LTTng to record an occuring event. Events still occur without
1738 satisfying event rules, but LTTng doesn't record them.
1739 ****
1740
1741
1742 [[plumbing]]
1743 == Components of noch:{LTTng}
1744
1745 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1746 to call LTTng a simple _tool_ since it is composed of multiple
1747 interacting components. This section describes those components,
1748 explains their respective roles, and shows how they connect together to
1749 form the LTTng ecosystem.
1750
1751 The following diagram shows how the most important components of LTTng
1752 interact with user applications, the Linux kernel, and you:
1753
1754 [role="img-100"]
1755 .Control and trace data paths between LTTng components.
1756 image::plumbing.png[]
1757
1758 The LTTng project incorporates:
1759
1760 * **LTTng-tools**: Libraries and command-line interface to
1761   control tracing sessions.
1762 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1763 ** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1764 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1765 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1766 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1767 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1768   applications.
1769 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1770    headers to instrument and trace any native user application.
1771 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1772 *** `liblttng-ust-libc-wrapper`
1773 *** `liblttng-ust-pthread-wrapper`
1774 *** `liblttng-ust-cyg-profile`
1775 *** `liblttng-ust-cyg-profile-fast`
1776 *** `liblttng-ust-dl`
1777 ** User space tracepoint provider source files generator command-line
1778    tool (man:lttng-gen-tp(1)).
1779 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1780    Java applications using `java.util.logging` or
1781    Apache log4j{nbsp}1.2 logging.
1782 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1783    Python applications using the standard `logging` package.
1784 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1785   the kernel.
1786 ** LTTng kernel tracer module.
1787 ** Tracing ring buffer kernel modules.
1788 ** Probe kernel modules.
1789 ** LTTng logger kernel module.
1790
1791
1792 [[lttng-cli]]
1793 === Tracing control command-line interface
1794
1795 [role="img-100"]
1796 .The tracing control command-line interface.
1797 image::plumbing-lttng-cli.png[]
1798
1799 The _man:lttng(1) command-line tool_ is the standard user interface to
1800 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1801 is part of LTTng-tools.
1802
1803 The cmd:lttng tool is linked with
1804 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1805 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1806
1807 The cmd:lttng tool has a Git-like interface:
1808
1809 [role="term"]
1810 ----
1811 $ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1812 ----
1813
1814 The <<controlling-tracing,Tracing control>> section explores the
1815 available features of LTTng using the cmd:lttng tool.
1816
1817
1818 [[liblttng-ctl-lttng]]
1819 === Tracing control library
1820
1821 [role="img-100"]
1822 .The tracing control library.
1823 image::plumbing-liblttng-ctl.png[]
1824
1825 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1826 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1827 underlying details of the protocol. `liblttng-ctl` is part of LTTng-tools.
1828
1829 The <<lttng-cli,cmd:lttng command-line tool>>
1830 is linked with `liblttng-ctl`.
1831
1832 Use `liblttng-ctl` in C or $$C++$$ source code by including its
1833 ``master'' header:
1834
1835 [source,c]
1836 ----
1837 #include <lttng/lttng.h>
1838 ----
1839
1840 Some objects are referenced by name (C string), such as tracing
1841 sessions, but most of them require to create a handle first using
1842 `lttng_create_handle()`.
1843
1844 As of LTTng{nbsp}{revision}, the best available developer documentation for
1845 `liblttng-ctl` is its installed header files. Every function and structure is
1846 thoroughly documented.
1847
1848
1849 [[lttng-ust]]
1850 === User space tracing library
1851
1852 [role="img-100"]
1853 .The user space tracing library.
1854 image::plumbing-liblttng-ust.png[]
1855
1856 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1857 is the LTTng user space tracer. It receives commands from a
1858 <<lttng-sessiond,session daemon>>, for example to
1859 enable and disable specific instrumentation points, and writes event
1860 records to ring buffers shared with a
1861 <<lttng-consumerd,consumer daemon>>.
1862 `liblttng-ust` is part of LTTng-UST.
1863
1864 Public C header files are installed beside `liblttng-ust` to
1865 instrument any <<c-application,C or $$C++$$ application>>.
1866
1867 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1868 packages, use their own library providing tracepoints which is
1869 linked with `liblttng-ust`.
1870
1871 An application or library doesn't have to initialize `liblttng-ust`
1872 manually: its constructor does the necessary tasks to properly register
1873 to a session daemon. The initialization phase also enables the
1874 instrumentation points matching the <<event,event rules>> that you
1875 already created.
1876
1877
1878 [[lttng-ust-agents]]
1879 === User space tracing agents
1880
1881 [role="img-100"]
1882 .The user space tracing agents.
1883 image::plumbing-lttng-ust-agents.png[]
1884
1885 The _LTTng-UST Java and Python agents_ are regular Java and Python
1886 packages which add LTTng tracing capabilities to the
1887 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1888
1889 In the case of Java, the
1890 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1891 core logging facilities] and
1892 https://logging.apache.org/log4j/1.2/[Apache log4j{nbsp}1.2] are supported.
1893 Note that Apache Log4{nbsp}2 isn't supported.
1894
1895 In the case of Python, the standard
1896 https://docs.python.org/3/library/logging.html[`logging`] package
1897 is supported. Both Python{nbsp}2 and Python{nbsp}3 modules can import the
1898 LTTng-UST Python agent package.
1899
1900 The applications using the LTTng-UST agents are in the
1901 `java.util.logging` (JUL),
1902 log4j, and Python <<domain,tracing domains>>.
1903
1904 Both agents use the same mechanism to trace the log statements. When an
1905 agent initializes, it creates a log handler that attaches to the root
1906 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1907 When the application executes a log statement, the root logger passes it
1908 to the log handler of the agent. The log handler of the agent calls a
1909 native function in a tracepoint provider package shared library linked
1910 with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
1911 other fields, like its logger name and its log level. This native
1912 function contains a user space instrumentation point, hence tracing the
1913 log statement.
1914
1915 The log level condition of an
1916 <<event,event rule>> is considered when tracing
1917 a Java or a Python application, and it's compatible with the standard
1918 JUL, log4j, and Python log levels.
1919
1920
1921 [[lttng-modules]]
1922 === LTTng kernel modules
1923
1924 [role="img-100"]
1925 .The LTTng kernel modules.
1926 image::plumbing-lttng-modules.png[]
1927
1928 The _LTTng kernel modules_ are a set of Linux kernel modules
1929 which implement the kernel tracer of the LTTng project. The LTTng
1930 kernel modules are part of LTTng-modules.
1931
1932 The LTTng kernel modules include:
1933
1934 * A set of _probe_ modules.
1935 +
1936 Each module attaches to a specific subsystem
1937 of the Linux kernel using its tracepoint instrument points. There are
1938 also modules to attach to the entry and return points of the Linux
1939 system call functions.
1940
1941 * _Ring buffer_ modules.
1942 +
1943 A ring buffer implementation is provided as kernel modules. The LTTng
1944 kernel tracer writes to the ring buffer; a
1945 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1946
1947 * The _LTTng kernel tracer_ module.
1948 * The _LTTng logger_ module.
1949 +
1950 The LTTng logger module implements the special path:{/proc/lttng-logger}
1951 (and path:{/dev/lttng-logger} since LTTng{nbsp}2.11) files so that any
1952 executable can generate LTTng events by opening and writing to those
1953 files.
1954 +
1955 See <<proc-lttng-logger-abi,LTTng logger>>.
1956
1957 Generally, you don't have to load the LTTng kernel modules manually
1958 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
1959 daemon>> loads the necessary modules when starting. If you have extra
1960 probe modules, you can specify to load them to the session daemon on the
1961 command line. See also
1962 <<linux-kernel-sig,Linux kernel module signature>>.
1963
1964 The LTTng kernel modules are installed in
1965 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
1966 the kernel release (see `uname --kernel-release`).
1967
1968
1969 [[lttng-sessiond]]
1970 === Session daemon
1971
1972 [role="img-100"]
1973 .The session daemon.
1974 image::plumbing-sessiond.png[]
1975
1976 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
1977 managing tracing sessions and for controlling the various components of
1978 LTTng. The session daemon is part of LTTng-tools.
1979
1980 The session daemon sends control requests to and receives control
1981 responses from:
1982
1983 * The <<lttng-ust,user space tracing library>>.
1984 +
1985 Any instance of the user space tracing library first registers to
1986 a session daemon. Then, the session daemon can send requests to
1987 this instance, such as:
1988 +
1989 --
1990 ** Get the list of tracepoints.
1991 ** Share an <<event,event rule>> so that the user space tracing library
1992    can enable or disable tracepoints. Amongst the possible conditions
1993    of an event rule is a filter expression which `liblttng-ust` evalutes
1994    when an event occurs.
1995 ** Share <<channel,channel>> attributes and ring buffer locations.
1996 --
1997 +
1998 The session daemon and the user space tracing library use a Unix
1999 domain socket for their communication.
2000
2001 * The <<lttng-ust-agents,user space tracing agents>>.
2002 +
2003 Any instance of a user space tracing agent first registers to
2004 a session daemon. Then, the session daemon can send requests to
2005 this instance, such as:
2006 +
2007 --
2008 ** Get the list of loggers.
2009 ** Enable or disable a specific logger.
2010 --
2011 +
2012 The session daemon and the user space tracing agent use a TCP connection
2013 for their communication.
2014
2015 * The <<lttng-modules,LTTng kernel tracer>>.
2016 * The <<lttng-consumerd,consumer daemon>>.
2017 +
2018 The session daemon sends requests to the consumer daemon to instruct
2019 it where to send the trace data streams, amongst other information.
2020
2021 * The <<lttng-relayd,relay daemon>>.
2022
2023 The session daemon receives commands from the
2024 <<liblttng-ctl-lttng,tracing control library>>.
2025
2026 The root session daemon loads the appropriate
2027 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
2028 a <<lttng-consumerd,consumer daemon>> as soon as you create
2029 an <<event,event rule>>.
2030
2031 The session daemon doesn't send and receive trace data: this is the
2032 role of the <<lttng-consumerd,consumer daemon>> and
2033 <<lttng-relayd,relay daemon>>. It does, however, generate the
2034 http://diamon.org/ctf/[CTF] metadata stream.
2035
2036 Each Unix user can have its own session daemon instance. The
2037 tracing sessions which different session daemons manage are completely
2038 independent.
2039
2040 The root user's session daemon is the only one which is
2041 allowed to control the LTTng kernel tracer, and its spawned consumer
2042 daemon is the only one which is allowed to consume trace data from the
2043 LTTng kernel tracer. Note, however, that any Unix user which is a member
2044 of the <<tracing-group,tracing group>> is allowed
2045 to create <<channel,channels>> in the
2046 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
2047 kernel.
2048
2049 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
2050 session daemon when using its `create` command if none is currently
2051 running. You can also start the session daemon manually.
2052
2053
2054 [[lttng-consumerd]]
2055 === Consumer daemon
2056
2057 [role="img-100"]
2058 .The consumer daemon.
2059 image::plumbing-consumerd.png[]
2060
2061 The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
2062 ring buffers with user applications or with the LTTng kernel modules to
2063 collect trace data and send it to some location (on disk or to a
2064 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
2065 is part of LTTng-tools.
2066
2067 You don't start a consumer daemon manually: a consumer daemon is always
2068 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
2069 <<event,event rule>>, that is, before you start tracing. When you kill
2070 its owner session daemon, the consumer daemon also exits because it is
2071 the child process of the session daemon. Command-line options of
2072 man:lttng-sessiond(8) target the consumer daemon process.
2073
2074 There are up to two running consumer daemons per Unix user, whereas only
2075 one session daemon can run per user. This is because each process can be
2076 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
2077 and 64-bit processes, it is more efficient to have separate
2078 corresponding 32-bit and 64-bit consumer daemons. The root user is an
2079 exception: it can have up to _three_ running consumer daemons: 32-bit
2080 and 64-bit instances for its user applications, and one more
2081 reserved for collecting kernel trace data.
2082
2083
2084 [[lttng-relayd]]
2085 === Relay daemon
2086
2087 [role="img-100"]
2088 .The relay daemon.
2089 image::plumbing-relayd.png[]
2090
2091 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
2092 between remote session and consumer daemons, local trace files, and a
2093 remote live trace viewer. The relay daemon is part of LTTng-tools.
2094
2095 The main purpose of the relay daemon is to implement a receiver of
2096 <<sending-trace-data-over-the-network,trace data over the network>>.
2097 This is useful when the target system doesn't have much file system
2098 space to record trace files locally.
2099
2100 The relay daemon is also a server to which a
2101 <<lttng-live,live trace viewer>> can
2102 connect. The live trace viewer sends requests to the relay daemon to
2103 receive trace data as the target system emits events. The
2104 communication protocol is named _LTTng live_; it is used over TCP
2105 connections.
2106
2107 Note that you can start the relay daemon on the target system directly.
2108 This is the setup of choice when the use case is to view events as
2109 the target system emits them without the need of a remote system.
2110
2111
2112 [[instrumenting]]
2113 == [[using-lttng]]Instrumentation
2114
2115 There are many examples of tracing and monitoring in our everyday life:
2116
2117 * You have access to real-time and historical weather reports and
2118   forecasts thanks to weather stations installed around the country.
2119 * You know your heart is safe thanks to an electrocardiogram.
2120 * You make sure not to drive your car too fast and to have enough fuel
2121   to reach your destination thanks to gauges visible on your dashboard.
2122
2123 All the previous examples have something in common: they rely on
2124 **instruments**. Without the electrodes attached to the surface of your
2125 body skin, cardiac monitoring is futile.
2126
2127 LTTng, as a tracer, is no different from those real life examples. If
2128 you're about to trace a software system or, in other words, record its
2129 history of execution, you better have **instrumentation points** in the
2130 subject you're tracing, that is, the actual software.
2131
2132 Various ways were developed to instrument a piece of software for LTTng
2133 tracing. The most straightforward one is to manually place
2134 instrumentation points, called _tracepoints_, in the source code of the
2135 software. It is also possible to add instrumentation points dynamically
2136 in the Linux kernel <<domain,tracing domain>>.
2137
2138 If you're only interested in tracing the Linux kernel, your
2139 instrumentation needs are probably already covered by the built-in
2140 <<lttng-modules,Linux kernel tracepoints>> of LTTng. You may also wish
2141 to trace a user application which is already instrumented for LTTng
2142 tracing. In such cases, skip this whole section and read the topics of
2143 the <<controlling-tracing,Tracing control>> section.
2144
2145 Many methods are available to instrument a piece of software for LTTng
2146 tracing. They are:
2147
2148 * <<c-application,User space instrumentation for C and $$C++$$
2149   applications>>.
2150 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
2151 * <<java-application,User space Java agent>>.
2152 * <<python-application,User space Python agent>>.
2153 * <<proc-lttng-logger-abi,LTTng logger>>.
2154 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
2155
2156
2157 [[c-application]]
2158 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
2159
2160 The procedure to instrument a C or $$C++$$ user application with
2161 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
2162
2163 . <<tracepoint-provider,Create the source files of a tracepoint provider
2164   package>>.
2165 . <<probing-the-application-source-code,Add tracepoints to
2166   the source code of the application>>.
2167 . <<building-tracepoint-providers-and-user-application,Build and link
2168   a tracepoint provider package and the user application>>.
2169
2170 If you need quick, man:printf(3)-like instrumentation, skip
2171 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
2172 instead.
2173
2174 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
2175 instrument a user application with `liblttng-ust`.
2176
2177
2178 [[tracepoint-provider]]
2179 ==== Create the source files of a tracepoint provider package
2180
2181 A _tracepoint provider_ is a set of compiled functions which provide
2182 **tracepoints** to an application, the type of instrumentation point
2183 supported by LTTng-UST. Those functions can emit events with
2184 user-defined fields and serialize those events as event records to one
2185 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
2186 macro, which you <<probing-the-application-source-code,insert in the
2187 source code of a user application>>, calls those functions.
2188
2189 A _tracepoint provider package_ is an object file (`.o`) or a shared
2190 library (`.so`) which contains one or more tracepoint providers.
2191 Its source files are:
2192
2193 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
2194 * A <<tpp-source,tracepoint provider package source>> (`.c`).
2195
2196 A tracepoint provider package is dynamically linked with `liblttng-ust`,
2197 the LTTng user space tracer, at run time.
2198
2199 [role="img-100"]
2200 .User application linked with `liblttng-ust` and containing a tracepoint provider.
2201 image::ust-app.png[]
2202
2203 NOTE: If you need quick, man:printf(3)-like instrumentation,
2204 skip creating and using a tracepoint provider and use
2205 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
2206
2207
2208 [[tpp-header]]
2209 ===== Create a tracepoint provider header file template
2210
2211 A _tracepoint provider header file_ contains the tracepoint
2212 definitions of a tracepoint provider.
2213
2214 To create a tracepoint provider header file:
2215
2216 . Start from this template:
2217 +
2218 --
2219 [source,c]
2220 .Tracepoint provider header file template (`.h` file extension).
2221 ----
2222 #undef TRACEPOINT_PROVIDER
2223 #define TRACEPOINT_PROVIDER provider_name
2224
2225 #undef TRACEPOINT_INCLUDE
2226 #define TRACEPOINT_INCLUDE "./tp.h"
2227
2228 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
2229 #define _TP_H
2230
2231 #include <lttng/tracepoint.h>
2232
2233 /*
2234  * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
2235  * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
2236  */
2237
2238 #endif /* _TP_H */
2239
2240 #include <lttng/tracepoint-event.h>
2241 ----
2242 --
2243
2244 . Replace:
2245 +
2246 * `provider_name` with the name of your tracepoint provider.
2247 * `"tp.h"` with the name of your tracepoint provider header file.
2248
2249 . Below the `#include <lttng/tracepoint.h>` line, put your
2250   <<defining-tracepoints,tracepoint definitions>>.
2251
2252 Your tracepoint provider name must be unique amongst all the possible
2253 tracepoint provider names used on the same target system. We
2254 suggest to include the name of your project or company in the name,
2255 for example, `org_lttng_my_project_tpp`.
2256
2257 TIP: [[lttng-gen-tp]]Use the man:lttng-gen-tp(1) tool to create
2258 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
2259 write are the <<defining-tracepoints,tracepoint definitions>>.
2260
2261
2262 [[defining-tracepoints]]
2263 ===== Create a tracepoint definition
2264
2265 A _tracepoint definition_ defines, for a given tracepoint:
2266
2267 * Its **input arguments**. They are the macro parameters that the
2268   `tracepoint()` macro accepts for this particular tracepoint
2269   in the source code of the user application.
2270 * Its **output event fields**. They are the sources of event fields
2271   that form the payload of any event that the execution of the
2272   `tracepoint()` macro emits for this particular tracepoint.
2273
2274 Create a tracepoint definition by using the
2275 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
2276 line in the
2277 <<tpp-header,tracepoint provider header file template>>.
2278
2279 The syntax of the `TRACEPOINT_EVENT()` macro is:
2280
2281 [source,c]
2282 .`TRACEPOINT_EVENT()` macro syntax.
2283 ----
2284 TRACEPOINT_EVENT(
2285     /* Tracepoint provider name */
2286     provider_name,
2287
2288     /* Tracepoint name */
2289     tracepoint_name,
2290
2291     /* Input arguments */
2292     TP_ARGS(
2293         arguments
2294     ),
2295
2296     /* Output event fields */
2297     TP_FIELDS(
2298         fields
2299     )
2300 )
2301 ----
2302
2303 Replace:
2304
2305 * `provider_name` with your tracepoint provider name.
2306 * `tracepoint_name` with your tracepoint name.
2307 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2308 * `fields` with the <<tpp-def-output-fields,output event field>>
2309   definitions.
2310
2311 This tracepoint emits events named `provider_name:tracepoint_name`.
2312
2313 [IMPORTANT]
2314 .Event name length limitation
2315 ====
2316 The concatenation of the tracepoint provider name and the
2317 tracepoint name must not exceed **254{nbsp}characters**. If it does, the
2318 instrumented application compiles and runs, but LTTng throws multiple
2319 warnings and you could experience serious issues.
2320 ====
2321
2322 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2323
2324 [source,c]
2325 .`TP_ARGS()` macro syntax.
2326 ----
2327 TP_ARGS(
2328     type, arg_name
2329 )
2330 ----
2331
2332 Replace:
2333
2334 * `type` with the C type of the argument.
2335 * `arg_name` with the argument name.
2336
2337 You can repeat `type` and `arg_name` up to 10{nbsp}times to have more
2338 than one argument.
2339
2340 .`TP_ARGS()` usage with three arguments.
2341 ====
2342 [source,c]
2343 ----
2344 TP_ARGS(
2345     int, count,
2346     float, ratio,
2347     const char*, query
2348 )
2349 ----
2350 ====
2351
2352 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2353 tracepoint definition with no input arguments.
2354
2355 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2356 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2357 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2358 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2359 one event field.
2360
2361 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2362 C expression that the tracer evalutes at the `tracepoint()` macro site
2363 in the source code of the application. This expression provides the
2364 source of data of a field. The argument expression can include input
2365 argument names listed in the `TP_ARGS()` macro.
2366
2367 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2368 must be unique within a given tracepoint definition.
2369
2370 Here's a complete tracepoint definition example:
2371
2372 .Tracepoint definition.
2373 ====
2374 The following tracepoint definition defines a tracepoint which takes
2375 three input arguments and has four output event fields.
2376
2377 [source,c]
2378 ----
2379 #include "my-custom-structure.h"
2380
2381 TRACEPOINT_EVENT(
2382     my_provider,
2383     my_tracepoint,
2384     TP_ARGS(
2385         const struct my_custom_structure*, my_custom_structure,
2386         float, ratio,
2387         const char*, query
2388     ),
2389     TP_FIELDS(
2390         ctf_string(query_field, query)
2391         ctf_float(double, ratio_field, ratio)
2392         ctf_integer(int, recv_size, my_custom_structure->recv_size)
2393         ctf_integer(int, send_size, my_custom_structure->send_size)
2394     )
2395 )
2396 ----
2397
2398 Refer to this tracepoint definition with the `tracepoint()` macro in
2399 the source code of your application like this:
2400
2401 [source,c]
2402 ----
2403 tracepoint(my_provider, my_tracepoint,
2404            my_structure, some_ratio, the_query);
2405 ----
2406 ====
2407
2408 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2409 if they satisfy an enabled <<event,event rule>>.
2410
2411
2412 [[using-tracepoint-classes]]
2413 ===== Use a tracepoint class
2414
2415 A _tracepoint class_ is a class of tracepoints which share the same
2416 output event field definitions. A _tracepoint instance_ is one
2417 instance of such a defined tracepoint class, with its own tracepoint
2418 name.
2419
2420 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2421 shorthand which defines both a tracepoint class and a tracepoint
2422 instance at the same time.
2423
2424 When you build a tracepoint provider package, the C or $$C++$$ compiler
2425 creates one serialization function for each **tracepoint class**. A
2426 serialization function is responsible for serializing the event fields
2427 of a tracepoint to a sub-buffer when tracing.
2428
2429 For various performance reasons, when your situation requires multiple
2430 tracepoint definitions with different names, but with the same event
2431 fields, we recommend that you manually create a tracepoint class
2432 and instantiate as many tracepoint instances as needed. One positive
2433 effect of such a design, amongst other advantages, is that all
2434 tracepoint instances of the same tracepoint class reuse the same
2435 serialization function, thus reducing
2436 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2437
2438 .Use a tracepoint class and tracepoint instances.
2439 ====
2440 Consider the following three tracepoint definitions:
2441
2442 [source,c]
2443 ----
2444 TRACEPOINT_EVENT(
2445     my_app,
2446     get_account,
2447     TP_ARGS(
2448         int, userid,
2449         size_t, len
2450     ),
2451     TP_FIELDS(
2452         ctf_integer(int, userid, userid)
2453         ctf_integer(size_t, len, len)
2454     )
2455 )
2456
2457 TRACEPOINT_EVENT(
2458     my_app,
2459     get_settings,
2460     TP_ARGS(
2461         int, userid,
2462         size_t, len
2463     ),
2464     TP_FIELDS(
2465         ctf_integer(int, userid, userid)
2466         ctf_integer(size_t, len, len)
2467     )
2468 )
2469
2470 TRACEPOINT_EVENT(
2471     my_app,
2472     get_transaction,
2473     TP_ARGS(
2474         int, userid,
2475         size_t, len
2476     ),
2477     TP_FIELDS(
2478         ctf_integer(int, userid, userid)
2479         ctf_integer(size_t, len, len)
2480     )
2481 )
2482 ----
2483
2484 In this case, we create three tracepoint classes, with one implicit
2485 tracepoint instance for each of them: `get_account`, `get_settings`, and
2486 `get_transaction`. However, they all share the same event field names
2487 and types. Hence three identical, yet independent serialization
2488 functions are created when you build the tracepoint provider package.
2489
2490 A better design choice is to define a single tracepoint class and three
2491 tracepoint instances:
2492
2493 [source,c]
2494 ----
2495 /* The tracepoint class */
2496 TRACEPOINT_EVENT_CLASS(
2497     /* Tracepoint provider name */
2498     my_app,
2499
2500     /* Tracepoint class name */
2501     my_class,
2502
2503     /* Input arguments */
2504     TP_ARGS(
2505         int, userid,
2506         size_t, len
2507     ),
2508
2509     /* Output event fields */
2510     TP_FIELDS(
2511         ctf_integer(int, userid, userid)
2512         ctf_integer(size_t, len, len)
2513     )
2514 )
2515
2516 /* The tracepoint instances */
2517 TRACEPOINT_EVENT_INSTANCE(
2518     /* Tracepoint provider name */
2519     my_app,
2520
2521     /* Tracepoint class name */
2522     my_class,
2523
2524     /* Tracepoint name */
2525     get_account,
2526
2527     /* Input arguments */
2528     TP_ARGS(
2529         int, userid,
2530         size_t, len
2531     )
2532 )
2533 TRACEPOINT_EVENT_INSTANCE(
2534     my_app,
2535     my_class,
2536     get_settings,
2537     TP_ARGS(
2538         int, userid,
2539         size_t, len
2540     )
2541 )
2542 TRACEPOINT_EVENT_INSTANCE(
2543     my_app,
2544     my_class,
2545     get_transaction,
2546     TP_ARGS(
2547         int, userid,
2548         size_t, len
2549     )
2550 )
2551 ----
2552 ====
2553
2554
2555 [[assigning-log-levels]]
2556 ===== Assign a log level to a tracepoint definition
2557
2558 Assign a _log level_ to a <<defining-tracepoints,tracepoint definition>>
2559 with the `TRACEPOINT_LOGLEVEL()` macro.
2560
2561 Assigning different levels of severity to tracepoint definitions can
2562 be useful: when you <<enabling-disabling-events,create an event rule>>,
2563 you can target tracepoints having a log level as severe as a specific
2564 value.
2565
2566 The concept of LTTng-UST log levels is similar to the levels found
2567 in typical logging frameworks:
2568
2569 * In a logging framework, the log level is given by the function
2570   or method name you use at the log statement site: `debug()`,
2571   `info()`, `warn()`, `error()`, and so on.
2572 * In LTTng-UST, you statically assign the log level to a tracepoint
2573   definition; any `tracepoint()` macro invocation which refers to
2574   this definition has this log level.
2575
2576 You must use `TRACEPOINT_LOGLEVEL()` _after_ the
2577 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2578 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2579 tracepoint.
2580
2581 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2582
2583 [source,c]
2584 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2585 ----
2586 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2587 ----
2588
2589 Replace:
2590
2591 * `provider_name` with the tracepoint provider name.
2592 * `tracepoint_name` with the tracepoint name.
2593 * `log_level` with the log level to assign to the tracepoint
2594   definition named `tracepoint_name` in the `provider_name`
2595   tracepoint provider.
2596 +
2597 See man:lttng-ust(3) for a list of available log level names.
2598
2599 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2600 ====
2601 [source,c]
2602 ----
2603 /* Tracepoint definition */
2604 TRACEPOINT_EVENT(
2605     my_app,
2606     get_transaction,
2607     TP_ARGS(
2608         int, userid,
2609         size_t, len
2610     ),
2611     TP_FIELDS(
2612         ctf_integer(int, userid, userid)
2613         ctf_integer(size_t, len, len)
2614     )
2615 )
2616
2617 /* Log level assignment */
2618 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2619 ----
2620 ====
2621
2622
2623 [[tpp-source]]
2624 ===== Create a tracepoint provider package source file
2625
2626 A _tracepoint provider package source file_ is a C source file which
2627 includes a <<tpp-header,tracepoint provider header file>> to expand its
2628 macros into event serialization and other functions.
2629
2630 Use the following tracepoint provider package source file template:
2631
2632 [source,c]
2633 .Tracepoint provider package source file template.
2634 ----
2635 #define TRACEPOINT_CREATE_PROBES
2636
2637 #include "tp.h"
2638 ----
2639
2640 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2641 header file>> name. You may also include more than one tracepoint
2642 provider header file here to create a tracepoint provider package
2643 holding more than one tracepoint providers.
2644
2645
2646 [[probing-the-application-source-code]]
2647 ==== Add tracepoints to the source code of an application
2648
2649 Once you <<tpp-header,create a tracepoint provider header file>>, use
2650 the `tracepoint()` macro in the source code of your application to
2651 insert the tracepoints that this header
2652 <<defining-tracepoints,defines>>.
2653
2654 The `tracepoint()` macro takes at least two parameters: the tracepoint
2655 provider name and the tracepoint name. The corresponding tracepoint
2656 definition defines the other parameters.
2657
2658 .`tracepoint()` usage.
2659 ====
2660 The following <<defining-tracepoints,tracepoint definition>> defines a
2661 tracepoint which takes two input arguments and has two output event
2662 fields.
2663
2664 [source,c]
2665 .Tracepoint provider header file.
2666 ----
2667 #include "my-custom-structure.h"
2668
2669 TRACEPOINT_EVENT(
2670     my_provider,
2671     my_tracepoint,
2672     TP_ARGS(
2673         int, argc,
2674         const char*, cmd_name
2675     ),
2676     TP_FIELDS(
2677         ctf_string(cmd_name, cmd_name)
2678         ctf_integer(int, number_of_args, argc)
2679     )
2680 )
2681 ----
2682
2683 Refer to this tracepoint definition with the `tracepoint()` macro in
2684 the source code of your application like this:
2685
2686 [source,c]
2687 .Application source file.
2688 ----
2689 #include "tp.h"
2690
2691 int main(int argc, char* argv[])
2692 {
2693     tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2694
2695     return 0;
2696 }
2697 ----
2698
2699 Note how the source code of the application includes
2700 the tracepoint provider header file containing the tracepoint
2701 definitions to use, path:{tp.h}.
2702 ====
2703
2704 .`tracepoint()` usage with a complex tracepoint definition.
2705 ====
2706 Consider this complex tracepoint definition, where multiple event
2707 fields refer to the same input arguments in their argument expression
2708 parameter:
2709
2710 [source,c]
2711 .Tracepoint provider header file.
2712 ----
2713 /* For `struct stat` */
2714 #include <sys/types.h>
2715 #include <sys/stat.h>
2716 #include <unistd.h>
2717
2718 TRACEPOINT_EVENT(
2719     my_provider,
2720     my_tracepoint,
2721     TP_ARGS(
2722         int, my_int_arg,
2723         char*, my_str_arg,
2724         struct stat*, st
2725     ),
2726     TP_FIELDS(
2727         ctf_integer(int, my_constant_field, 23 + 17)
2728         ctf_integer(int, my_int_arg_field, my_int_arg)
2729         ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2730         ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2731                                      my_str_arg[2] + my_str_arg[3])
2732         ctf_string(my_str_arg_field, my_str_arg)
2733         ctf_integer_hex(off_t, size_field, st->st_size)
2734         ctf_float(double, size_dbl_field, (double) st->st_size)
2735         ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2736                           size_t, strlen(my_str_arg) / 2)
2737     )
2738 )
2739 ----
2740
2741 Refer to this tracepoint definition with the `tracepoint()` macro in
2742 the source code of your application like this:
2743
2744 [source,c]
2745 .Application source file.
2746 ----
2747 #define TRACEPOINT_DEFINE
2748 #include "tp.h"
2749
2750 int main(void)
2751 {
2752     struct stat s;
2753
2754     stat("/etc/fstab", &s);
2755     tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2756
2757     return 0;
2758 }
2759 ----
2760
2761 If you look at the event record that LTTng writes when tracing this
2762 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2763 it should look like this:
2764
2765 .Event record fields
2766 |====
2767 |Field name              |Field value
2768 |`my_constant_field`     |40
2769 |`my_int_arg_field`      |23
2770 |`my_int_arg_field2`     |529
2771 |`sum4_field`            |389
2772 |`my_str_arg_field`      |`Hello, World!`
2773 |`size_field`            |0x12d
2774 |`size_dbl_field`        |301.0
2775 |`half_my_str_arg_field` |`Hello,`
2776 |====
2777 ====
2778
2779 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2780 compute--they use the call stack, for example. To avoid this computation
2781 when the tracepoint is disabled, use the `tracepoint_enabled()` and
2782 `do_tracepoint()` macros.
2783
2784 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2785 is:
2786
2787 [source,c]
2788 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2789 ----
2790 tracepoint_enabled(provider_name, tracepoint_name)
2791 do_tracepoint(provider_name, tracepoint_name, ...)
2792 ----
2793
2794 Replace:
2795
2796 * `provider_name` with the tracepoint provider name.
2797 * `tracepoint_name` with the tracepoint name.
2798
2799 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2800 `tracepoint_name` from the provider named `provider_name` is enabled
2801 **at run time**.
2802
2803 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2804 if the tracepoint is enabled. Using `tracepoint()` with
2805 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2806 the `tracepoint_enabled()` check, thus a race condition is
2807 possible in this situation:
2808
2809 [source,c]
2810 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2811 ----
2812 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2813     stuff = prepare_stuff();
2814 }
2815
2816 tracepoint(my_provider, my_tracepoint, stuff);
2817 ----
2818
2819 If the tracepoint is enabled after the condition, then `stuff` isn't
2820 prepared: the emitted event will either contain wrong data, or the whole
2821 application could crash (segmentation fault, for example).
2822
2823 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2824 `STAP_PROBEV()` call. If you need it, you must emit
2825 this call yourself.
2826
2827
2828 [[building-tracepoint-providers-and-user-application]]
2829 ==== Build and link a tracepoint provider package and an application
2830
2831 Once you have one or more <<tpp-header,tracepoint provider header
2832 files>> and a <<tpp-source,tracepoint provider package source file>>,
2833 create the tracepoint provider package by compiling its source
2834 file. From here, multiple build and run scenarios are possible. The
2835 following table shows common application and library configurations
2836 along with the required command lines to achieve them.
2837
2838 In the following diagrams, we use the following file names:
2839
2840 `app`::
2841   Executable application.
2842
2843 `app.o`::
2844   Application object file.
2845
2846 `tpp.o`::
2847   Tracepoint provider package object file.
2848
2849 `tpp.a`::
2850   Tracepoint provider package archive file.
2851
2852 `libtpp.so`::
2853   Tracepoint provider package shared object file.
2854
2855 `emon.o`::
2856   User library object file.
2857
2858 `libemon.so`::
2859   User library shared object file.
2860
2861 We use the following symbols in the diagrams of table below:
2862
2863 [role="img-100"]
2864 .Symbols used in the build scenario diagrams.
2865 image::ust-sit-symbols.png[]
2866
2867 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2868 variable in the following instructions.
2869
2870 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2871 .Common tracepoint provider package scenarios.
2872 |====
2873 |Scenario |Instructions
2874
2875 |
2876 The instrumented application is statically linked with
2877 the tracepoint provider package object.
2878
2879 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2880
2881 |
2882 include::../common/ust-sit-step-tp-o.txt[]
2883
2884 To build the instrumented application:
2885
2886 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2887 +
2888 --
2889 [source,c]
2890 ----
2891 #define TRACEPOINT_DEFINE
2892 ----
2893 --
2894
2895 . Compile the application source file:
2896 +
2897 --
2898 [role="term"]
2899 ----
2900 $ gcc -c app.c
2901 ----
2902 --
2903
2904 . Build the application:
2905 +
2906 --
2907 [role="term"]
2908 ----
2909 $ gcc -o app app.o tpp.o -llttng-ust -ldl
2910 ----
2911 --
2912
2913 To run the instrumented application:
2914
2915 * Start the application:
2916 +
2917 --
2918 [role="term"]
2919 ----
2920 $ ./app
2921 ----
2922 --
2923
2924 |
2925 The instrumented application is statically linked with the
2926 tracepoint provider package archive file.
2927
2928 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2929
2930 |
2931 To create the tracepoint provider package archive file:
2932
2933 . Compile the <<tpp-source,tracepoint provider package source file>>:
2934 +
2935 --
2936 [role="term"]
2937 ----
2938 $ gcc -I. -c tpp.c
2939 ----
2940 --
2941
2942 . Create the tracepoint provider package archive file:
2943 +
2944 --
2945 [role="term"]
2946 ----
2947 $ ar rcs tpp.a tpp.o
2948 ----
2949 --
2950
2951 To build the instrumented application:
2952
2953 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2954 +
2955 --
2956 [source,c]
2957 ----
2958 #define TRACEPOINT_DEFINE
2959 ----
2960 --
2961
2962 . Compile the application source file:
2963 +
2964 --
2965 [role="term"]
2966 ----
2967 $ gcc -c app.c
2968 ----
2969 --
2970
2971 . Build the application:
2972 +
2973 --
2974 [role="term"]
2975 ----
2976 $ gcc -o app app.o tpp.a -llttng-ust -ldl
2977 ----
2978 --
2979
2980 To run the instrumented application:
2981
2982 * Start the application:
2983 +
2984 --
2985 [role="term"]
2986 ----
2987 $ ./app
2988 ----
2989 --
2990
2991 |
2992 The instrumented application is linked with the tracepoint provider
2993 package shared object.
2994
2995 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
2996
2997 |
2998 include::../common/ust-sit-step-tp-so.txt[]
2999
3000 To build the instrumented application:
3001
3002 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3003 +
3004 --
3005 [source,c]
3006 ----
3007 #define TRACEPOINT_DEFINE
3008 ----
3009 --
3010
3011 . Compile the application source file:
3012 +
3013 --
3014 [role="term"]
3015 ----
3016 $ gcc -c app.c
3017 ----
3018 --
3019
3020 . Build the application:
3021 +
3022 --
3023 [role="term"]
3024 ----
3025 $ gcc -o app app.o -ldl -L. -ltpp
3026 ----
3027 --
3028
3029 To run the instrumented application:
3030
3031 * Start the application:
3032 +
3033 --
3034 [role="term"]
3035 ----
3036 $ ./app
3037 ----
3038 --
3039
3040 |
3041 The tracepoint provider package shared object is preloaded before the
3042 instrumented application starts.
3043
3044 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
3045
3046 |
3047 include::../common/ust-sit-step-tp-so.txt[]
3048
3049 To build the instrumented application:
3050
3051 . In path:{app.c}, before including path:{tpp.h}, add the
3052   following lines:
3053 +
3054 --
3055 [source,c]
3056 ----
3057 #define TRACEPOINT_DEFINE
3058 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3059 ----
3060 --
3061
3062 . Compile the application source file:
3063 +
3064 --
3065 [role="term"]
3066 ----
3067 $ gcc -c app.c
3068 ----
3069 --
3070
3071 . Build the application:
3072 +
3073 --
3074 [role="term"]
3075 ----
3076 $ gcc -o app app.o -ldl
3077 ----
3078 --
3079
3080 To run the instrumented application with tracing support:
3081
3082 * Preload the tracepoint provider package shared object and
3083   start the application:
3084 +
3085 --
3086 [role="term"]
3087 ----
3088 $ LD_PRELOAD=./libtpp.so ./app
3089 ----
3090 --
3091
3092 To run the instrumented application without tracing support:
3093
3094 * Start the application:
3095 +
3096 --
3097 [role="term"]
3098 ----
3099 $ ./app
3100 ----
3101 --
3102
3103 |
3104 The instrumented application dynamically loads the tracepoint provider
3105 package shared object.
3106
3107 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
3108
3109 |
3110 include::../common/ust-sit-step-tp-so.txt[]
3111
3112 To build the instrumented application:
3113
3114 . In path:{app.c}, before including path:{tpp.h}, add the
3115   following lines:
3116 +
3117 --
3118 [source,c]
3119 ----
3120 #define TRACEPOINT_DEFINE
3121 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3122 ----
3123 --
3124
3125 . Compile the application source file:
3126 +
3127 --
3128 [role="term"]
3129 ----
3130 $ gcc -c app.c
3131 ----
3132 --
3133
3134 . Build the application:
3135 +
3136 --
3137 [role="term"]
3138 ----
3139 $ gcc -o app app.o -ldl
3140 ----
3141 --
3142
3143 To run the instrumented application:
3144
3145 * Start the application:
3146 +
3147 --
3148 [role="term"]
3149 ----
3150 $ ./app
3151 ----
3152 --
3153
3154 |
3155 The application is linked with the instrumented user library.
3156
3157 The instrumented user library is statically linked with the tracepoint
3158 provider package object file.
3159
3160 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
3161
3162 |
3163 include::../common/ust-sit-step-tp-o-fpic.txt[]
3164
3165 To build the instrumented user library:
3166
3167 . In path:{emon.c}, before including path:{tpp.h}, add the
3168   following line:
3169 +
3170 --
3171 [source,c]
3172 ----
3173 #define TRACEPOINT_DEFINE
3174 ----
3175 --
3176
3177 . Compile the user library source file:
3178 +
3179 --
3180 [role="term"]
3181 ----
3182 $ gcc -I. -fpic -c emon.c
3183 ----
3184 --
3185
3186 . Build the user library shared object:
3187 +
3188 --
3189 [role="term"]
3190 ----
3191 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
3192 ----
3193 --
3194
3195 To build the application:
3196
3197 . Compile the application source file:
3198 +
3199 --
3200 [role="term"]
3201 ----
3202 $ gcc -c app.c
3203 ----
3204 --
3205
3206 . Build the application:
3207 +
3208 --
3209 [role="term"]
3210 ----
3211 $ gcc -o app app.o -L. -lemon
3212 ----
3213 --
3214
3215 To run the application:
3216
3217 * Start the application:
3218 +
3219 --
3220 [role="term"]
3221 ----
3222 $ ./app
3223 ----
3224 --
3225
3226 |
3227 The application is linked with the instrumented user library.
3228
3229 The instrumented user library is linked with the tracepoint provider
3230 package shared object.
3231
3232 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3233
3234 |
3235 include::../common/ust-sit-step-tp-so.txt[]
3236
3237 To build the instrumented user library:
3238
3239 . In path:{emon.c}, before including path:{tpp.h}, add the
3240   following line:
3241 +
3242 --
3243 [source,c]
3244 ----
3245 #define TRACEPOINT_DEFINE
3246 ----
3247 --
3248
3249 . Compile the user library source file:
3250 +
3251 --
3252 [role="term"]
3253 ----
3254 $ gcc -I. -fpic -c emon.c
3255 ----
3256 --
3257
3258 . Build the user library shared object:
3259 +
3260 --
3261 [role="term"]
3262 ----
3263 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3264 ----
3265 --
3266
3267 To build the application:
3268
3269 . Compile the application source file:
3270 +
3271 --
3272 [role="term"]
3273 ----
3274 $ gcc -c app.c
3275 ----
3276 --
3277
3278 . Build the application:
3279 +
3280 --
3281 [role="term"]
3282 ----
3283 $ gcc -o app app.o -L. -lemon
3284 ----
3285 --
3286
3287 To run the application:
3288
3289 * Start the application:
3290 +
3291 --
3292 [role="term"]
3293 ----
3294 $ ./app
3295 ----
3296 --
3297
3298 |
3299 The tracepoint provider package shared object is preloaded before the
3300 application starts.
3301
3302 The application is linked with the instrumented user library.
3303
3304 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3305
3306 |
3307 include::../common/ust-sit-step-tp-so.txt[]
3308
3309 To build the instrumented user library:
3310
3311 . In path:{emon.c}, before including path:{tpp.h}, add the
3312   following lines:
3313 +
3314 --
3315 [source,c]
3316 ----
3317 #define TRACEPOINT_DEFINE
3318 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3319 ----
3320 --
3321
3322 . Compile the user library source file:
3323 +
3324 --
3325 [role="term"]
3326 ----
3327 $ gcc -I. -fpic -c emon.c
3328 ----
3329 --
3330
3331 . Build the user library shared object:
3332 +
3333 --
3334 [role="term"]
3335 ----
3336 $ gcc -shared -o libemon.so emon.o -ldl
3337 ----
3338 --
3339
3340 To build the application:
3341
3342 . Compile the application source file:
3343 +
3344 --
3345 [role="term"]
3346 ----
3347 $ gcc -c app.c
3348 ----
3349 --
3350
3351 . Build the application:
3352 +
3353 --
3354 [role="term"]
3355 ----
3356 $ gcc -o app app.o -L. -lemon
3357 ----
3358 --
3359
3360 To run the application with tracing support:
3361
3362 * Preload the tracepoint provider package shared object and
3363   start the application:
3364 +
3365 --
3366 [role="term"]
3367 ----
3368 $ LD_PRELOAD=./libtpp.so ./app
3369 ----
3370 --
3371
3372 To run the application without tracing support:
3373
3374 * Start the application:
3375 +
3376 --
3377 [role="term"]
3378 ----
3379 $ ./app
3380 ----
3381 --
3382
3383 |
3384 The application is linked with the instrumented user library.
3385
3386 The instrumented user library dynamically loads the tracepoint provider
3387 package shared object.
3388
3389 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3390
3391 |
3392 include::../common/ust-sit-step-tp-so.txt[]
3393
3394 To build the instrumented user library:
3395
3396 . In path:{emon.c}, before including path:{tpp.h}, add the
3397   following lines:
3398 +
3399 --
3400 [source,c]
3401 ----
3402 #define TRACEPOINT_DEFINE
3403 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3404 ----
3405 --
3406
3407 . Compile the user library source file:
3408 +
3409 --
3410 [role="term"]
3411 ----
3412 $ gcc -I. -fpic -c emon.c
3413 ----
3414 --
3415
3416 . Build the user library shared object:
3417 +
3418 --
3419 [role="term"]
3420 ----
3421 $ gcc -shared -o libemon.so emon.o -ldl
3422 ----
3423 --
3424
3425 To build the application:
3426
3427 . Compile the application source file:
3428 +
3429 --
3430 [role="term"]
3431 ----
3432 $ gcc -c app.c
3433 ----
3434 --
3435
3436 . Build the application:
3437 +
3438 --
3439 [role="term"]
3440 ----
3441 $ gcc -o app app.o -L. -lemon
3442 ----
3443 --
3444
3445 To run the application:
3446
3447 * Start the application:
3448 +
3449 --
3450 [role="term"]
3451 ----
3452 $ ./app
3453 ----
3454 --
3455
3456 |
3457 The application dynamically loads the instrumented user library.
3458
3459 The instrumented user library is linked with the tracepoint provider
3460 package shared object.
3461
3462 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3463
3464 |
3465 include::../common/ust-sit-step-tp-so.txt[]
3466
3467 To build the instrumented user library:
3468
3469 . In path:{emon.c}, before including path:{tpp.h}, add the
3470   following line:
3471 +
3472 --
3473 [source,c]
3474 ----
3475 #define TRACEPOINT_DEFINE
3476 ----
3477 --
3478
3479 . Compile the user library source file:
3480 +
3481 --
3482 [role="term"]
3483 ----
3484 $ gcc -I. -fpic -c emon.c
3485 ----
3486 --
3487
3488 . Build the user library shared object:
3489 +
3490 --
3491 [role="term"]
3492 ----
3493 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3494 ----
3495 --
3496
3497 To build the application:
3498
3499 . Compile the application source file:
3500 +
3501 --
3502 [role="term"]
3503 ----
3504 $ gcc -c app.c
3505 ----
3506 --
3507
3508 . Build the application:
3509 +
3510 --
3511 [role="term"]
3512 ----
3513 $ gcc -o app app.o -ldl -L. -lemon
3514 ----
3515 --
3516
3517 To run the application:
3518
3519 * Start the application:
3520 +
3521 --
3522 [role="term"]
3523 ----
3524 $ ./app
3525 ----
3526 --
3527
3528 |
3529 The application dynamically loads the instrumented user library.
3530
3531 The instrumented user library dynamically loads the tracepoint provider
3532 package shared object.
3533
3534 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3535
3536 |
3537 include::../common/ust-sit-step-tp-so.txt[]
3538
3539 To build the instrumented user library:
3540
3541 . In path:{emon.c}, before including path:{tpp.h}, add the
3542   following lines:
3543 +
3544 --
3545 [source,c]
3546 ----
3547 #define TRACEPOINT_DEFINE
3548 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3549 ----
3550 --
3551
3552 . Compile the user library source file:
3553 +
3554 --
3555 [role="term"]
3556 ----
3557 $ gcc -I. -fpic -c emon.c
3558 ----
3559 --
3560
3561 . Build the user library shared object:
3562 +
3563 --
3564 [role="term"]
3565 ----
3566 $ gcc -shared -o libemon.so emon.o -ldl
3567 ----
3568 --
3569
3570 To build the application:
3571
3572 . Compile the application source file:
3573 +
3574 --
3575 [role="term"]
3576 ----
3577 $ gcc -c app.c
3578 ----
3579 --
3580
3581 . Build the application:
3582 +
3583 --
3584 [role="term"]
3585 ----
3586 $ gcc -o app app.o -ldl -L. -lemon
3587 ----
3588 --
3589
3590 To run the application:
3591
3592 * Start the application:
3593 +
3594 --
3595 [role="term"]
3596 ----
3597 $ ./app
3598 ----
3599 --
3600
3601 |
3602 The tracepoint provider package shared object is preloaded before the
3603 application starts.
3604
3605 The application dynamically loads the instrumented user library.
3606
3607 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3608
3609 |
3610 include::../common/ust-sit-step-tp-so.txt[]
3611
3612 To build the instrumented user library:
3613
3614 . In path:{emon.c}, before including path:{tpp.h}, add the
3615   following lines:
3616 +
3617 --
3618 [source,c]
3619 ----
3620 #define TRACEPOINT_DEFINE
3621 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3622 ----
3623 --
3624
3625 . Compile the user library source file:
3626 +
3627 --
3628 [role="term"]
3629 ----
3630 $ gcc -I. -fpic -c emon.c
3631 ----
3632 --
3633
3634 . Build the user library shared object:
3635 +
3636 --
3637 [role="term"]
3638 ----
3639 $ gcc -shared -o libemon.so emon.o -ldl
3640 ----
3641 --
3642
3643 To build the application:
3644
3645 . Compile the application source file:
3646 +
3647 --
3648 [role="term"]
3649 ----
3650 $ gcc -c app.c
3651 ----
3652 --
3653
3654 . Build the application:
3655 +
3656 --
3657 [role="term"]
3658 ----
3659 $ gcc -o app app.o -L. -lemon
3660 ----
3661 --
3662
3663 To run the application with tracing support:
3664
3665 * Preload the tracepoint provider package shared object and
3666   start the application:
3667 +
3668 --
3669 [role="term"]
3670 ----
3671 $ LD_PRELOAD=./libtpp.so ./app
3672 ----
3673 --
3674
3675 To run the application without tracing support:
3676
3677 * Start the application:
3678 +
3679 --
3680 [role="term"]
3681 ----
3682 $ ./app
3683 ----
3684 --
3685
3686 |
3687 The application is statically linked with the tracepoint provider
3688 package object file.
3689
3690 The application is linked with the instrumented user library.
3691
3692 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3693
3694 |
3695 include::../common/ust-sit-step-tp-o.txt[]
3696
3697 To build the instrumented user library:
3698
3699 . In path:{emon.c}, before including path:{tpp.h}, add the
3700   following line:
3701 +
3702 --
3703 [source,c]
3704 ----
3705 #define TRACEPOINT_DEFINE
3706 ----
3707 --
3708
3709 . Compile the user library source file:
3710 +
3711 --
3712 [role="term"]
3713 ----
3714 $ gcc -I. -fpic -c emon.c
3715 ----
3716 --
3717
3718 . Build the user library shared object:
3719 +
3720 --
3721 [role="term"]
3722 ----
3723 $ gcc -shared -o libemon.so emon.o
3724 ----
3725 --
3726
3727 To build the application:
3728
3729 . Compile the application source file:
3730 +
3731 --
3732 [role="term"]
3733 ----
3734 $ gcc -c app.c
3735 ----
3736 --
3737
3738 . Build the application:
3739 +
3740 --
3741 [role="term"]
3742 ----
3743 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3744 ----
3745 --
3746
3747 To run the instrumented application:
3748
3749 * Start the application:
3750 +
3751 --
3752 [role="term"]
3753 ----
3754 $ ./app
3755 ----
3756 --
3757
3758 |
3759 The application is statically linked with the tracepoint provider
3760 package object file.
3761
3762 The application dynamically loads the instrumented user library.
3763
3764 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3765
3766 |
3767 include::../common/ust-sit-step-tp-o.txt[]
3768
3769 To build the application:
3770
3771 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3772 +
3773 --
3774 [source,c]
3775 ----
3776 #define TRACEPOINT_DEFINE
3777 ----
3778 --
3779
3780 . Compile the application source file:
3781 +
3782 --
3783 [role="term"]
3784 ----
3785 $ gcc -c app.c
3786 ----
3787 --
3788
3789 . Build the application:
3790 +
3791 --
3792 [role="term"]
3793 ----
3794 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3795     -llttng-ust -ldl
3796 ----
3797 --
3798 +
3799 The `--export-dynamic` option passed to the linker is necessary for the
3800 dynamically loaded library to ``see'' the tracepoint symbols defined in
3801 the application.
3802
3803 To build the instrumented user library:
3804
3805 . Compile the user library source file:
3806 +
3807 --
3808 [role="term"]
3809 ----
3810 $ gcc -I. -fpic -c emon.c
3811 ----
3812 --
3813
3814 . Build the user library shared object:
3815 +
3816 --
3817 [role="term"]
3818 ----
3819 $ gcc -shared -o libemon.so emon.o
3820 ----
3821 --
3822
3823 To run the application:
3824
3825 * Start the application:
3826 +
3827 --
3828 [role="term"]
3829 ----
3830 $ ./app
3831 ----
3832 --
3833 |====
3834
3835
3836 [[using-lttng-ust-with-daemons]]
3837 ===== Use noch:{LTTng-UST} with daemons
3838
3839 If your instrumented application calls man:fork(2), man:clone(2),
3840 or BSD's man:rfork(2), without a following man:exec(3)-family
3841 system call, you must preload the path:{liblttng-ust-fork.so} shared
3842 object when you start the application.
3843
3844 [role="term"]
3845 ----
3846 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3847 ----
3848
3849 If your tracepoint provider package is
3850 a shared library which you also preload, you must put both
3851 shared objects in env:LD_PRELOAD:
3852
3853 [role="term"]
3854 ----
3855 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3856 ----
3857
3858
3859 [role="since-2.9"]
3860 [[liblttng-ust-fd]]
3861 ===== Use noch:{LTTng-UST} with applications which close file descriptors that don't belong to them
3862
3863 If your instrumented application closes one or more file descriptors
3864 which it did not open itself, you must preload the
3865 path:{liblttng-ust-fd.so} shared object when you start the application:
3866
3867 [role="term"]
3868 ----
3869 $ LD_PRELOAD=liblttng-ust-fd.so ./my-app
3870 ----
3871
3872 Typical use cases include closing all the file descriptors after
3873 man:fork(2) or man:rfork(2) and buggy applications doing
3874 ``double closes''.
3875
3876
3877 [[lttng-ust-pkg-config]]
3878 ===== Use noch:{pkg-config}
3879
3880 On some distributions, LTTng-UST ships with a
3881 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3882 metadata file. If this is your case, then use cmd:pkg-config to
3883 build an application on the command line:
3884
3885 [role="term"]
3886 ----
3887 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3888 ----
3889
3890
3891 [[instrumenting-32-bit-app-on-64-bit-system]]
3892 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3893
3894 In order to trace a 32-bit application running on a 64-bit system,
3895 LTTng must use a dedicated 32-bit
3896 <<lttng-consumerd,consumer daemon>>.
3897
3898 The following steps show how to build and install a 32-bit consumer
3899 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3900 build and install the 32-bit LTTng-UST libraries, and how to build and
3901 link an instrumented 32-bit application in that context.
3902
3903 To build a 32-bit instrumented application for a 64-bit target system,
3904 assuming you have a fresh target system with no installed Userspace RCU
3905 or LTTng packages:
3906
3907 . Download, build, and install a 32-bit version of Userspace RCU:
3908 +
3909 --
3910 [role="term"]
3911 ----
3912 $ cd $(mktemp -d) &&
3913 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3914 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3915 cd userspace-rcu-0.9.* &&
3916 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3917 make &&
3918 sudo make install &&
3919 sudo ldconfig
3920 ----
3921 --
3922
3923 . Using the package manager of your distribution, or from source,
3924   install the following 32-bit versions of the following dependencies of
3925   LTTng-tools and LTTng-UST:
3926 +
3927 --
3928 * https://sourceforge.net/projects/libuuid/[libuuid]
3929 * http://directory.fsf.org/wiki/Popt[popt]
3930 * http://www.xmlsoft.org/[libxml2]
3931 --
3932
3933 . Download, build, and install a 32-bit version of the latest
3934   LTTng-UST{nbsp}{revision}:
3935 +
3936 --
3937 [role="term"]
3938 ----
3939 $ cd $(mktemp -d) &&
3940 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.12.tar.bz2 &&
3941 tar -xf lttng-ust-latest-2.12.tar.bz2 &&
3942 cd lttng-ust-2.12.* &&
3943 ./configure --libdir=/usr/local/lib32 \
3944             CFLAGS=-m32 CXXFLAGS=-m32 \
3945             LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
3946 make &&
3947 sudo make install &&
3948 sudo ldconfig
3949 ----
3950 --
3951 +
3952 [NOTE]
3953 ====
3954 Depending on your distribution,
3955 32-bit libraries could be installed at a different location than
3956 `/usr/lib32`. For example, Debian is known to install
3957 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
3958
3959 In this case, make sure to set `LDFLAGS` to all the
3960 relevant 32-bit library paths, for example:
3961
3962 [role="term"]
3963 ----
3964 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
3965 ----
3966 ====
3967
3968 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
3969   the 32-bit consumer daemon:
3970 +
3971 --
3972 [role="term"]
3973 ----
3974 $ cd $(mktemp -d) &&
3975 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
3976 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
3977 cd lttng-tools-2.12.* &&
3978 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3979             LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
3980             --disable-bin-lttng --disable-bin-lttng-crash \
3981             --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
3982 make &&
3983 cd src/bin/lttng-consumerd &&
3984 sudo make install &&
3985 sudo ldconfig
3986 ----
3987 --
3988
3989 . From your distribution or from source,
3990   <<installing-lttng,install>> the 64-bit versions of
3991   LTTng-UST and Userspace RCU.
3992 . Download, build, and install the 64-bit version of the
3993   latest LTTng-tools{nbsp}{revision}:
3994 +
3995 --
3996 [role="term"]
3997 ----
3998 $ cd $(mktemp -d) &&
3999 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.12.tar.bz2 &&
4000 tar -xf lttng-tools-latest-2.12.tar.bz2 &&
4001 cd lttng-tools-2.12.* &&
4002 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
4003             --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
4004 make &&
4005 sudo make install &&
4006 sudo ldconfig
4007 ----
4008 --
4009
4010 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
4011   when linking your 32-bit application:
4012 +
4013 ----
4014 -m32 -L/usr/lib32 -L/usr/local/lib32 \
4015 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
4016 ----
4017 +
4018 For example, let's rebuild the quick start example in
4019 <<tracing-your-own-user-application,Trace a user application>> as an
4020 instrumented 32-bit application:
4021 +
4022 --
4023 [role="term"]
4024 ----
4025 $ gcc -m32 -c -I. hello-tp.c
4026 $ gcc -m32 -c hello.c
4027 $ gcc -m32 -o hello hello.o hello-tp.o \
4028     -L/usr/lib32 -L/usr/local/lib32 \
4029     -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
4030     -llttng-ust -ldl
4031 ----
4032 --
4033
4034 No special action is required to execute the 32-bit application and
4035 to trace it: use the command-line man:lttng(1) tool as usual.
4036
4037
4038 [role="since-2.5"]
4039 [[tracef]]
4040 ==== Use `tracef()`
4041
4042 man:tracef(3) is a small LTTng-UST API designed for quick,
4043 man:printf(3)-like instrumentation without the burden of
4044 <<tracepoint-provider,creating>> and
4045 <<building-tracepoint-providers-and-user-application,building>>
4046 a tracepoint provider package.
4047
4048 To use `tracef()` in your application:
4049
4050 . In the C or C++ source files where you need to use `tracef()`,
4051   include `<lttng/tracef.h>`:
4052 +
4053 --
4054 [source,c]
4055 ----
4056 #include <lttng/tracef.h>
4057 ----
4058 --
4059
4060 . In the source code of the application, use `tracef()` like you would
4061   use man:printf(3):
4062 +
4063 --
4064 [source,c]
4065 ----
4066     /* ... */
4067
4068     tracef("my message: %d (%s)", my_integer, my_string);
4069
4070     /* ... */
4071 ----
4072 --
4073
4074 . Link your application with `liblttng-ust`:
4075 +
4076 --
4077 [role="term"]
4078 ----
4079 $ gcc -o app app.c -llttng-ust
4080 ----
4081 --
4082
4083 To trace the events that `tracef()` calls emit:
4084
4085 * <<enabling-disabling-events,Create an event rule>> which matches the
4086   `lttng_ust_tracef:*` event name:
4087 +
4088 --
4089 [role="term"]
4090 ----
4091 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
4092 ----
4093 --
4094
4095 [IMPORTANT]
4096 .Limitations of `tracef()`
4097 ====
4098 The `tracef()` utility function was developed to make user space tracing
4099 super simple, albeit with notable disadvantages compared to
4100 <<defining-tracepoints,user-defined tracepoints>>:
4101
4102 * All the emitted events have the same tracepoint provider and
4103   tracepoint names, respectively `lttng_ust_tracef` and `event`.
4104 * There is no static type checking.
4105 * The only event record field you actually get, named `msg`, is a string
4106   potentially containing the values you passed to `tracef()`
4107   using your own format string. This also means that you can't filter
4108   events with a custom expression at run time because there are no
4109   isolated fields.
4110 * Since `tracef()` uses the man:vasprintf(3) function of the
4111   C{nbsp}standard library behind the scenes to format the strings at run
4112   time, its expected performance is lower than with user-defined
4113   tracepoints, which don't require a conversion to a string.
4114
4115 Taking this into consideration, `tracef()` is useful for some quick
4116 prototyping and debugging, but you shouldn't consider it for any
4117 permanent and serious applicative instrumentation.
4118 ====
4119
4120
4121 [role="since-2.7"]
4122 [[tracelog]]
4123 ==== Use `tracelog()`
4124
4125 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
4126 the difference that it accepts an additional log level parameter.
4127
4128 The goal of `tracelog()` is to ease the migration from logging to
4129 tracing.
4130
4131 To use `tracelog()` in your application:
4132
4133 . In the C or C++ source files where you need to use `tracelog()`,
4134   include `<lttng/tracelog.h>`:
4135 +
4136 --
4137 [source,c]
4138 ----
4139 #include <lttng/tracelog.h>
4140 ----
4141 --
4142
4143 . In the source code of the application, use `tracelog()` like you would
4144   use man:printf(3), except for the first parameter which is the log
4145   level:
4146 +
4147 --
4148 [source,c]
4149 ----
4150     /* ... */
4151
4152     tracelog(TRACE_WARNING, "my message: %d (%s)",
4153              my_integer, my_string);
4154
4155     /* ... */
4156 ----
4157 --
4158 +
4159 See man:lttng-ust(3) for a list of available log level names.
4160
4161 . Link your application with `liblttng-ust`:
4162 +
4163 --
4164 [role="term"]
4165 ----
4166 $ gcc -o app app.c -llttng-ust
4167 ----
4168 --
4169
4170 To trace the events that `tracelog()` calls emit with a log level
4171 _as severe as_ a specific log level:
4172
4173 * <<enabling-disabling-events,Create an event rule>> which matches the
4174   `lttng_ust_tracelog:*` event name and a minimum level
4175   of severity:
4176 +
4177 --
4178 [role="term"]
4179 ----
4180 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4181                    --loglevel=TRACE_WARNING
4182 ----
4183 --
4184
4185 To trace the events that `tracelog()` calls emit with a
4186 _specific log level_:
4187
4188 * Create an event rule which matches the `lttng_ust_tracelog:*`
4189   event name and a specific log level:
4190 +
4191 --
4192 [role="term"]
4193 ----
4194 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
4195                    --loglevel-only=TRACE_INFO
4196 ----
4197 --
4198
4199
4200 [[prebuilt-ust-helpers]]
4201 === Prebuilt user space tracing helpers
4202
4203 The LTTng-UST package provides a few helpers in the form of preloadable
4204 shared objects which automatically instrument system functions and
4205 calls.
4206
4207 The helper shared objects are normally found in dir:{/usr/lib}. If you
4208 built LTTng-UST <<building-from-source,from source>>, they are probably
4209 located in dir:{/usr/local/lib}.
4210
4211 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
4212 are:
4213
4214 path:{liblttng-ust-libc-wrapper.so}::
4215 path:{liblttng-ust-pthread-wrapper.so}::
4216   <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
4217   memory and POSIX threads function tracing>>.
4218
4219 path:{liblttng-ust-cyg-profile.so}::
4220 path:{liblttng-ust-cyg-profile-fast.so}::
4221   <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
4222
4223 path:{liblttng-ust-dl.so}::
4224   <<liblttng-ust-dl,Dynamic linker tracing>>.
4225
4226 To use a user space tracing helper with any user application:
4227
4228 * Preload the helper shared object when you start the application:
4229 +
4230 --
4231 [role="term"]
4232 ----
4233 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
4234 ----
4235 --
4236 +
4237 You can preload more than one helper:
4238 +
4239 --
4240 [role="term"]
4241 ----
4242 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
4243 ----
4244 --
4245
4246
4247 [role="since-2.3"]
4248 [[liblttng-ust-libc-pthread-wrapper]]
4249 ==== Instrument C standard library memory and POSIX threads functions
4250
4251 The path:{liblttng-ust-libc-wrapper.so} and
4252 path:{liblttng-ust-pthread-wrapper.so} helpers
4253 add instrumentation to some C standard library and POSIX
4254 threads functions.
4255
4256 [role="growable"]
4257 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
4258 |====
4259 |TP provider name |TP name |Instrumented function
4260
4261 .6+|`lttng_ust_libc` |`malloc`         |man:malloc(3)
4262                      |`calloc`         |man:calloc(3)
4263                      |`realloc`        |man:realloc(3)
4264                      |`free`           |man:free(3)
4265                      |`memalign`       |man:memalign(3)
4266                      |`posix_memalign` |man:posix_memalign(3)
4267 |====
4268
4269 [role="growable"]
4270 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
4271 |====
4272 |TP provider name |TP name |Instrumented function
4273
4274 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
4275                         |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
4276                         |`pthread_mutex_trylock`  |man:pthread_mutex_trylock(3p)
4277                         |`pthread_mutex_unlock`   |man:pthread_mutex_unlock(3p)
4278 |====
4279
4280 When you preload the shared object, it replaces the functions listed
4281 in the previous tables by wrappers which contain tracepoints and call
4282 the replaced functions.
4283
4284
4285 [[liblttng-ust-cyg-profile]]
4286 ==== Instrument function entry and exit
4287
4288 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4289 to the entry and exit points of functions.
4290
4291 man:gcc(1) and man:clang(1) have an option named
4292 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4293 which generates instrumentation calls for entry and exit to functions.
4294 The LTTng-UST function tracing helpers,
4295 path:{liblttng-ust-cyg-profile.so} and
4296 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4297 to add tracepoints to the two generated functions (which contain
4298 `cyg_profile` in their names, hence the name of the helper).
4299
4300 To use the LTTng-UST function tracing helper, the source files to
4301 instrument must be built using the `-finstrument-functions` compiler
4302 flag.
4303
4304 There are two versions of the LTTng-UST function tracing helper:
4305
4306 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4307   that you should only use when it can be _guaranteed_ that the
4308   complete event stream is recorded without any lost event record.
4309   Any kind of duplicate information is left out.
4310 +
4311 Assuming no event record is lost, having only the function addresses on
4312 entry is enough to create a call graph, since an event record always
4313 contains the ID of the CPU that generated it.
4314 +
4315 Use a tool like man:addr2line(1) to convert function addresses back to
4316 source file names and line numbers.
4317
4318 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4319 which also works in use cases where event records might get discarded or
4320 not recorded from application startup.
4321 In these cases, the trace analyzer needs more information to be
4322 able to reconstruct the program flow.
4323
4324 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4325 points of this helper.
4326
4327 All the tracepoints that this helper provides have the
4328 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4329
4330 TIP: It's sometimes a good idea to limit the number of source files that
4331 you compile with the `-finstrument-functions` option to prevent LTTng
4332 from writing an excessive amount of trace data at run time. When using
4333 man:gcc(1), use the
4334 `-finstrument-functions-exclude-function-list` option to avoid
4335 instrument entries and exits of specific function names.
4336
4337
4338 [role="since-2.4"]
4339 [[liblttng-ust-dl]]
4340 ==== Instrument the dynamic linker
4341
4342 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4343 man:dlopen(3) and man:dlclose(3) function calls.
4344
4345 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4346 of this helper.
4347
4348
4349 [role="since-2.4"]
4350 [[java-application]]
4351 === User space Java agent
4352
4353 You can instrument any Java application which uses one of the following
4354 logging frameworks:
4355
4356 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4357   (JUL) core logging facilities.
4358 * http://logging.apache.org/log4j/1.2/[**Apache log4j{nbsp}1.2**], since
4359   LTTng{nbsp}2.6. Note that Apache Log4j{nbsp}2 isn't supported.
4360
4361 [role="img-100"]
4362 .LTTng-UST Java agent imported by a Java application.
4363 image::java-app.png[]
4364
4365 Note that the methods described below are new in LTTng{nbsp}2.8.
4366 Previous LTTng versions use another technique.
4367
4368 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4369 and https://ci.lttng.org/[continuous integration], thus this version is
4370 directly supported. However, the LTTng-UST Java agent is also tested
4371 with OpenJDK{nbsp}7.
4372
4373
4374 [role="since-2.8"]
4375 [[jul]]
4376 ==== Use the LTTng-UST Java agent for `java.util.logging`
4377
4378 To use the LTTng-UST Java agent in a Java application which uses
4379 `java.util.logging` (JUL):
4380
4381 . In the source code of the Java application, import the LTTng-UST log
4382   handler package for `java.util.logging`:
4383 +
4384 --
4385 [source,java]
4386 ----
4387 import org.lttng.ust.agent.jul.LttngLogHandler;
4388 ----
4389 --
4390
4391 . Create an LTTng-UST JUL log handler:
4392 +
4393 --
4394 [source,java]
4395 ----
4396 Handler lttngUstLogHandler = new LttngLogHandler();
4397 ----
4398 --
4399
4400 . Add this handler to the JUL loggers which should emit LTTng events:
4401 +
4402 --
4403 [source,java]
4404 ----
4405 Logger myLogger = Logger.getLogger("some-logger");
4406
4407 myLogger.addHandler(lttngUstLogHandler);
4408 ----
4409 --
4410
4411 . Use `java.util.logging` log statements and configuration as usual.
4412   The loggers with an attached LTTng-UST log handler can emit
4413   LTTng events.
4414
4415 . Before exiting the application, remove the LTTng-UST log handler from
4416   the loggers attached to it and call its `close()` method:
4417 +
4418 --
4419 [source,java]
4420 ----
4421 myLogger.removeHandler(lttngUstLogHandler);
4422 lttngUstLogHandler.close();
4423 ----
4424 --
4425 +
4426 This isn't strictly necessary, but it is recommended for a clean
4427 disposal of the resources of the handler.
4428
4429 . Include the common and JUL-specific JAR files of the LTTng-UST Java agent,
4430   path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4431   in the
4432   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4433   path] when you build the Java application.
4434 +
4435 The JAR files are typically located in dir:{/usr/share/java}.
4436 +
4437 IMPORTANT: The LTTng-UST Java agent must be
4438 <<installing-lttng,installed>> for the logging framework your
4439 application uses.
4440
4441 .Use the LTTng-UST Java agent for `java.util.logging`.
4442 ====
4443 [source,java]
4444 .path:{Test.java}
4445 ----
4446 import java.io.IOException;
4447 import java.util.logging.Handler;
4448 import java.util.logging.Logger;
4449 import org.lttng.ust.agent.jul.LttngLogHandler;
4450
4451 public class Test
4452 {
4453     private static final int answer = 42;
4454
4455     public static void main(String[] argv) throws Exception
4456     {
4457         // Create a logger
4458         Logger logger = Logger.getLogger("jello");
4459
4460         // Create an LTTng-UST log handler
4461         Handler lttngUstLogHandler = new LttngLogHandler();
4462
4463         // Add the LTTng-UST log handler to our logger
4464         logger.addHandler(lttngUstLogHandler);
4465
4466         // Log at will!
4467         logger.info("some info");
4468         logger.warning("some warning");
4469         Thread.sleep(500);
4470         logger.finer("finer information; the answer is " + answer);
4471         Thread.sleep(123);
4472         logger.severe("error!");
4473
4474         // Not mandatory, but cleaner
4475         logger.removeHandler(lttngUstLogHandler);
4476         lttngUstLogHandler.close();
4477     }
4478 }
4479 ----
4480
4481 Build this example:
4482
4483 [role="term"]
4484 ----
4485 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4486 ----
4487
4488 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4489 <<enabling-disabling-events,create an event rule>> matching the
4490 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4491
4492 [role="term"]
4493 ----
4494 $ lttng create
4495 $ lttng enable-event --jul jello
4496 $ lttng start
4497 ----
4498
4499 Run the compiled class:
4500
4501 [role="term"]
4502 ----
4503 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4504 ----
4505
4506 <<basic-tracing-session-control,Stop tracing>> and inspect the
4507 recorded events:
4508
4509 [role="term"]
4510 ----
4511 $ lttng stop
4512 $ lttng view
4513 ----
4514 ====
4515
4516 In the resulting trace, an <<event,event record>> generated by a Java
4517 application using `java.util.logging` is named `lttng_jul:event` and
4518 has the following fields:
4519
4520 `msg`::
4521   Log record message.
4522
4523 `logger_name`::
4524   Logger name.
4525
4526 `class_name`::
4527   Name of the class in which the log statement was executed.
4528
4529 `method_name`::
4530   Name of the method in which the log statement was executed.
4531
4532 `long_millis`::
4533   Logging time (timestamp in milliseconds).
4534
4535 `int_loglevel`::
4536   Log level integer value.
4537
4538 `int_threadid`::
4539   ID of the thread in which the log statement was executed.
4540
4541 Use the opt:lttng-enable-event(1):--loglevel or
4542 opt:lttng-enable-event(1):--loglevel-only option of the
4543 man:lttng-enable-event(1) command to target a range of JUL log levels
4544 or a specific JUL log level.
4545
4546
4547 [role="since-2.8"]
4548 [[log4j]]
4549 ==== Use the LTTng-UST Java agent for Apache log4j
4550
4551 To use the LTTng-UST Java agent in a Java application which uses
4552 Apache log4j{nbsp}1.2:
4553
4554 . In the source code of the Java application, import the LTTng-UST log
4555   appender package for Apache log4j:
4556 +
4557 --
4558 [source,java]
4559 ----
4560 import org.lttng.ust.agent.log4j.LttngLogAppender;
4561 ----
4562 --
4563
4564 . Create an LTTng-UST log4j log appender:
4565 +
4566 --
4567 [source,java]
4568 ----
4569 Appender lttngUstLogAppender = new LttngLogAppender();
4570 ----
4571 --
4572
4573 . Add this appender to the log4j loggers which should emit LTTng events:
4574 +
4575 --
4576 [source,java]
4577 ----
4578 Logger myLogger = Logger.getLogger("some-logger");
4579
4580 myLogger.addAppender(lttngUstLogAppender);
4581 ----
4582 --
4583
4584 . Use Apache log4j log statements and configuration as usual. The
4585   loggers with an attached LTTng-UST log appender can emit LTTng events.
4586
4587 . Before exiting the application, remove the LTTng-UST log appender from
4588   the loggers attached to it and call its `close()` method:
4589 +
4590 --
4591 [source,java]
4592 ----
4593 myLogger.removeAppender(lttngUstLogAppender);
4594 lttngUstLogAppender.close();
4595 ----
4596 --
4597 +
4598 This isn't strictly necessary, but it is recommended for a clean
4599 disposal of the resources of the appender.
4600
4601 . Include the common and log4j-specific JAR
4602   files of the LTTng-UST Java agent, path:{lttng-ust-agent-common.jar} and
4603   path:{lttng-ust-agent-log4j.jar}, in the
4604   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4605   path] when you build the Java application.
4606 +
4607 The JAR files are typically located in dir:{/usr/share/java}.
4608 +
4609 IMPORTANT: The LTTng-UST Java agent must be
4610 <<installing-lttng,installed>> for the logging framework your
4611 application uses.
4612
4613 .Use the LTTng-UST Java agent for Apache log4j.
4614 ====
4615 [source,java]
4616 .path:{Test.java}
4617 ----
4618 import org.apache.log4j.Appender;
4619 import org.apache.log4j.Logger;
4620 import org.lttng.ust.agent.log4j.LttngLogAppender;
4621
4622 public class Test
4623 {
4624     private static final int answer = 42;
4625
4626     public static void main(String[] argv) throws Exception
4627     {
4628         // Create a logger
4629         Logger logger = Logger.getLogger("jello");
4630
4631         // Create an LTTng-UST log appender
4632         Appender lttngUstLogAppender = new LttngLogAppender();
4633
4634         // Add the LTTng-UST log appender to our logger
4635         logger.addAppender(lttngUstLogAppender);
4636
4637         // Log at will!
4638         logger.info("some info");
4639         logger.warn("some warning");
4640         Thread.sleep(500);
4641         logger.debug("debug information; the answer is " + answer);
4642         Thread.sleep(123);
4643         logger.fatal("error!");
4644
4645         // Not mandatory, but cleaner
4646         logger.removeAppender(lttngUstLogAppender);
4647         lttngUstLogAppender.close();
4648     }
4649 }
4650
4651 ----
4652
4653 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4654 file):
4655
4656 [role="term"]
4657 ----
4658 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4659 ----
4660
4661 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4662 <<enabling-disabling-events,create an event rule>> matching the
4663 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4664
4665 [role="term"]
4666 ----
4667 $ lttng create
4668 $ lttng enable-event --log4j jello
4669 $ lttng start
4670 ----
4671
4672 Run the compiled class:
4673
4674 [role="term"]
4675 ----
4676 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4677 ----
4678
4679 <<basic-tracing-session-control,Stop tracing>> and inspect the
4680 recorded events:
4681
4682 [role="term"]
4683 ----
4684 $ lttng stop
4685 $ lttng view
4686 ----
4687 ====
4688
4689 In the resulting trace, an <<event,event record>> generated by a Java
4690 application using log4j is named `lttng_log4j:event` and
4691 has the following fields:
4692
4693 `msg`::
4694   Log record message.
4695
4696 `logger_name`::
4697   Logger name.
4698
4699 `class_name`::
4700   Name of the class in which the log statement was executed.
4701
4702 `method_name`::
4703   Name of the method in which the log statement was executed.
4704
4705 `filename`::
4706   Name of the file in which the executed log statement is located.
4707
4708 `line_number`::
4709   Line number at which the log statement was executed.
4710
4711 `timestamp`::
4712   Logging timestamp.
4713
4714 `int_loglevel`::
4715   Log level integer value.
4716
4717 `thread_name`::
4718   Name of the Java thread in which the log statement was executed.
4719
4720 Use the opt:lttng-enable-event(1):--loglevel or
4721 opt:lttng-enable-event(1):--loglevel-only option of the
4722 man:lttng-enable-event(1) command to target a range of Apache log4j
4723 log levels or a specific log4j log level.
4724
4725
4726 [role="since-2.8"]
4727 [[java-application-context]]
4728 ==== Provide application-specific context fields in a Java application
4729
4730 A Java application-specific context field is a piece of state provided
4731 by the application which <<adding-context,you can add>>, using the
4732 man:lttng-add-context(1) command, to each <<event,event record>>
4733 produced by the log statements of this application.
4734
4735 For example, a given object might have a current request ID variable.
4736 You can create a context information retriever for this object and
4737 assign a name to this current request ID. You can then, using the
4738 man:lttng-add-context(1) command, add this context field by name to
4739 the JUL or log4j <<channel,channel>>.
4740
4741 To provide application-specific context fields in a Java application:
4742
4743 . In the source code of the Java application, import the LTTng-UST
4744   Java agent context classes and interfaces:
4745 +
4746 --
4747 [source,java]
4748 ----
4749 import org.lttng.ust.agent.context.ContextInfoManager;
4750 import org.lttng.ust.agent.context.IContextInfoRetriever;
4751 ----
4752 --
4753
4754 . Create a context information retriever class, that is, a class which
4755   implements the `IContextInfoRetriever` interface:
4756 +
4757 --
4758 [source,java]
4759 ----
4760 class MyContextInfoRetriever implements IContextInfoRetriever
4761 {
4762     @Override
4763     public Object retrieveContextInfo(String key)
4764     {
4765         if (key.equals("intCtx")) {
4766             return (short) 17;
4767         } else if (key.equals("strContext")) {
4768             return "context value!";
4769         } else {
4770             return null;
4771         }
4772     }
4773 }
4774 ----
4775 --
4776 +
4777 This `retrieveContextInfo()` method is the only member of the
4778 `IContextInfoRetriever` interface. Its role is to return the current
4779 value of a state by name to create a context field. The names of the
4780 context fields and which state variables they return depends on your
4781 specific scenario.
4782 +
4783 All primitive types and objects are supported as context fields.
4784 When `retrieveContextInfo()` returns an object, the context field
4785 serializer calls its `toString()` method to add a string field to
4786 event records. The method can also return `null`, which means that
4787 no context field is available for the required name.
4788
4789 . Register an instance of your context information retriever class to
4790   the context information manager singleton:
4791 +
4792 --
4793 [source,java]
4794 ----
4795 IContextInfoRetriever cir = new MyContextInfoRetriever();
4796 ContextInfoManager cim = ContextInfoManager.getInstance();
4797 cim.registerContextInfoRetriever("retrieverName", cir);
4798 ----
4799 --
4800
4801 . Before exiting the application, remove your context information
4802   retriever from the context information manager singleton:
4803 +
4804 --
4805 [source,java]
4806 ----
4807 ContextInfoManager cim = ContextInfoManager.getInstance();
4808 cim.unregisterContextInfoRetriever("retrieverName");
4809 ----
4810 --
4811 +
4812 This isn't strictly necessary, but it is recommended for a clean
4813 disposal of some resources of the manager.
4814
4815 . Build your Java application with LTTng-UST Java agent support as
4816   usual, following the procedure for either the <<jul,JUL>> or
4817   <<log4j,Apache log4j>> framework.
4818
4819
4820 .Provide application-specific context fields in a Java application.
4821 ====
4822 [source,java]
4823 .path:{Test.java}
4824 ----
4825 import java.util.logging.Handler;
4826 import java.util.logging.Logger;
4827 import org.lttng.ust.agent.jul.LttngLogHandler;
4828 import org.lttng.ust.agent.context.ContextInfoManager;
4829 import org.lttng.ust.agent.context.IContextInfoRetriever;
4830
4831 public class Test
4832 {
4833     // Our context information retriever class
4834     private static class MyContextInfoRetriever
4835     implements IContextInfoRetriever
4836     {
4837         @Override
4838         public Object retrieveContextInfo(String key) {
4839             if (key.equals("intCtx")) {
4840                 return (short) 17;
4841             } else if (key.equals("strContext")) {
4842                 return "context value!";
4843             } else {
4844                 return null;
4845             }
4846         }
4847     }
4848
4849     private static final int answer = 42;
4850
4851     public static void main(String args[]) throws Exception
4852     {
4853         // Get the context information manager instance
4854         ContextInfoManager cim = ContextInfoManager.getInstance();
4855
4856         // Create and register our context information retriever
4857         IContextInfoRetriever cir = new MyContextInfoRetriever();
4858         cim.registerContextInfoRetriever("myRetriever", cir);
4859
4860         // Create a logger
4861         Logger logger = Logger.getLogger("jello");
4862
4863         // Create an LTTng-UST log handler
4864         Handler lttngUstLogHandler = new LttngLogHandler();
4865
4866         // Add the LTTng-UST log handler to our logger
4867         logger.addHandler(lttngUstLogHandler);
4868
4869         // Log at will!
4870         logger.info("some info");
4871         logger.warning("some warning");
4872         Thread.sleep(500);
4873         logger.finer("finer information; the answer is " + answer);
4874         Thread.sleep(123);
4875         logger.severe("error!");
4876
4877         // Not mandatory, but cleaner
4878         logger.removeHandler(lttngUstLogHandler);
4879         lttngUstLogHandler.close();
4880         cim.unregisterContextInfoRetriever("myRetriever");
4881     }
4882 }
4883 ----
4884
4885 Build this example:
4886
4887 [role="term"]
4888 ----
4889 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4890 ----
4891
4892 <<creating-destroying-tracing-sessions,Create a tracing session>>
4893 and <<enabling-disabling-events,create an event rule>> matching the
4894 `jello` JUL logger:
4895
4896 [role="term"]
4897 ----
4898 $ lttng create
4899 $ lttng enable-event --jul jello
4900 ----
4901
4902 <<adding-context,Add the application-specific context fields>> to the
4903 JUL channel:
4904
4905 [role="term"]
4906 ----
4907 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
4908 $ lttng add-context --jul --type='$app.myRetriever:strContext'
4909 ----
4910
4911 <<basic-tracing-session-control,Start tracing>>:
4912
4913 [role="term"]
4914 ----
4915 $ lttng start
4916 ----
4917
4918 Run the compiled class:
4919
4920 [role="term"]
4921 ----
4922 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4923 ----
4924
4925 <<basic-tracing-session-control,Stop tracing>> and inspect the
4926 recorded events:
4927
4928 [role="term"]
4929 ----
4930 $ lttng stop
4931 $ lttng view
4932 ----
4933 ====
4934
4935
4936 [role="since-2.7"]
4937 [[python-application]]
4938 === User space Python agent
4939
4940 You can instrument a Python{nbsp}2 or Python{nbsp}3 application which
4941 uses the standard
4942 https://docs.python.org/3/library/logging.html[`logging`] package.
4943
4944 Each log statement emits an LTTng event once the
4945 application module imports the
4946 <<lttng-ust-agents,LTTng-UST Python agent>> package.
4947
4948 [role="img-100"]
4949 .A Python application importing the LTTng-UST Python agent.
4950 image::python-app.png[]
4951
4952 To use the LTTng-UST Python agent:
4953
4954 . In the source code of the Python application, import the LTTng-UST
4955   Python agent:
4956 +
4957 --
4958 [source,python]
4959 ----
4960 import lttngust
4961 ----
4962 --
4963 +
4964 The LTTng-UST Python agent automatically adds its logging handler to the
4965 root logger at import time.
4966 +
4967 Any log statement that the application executes before this import does
4968 not emit an LTTng event.
4969 +
4970 IMPORTANT: The LTTng-UST Python agent must be
4971 <<installing-lttng,installed>>.
4972
4973 . Use log statements and logging configuration as usual.
4974   Since the LTTng-UST Python agent adds a handler to the _root_
4975   logger, you can trace any log statement from any logger.
4976
4977 .Use the LTTng-UST Python agent.
4978 ====
4979 [source,python]
4980 .path:{test.py}
4981 ----
4982 import lttngust
4983 import logging
4984 import time
4985
4986
4987 def example():
4988     logging.basicConfig()
4989     logger = logging.getLogger('my-logger')
4990
4991     while True:
4992         logger.debug('debug message')
4993         logger.info('info message')
4994         logger.warn('warn message')
4995         logger.error('error message')
4996         logger.critical('critical message')
4997         time.sleep(1)
4998
4999
5000 if __name__ == '__main__':
5001     example()
5002 ----
5003
5004 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
5005 logging handler which prints to the standard error stream, isn't
5006 strictly required for LTTng-UST tracing to work, but in versions of
5007 Python preceding{nbsp}3.2, you could see a warning message which indicates
5008 that no handler exists for the logger `my-logger`.
5009
5010 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5011 <<enabling-disabling-events,create an event rule>> matching the
5012 `my-logger` Python logger, and <<basic-tracing-session-control,start
5013 tracing>>:
5014
5015 [role="term"]
5016 ----
5017 $ lttng create
5018 $ lttng enable-event --python my-logger
5019 $ lttng start
5020 ----
5021
5022 Run the Python script:
5023
5024 [role="term"]
5025 ----
5026 $ python test.py
5027 ----
5028
5029 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5030 events:
5031
5032 [role="term"]
5033 ----
5034 $ lttng stop
5035 $ lttng view
5036 ----
5037 ====
5038
5039 In the resulting trace, an <<event,event record>> generated by a Python
5040 application is named `lttng_python:event` and has the following fields:
5041
5042 `asctime`::
5043   Logging time (string).
5044
5045 `msg`::
5046   Log record message.
5047
5048 `logger_name`::
5049   Logger name.
5050
5051 `funcName`::
5052   Name of the function in which the log statement was executed.
5053
5054 `lineno`::
5055   Line number at which the log statement was executed.
5056
5057 `int_loglevel`::
5058   Log level integer value.
5059
5060 `thread`::
5061   ID of the Python thread in which the log statement was executed.
5062
5063 `threadName`::
5064   Name of the Python thread in which the log statement was executed.
5065
5066 Use the opt:lttng-enable-event(1):--loglevel or
5067 opt:lttng-enable-event(1):--loglevel-only option of the
5068 man:lttng-enable-event(1) command to target a range of Python log levels
5069 or a specific Python log level.
5070
5071 When an application imports the LTTng-UST Python agent, the agent tries
5072 to register to a <<lttng-sessiond,session daemon>>. Note that you must
5073 <<start-sessiond,start the session daemon>> _before_ you run the Python
5074 application. If a session daemon is found, the agent tries to register
5075 to it during five seconds, after which the application continues
5076 without LTTng tracing support. Override this timeout value with
5077 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
5078 (milliseconds).
5079
5080 If the session daemon stops while a Python application with an imported
5081 LTTng-UST Python agent runs, the agent retries to connect and to
5082 register to a session daemon every three seconds. Override this
5083 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
5084 variable.
5085
5086
5087 [role="since-2.5"]
5088 [[proc-lttng-logger-abi]]
5089 === LTTng logger
5090
5091 The `lttng-tracer` Linux kernel module, part of
5092 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger files
5093 path:{/proc/lttng-logger} and path:{/dev/lttng-logger} (since
5094 LTTng{nbsp}2.11) when it's loaded. Any application can write text data
5095 to any of those files to emit an LTTng event.
5096
5097 [role="img-100"]
5098 .An application writes to the LTTng logger file to emit an LTTng event.
5099 image::lttng-logger.png[]
5100
5101 The LTTng logger is the quickest method--not the most efficient,
5102 however--to add instrumentation to an application. It is designed
5103 mostly to instrument shell scripts:
5104
5105 [role="term"]
5106 ----
5107 $ echo "Some message, some $variable" > /dev/lttng-logger
5108 ----
5109
5110 Any event that the LTTng logger emits is named `lttng_logger` and
5111 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
5112 other instrumentation points in the kernel tracing domain, **any Unix
5113 user** can <<enabling-disabling-events,create an event rule>> which
5114 matches its event name, not only the root user or users in the
5115 <<tracing-group,tracing group>>.
5116
5117 To use the LTTng logger:
5118
5119 * From any application, write text data to the path:{/dev/lttng-logger}
5120   file.
5121
5122 The `msg` field of `lttng_logger` event records contains the
5123 recorded message.
5124
5125 NOTE: The maximum message length of an LTTng logger event is
5126 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
5127 than one event to contain the remaining data.
5128
5129 You shouldn't use the LTTng logger to trace a user application which
5130 can be instrumented in a more efficient way, namely:
5131
5132 * <<c-application,C and $$C++$$ applications>>.
5133 * <<java-application,Java applications>>.
5134 * <<python-application,Python applications>>.
5135
5136 .Use the LTTng logger.
5137 ====
5138 [source,bash]
5139 .path:{test.bash}
5140 ----
5141 echo 'Hello, World!' > /dev/lttng-logger
5142 sleep 2
5143 df --human-readable --print-type / > /dev/lttng-logger
5144 ----
5145
5146 <<creating-destroying-tracing-sessions,Create a tracing session>>,
5147 <<enabling-disabling-events,create an event rule>> matching the
5148 `lttng_logger` Linux kernel tracepoint, and
5149 <<basic-tracing-session-control,start tracing>>:
5150
5151 [role="term"]
5152 ----
5153 $ lttng create
5154 $ lttng enable-event --kernel lttng_logger
5155 $ lttng start
5156 ----
5157
5158 Run the Bash script:
5159
5160 [role="term"]
5161 ----
5162 $ bash test.bash
5163 ----
5164
5165 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
5166 events:
5167
5168 [role="term"]
5169 ----
5170 $ lttng stop
5171 $ lttng view
5172 ----
5173 ====
5174
5175
5176 [[instrumenting-linux-kernel]]
5177 === LTTng kernel tracepoints
5178
5179 NOTE: This section shows how to _add_ instrumentation points to the
5180 Linux kernel. The subsystems of the kernel are already thoroughly
5181 instrumented at strategic places for LTTng when you
5182 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
5183 package.
5184
5185 ////
5186 There are two methods to instrument the Linux kernel:
5187
5188 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
5189   tracepoint which uses the `TRACE_EVENT()` API.
5190 +
5191 Choose this if you want to instrumentation a Linux kernel tree with an
5192 instrumentation point compatible with ftrace, perf, and SystemTap.
5193
5194 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
5195   instrument an out-of-tree kernel module.
5196 +
5197 Choose this if you don't need ftrace, perf, or SystemTap support.
5198 ////
5199
5200
5201 [[linux-add-lttng-layer]]
5202 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
5203
5204 This section shows how to add an LTTng layer to existing ftrace
5205 instrumentation using the `TRACE_EVENT()` API.
5206
5207 This section doesn't document the `TRACE_EVENT()` macro. Read the
5208 following articles to learn more about this API:
5209
5210 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part{nbsp}1)]
5211 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part{nbsp}2)]
5212 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part{nbsp}3)]
5213
5214 The following procedure assumes that your ftrace tracepoints are
5215 correctly defined in their own header and that they are created in
5216 one source file using the `CREATE_TRACE_POINTS` definition.
5217
5218 To add an LTTng layer over an existing ftrace tracepoint:
5219
5220 . Make sure the following kernel configuration options are
5221   enabled:
5222 +
5223 --
5224 * `CONFIG_MODULES`
5225 * `CONFIG_KALLSYMS`
5226 * `CONFIG_HIGH_RES_TIMERS`
5227 * `CONFIG_TRACEPOINTS`
5228 --
5229
5230 . Build the Linux source tree with your custom ftrace tracepoints.
5231 . Boot the resulting Linux image on your target system.
5232 +
5233 Confirm that the tracepoints exist by looking for their names in the
5234 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
5235 is your subsystem name.
5236
5237 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
5238 +
5239 --
5240 [role="term"]
5241 ----
5242 $ cd $(mktemp -d) &&
5243 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.12.tar.bz2 &&
5244 tar -xf lttng-modules-latest-2.12.tar.bz2 &&
5245 cd lttng-modules-2.12.*
5246 ----
5247 --
5248
5249 . In dir:{instrumentation/events/lttng-module}, relative to the root
5250   of the LTTng-modules source tree, create a header file named
5251   +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
5252   LTTng-modules tracepoint definitions using the LTTng-modules
5253   macros in it.
5254 +
5255 Start with this template:
5256 +
5257 --
5258 [source,c]
5259 .path:{instrumentation/events/lttng-module/my_subsys.h}
5260 ----
5261 #undef TRACE_SYSTEM
5262 #define TRACE_SYSTEM my_subsys
5263
5264 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
5265 #define _LTTNG_MY_SUBSYS_H
5266
5267 #include "../../../probes/lttng-tracepoint-event.h"
5268 #include <linux/tracepoint.h>
5269
5270 LTTNG_TRACEPOINT_EVENT(
5271     /*
5272      * Format is identical to the TRACE_EVENT() version for the three
5273      * following macro parameters:
5274      */
5275     my_subsys_my_event,
5276     TP_PROTO(int my_int, const char *my_string),
5277     TP_ARGS(my_int, my_string),
5278
5279     /* LTTng-modules specific macros */
5280     TP_FIELDS(
5281         ctf_integer(int, my_int_field, my_int)
5282         ctf_string(my_bar_field, my_bar)
5283     )
5284 )
5285
5286 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5287
5288 #include "../../../probes/define_trace.h"
5289 ----
5290 --
5291 +
5292 The entries in the `TP_FIELDS()` section are the list of fields for the
5293 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5294 the `TRACE_EVENT()` ftrace macro.
5295 +
5296 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5297 complete description of the available `ctf_*()` macros.
5298
5299 . Create the kernel module C{nbsp}source file of the LTTng-modules
5300   probe, +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5301   subsystem name:
5302 +
5303 --
5304 [source,c]
5305 .path:{probes/lttng-probe-my-subsys.c}
5306 ----
5307 #include <linux/module.h>
5308 #include "../lttng-tracer.h"
5309
5310 /*
5311  * Build-time verification of mismatch between mainline
5312  * TRACE_EVENT() arguments and the LTTng-modules adaptation
5313  * layer LTTNG_TRACEPOINT_EVENT() arguments.
5314  */
5315 #include <trace/events/my_subsys.h>
5316
5317 /* Create LTTng tracepoint probes */
5318 #define LTTNG_PACKAGE_BUILD
5319 #define CREATE_TRACE_POINTS
5320 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5321
5322 #include "../instrumentation/events/lttng-module/my_subsys.h"
5323
5324 MODULE_LICENSE("GPL and additional rights");
5325 MODULE_AUTHOR("Your name <your-email>");
5326 MODULE_DESCRIPTION("LTTng my_subsys probes");
5327 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5328     __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5329     __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5330     LTTNG_MODULES_EXTRAVERSION);
5331 ----
5332 --
5333
5334 . Edit path:{probes/KBuild} and add your new kernel module object
5335   next to the existing ones:
5336 +
5337 --
5338 [source,make]
5339 .path:{probes/KBuild}
5340 ----
5341 # ...
5342
5343 obj-m += lttng-probe-module.o
5344 obj-m += lttng-probe-power.o
5345
5346 obj-m += lttng-probe-my-subsys.o
5347
5348 # ...
5349 ----
5350 --
5351
5352 . Build and install the LTTng kernel modules:
5353 +
5354 --
5355 [role="term"]
5356 ----
5357 $ make KERNELDIR=/path/to/linux
5358 # make modules_install && depmod -a
5359 ----
5360 --
5361 +
5362 Replace `/path/to/linux` with the path to the Linux source tree where
5363 you defined and used tracepoints with the `TRACE_EVENT()` ftrace macro.
5364
5365 Note that you can also use the
5366 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5367 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5368 C code that need to be executed before the event fields are recorded.
5369
5370 The best way to learn how to use the previous LTTng-modules macros is to
5371 inspect the existing LTTng-modules tracepoint definitions in the
5372 dir:{instrumentation/events/lttng-module} header files. Compare them
5373 with the Linux kernel mainline versions in the
5374 dir:{include/trace/events} directory of the Linux source tree.
5375
5376
5377 [role="since-2.7"]
5378 [[lttng-tracepoint-event-code]]
5379 ===== Use custom C code to access the data for tracepoint fields
5380
5381 Although we recommended to always use the
5382 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5383 the arguments and fields of an LTTng-modules tracepoint when possible,
5384 sometimes you need a more complex process to access the data that the
5385 tracer records as event record fields. In other words, you need local
5386 variables and multiple C{nbsp}statements instead of simple
5387 argument-based expressions that you pass to the
5388 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5389
5390 Use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5391 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5392 a block of C{nbsp}code to be executed before LTTng records the fields.
5393 The structure of this macro is:
5394
5395 [source,c]
5396 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5397 ----
5398 LTTNG_TRACEPOINT_EVENT_CODE(
5399     /*
5400      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5401      * version for the following three macro parameters:
5402      */
5403     my_subsys_my_event,
5404     TP_PROTO(int my_int, const char *my_string),
5405     TP_ARGS(my_int, my_string),
5406
5407     /* Declarations of custom local variables */
5408     TP_locvar(
5409         int a = 0;
5410         unsigned long b = 0;
5411         const char *name = "(undefined)";
5412         struct my_struct *my_struct;
5413     ),
5414
5415     /*
5416      * Custom code which uses both tracepoint arguments
5417      * (in TP_ARGS()) and local variables (in TP_locvar()).
5418      *
5419      * Local variables are actually members of a structure pointed
5420      * to by the special variable tp_locvar.
5421      */
5422     TP_code(
5423         if (my_int) {
5424             tp_locvar->a = my_int + 17;
5425             tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5426             tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5427             tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5428             put_my_struct(tp_locvar->my_struct);
5429
5430             if (tp_locvar->b) {
5431                 tp_locvar->a = 1;
5432             }
5433         }
5434     ),
5435
5436     /*
5437      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5438      * version for this, except that tp_locvar members can be
5439      * used in the argument expression parameters of
5440      * the ctf_*() macros.
5441      */
5442     TP_FIELDS(
5443         ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5444         ctf_integer(int, my_struct_a, tp_locvar->a)
5445         ctf_string(my_string_field, my_string)
5446         ctf_string(my_struct_name, tp_locvar->name)
5447     )
5448 )
5449 ----
5450
5451 IMPORTANT: The C code defined in `TP_code()` must not have any side
5452 effects when executed. In particular, the code must not allocate
5453 memory or get resources without deallocating this memory or putting
5454 those resources afterwards.
5455
5456
5457 [[instrumenting-linux-kernel-tracing]]
5458 ==== Load and unload a custom probe kernel module
5459
5460 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5461 kernel module>> in the kernel before it can emit LTTng events.
5462
5463 To load the default probe kernel modules and a custom probe kernel
5464 module:
5465
5466 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5467   probe modules to load when starting a root <<lttng-sessiond,session
5468   daemon>>:
5469 +
5470 --
5471 .Load the `my_subsys`, `usb`, and the default probe modules.
5472 ====
5473 [role="term"]
5474 ----
5475 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5476 ----
5477 ====
5478 --
5479 +
5480 You only need to pass the subsystem name, not the whole kernel module
5481 name.
5482
5483 To load _only_ a given custom probe kernel module:
5484
5485 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5486   modules to load when starting a root session daemon:
5487 +
5488 --
5489 .Load only the `my_subsys` and `usb` probe modules.
5490 ====
5491 [role="term"]
5492 ----
5493 # lttng-sessiond --kmod-probes=my_subsys,usb
5494 ----
5495 ====
5496 --
5497
5498 To confirm that a probe module is loaded:
5499
5500 * Use man:lsmod(8):
5501 +
5502 --
5503 [role="term"]
5504 ----
5505 $ lsmod | grep lttng_probe_usb
5506 ----
5507 --
5508
5509 To unload the loaded probe modules:
5510
5511 * Kill the session daemon with `SIGTERM`:
5512 +
5513 --
5514 [role="term"]
5515 ----
5516 # pkill lttng-sessiond
5517 ----
5518 --
5519 +
5520 You can also use the man:modprobe(8) `--remove` option if the session
5521 daemon terminates abnormally.
5522
5523
5524 [[controlling-tracing]]
5525 == Tracing control
5526
5527 Once an application or a Linux kernel is
5528 <<instrumenting,instrumented>> for LTTng tracing,
5529 you can _trace_ it.
5530
5531 This section is divided in topics on how to use the various
5532 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5533 command-line tool>>, to _control_ the LTTng daemons and tracers.
5534
5535 NOTE: In the following subsections, we refer to an man:lttng(1) command
5536 using its man page name. For example, instead of _Run the `create`
5537 command to..._, we use _Run the man:lttng-create(1) command to..._.
5538
5539
5540 [[start-sessiond]]
5541 === Start a session daemon
5542
5543 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5544 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5545 command-line tool.
5546
5547 You will see the following error when you run a command while no session
5548 daemon is running:
5549
5550 ----
5551 Error: No session daemon is available
5552 ----
5553
5554 The only command that automatically runs a session daemon is
5555 man:lttng-create(1), which you use to
5556 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5557 this is most of the time the first operation that you do, sometimes it's
5558 not. Some examples are:
5559
5560 * <<list-instrumentation-points,List the available instrumentation points>>.
5561 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5562
5563 [[tracing-group]] Each Unix user must have its own running session
5564 daemon to trace user applications. The session daemon that the root user
5565 starts is the only one allowed to control the LTTng kernel tracer. Users
5566 that are part of the _tracing group_ can control the root session
5567 daemon. The default tracing group name is `tracing`; set it to something
5568 else with the opt:lttng-sessiond(8):--group option when you start the
5569 root session daemon.
5570
5571 To start a user session daemon:
5572
5573 * Run man:lttng-sessiond(8):
5574 +
5575 --
5576 [role="term"]
5577 ----
5578 $ lttng-sessiond --daemonize
5579 ----
5580 --
5581
5582 To start the root session daemon:
5583
5584 * Run man:lttng-sessiond(8) as the root user:
5585 +
5586 --
5587 [role="term"]
5588 ----
5589 # lttng-sessiond --daemonize
5590 ----
5591 --
5592
5593 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5594 start the session daemon in foreground.
5595
5596 To stop a session daemon, use man:kill(1) on its process ID (standard
5597 `TERM` signal).
5598
5599 Note that some Linux distributions could manage the LTTng session daemon
5600 as a service. In this case, you should use the service manager to
5601 start, restart, and stop session daemons.
5602
5603
5604 [[creating-destroying-tracing-sessions]]
5605 === Create and destroy a tracing session
5606
5607 Almost all the LTTng control operations happen in the scope of
5608 a <<tracing-session,tracing session>>, which is the dialogue between the
5609 <<lttng-sessiond,session daemon>> and you.
5610
5611 To create a tracing session with a generated name:
5612
5613 * Use the man:lttng-create(1) command:
5614 +
5615 --
5616 [role="term"]
5617 ----
5618 $ lttng create
5619 ----
5620 --
5621
5622 The name of the created tracing session is `auto` followed by the
5623 creation date.
5624
5625 To create a tracing session with a specific name:
5626
5627 * Use the optional argument of the man:lttng-create(1) command:
5628 +
5629 --
5630 [role="term"]
5631 ----
5632 $ lttng create my-session
5633 ----
5634 --
5635 +
5636 Replace `my-session` with the specific tracing session name.
5637
5638 LTTng appends the creation date to the name of the created tracing
5639 session.
5640
5641 LTTng writes the traces of a tracing session in
5642 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5643 name of the tracing session. Note that the env:LTTNG_HOME environment
5644 variable defaults to `$HOME` if not set.
5645
5646 To output LTTng traces to a non-default location:
5647
5648 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5649 +
5650 --
5651 [role="term"]
5652 ----
5653 $ lttng create my-session --output=/tmp/some-directory
5654 ----
5655 --
5656
5657 You may create as many tracing sessions as you wish.
5658
5659 To list all the existing tracing sessions for your Unix user:
5660
5661 * Use the man:lttng-list(1) command:
5662 +
5663 --
5664 [role="term"]
5665 ----
5666 $ lttng list
5667 ----
5668 --
5669
5670 [[cur-tracing-session]]When you create a tracing session, it is set as
5671 the _current tracing session_. The following man:lttng(1) commands
5672 operate on the current tracing session when you don't specify one:
5673
5674 [role="list-3-cols"]
5675 * man:lttng-add-context(1)
5676 * man:lttng-clear(1)
5677 * man:lttng-destroy(1)
5678 * man:lttng-disable-channel(1)
5679 * man:lttng-disable-event(1)
5680 * man:lttng-disable-rotation(1)
5681 * man:lttng-enable-channel(1)
5682 * man:lttng-enable-event(1)
5683 * man:lttng-enable-rotation(1)
5684 * man:lttng-load(1)
5685 * man:lttng-regenerate(1)
5686 * man:lttng-rotate(1)
5687 * man:lttng-save(1)
5688 * man:lttng-snapshot(1)
5689 * man:lttng-start(1)
5690 * man:lttng-status(1)
5691 * man:lttng-stop(1)
5692 * man:lttng-track(1)
5693 * man:lttng-untrack(1)
5694 * man:lttng-view(1)
5695
5696 To change the current tracing session:
5697
5698 * Use the man:lttng-set-session(1) command:
5699 +
5700 --
5701 [role="term"]
5702 ----
5703 $ lttng set-session new-session
5704 ----
5705 --
5706 +
5707 Replace `new-session` by the name of the new current tracing session.
5708
5709 When you're done tracing in a given tracing session, destroy it. This
5710 operation frees the resources taken by the tracing session to destroy;
5711 it doesn't destroy the trace data that LTTng wrote for this tracing
5712 session (see <<clear,Clear a tracing session>> for one way to do this).
5713
5714 To destroy the current tracing session:
5715
5716 * Use the man:lttng-destroy(1) command:
5717 +
5718 --
5719 [role="term"]
5720 ----
5721 $ lttng destroy
5722 ----
5723 --
5724
5725 The man:lttng-destroy(1) command also runs the man:lttng-stop(1)
5726 command implicitly (see <<basic-tracing-session-control,Start and stop a
5727 tracing session>>). You need to stop tracing to make LTTng flush the
5728 remaining trace data and make the trace readable.
5729
5730
5731 [[list-instrumentation-points]]
5732 === List the available instrumentation points
5733
5734 The <<lttng-sessiond,session daemon>> can query the running instrumented
5735 user applications and the Linux kernel to get a list of available
5736 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5737 they are tracepoints and system calls. For the user space tracing
5738 domain, they are tracepoints. For the other tracing domains, they are
5739 logger names.
5740
5741 To list the available instrumentation points:
5742
5743 * Use the man:lttng-list(1) command with the option of the requested
5744   tracing domain amongst:
5745 +
5746 --
5747 opt:lttng-list(1):--kernel::
5748     Linux kernel tracepoints (your Unix user must be a root user, or it
5749     must be a member of the <<tracing-group,tracing group>>).
5750
5751 opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall::
5752     Linux kernel system calls (your Unix user must be a root user, or it
5753     must be a member of the tracing group).
5754
5755 opt:lttng-list(1):--userspace::
5756     User space tracepoints.
5757
5758 opt:lttng-list(1):--jul::
5759    `java.util.logging` loggers.
5760
5761 opt:lttng-list(1):--log4j::
5762     Apache log4j loggers.
5763
5764 opt:lttng-list(1):--python::
5765     Python loggers.
5766 --
5767
5768 .List the available user space tracepoints.
5769 ====
5770 [role="term"]
5771 ----
5772 $ lttng list --userspace
5773 ----
5774 ====
5775
5776 .List the available Linux kernel system call tracepoints.
5777 ====
5778 [role="term"]
5779 ----
5780 $ lttng list --kernel --syscall
5781 ----
5782 ====
5783
5784
5785 [[enabling-disabling-events]]
5786 === Create and enable an event rule
5787
5788 Once you <<creating-destroying-tracing-sessions,create a tracing
5789 session>>, you can create <<event,event rules>> with the
5790 man:lttng-enable-event(1) command.
5791
5792 You specify each condition with a command-line option. The available
5793 condition arguments are shown in the following table.
5794
5795 [role="growable",cols="asciidoc,asciidoc,default"]
5796 .Condition command-line arguments for the man:lttng-enable-event(1) command.
5797 |====
5798 |Argument |Description |Applicable tracing domains
5799
5800 |
5801 One of:
5802
5803 . `--syscall`
5804 . +--probe=__ADDR__+
5805 . +--function=__ADDR__+
5806 . +--userspace-probe=__PATH__:__SYMBOL__+
5807 . +--userspace-probe=sdt:__PATH__:__PROVIDER__:__NAME__+
5808
5809 |
5810 Instead of using the default _tracepoint_ instrumentation type, use:
5811
5812 . A Linux system call (entry and exit).
5813 . A Linux https://lwn.net/Articles/132196/[kprobe] (symbol or address).
5814 . The entry and return points of a Linux function (symbol or address).
5815 . The entry point of a user application or library function (path to
5816   application/library and symbol).
5817 . A https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[SystemTap
5818   Statically Defined Tracing] (USDT) probe (path to application/library,
5819   provider and probe names).
5820
5821 |Linux kernel.
5822
5823 |First positional argument.
5824
5825 |
5826 Tracepoint or system call name.
5827
5828 With the opt:lttng-enable-event(1):--probe,
5829 opt:lttng-enable-event(1):--function, and
5830 opt:lttng-enable-event(1):--userspace-probe options, this is a custom
5831 name given to the event rule. With the JUL, log4j, and Python domains,
5832 this is a logger name.
5833
5834 With a tracepoint, logger, or system call name, use the special
5835 `*` globbing character to match anything (for example, `sched_*`,
5836 `my_comp*:*msg_*`).
5837
5838 |All.
5839
5840 |
5841 One of:
5842
5843 . +--loglevel=__LEVEL__+
5844 . +--loglevel-only=__LEVEL__+
5845
5846 |
5847 . Match only tracepoints or log statements with a logging level at
5848   least as severe as +__LEVEL__+.
5849 . Match only tracepoints or log statements with a logging level
5850   equal to +__LEVEL__+.
5851
5852 See man:lttng-enable-event(1) for the list of available logging level
5853 names.
5854
5855 |User space, JUL, log4j, and Python.
5856
5857 |+--exclude=__EXCLUSIONS__+
5858
5859 |
5860 When you use a `*` character at the end of the tracepoint or logger
5861 name (first positional argument), exclude the specific names in the
5862 comma-delimited list +__EXCLUSIONS__+.
5863
5864 |
5865 User space, JUL, log4j, and Python.
5866
5867 |+--filter=__EXPR__+
5868
5869 |
5870 Match only events which satisfy the expression +__EXPR__+.
5871
5872 See man:lttng-enable-event(1) to learn more about the syntax of a
5873 filter expression.
5874
5875 |All.
5876
5877 |====
5878
5879 You attach an event rule to a <<channel,channel>> on creation. If you do
5880 not specify the channel with the opt:lttng-enable-event(1):--channel
5881 option, and if the event rule to create is the first in its
5882 <<domain,tracing domain>> for a given tracing session, then LTTng
5883 creates a _default channel_ for you. This default channel is reused in
5884 subsequent invocations of the man:lttng-enable-event(1) command for the
5885 same tracing domain.
5886
5887 An event rule is always enabled at creation time.
5888
5889 The following examples show how to combine the previous
5890 command-line options to create simple to more complex event rules.
5891
5892 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5893 ====
5894 [role="term"]
5895 ----
5896 $ lttng enable-event --kernel sched_switch
5897 ----
5898 ====
5899
5900 .Create an event rule matching four Linux kernel system calls (default channel).
5901 ====
5902 [role="term"]
5903 ----
5904 $ lttng enable-event --kernel --syscall open,write,read,close
5905 ----
5906 ====
5907
5908 .Create event rules matching tracepoints with filter expressions (default channel).
5909 ====
5910 [role="term"]
5911 ----
5912 $ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5913 ----
5914
5915 [role="term"]
5916 ----
5917 $ lttng enable-event --kernel --all \
5918                    --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5919 ----
5920
5921 [role="term"]
5922 ----
5923 $ lttng enable-event --jul my_logger \
5924                    --filter='$app.retriever:cur_msg_id > 3'
5925 ----
5926
5927 IMPORTANT: Make sure to always quote the filter string when you
5928 use man:lttng(1) from a shell.
5929
5930 See also <<pid-tracking,Track process attributes>> which offers another,
5931 more efficient filtering mechanism for process ID, user ID, and group
5932 ID attributes.
5933 ====
5934
5935 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5936 ====
5937 [role="term"]
5938 ----
5939 $ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5940 ----
5941
5942 IMPORTANT: Make sure to always quote the wildcard character when you
5943 use man:lttng(1) from a shell.
5944 ====
5945
5946 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5947 ====
5948 [role="term"]
5949 ----
5950 $ lttng enable-event --python my-app.'*' \
5951                    --exclude='my-app.module,my-app.hello'
5952 ----
5953 ====
5954
5955 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5956 ====
5957 [role="term"]
5958 ----
5959 $ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5960 ----
5961 ====
5962
5963 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5964 ====
5965 [role="term"]
5966 ----
5967 $ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5968 ----
5969 ====
5970
5971 .Create an event rule matching the `malloc` function entry in path:{/usr/lib/libc.so.6}:
5972 ====
5973 [role="term"]
5974 ----
5975 $ lttng enable-event --kernel --userspace-probe=/usr/lib/libc.so.6:malloc \
5976                    libc_malloc
5977 ----
5978 ====
5979
5980 .Create an event rule matching the `server`/`accept_request` https://www.sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps[USDT probe] in path:{/usr/bin/serv}:
5981 ====
5982 [role="term"]
5983 ----
5984 $ lttng enable-event --kernel --userspace-probe=sdt:serv:server:accept_request \
5985                   server_accept_request
5986 ----
5987 ====
5988
5989 The event rules of a given channel form a whitelist: as soon as an
5990 emitted event passes one of them, LTTng can record the event. For
5991 example, an event named `my_app:my_tracepoint` emitted from a user space
5992 tracepoint with a `TRACE_ERROR` log level passes both of the following
5993 rules:
5994
5995 [role="term"]
5996 ----
5997 $ lttng enable-event --userspace my_app:my_tracepoint
5998 $ lttng enable-event --userspace my_app:my_tracepoint \
5999                    --loglevel=TRACE_INFO
6000 ----
6001
6002 The second event rule is redundant: the first one includes
6003 the second one.
6004
6005
6006 [[disable-event-rule]]
6007 === Disable an event rule
6008
6009 To disable an event rule that you <<enabling-disabling-events,created>>
6010 previously, use the man:lttng-disable-event(1) command. This command
6011 disables _all_ the event rules (of a given tracing domain and channel)
6012 which match an instrumentation point. The other conditions aren't
6013 supported as of LTTng{nbsp}{revision}.
6014
6015 The LTTng tracer doesn't record an emitted event which passes
6016 a _disabled_ event rule.
6017
6018 .Disable an event rule matching a Python logger (default channel).
6019 ====
6020 [role="term"]
6021 ----
6022 $ lttng disable-event --python my-logger
6023 ----
6024 ====
6025
6026 .Disable an event rule matching all `java.util.logging` loggers (default channel).
6027 ====
6028 [role="term"]
6029 ----
6030 $ lttng disable-event --jul '*'
6031 ----
6032 ====
6033
6034 .Disable _all_ the event rules of the default channel.
6035 ====
6036 The opt:lttng-disable-event(1):--all-events option isn't, like the
6037 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
6038 equivalent of the event name `*` (wildcard): it disables _all_ the event
6039 rules of a given channel.
6040
6041 [role="term"]
6042 ----
6043 $ lttng disable-event --jul --all-events
6044 ----
6045 ====
6046
6047 NOTE: You can't delete an event rule once you create it.
6048
6049
6050 [[status]]
6051 === Get the status of a tracing session
6052
6053 To get the status of the <<cur-tracing-session,current tracing
6054 session>>, that is, its parameters, its channels, event rules, and their
6055 attributes:
6056
6057 * Use the man:lttng-status(1) command:
6058 +
6059 --
6060 [role="term"]
6061 ----
6062 $ lttng status
6063 ----
6064 --
6065
6066 To get the status of any tracing session:
6067
6068 * Use the man:lttng-list(1) command with the name of the tracing
6069   session:
6070 +
6071 --
6072 [role="term"]
6073 ----
6074 $ lttng list my-session
6075 ----
6076 --
6077 +
6078 Replace `my-session` with the desired tracing session name.
6079
6080
6081 [[basic-tracing-session-control]]
6082 === Start and stop a tracing session
6083
6084 Once you <<creating-destroying-tracing-sessions,create a tracing
6085 session>> and
6086 <<enabling-disabling-events,create one or more event rules>>,
6087 you can start and stop the tracers for this tracing session.
6088
6089 To start tracing in the <<cur-tracing-session,current tracing session>>:
6090
6091 * Use the man:lttng-start(1) command:
6092 +
6093 --
6094 [role="term"]
6095 ----
6096 $ lttng start
6097 ----
6098 --
6099
6100 LTTng is very flexible: you can launch user applications before
6101 or after the you start the tracers. The tracers only record the events
6102 if they pass enabled event rules and if they occur while the tracers are
6103 started.
6104
6105 To stop tracing in the current tracing session:
6106
6107 * Use the man:lttng-stop(1) command:
6108 +
6109 --
6110 [role="term"]
6111 ----
6112 $ lttng stop
6113 ----
6114 --
6115 +
6116 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
6117 records>> or lost sub-buffers since the last time you ran
6118 man:lttng-start(1), warnings are printed when you run the
6119 man:lttng-stop(1) command.
6120
6121 IMPORTANT: You need to stop tracing to make LTTng flush the remaining
6122 trace data and make the trace readable. Note that the
6123 man:lttng-destroy(1) command (see
6124 <<creating-destroying-tracing-sessions,Create and destroy a tracing
6125 session>>) also runs the man:lttng-stop(1) command implicitly.
6126
6127 [role="since-2.12"]
6128 [[clear]]
6129 === Clear a tracing session
6130
6131 You might need to remove all the current tracing data of one or more
6132 <<tracing-session,tracing sessions>> between multiple attempts to
6133 reproduce a problem without interrupting the LTTng tracing activity.
6134
6135 To clear the tracing data of the
6136 <<cur-tracing-session,current tracing session>>:
6137
6138 * Use the man:lttng-clear(1) command:
6139 +
6140 --
6141 [role="term"]
6142 ----
6143 $ lttng clear
6144 ----
6145 --
6146
6147 To clear the tracing data of all the tracing sessions:
6148
6149 * Use the `lttng clear` command with the opt:lttng-clear(1):--all
6150   option:
6151 +
6152 --
6153 [role="term"]
6154 ----
6155 $ lttng clear --all
6156 ----
6157 --
6158
6159
6160 [[enabling-disabling-channels]]
6161 === Create a channel
6162
6163 Once you create a tracing session, you can create a <<channel,channel>>
6164 with the man:lttng-enable-channel(1) command.
6165
6166 Note that LTTng automatically creates a default channel when, for a
6167 given <<domain,tracing domain>>, no channels exist and you
6168 <<enabling-disabling-events,create>> the first event rule. This default
6169 channel is named `channel0` and its attributes are set to reasonable
6170 values. Therefore, you only need to create a channel when you need
6171 non-default attributes.
6172
6173 You specify each non-default channel attribute with a command-line
6174 option when you use the man:lttng-enable-channel(1) command. The
6175 available command-line options are:
6176
6177 [role="growable",cols="asciidoc,asciidoc"]
6178 .Command-line options for the man:lttng-enable-channel(1) command.
6179 |====
6180 |Option |Description
6181
6182 |`--overwrite`
6183
6184 |
6185 Use the _overwrite_
6186 <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> instead
6187 of the default _discard_ mode.
6188
6189 |`--buffers-pid` (user space tracing domain only)
6190
6191 |
6192 Use the per-process <<channel-buffering-schemes,buffering scheme>>
6193 instead of the default per-user buffering scheme.
6194
6195 |+--subbuf-size=__SIZE__+
6196
6197 |
6198 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
6199 either for each Unix user (default), or for each instrumented process.
6200
6201 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6202
6203 |+--num-subbuf=__COUNT__+
6204
6205 |
6206 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
6207 for each Unix user (default), or for each instrumented process.
6208
6209 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
6210
6211 |+--tracefile-size=__SIZE__+
6212
6213 |
6214 Set the maximum size of each trace file that this channel writes within
6215 a stream to +__SIZE__+ bytes instead of no maximum.
6216
6217 See <<tracefile-rotation,Trace file count and size>>.
6218
6219 |+--tracefile-count=__COUNT__+
6220
6221 |
6222 Limit the number of trace files that this channel creates to
6223 +__COUNT__+ channels instead of no limit.
6224
6225 See <<tracefile-rotation,Trace file count and size>>.
6226
6227 |+--switch-timer=__PERIODUS__+
6228
6229 |
6230 Set the <<channel-switch-timer,switch timer period>>
6231 to +__PERIODUS__+{nbsp}µs.
6232
6233 |+--read-timer=__PERIODUS__+
6234
6235 |
6236 Set the <<channel-read-timer,read timer period>>
6237 to +__PERIODUS__+{nbsp}µs.
6238
6239 |[[opt-blocking-timeout]]+--blocking-timeout=__TIMEOUTUS__+
6240
6241 |
6242 Set the timeout of user space applications which load LTTng-UST
6243 in blocking mode to +__TIMEOUTUS__+:
6244
6245 0 (default)::
6246     Never block (non-blocking mode).
6247
6248 `inf`::
6249     Block forever until space is available in a sub-buffer to record
6250     the event.
6251
6252 __n__, a positive value::
6253     Wait for at most __n__ µs when trying to write into a sub-buffer.
6254
6255 Note that, for this option to have any effect on an instrumented
6256 user space application, you need to run the application with a set
6257 env:LTTNG_UST_ALLOW_BLOCKING environment variable.
6258
6259 |+--output=__TYPE__+ (Linux kernel tracing domain only)
6260
6261 |
6262 Set the output type of the channel to +__TYPE__+, either `mmap` or
6263 `splice`.
6264
6265 |====
6266
6267 You can only create a channel in the Linux kernel and user space
6268 <<domain,tracing domains>>: other tracing domains have their own channel
6269 created on the fly when <<enabling-disabling-events,creating event
6270 rules>>.
6271
6272 [IMPORTANT]
6273 ====
6274 Because of a current LTTng limitation, you must create all channels
6275 _before_ you <<basic-tracing-session-control,start tracing>> in a given
6276 tracing session, that is, before the first time you run
6277 man:lttng-start(1).
6278
6279 Since LTTng automatically creates a default channel when you use the
6280 man:lttng-enable-event(1) command with a specific tracing domain, you
6281 can't, for example, create a Linux kernel event rule, start tracing,
6282 and then create a user space event rule, because no user space channel
6283 exists yet and it's too late to create one.
6284
6285 For this reason, make sure to configure your channels properly
6286 before starting the tracers for the first time!
6287 ====
6288
6289 The following examples show how to combine the previous
6290 command-line options to create simple to more complex channels.
6291
6292 .Create a Linux kernel channel with default attributes.
6293 ====
6294 [role="term"]
6295 ----
6296 $ lttng enable-channel --kernel my-channel
6297 ----
6298 ====
6299
6300 .Create a user space channel with four sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
6301 ====
6302 [role="term"]
6303 ----
6304 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
6305                      --buffers-pid my-channel
6306 ----
6307 ====
6308
6309 .[[blocking-timeout-example]]Create a default user space channel with an infinite blocking timeout.
6310 ====
6311 <<creating-destroying-tracing-sessions,Create a tracing-session>>,
6312 create the channel, <<enabling-disabling-events,create an event rule>>,
6313 and <<basic-tracing-session-control,start tracing>>:
6314
6315 [role="term"]
6316 ----
6317 $ lttng create
6318 $ lttng enable-channel --userspace --blocking-timeout=inf blocking-channel
6319 $ lttng enable-event --userspace --channel=blocking-channel --all
6320 $ lttng start
6321 ----
6322
6323 Run an application instrumented with LTTng-UST and allow it to block:
6324
6325 [role="term"]
6326 ----
6327 $ LTTNG_UST_ALLOW_BLOCKING=1 my-app
6328 ----
6329 ====
6330
6331 .Create a Linux kernel channel which rotates eight trace files of 4{nbsp}MiB each for each stream
6332 ====
6333 [role="term"]
6334 ----
6335 $ lttng enable-channel --kernel --tracefile-count=8 \
6336                      --tracefile-size=4194304 my-channel
6337 ----
6338 ====
6339
6340 .Create a user space channel in overwrite (or _flight recorder_) mode.
6341 ====
6342 [role="term"]
6343 ----
6344 $ lttng enable-channel --userspace --overwrite my-channel
6345 ----
6346 ====
6347
6348 <<enabling-disabling-events,Create>> the same event rule in
6349 two different channels:
6350
6351 [role="term"]
6352 ----
6353 $ lttng enable-event --userspace --channel=my-channel app:tp
6354 $ lttng enable-event --userspace --channel=other-channel app:tp
6355 ----
6356
6357 If both channels are enabled, when a tracepoint named `app:tp` is
6358 reached, LTTng records two events, one for each channel.
6359
6360
6361 [[disable-channel]]
6362 === Disable a channel
6363
6364 To disable a specific channel that you <<enabling-disabling-channels,created>>
6365 previously, use the man:lttng-disable-channel(1) command.
6366
6367 .Disable a specific Linux kernel channel.
6368 ====
6369 [role="term"]
6370 ----
6371 $ lttng disable-channel --kernel my-channel
6372 ----
6373 ====
6374
6375 The state of a channel precedes the individual states of event rules
6376 attached to it: event rules which belong to a disabled channel, even if
6377 they are enabled, are also considered disabled.
6378
6379
6380 [[adding-context]]
6381 === Add context fields to a channel
6382
6383 Event record fields in trace files provide important information about
6384 events that occured previously, but sometimes some external context may
6385 help you solve a problem faster.
6386
6387 Examples of context fields are:
6388
6389 * The **process ID**, **thread ID**, **process name**, and
6390   **process priority** of the thread in which the event occurs.
6391 * The **hostname** of the system on which the event occurs.
6392 * The Linux kernel and user call stacks (since
6393   LTTng{nbsp}2.11).
6394 * The current values of many possible **performance counters** using
6395   perf, for example:
6396 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
6397 ** Cache misses.
6398 ** Branch instructions, misses, and loads.
6399 ** CPU faults.
6400 * Any context defined at the application level (supported for the
6401   JUL and log4j <<domain,tracing domains>>).
6402
6403 To get the full list of available context fields, see
6404 `lttng add-context --list`. Some context fields are reserved for a
6405 specific <<domain,tracing domain>> (Linux kernel or user space).
6406
6407 You add context fields to <<channel,channels>>. All the events
6408 that a channel with added context fields records contain those fields.
6409
6410 To add context fields to one or all the channels of a given tracing
6411 session:
6412
6413 * Use the man:lttng-add-context(1) command.
6414
6415 .Add context fields to all the channels of the current tracing session.
6416 ====
6417 The following command line adds the virtual process identifier and
6418 the per-thread CPU cycles count fields to all the user space channels
6419 of the
6420 <<cur-tracing-session,current tracing session>>.
6421
6422 [role="term"]
6423 ----
6424 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6425 ----
6426 ====
6427
6428 .Add performance counter context fields by raw ID
6429 ====
6430 See man:lttng-add-context(1) for the exact format of the context field
6431 type, which is partly compatible with the format used in
6432 man:perf-record(1).
6433
6434 [role="term"]
6435 ----
6436 $ lttng add-context --userspace --type=perf:thread:raw:r0110:test
6437 $ lttng add-context --kernel --type=perf:cpu:raw:r0013c:x86unhalted
6438 ----
6439 ====
6440
6441 .Add context fields to a specific channel.
6442 ====
6443 The following command line adds the thread identifier and user call
6444 stack context fields to the Linux kernel channel named `my-channel` in
6445 the current tracing session.
6446
6447 [role="term"]
6448 ----
6449 $ lttng add-context --kernel --channel=my-channel \
6450      --type=tid --type=callstack-user
6451 ----
6452 ====
6453
6454 .Add an application-specific context field to a specific channel.
6455 ====
6456 The following command line adds the `cur_msg_id` context field of the
6457 `retriever` context retriever for all the instrumented
6458 <<java-application,Java applications>> recording <<event,event records>>
6459 in the channel named `my-channel`:
6460
6461 [role="term"]
6462 ----
6463 $ lttng add-context --kernel --channel=my-channel \
6464                   --type='$app:retriever:cur_msg_id'
6465 ----
6466
6467 IMPORTANT: Make sure to always quote the `$` character when you
6468 use man:lttng-add-context(1) from a shell.
6469 ====
6470
6471 NOTE: You can't remove context fields from a channel once you add it.
6472
6473
6474 [role="since-2.7"]
6475 [[pid-tracking]]
6476 === Track process attributes
6477
6478 It's often useful to only allow processes with specific attributes to
6479 emit events. For example, you may wish to record all the system calls
6480 which a given process makes (à la
6481 http://linux.die.net/man/1/strace[strace]).
6482
6483 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6484 purpose. Both commands operate on _inclusion sets_ of process attribute
6485 values. The available process attribute types are:
6486
6487 Linux kernel <<domain,tracing domain>> only::
6488 +
6489 * Process ID (PID).
6490
6491 * Virtual process ID (VPID).
6492 +
6493 This is the PID as seen by the application.
6494
6495 * Unix user ID (UID) (since LTTng{nbsp}2.12).
6496
6497 * Virtual Unix user ID (VUID) (since LTTng{nbsp}2.12).
6498 +
6499 This is the UID as seen by the application.
6500
6501 * Unix group ID (GID) (since LTTng{nbsp}2.12).
6502
6503 * Virtual Unix group ID (VGID) (since LTTng{nbsp}2.12).
6504 +
6505 This is the GID as seen by the application.
6506
6507
6508 User space tracing domain::
6509 +
6510 * VPID.
6511 * VUID (since LTTng{nbsp}2.12).
6512 * VGID (since LTTng{nbsp}2.12).
6513
6514 Each tracing domain has one inclusion set per process attribute type:
6515 the Linux kernel tracing domain has six while the user space tracing
6516 domain has three.
6517
6518 For a given event which passes an enabled <<event,event rule>> to be
6519 recorded, _all_ the attributes of its executing process must be part of
6520 the inclusion sets of the tracing domain of the event rule.
6521
6522 Add entries to an inclusion set with the man:lttng-track(1) command and
6523 remove entries with the man:lttng-untrack(1) command. A process
6524 attribute is _tracked_ when it's part of an inclusion set and
6525 _untracked_ otherwise.
6526
6527 [NOTE]
6528 ====
6529 The process attribute values are _numeric_.
6530
6531 Should a process with a given tracked process ID, for example, exit, and
6532 then a new process be given this ID, then the latter would also be
6533 allowed to emit events.
6534
6535 With the `lttng track` command, you can add Unix user and group _names_
6536 to the user and group inclusion sets: the <<lttng-sessiond,session
6537 daemon>> finds the corresponding UID, VUID, GID, or VGID once on
6538 _addition_ to the inclusion set. This means that if you rename the user
6539 or group after you run `lttng track`, its user/group ID remains tracked.
6540 ====
6541
6542 .Track and untrack virtual process IDs.
6543 ====
6544 For the sake of the following example, assume the target system has
6545 16{nbsp}possible VPIDs.
6546
6547 When you
6548 <<creating-destroying-tracing-sessions,create a tracing session>>,
6549 the user space VPID inclusion set contains _all_ the possible VPIDs:
6550
6551 [role="img-100"]
6552 .All VPIDs are tracked.
6553 image::track-all.png[]
6554
6555 When the inclusion set is full and you use the man:lttng-track(1)
6556 command to specify some VPIDs to track, LTTng first clears the inclusion
6557 set, and then it adds the specific VPIDs to track. After:
6558
6559 [role="term"]
6560 ----
6561 $ lttng track --userspace --vpid=3,4,7,10,13
6562 ----
6563
6564 the VPID inclusion set is:
6565
6566 [role="img-100"]
6567 .VPIDs 3, 4, 7, 10, and 13 are tracked.
6568 image::track-3-4-7-10-13.png[]
6569
6570 Add more VPIDs to the inclusion set afterwards:
6571
6572 [role="term"]
6573 ----
6574 $ lttng track --userspace --vpid=1,15,16
6575 ----
6576
6577 The result is:
6578
6579 [role="img-100"]
6580 .VPIDs 1, 15, and 16 are added to the inclusion set.
6581 image::track-1-3-4-7-10-13-15-16.png[]
6582
6583 The man:lttng-untrack(1) command removes entries from process attribute
6584 inclusion sets. Given the previous example, the following command:
6585
6586 [role="term"]
6587 ----
6588 $ lttng untrack --userspace --vpid=3,7,10,13
6589 ----
6590
6591 leads to this VPID inclusion set:
6592
6593 [role="img-100"]
6594 .VPIDs 3, 7, 10, and 13 are removed from the inclusion set.
6595 image::track-1-4-15-16.png[]
6596
6597 LTTng can track all the possible VPIDs again using the
6598 opt:lttng-track(1):--all option:
6599
6600 [role="term"]
6601 ----
6602 $ lttng track --userspace --vpid --all
6603 ----
6604
6605 The result is, again:
6606
6607 [role="img-100"]
6608 .All VPIDs are tracked.
6609 image::track-all.png[]
6610 ====
6611
6612 .Track only specific process attributes.
6613 ====
6614 A typical use case with process attribute tracking is to start with an
6615 empty inclusion set, then <<basic-tracing-session-control,start the
6616 tracers>>, and then add entries manually while the tracers are active.
6617
6618 Use the opt:lttng-untrack(1):--all option of the
6619 man:lttng-untrack(1) command to clear the inclusion set after you
6620 <<creating-destroying-tracing-sessions,create a tracing session>>, for
6621 example (with UIDs):
6622
6623 [role="term"]
6624 ----
6625 $ lttng untrack --kernel --uid --all
6626 ----
6627
6628 gives:
6629
6630 [role="img-100"]
6631 .No UIDs are tracked.
6632 image::untrack-all.png[]
6633
6634 If you trace with this inclusion set configuration, the LTTng kernel
6635 tracer records no events within the <<cur-tracing-session,current
6636 tracing session>> because it doesn't track any UID. Use the
6637 man:lttng-track(1) command as usual to track specific UIDs when you need
6638 to, for example:
6639
6640 [role="term"]
6641 ----
6642 $ lttng track --kernel --uid=http,11
6643 ----
6644
6645 Result:
6646
6647 [role="img-100"]
6648 .UIDs 6 (`http`) and 11 are tracked.
6649 image::track-6-11.png[]
6650 ====
6651
6652
6653 [role="since-2.5"]
6654 [[saving-loading-tracing-session]]
6655 === Save and load tracing session configurations
6656
6657 Configuring a <<tracing-session,tracing session>> can be long. Some of
6658 the tasks involved are:
6659
6660 * <<enabling-disabling-channels,Create channels>> with
6661   specific attributes.
6662 * <<adding-context,Add context fields>> to specific channels.
6663 * <<enabling-disabling-events,Create event rules>> with specific log
6664   level and filter conditions.
6665
6666 If you use LTTng to solve real world problems, chances are you have to
6667 record events using the same tracing session setup over and over,
6668 modifying a few variables each time in your instrumented program
6669 or environment. To avoid constant tracing session reconfiguration,
6670 the man:lttng(1) command-line tool can save and load tracing session
6671 configurations to/from XML files.
6672
6673 To save a given tracing session configuration:
6674
6675 * Use the man:lttng-save(1) command:
6676 +
6677 --
6678 [role="term"]
6679 ----
6680 $ lttng save my-session
6681 ----
6682 --
6683 +
6684 Replace `my-session` with the name of the tracing session to save.
6685
6686 LTTng saves tracing session configurations to
6687 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6688 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6689 the opt:lttng-save(1):--output-path option to change this destination
6690 directory.
6691
6692 LTTng saves all configuration parameters, for example:
6693
6694 * The tracing session name.
6695 * The trace data output path.
6696 * The channels with their state and all their attributes.
6697 * The context fields you added to channels.
6698 * The event rules with their state, log level and filter conditions.
6699
6700 To load a tracing session:
6701
6702 * Use the man:lttng-load(1) command:
6703 +
6704 --
6705 [role="term"]
6706 ----
6707 $ lttng load my-session
6708 ----
6709 --
6710 +
6711 Replace `my-session` with the name of the tracing session to load.
6712
6713 When LTTng loads a configuration, it restores your saved tracing session
6714 as if you just configured it manually.
6715
6716 See man:lttng-load(1) for the complete list of command-line options. You
6717 can also save and load many sessions at a time, and decide in which
6718 directory to output the XML files.
6719
6720
6721 [[sending-trace-data-over-the-network]]
6722 === Send trace data over the network
6723
6724 LTTng can send the recorded trace data to a remote system over the
6725 network instead of writing it to the local file system.
6726
6727 To send the trace data over the network:
6728
6729 . On the _remote_ system (which can also be the target system),
6730   start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6731 +
6732 --
6733 [role="term"]
6734 ----
6735 $ lttng-relayd
6736 ----
6737 --
6738
6739 . On the _target_ system, create a tracing session configured to
6740   send trace data over the network:
6741 +
6742 --
6743 [role="term"]
6744 ----
6745 $ lttng create my-session --set-url=net://remote-system
6746 ----
6747 --
6748 +
6749 Replace `remote-system` by the host name or IP address of the
6750 remote system. See man:lttng-create(1) for the exact URL format.
6751
6752 . On the target system, use the man:lttng(1) command-line tool as usual.
6753   When tracing is active, the consumer daemon of the target sends
6754   sub-buffers to the relay daemon running on the remote system instead
6755   of flushing them to the local file system. The relay daemon writes the
6756   received packets to the local file system.
6757
6758 The relay daemon writes trace files to
6759 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6760 +__hostname__+ is the host name of the target system and +__session__+
6761 is the tracing session name. Note that the env:LTTNG_HOME environment
6762 variable defaults to `$HOME` if not set. Use the
6763 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6764 trace files to another base directory.
6765
6766
6767 [role="since-2.4"]
6768 [[lttng-live]]
6769 === View events as LTTng emits them (noch:{LTTng} live)
6770
6771 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6772 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6773 display events as LTTng emits them on the target system while tracing is
6774 active.
6775
6776 The relay daemon creates a _tee_: it forwards the trace data to both
6777 the local file system and to connected live viewers:
6778
6779 [role="img-90"]
6780 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6781 image::live.png[]
6782
6783 To use LTTng live:
6784
6785 . On the _target system_, create a <<tracing-session,tracing session>>
6786   in _live mode_:
6787 +
6788 --
6789 [role="term"]
6790 ----
6791 $ lttng create my-session --live
6792 ----
6793 --
6794 +
6795 This spawns a local relay daemon.
6796
6797 . Start the live viewer and configure it to connect to the relay
6798   daemon. For example, with
6799   https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]:
6800 +
6801 --
6802 [role="term"]
6803 ----
6804 $ babeltrace2 net://localhost/host/hostname/my-session
6805 ----
6806 --
6807 +
6808 Replace:
6809 +
6810 --
6811 * `hostname` with the host name of the target system.
6812 * `my-session` with the name of the tracing session to view.
6813 --
6814
6815 . Configure the tracing session as usual with the man:lttng(1)
6816   command-line tool, and <<basic-tracing-session-control,start tracing>>.
6817
6818 List the available live tracing sessions with Babeltrace{nbsp}2:
6819
6820 [role="term"]
6821 ----
6822 $ babeltrace2 net://localhost
6823 ----
6824
6825 You can start the relay daemon on another system. In this case, you need
6826 to specify the URL of the relay daemon when you create the tracing
6827 session with the opt:lttng-create(1):--set-url option. You also need to
6828 replace `localhost` in the procedure above with the host name of the
6829 system on which the relay daemon is running.
6830
6831 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6832 command-line options.
6833
6834
6835 [role="since-2.3"]
6836 [[taking-a-snapshot]]
6837 === Take a snapshot of the current sub-buffers of a tracing session
6838
6839 The normal behavior of LTTng is to append full sub-buffers to growing
6840 trace data files. This is ideal to keep a full history of the events
6841 that occurred on the target system, but it can
6842 represent too much data in some situations. For example, you may wish
6843 to trace your application continuously until some critical situation
6844 happens, in which case you only need the latest few recorded
6845 events to perform the desired analysis, not multi-gigabyte trace files.
6846
6847 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6848 current sub-buffers of a given <<tracing-session,tracing session>>.
6849 LTTng can write the snapshot to the local file system or send it over
6850 the network.
6851
6852 [role="img-100"]
6853 .A snapshot is a copy of the current sub-buffers, which aren't cleared after the operation.
6854 image::snapshot.png[]
6855
6856 If you wish to create unmanaged, self-contained, non-overlapping
6857 trace chunk archives instead of a simple copy of the current
6858 sub-buffers, see the <<session-rotation,tracing session rotation>>
6859 feature (available since LTTng{nbsp}2.11).
6860
6861 To take a snapshot:
6862
6863 . Create a tracing session in _snapshot mode_:
6864 +
6865 --
6866 [role="term"]
6867 ----
6868 $ lttng create my-session --snapshot
6869 ----
6870 --
6871 +
6872 The <<channel-overwrite-mode-vs-discard-mode,event record loss mode>> of
6873 <<channel,channels>> created in this mode is automatically set to
6874 _overwrite_ (flight recorder mode).
6875
6876 . Configure the tracing session as usual with the man:lttng(1)
6877   command-line tool, and <<basic-tracing-session-control,start tracing>>.
6878
6879 . **Optional**: When you need to take a snapshot,
6880   <<basic-tracing-session-control,stop tracing>>.
6881 +
6882 You can take a snapshot when the tracers are active, but if you stop
6883 them first, you're sure that the data in the sub-buffers doesn't
6884 change before you actually take the snapshot.
6885
6886 . Take a snapshot:
6887 +
6888 --
6889 [role="term"]
6890 ----
6891 $ lttng snapshot record --name=my-first-snapshot
6892 ----
6893 --
6894 +
6895 LTTng writes the current sub-buffers of all the channels of the
6896 <<cur-tracing-session,current tracing session>> to
6897 trace files on the local file system. Those trace files have
6898 `my-first-snapshot` in their name.
6899
6900 There is no difference between the format of a normal trace file and the
6901 format of a snapshot: viewers of LTTng traces also support LTTng
6902 snapshots.
6903
6904 By default, LTTng writes snapshot files to the path shown by
6905 `lttng snapshot list-output`. You can change this path or decide to send
6906 snapshots over the network using either:
6907
6908 . An output path or URL that you specify when you
6909   <<creating-destroying-tracing-sessions,create the tracing session>>.
6910 . A snapshot output path or URL that you add using
6911   `lttng snapshot add-output`.
6912 . An output path or URL that you provide directly to the
6913   `lttng snapshot record` command.
6914
6915 Method{nbsp}3 overrides method{nbsp}2, which overrides method 1. When
6916 you specify a URL, a relay daemon must listen on a remote system (see
6917 <<sending-trace-data-over-the-network,Send trace data over the
6918 network>>).
6919
6920
6921 [role="since-2.11"]
6922 [[session-rotation]]
6923 === Archive the current trace chunk (rotate a tracing session)
6924
6925 The <<taking-a-snapshot,snapshot user guide>> shows how to dump the
6926 current sub-buffers of a tracing session to the file system or send them
6927 over the network. When you take a snapshot, LTTng doesn't clear the ring
6928 buffers of the tracing session: if you take another snapshot immediately
6929 after, both snapshots could contain overlapping trace data.
6930
6931 Inspired by https://en.wikipedia.org/wiki/Log_rotation[log rotation],
6932 _tracing session rotation_ is a feature which appends the content of the
6933 ring buffers to what's already on the file system or sent over the
6934 network since the creation of the tracing session or since the last
6935 rotation, and then clears those ring buffers to avoid trace data
6936 overlaps.
6937
6938 What LTTng is about to write when performing a tracing session rotation
6939 is called the _current trace chunk_. When this current trace chunk is
6940 written to the file system or sent over the network, it becomes a _trace
6941 chunk archive_. Therefore, a tracing session rotation _archives_ the
6942 current trace chunk.
6943
6944 [role="img-100"]
6945 .A tracing session rotation operation _archives_ the current trace chunk.
6946 image::rotation.png[]
6947
6948 A trace chunk archive is a self-contained LTTng trace which LTTng
6949 doesn't manage anymore: you can read it, modify it, move it, or remove
6950 it.
6951
6952 There are two methods to perform a tracing session rotation: immediately
6953 or with a rotation schedule.
6954
6955 To perform an immediate tracing session rotation:
6956
6957 . <<creating-destroying-tracing-sessions,Create a tracing session>>
6958   in _normal mode_ or _network streaming mode_
6959   (only those two creation modes support tracing session rotation):
6960 +
6961 --
6962 [role="term"]
6963 ----
6964 $ lttng create my-session
6965 ----
6966 --
6967
6968 . <<enabling-disabling-events,Create one or more event rules>>
6969   and <<basic-tracing-session-control,start tracing>>:
6970 +
6971 --
6972 [role="term"]
6973 ----
6974 $ lttng enable-event --kernel sched_'*'
6975 $ lttng start
6976 ----
6977 --
6978
6979 . When needed, immediately rotate the
6980   <<cur-tracing-session,current tracing session>>:
6981 +
6982 --
6983 [role="term"]
6984 ----
6985 $ lttng rotate
6986 ----
6987 --
6988 +
6989 The cmd:lttng-rotate command prints the path to the created trace
6990 chunk archive. See man:lttng-rotate(1) to learn about the format
6991 of trace chunk archive directory names.
6992 +
6993 Perform other immediate rotations while the tracing session is
6994 active. It is guaranteed that all the trace chunk archives don't
6995 contain overlapping trace data. You can also perform an immediate
6996 rotation once you have <<basic-tracing-session-control,stopped>> the
6997 tracing session.
6998
6999 . When you're done tracing,
7000   <<creating-destroying-tracing-sessions,destroy the current tracing
7001   session>>:
7002 +
7003 --
7004 [role="term"]
7005 ----
7006 $ lttng destroy
7007 ----
7008 --
7009 +
7010 The tracing session destruction operation creates one last trace
7011 chunk archive from the current trace chunk.
7012
7013 A tracing session rotation schedule is a planned rotation which LTTng
7014 performs automatically based on one of the following conditions:
7015
7016 * A timer with a configured period times out.
7017
7018 * The total size of the flushed part of the current trace chunk
7019   becomes greater than or equal to a configured value.
7020
7021 To schedule a tracing session rotation, set a _rotation schedule_:
7022
7023 . <<creating-destroying-tracing-sessions,Create a tracing session>>
7024   in _normal mode_ or _network streaming mode_
7025   (only those two creation modes support tracing session rotation):
7026 +
7027 --
7028 [role="term"]
7029 ----
7030 $ lttng create my-session
7031 ----
7032 --
7033
7034 . <<enabling-disabling-events,Create one or more event rules>>:
7035 +
7036 --
7037 [role="term"]
7038 ----
7039 $ lttng enable-event --kernel sched_'*'
7040 ----
7041 --
7042
7043 . Set a tracing session rotation schedule:
7044 +
7045 --
7046 [role="term"]
7047 ----
7048 $ lttng enable-rotation --timer=10s
7049 ----
7050 --
7051 +
7052 In this example, we set a rotation schedule so that LTTng performs a
7053 tracing session rotation every ten seconds.
7054 +
7055 See man:lttng-enable-rotation(1) to learn more about other ways to set a
7056 rotation schedule.
7057
7058 . <<basic-tracing-session-control,Start tracing>>:
7059 +
7060 --
7061 [role="term"]
7062 ----
7063 $ lttng start
7064 ----
7065 --
7066 +
7067 LTTng performs tracing session rotations automatically while the tracing
7068 session is active thanks to the rotation schedule.
7069
7070 . When you're done tracing,
7071   <<creating-destroying-tracing-sessions,destroy the current tracing
7072   session>>:
7073 +
7074 --
7075 [role="term"]
7076 ----
7077 $ lttng destroy
7078 ----
7079 --
7080 +
7081 The tracing session destruction operation creates one last trace chunk
7082 archive from the current trace chunk.
7083
7084 Use man:lttng-disable-rotation(1) to unset a tracing session
7085 rotation schedule.
7086
7087 NOTE: man:lttng-rotate(1) and man:lttng-enable-rotation(1) list
7088 limitations regarding those two commands.
7089
7090
7091 [role="since-2.6"]
7092 [[mi]]
7093 === Use the machine interface
7094
7095 With any command of the man:lttng(1) command-line tool, set the
7096 opt:lttng(1):--mi option to `xml` (before the command name) to get an
7097 XML machine interface output, for example:
7098
7099 [role="term"]
7100 ----
7101 $ lttng --mi=xml enable-event --kernel --syscall open
7102 ----
7103
7104 A schema definition (XSD) is
7105 https://github.com/lttng/lttng-tools/blob/stable-{revision}/src/common/src/common/mi-lttng-4.0.xsd[available]
7106 to ease the integration with external tools as much as possible.
7107
7108
7109 [role="since-2.8"]
7110 [[metadata-regenerate]]
7111 === Regenerate the metadata of an LTTng trace
7112
7113 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
7114 data stream files and a metadata file. This metadata file contains,
7115 amongst other things, information about the offset of the clock sources
7116 used to timestamp <<event,event records>> when tracing.
7117
7118 If, once a <<tracing-session,tracing session>> is
7119 <<basic-tracing-session-control,started>>, a major
7120 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
7121 happens, the clock offset of the trace also needs to be updated. Use
7122 the `metadata` item of the man:lttng-regenerate(1) command to do so.
7123
7124 The main use case of this command is to allow a system to boot with
7125 an incorrect wall time and trace it with LTTng before its wall time
7126 is corrected. Once the system is known to be in a state where its
7127 wall time is correct, it can run `lttng regenerate metadata`.
7128
7129 To regenerate the metadata of an LTTng trace:
7130
7131 * Use the `metadata` item of the man:lttng-regenerate(1) command:
7132 +
7133 --
7134 [role="term"]
7135 ----
7136 $ lttng regenerate metadata
7137 ----
7138 --
7139
7140 [IMPORTANT]
7141 ====
7142 `lttng regenerate metadata` has the following limitations:
7143
7144 * Tracing session <<creating-destroying-tracing-sessions,created>>
7145   in non-live mode.
7146 * User space <<channel,channels>>, if any, are using
7147   <<channel-buffering-schemes,per-user buffering>>.
7148 ====
7149
7150
7151 [role="since-2.9"]
7152 [[regenerate-statedump]]
7153 === Regenerate the state dump of a tracing session
7154
7155 The LTTng kernel and user space tracers generate state dump
7156 <<event,event records>> when the application starts or when you
7157 <<basic-tracing-session-control,start a tracing session>>. An analysis
7158 can use the state dump event records to set an initial state before it
7159 builds the rest of the state from the following event records.
7160 http://tracecompass.org/[Trace Compass] is a notable example of an
7161 application which uses the state dump of an LTTng trace.
7162
7163 When you <<taking-a-snapshot,take a snapshot>>, it's possible that the
7164 state dump event records aren't included in the snapshot because they
7165 were recorded to a sub-buffer that has been consumed or overwritten
7166 already.
7167
7168 Use the `lttng regenerate statedump` command to emit the state
7169 dump event records again.
7170
7171 To regenerate the state dump of the current tracing session, provided
7172 create it in snapshot mode, before you take a snapshot:
7173
7174 . Use the `statedump` item of the man:lttng-regenerate(1) command:
7175 +
7176 --
7177 [role="term"]
7178 ----
7179 $ lttng regenerate statedump
7180 ----
7181 --
7182
7183 . <<basic-tracing-session-control,Stop the tracing session>>:
7184 +
7185 --
7186 [role="term"]
7187 ----
7188 $ lttng stop
7189 ----
7190 --
7191
7192 . <<taking-a-snapshot,Take a snapshot>>:
7193 +
7194 --
7195 [role="term"]
7196 ----
7197 $ lttng snapshot record --name=my-snapshot
7198 ----
7199 --
7200
7201 Depending on the event throughput, you should run steps 1 and 2
7202 as closely as possible.
7203
7204 NOTE: To record the state dump events, you need to
7205 <<enabling-disabling-events,create event rules>> which enable them.
7206 LTTng-UST state dump tracepoints start with `lttng_ust_statedump:`.
7207 LTTng-modules state dump tracepoints start with `lttng_statedump_`.
7208
7209
7210 [role="since-2.7"]
7211 [[persistent-memory-file-systems]]
7212 === Record trace data on persistent memory file systems
7213
7214 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
7215 (NVRAM) is random-access memory that retains its information when power
7216 is turned off (non-volatile). Systems with such memory can store data
7217 structures in RAM and retrieve them after a reboot, without flushing
7218 to typical _storage_.
7219
7220 Linux supports NVRAM file systems thanks to either
7221 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
7222 (requires Linux{nbsp}4.1+) or http://pramfs.sourceforge.net/[PRAMFS] (requires Linux{nbsp}<{nbsp}4).
7223
7224 This section doesn't describe how to operate such file systems;
7225 we assume that you have a working persistent memory file system.
7226
7227 When you create a <<tracing-session,tracing session>>, you can specify
7228 the path of the shared memory holding the sub-buffers. If you specify a
7229 location on an NVRAM file system, then you can retrieve the latest
7230 recorded trace data when the system reboots after a crash.
7231
7232 To record trace data on a persistent memory file system and retrieve the
7233 trace data after a system crash:
7234
7235 . Create a tracing session with a sub-buffer shared memory path located
7236   on an NVRAM file system:
7237 +
7238 --
7239 [role="term"]
7240 ----
7241 $ lttng create my-session --shm-path=/path/to/shm
7242 ----
7243 --
7244
7245 . Configure the tracing session as usual with the man:lttng(1)
7246   command-line tool, and <<basic-tracing-session-control,start tracing>>.
7247
7248 . After a system crash, use the man:lttng-crash(1) command-line tool to
7249   view the trace data recorded on the NVRAM file system:
7250 +
7251 --
7252 [role="term"]
7253 ----
7254 $ lttng-crash /path/to/shm
7255 ----
7256 --
7257
7258 The binary layout of the ring buffer files isn't exactly the same as
7259 the trace files layout. This is why you need to use man:lttng-crash(1)
7260 instead of your preferred trace viewer directly.
7261
7262 To convert the ring buffer files to LTTng trace files:
7263
7264 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
7265 +
7266 --
7267 [role="term"]
7268 ----
7269 $ lttng-crash --extract=/path/to/trace /path/to/shm
7270 ----
7271 --
7272
7273
7274 [role="since-2.10"]
7275 [[notif-trigger-api]]
7276 === Get notified when the buffer usage of a channel is too high or too low
7277
7278 With the $$C/C++$$ notification and trigger API of LTTng, your user
7279 application can get notified when the buffer usage of one or more
7280 <<channel,channels>> becomes too low or too high. Use this API
7281 and enable or disable <<event,event rules>> during tracing to avoid
7282 <<channel-overwrite-mode-vs-discard-mode,discarded event records>>.
7283
7284 .Have a user application get notified when the buffer usage of an LTTng channel is too high.
7285 ====
7286 In this example, we create and build an application which gets notified
7287 when the buffer usage of a specific LTTng channel is higher than
7288 75{nbsp}%. We only print that it is the case in the example, but we
7289 could as well use the API of <<liblttng-ctl-lttng,`liblttng-ctl`>> to
7290 disable event rules when this happens.
7291
7292 . Create the C{nbsp}source file of application:
7293 +
7294 --
7295 [source,c]
7296 .path:{notif-app.c}
7297 ----
7298 #include <stdio.h>
7299 #include <assert.h>
7300 #include <lttng/domain.h>
7301 #include <lttng/action/action.h>
7302 #include <lttng/action/notify.h>
7303 #include <lttng/condition/condition.h>
7304 #include <lttng/condition/buffer-usage.h>
7305 #include <lttng/condition/evaluation.h>
7306 #include <lttng/notification/channel.h>
7307 #include <lttng/notification/notification.h>
7308 #include <lttng/trigger/trigger.h>
7309 #include <lttng/endpoint.h>
7310
7311 int main(int argc, char *argv[])
7312 {
7313     int exit_status = 0;
7314     struct lttng_notification_channel *notification_channel;
7315     struct lttng_condition *condition;
7316     struct lttng_action *action;
7317     struct lttng_trigger *trigger;
7318     const char *tracing_session_name;
7319     const char *channel_name;
7320
7321     assert(argc >= 3);
7322     tracing_session_name = argv[1];
7323     channel_name = argv[2];
7324
7325     /*
7326      * Create a notification channel. A notification channel
7327      * connects the user application to the LTTng session daemon.
7328      * This notification channel can be used to listen to various
7329      * types of notifications.
7330      */
7331     notification_channel = lttng_notification_channel_create(
7332         lttng_session_daemon_notification_endpoint);
7333
7334     /*
7335      * Create a "high buffer usage" condition. In this case, the
7336      * condition is reached when the buffer usage is greater than or
7337      * equal to 75 %. We create the condition for a specific tracing
7338      * session name, channel name, and for the user space tracing
7339      * domain.
7340      *
7341      * The "low buffer usage" condition type also exists.
7342      */
7343     condition = lttng_condition_buffer_usage_high_create();
7344     lttng_condition_buffer_usage_set_threshold_ratio(condition, .75);
7345     lttng_condition_buffer_usage_set_session_name(
7346         condition, tracing_session_name);
7347     lttng_condition_buffer_usage_set_channel_name(condition,
7348         channel_name);
7349     lttng_condition_buffer_usage_set_domain_type(condition,
7350         LTTNG_DOMAIN_UST);
7351
7352     /*
7353      * Create an action (get a notification) to take when the
7354      * condition created above is reached.
7355      */
7356     action = lttng_action_notify_create();
7357
7358     /*
7359      * Create a trigger. A trigger associates a condition to an
7360      * action: the action is executed when the condition is reached.
7361      */
7362     trigger = lttng_trigger_create(condition, action);
7363
7364     /* Register the trigger to LTTng. */
7365     lttng_register_trigger(trigger);
7366
7367     /*
7368      * Now that we have registered a trigger, a notification will be
7369      * emitted everytime its condition is met. To receive this
7370      * notification, we must subscribe to notifications that match
7371      * the same condition.
7372      */
7373     lttng_notification_channel_subscribe(notification_channel,
7374         condition);
7375
7376     /*
7377      * Notification loop. Put this in a dedicated thread to avoid
7378      * blocking the main thread.
7379      */
7380     for (;;) {
7381         struct lttng_notification *notification;
7382         enum lttng_notification_channel_status status;
7383         const struct lttng_evaluation *notification_evaluation;
7384         const struct lttng_condition *notification_condition;
7385         double buffer_usage;
7386
7387         /* Receive the next notification. */
7388         status = lttng_notification_channel_get_next_notification(
7389             notification_channel, &notification);
7390
7391         switch (status) {
7392         case LTTNG_NOTIFICATION_CHANNEL_STATUS_OK:
7393             break;
7394         case LTTNG_NOTIFICATION_CHANNEL_STATUS_NOTIFICATIONS_DROPPED:
7395             /*
7396              * The session daemon can drop notifications if a monitoring
7397              * application isn't consuming the notifications fast
7398              * enough.
7399              */
7400             continue;
7401         case LTTNG_NOTIFICATION_CHANNEL_STATUS_CLOSED:
7402             /*
7403              * The notification channel has been closed by the
7404              * session daemon. This is typically caused by a session
7405              * daemon shutting down.
7406              */
7407             goto end;
7408         default:
7409             /* Unhandled conditions or errors. */
7410             exit_status = 1;
7411             goto end;
7412         }
7413
7414         /*
7415          * A notification provides, amongst other things:
7416          *
7417          * * The condition that caused this notification to be
7418          *   emitted.
7419          * * The condition evaluation, which provides more
7420          *   specific information on the evaluation of the
7421          *   condition.
7422          *
7423          * The condition evaluation provides the buffer usage
7424          * value at the moment the condition was reached.
7425          */
7426         notification_condition = lttng_notification_get_condition(
7427             notification);
7428         notification_evaluation = lttng_notification_get_evaluation(
7429             notification);
7430
7431         /* We're subscribed to only one condition. */
7432         assert(lttng_condition_get_type(notification_condition) ==
7433             LTTNG_CONDITION_TYPE_BUFFER_USAGE_HIGH);
7434
7435         /*
7436          * Get the exact sampled buffer usage from the
7437          * condition evaluation.
7438          */
7439         lttng_evaluation_buffer_usage_get_usage_ratio(
7440             notification_evaluation, &buffer_usage);
7441
7442         /*
7443          * At this point, instead of printing a message, we
7444          * could do something to reduce the buffer usage of the channel,
7445          * like disable specific events.
7446          */
7447         printf("Buffer usage is %f %% in tracing session \"%s\", "
7448             "user space channel \"%s\".\n", buffer_usage * 100,
7449             tracing_session_name, channel_name);
7450         lttng_notification_destroy(notification);
7451     }
7452
7453 end:
7454     lttng_action_destroy(action);
7455     lttng_condition_destroy(condition);
7456     lttng_trigger_destroy(trigger);
7457     lttng_notification_channel_destroy(notification_channel);
7458     return exit_status;
7459 }
7460 ----
7461 --
7462
7463 . Build the `notif-app` application, linking it to `liblttng-ctl`:
7464 +
7465 --
7466 [role="term"]
7467 ----
7468 $ gcc -o notif-app notif-app.c -llttng-ctl
7469 ----
7470 --
7471
7472 . <<creating-destroying-tracing-sessions,Create a tracing session>>,
7473   <<enabling-disabling-events,create an event rule>> matching all the
7474   user space tracepoints, and
7475   <<basic-tracing-session-control,start tracing>>:
7476 +
7477 --
7478 [role="term"]
7479 ----
7480 $ lttng create my-session
7481 $ lttng enable-event --userspace --all
7482 $ lttng start
7483 ----
7484 --
7485 +
7486 If you create the channel manually with the man:lttng-enable-channel(1)
7487 command, control how frequently LTTng samples the current values of the
7488 channel properties to evaluate user conditions with the
7489 opt:lttng-enable-channel(1):--monitor-timer option.
7490
7491 . Run the `notif-app` application. This program accepts the
7492   <<tracing-session,tracing session>> name and the user space channel
7493   name as its two first arguments. The channel which LTTng automatically
7494   creates with the man:lttng-enable-event(1) command above is named
7495   `channel0`:
7496 +
7497 --
7498 [role="term"]
7499 ----
7500 $ ./notif-app my-session channel0
7501 ----
7502 --
7503
7504 . In another terminal, run an application with a very high event
7505   throughput so that the 75{nbsp}% buffer usage condition is reached.
7506 +
7507 In the first terminal, the application should print lines like this:
7508 +
7509 ----
7510 Buffer usage is 81.45197 % in tracing session "my-session", user space
7511 channel "channel0".
7512 ----
7513 +
7514 If you don't see anything, try modifying the condition in
7515 path:{notif-app.c} to a lower value (0.1, for example), rebuilding it
7516 (step{nbsp}2) and running it again (step{nbsp}4).
7517 ====
7518
7519
7520 [[reference]]
7521 == Reference
7522
7523 [[lttng-modules-ref]]
7524 === noch:{LTTng-modules}
7525
7526
7527 [role="since-2.9"]
7528 [[lttng-tracepoint-enum]]
7529 ==== `LTTNG_TRACEPOINT_ENUM()` usage
7530
7531 Use the `LTTNG_TRACEPOINT_ENUM()` macro to define an enumeration:
7532
7533 [source,c]
7534 ----
7535 LTTNG_TRACEPOINT_ENUM(name, TP_ENUM_VALUES(entries))
7536 ----
7537
7538 Replace:
7539
7540 * `name` with the name of the enumeration (C identifier, unique
7541   amongst all the defined enumerations).
7542 * `entries` with a list of enumeration entries.
7543
7544 The available enumeration entry macros are:
7545
7546 +ctf_enum_value(__name__, __value__)+::
7547   Entry named +__name__+ mapped to the integral value +__value__+.
7548
7549 +ctf_enum_range(__name__, __begin__, __end__)+::
7550   Entry named +__name__+ mapped to the range of integral values between
7551   +__begin__+ (included) and +__end__+ (included).
7552
7553 +ctf_enum_auto(__name__)+::
7554   Entry named +__name__+ mapped to the integral value following the
7555   last mapping value.
7556 +
7557 The last value of a `ctf_enum_value()` entry is its +__value__+
7558 parameter.
7559 +
7560 The last value of a `ctf_enum_range()` entry is its +__end__+ parameter.
7561 +
7562 If `ctf_enum_auto()` is the first entry in the list, its integral
7563 value is 0.
7564
7565 Use the `ctf_enum()` <<lttng-modules-tp-fields,field definition macro>>
7566 to use a defined enumeration as a tracepoint field.
7567
7568 .Define an enumeration with `LTTNG_TRACEPOINT_ENUM()`.
7569 ====
7570 [source,c]
7571 ----
7572 LTTNG_TRACEPOINT_ENUM(
7573     my_enum,
7574     TP_ENUM_VALUES(
7575         ctf_enum_auto("AUTO: EXPECT 0")
7576         ctf_enum_value("VALUE: 23", 23)
7577         ctf_enum_value("VALUE: 27", 27)
7578         ctf_enum_auto("AUTO: EXPECT 28")
7579         ctf_enum_range("RANGE: 101 TO 303", 101, 303)
7580         ctf_enum_auto("AUTO: EXPECT 304")
7581     )
7582 )
7583 ----
7584 ====
7585
7586
7587 [role="since-2.7"]
7588 [[lttng-modules-tp-fields]]
7589 ==== Tracepoint fields macros (for `TP_FIELDS()`)
7590
7591 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
7592 tracepoint fields, which must be listed within `TP_FIELDS()` in
7593 `LTTNG_TRACEPOINT_EVENT()`, are:
7594
7595 [role="func-desc growable",cols="asciidoc,asciidoc"]
7596 .Available macros to define LTTng-modules tracepoint fields
7597 |====
7598 |Macro |Description and parameters
7599
7600 |
7601 +ctf_integer(__t__, __n__, __e__)+
7602
7603 +ctf_integer_nowrite(__t__, __n__, __e__)+
7604
7605 +ctf_user_integer(__t__, __n__, __e__)+
7606
7607 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
7608 |
7609 Standard integer, displayed in base{nbsp}10.
7610
7611 +__t__+::
7612   Integer C type (`int`, `long`, `size_t`, ...).
7613
7614 +__n__+::
7615   Field name.
7616
7617 +__e__+::
7618   Argument expression.
7619
7620 |
7621 +ctf_integer_hex(__t__, __n__, __e__)+
7622
7623 +ctf_user_integer_hex(__t__, __n__, __e__)+
7624 |
7625 Standard integer, displayed in base{nbsp}16.
7626
7627 +__t__+::
7628   Integer C type.
7629
7630 +__n__+::
7631   Field name.
7632
7633 +__e__+::
7634   Argument expression.
7635
7636 |+ctf_integer_oct(__t__, __n__, __e__)+
7637 |
7638 Standard integer, displayed in base{nbsp}8.
7639
7640 +__t__+::
7641   Integer C type.
7642
7643 +__n__+::
7644   Field name.
7645
7646 +__e__+::
7647   Argument expression.
7648
7649 |
7650 +ctf_integer_network(__t__, __n__, __e__)+
7651
7652 +ctf_user_integer_network(__t__, __n__, __e__)+
7653 |
7654 Integer in network byte order (big-endian), displayed in base{nbsp}10.
7655
7656 +__t__+::
7657   Integer C type.
7658
7659 +__n__+::
7660   Field name.
7661
7662 +__e__+::
7663   Argument expression.
7664
7665 |
7666 +ctf_integer_network_hex(__t__, __n__, __e__)+
7667
7668 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
7669 |
7670 Integer in network byte order, displayed in base{nbsp}16.
7671
7672 +__t__+::
7673   Integer C type.
7674
7675 +__n__+::
7676   Field name.
7677
7678 +__e__+::
7679   Argument expression.
7680
7681 |
7682 +ctf_enum(__N__, __t__, __n__, __e__)+
7683
7684 +ctf_enum_nowrite(__N__, __t__, __n__, __e__)+
7685
7686 +ctf_user_enum(__N__, __t__, __n__, __e__)+
7687
7688 +ctf_user_enum_nowrite(__N__, __t__, __n__, __e__)+
7689 |
7690 Enumeration.
7691
7692 +__N__+::
7693   Name of a <<lttng-tracepoint-enum,previously defined enumeration>>.
7694
7695 +__t__+::
7696   Integer C type (`int`, `long`, `size_t`, ...).
7697
7698 +__n__+::
7699   Field name.
7700
7701 +__e__+::
7702   Argument expression.
7703
7704 |
7705 +ctf_string(__n__, __e__)+
7706
7707 +ctf_string_nowrite(__n__, __e__)+
7708
7709 +ctf_user_string(__n__, __e__)+
7710
7711 +ctf_user_string_nowrite(__n__, __e__)+
7712 |
7713 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
7714
7715 +__n__+::
7716   Field name.
7717
7718 +__e__+::
7719   Argument expression.
7720
7721 |
7722 +ctf_array(__t__, __n__, __e__, __s__)+
7723
7724 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
7725
7726 +ctf_user_array(__t__, __n__, __e__, __s__)+
7727
7728 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
7729 |
7730 Statically-sized array of integers.
7731
7732 +__t__+::
7733   Array element C type.
7734
7735 +__n__+::
7736   Field name.
7737
7738 +__e__+::
7739   Argument expression.
7740
7741 +__s__+::
7742   Number of elements.
7743
7744 |
7745 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
7746
7747 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7748
7749 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
7750
7751 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
7752 |
7753 Statically-sized array of bits.
7754
7755 The type of +__e__+ must be an integer type. +__s__+ is the number
7756 of elements of such type in +__e__+, not the number of bits.
7757
7758 +__t__+::
7759   Array element C type.
7760
7761 +__n__+::
7762   Field name.
7763
7764 +__e__+::
7765   Argument expression.
7766
7767 +__s__+::
7768   Number of elements.
7769
7770 |
7771 +ctf_array_text(__t__, __n__, __e__, __s__)+
7772
7773 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
7774
7775 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
7776
7777 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
7778 |
7779 Statically-sized array, printed as text.
7780
7781 The string doesn't need to be null-terminated.
7782
7783 +__t__+::
7784   Array element C type (always `char`).
7785
7786 +__n__+::
7787   Field name.
7788
7789 +__e__+::
7790   Argument expression.
7791
7792 +__s__+::
7793   Number of elements.
7794
7795 |
7796 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
7797
7798 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7799
7800 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
7801
7802 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
7803 |
7804 Dynamically-sized array of integers.
7805
7806 The type of +__E__+ must be unsigned.
7807
7808 +__t__+::
7809   Array element C type.
7810
7811 +__n__+::
7812   Field name.
7813
7814 +__e__+::
7815   Argument expression.
7816
7817 +__T__+::
7818   Length expression C type.
7819
7820 +__E__+::
7821   Length expression.
7822
7823 |
7824 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7825
7826 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
7827 |
7828 Dynamically-sized array of integers, displayed in base{nbsp}16.
7829
7830 The type of +__E__+ must be unsigned.
7831
7832 +__t__+::
7833   Array element C type.
7834
7835 +__n__+::
7836   Field name.
7837
7838 +__e__+::
7839   Argument expression.
7840
7841 +__T__+::
7842   Length expression C type.
7843
7844 +__E__+::
7845   Length expression.
7846
7847 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
7848 |
7849 Dynamically-sized array of integers in network byte order (big-endian),
7850 displayed in base{nbsp}10.
7851
7852 The type of +__E__+ must be unsigned.
7853
7854 +__t__+::
7855   Array element C type.
7856
7857 +__n__+::
7858   Field name.
7859
7860 +__e__+::
7861   Argument expression.
7862
7863 +__T__+::
7864   Length expression C type.
7865
7866 +__E__+::
7867   Length expression.
7868
7869 |
7870 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7871
7872 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7873
7874 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
7875
7876 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
7877 |
7878 Dynamically-sized array of bits.
7879
7880 The type of +__e__+ must be an integer type. +__s__+ is the number
7881 of elements of such type in +__e__+, not the number of bits.
7882
7883 The type of +__E__+ must be unsigned.
7884
7885 +__t__+::
7886   Array element C type.
7887
7888 +__n__+::
7889   Field name.
7890
7891 +__e__+::
7892   Argument expression.
7893
7894 +__T__+::
7895   Length expression C type.
7896
7897 +__E__+::
7898   Length expression.
7899
7900 |
7901 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7902
7903 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7904
7905 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
7906
7907 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
7908 |
7909 Dynamically-sized array, displayed as text.
7910
7911 The string doesn't need to be null-terminated.
7912
7913 The type of +__E__+ must be unsigned.
7914
7915 The behaviour is undefined if +__e__+ is `NULL`.
7916
7917 +__t__+::
7918   Sequence element C type (always `char`).
7919
7920 +__n__+::
7921   Field name.
7922
7923 +__e__+::
7924   Argument expression.
7925
7926 +__T__+::
7927   Length expression C type.
7928
7929 +__E__+::
7930   Length expression.
7931 |====
7932
7933 Use the `_user` versions when the argument expression, `e`, is
7934 a user space address. In the cases of `ctf_user_integer*()` and
7935 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
7936 be addressable.
7937
7938 The `_nowrite` versions omit themselves from the session trace, but are
7939 otherwise identical. This means the `_nowrite` fields won't be written
7940 in the recorded trace. Their primary purpose is to make some
7941 of the event context available to the
7942 <<enabling-disabling-events,event filters>> without having to
7943 commit the data to sub-buffers.
7944
7945
7946 [[glossary]]
7947 == Glossary
7948
7949 Terms related to LTTng and to tracing in general:
7950
7951 Babeltrace::
7952   The http://diamon.org/babeltrace[Babeltrace] project, which includes:
7953 +
7954 * The
7955   https://babeltrace.org/docs/v2.0/man1/babeltrace2.1/[cmd:babeltrace2]
7956   command-line interface.
7957 * The libbabeltrace2 library which offers a
7958   https://babeltrace.org/docs/v2.0/libbabeltrace2/[C API].
7959 * https://babeltrace.org/docs/v2.0/python/bt2/[Python{nbsp}3 bindings].
7960 * Plugins.
7961
7962 [[def-buffering-scheme]]<<channel-buffering-schemes,buffering scheme>>::
7963   A layout of <<def-sub-buffer,sub-buffers>> applied to a given channel.
7964
7965 [[def-channel]]<<channel,channel>>::
7966   An entity which is responsible for a set of
7967   <<def-ring-buffer,ring buffers>>.
7968 +
7969 <<def-event-rule,Event rules>> are always attached to a specific
7970 channel.
7971
7972 clock::
7973   A source of time for a <<def-tracer,tracer>>.
7974
7975 [[def-consumer-daemon]]<<lttng-consumerd,consumer daemon>>::
7976   A process which is responsible for consuming the full
7977   <<def-sub-buffer,sub-buffers>> and write them to a file system or
7978   send them over the network.
7979
7980 [[def-current-trace-chunk]]current trace chunk::
7981   A <<def-trace-chunk,trace chunk>> which includes the current content
7982   of all the <<def-sub-buffer,sub-buffers>> of the
7983   <<def-tracing-session-rotation,tracing session>> and the stream files
7984   produced since the latest event amongst:
7985 +
7986 * The creation of the <<def-tracing-session,tracing session>>.
7987 * The last tracing session rotation, if any.
7988
7989 <<channel-overwrite-mode-vs-discard-mode,discard mode>>::
7990   The <<def-event-record-loss-mode,event record loss mode>> in which
7991   the <<def-tracer,tracer>> _discards_ new event records when there's no
7992   <<def-sub-buffer,sub-buffer>> space left to store them.
7993
7994 [[def-event]]event::
7995   The consequence of the execution of an
7996   <<def-instrumentation-point,instrumentation point>>, like a
7997   <<def-tracepoint,tracepoint>> that you manually place in some source
7998   code, or a Linux kernel kprobe.
7999 +
8000 An event is said to _occur_ at a specific time. <<def-lttng,LTTng>> can
8001 take various actions upon the occurrence of an event, like record its
8002 payload to a <<def-sub-buffer,sub-buffer>>.
8003
8004 [[def-event-name]]event name::
8005   The name of an <<def-event,event>>, which is also the name of the
8006   <<def-event-record,event record>>.
8007 +
8008 This is also called the _instrumentation point name_.
8009
8010 [[def-event-record]]event record::
8011   A record, in a <<def-trace,trace>>, of the payload of an
8012   <<def-event,event>> which occured.
8013
8014 [[def-event-record-loss-mode]]<<channel-overwrite-mode-vs-discard-mode,event record loss mode>>::
8015   The mechanism by which event records of a given
8016   <<def-channel,channel>> are lost (not recorded) when there is no
8017   <<def-sub-buffer,sub-buffer>> space left to store them.
8018
8019 [[def-event-rule]]<<event,event rule>>::
8020   Set of conditions which must be satisfied for one or more occuring
8021   <<def-event,events>> to be recorded.
8022
8023 [[def-incl-set]]inclusion set::
8024   In the <<pid-tracking,process attribute tracking>> context: a
8025   set of <<def-proc-attr,process attributes>> of a given type.
8026
8027 <<instrumenting,instrumentation>>::
8028   The use of <<def-lttng,LTTng>> probes to make a piece of software
8029   traceable.
8030
8031 [[def-instrumentation-point]]instrumentation point::
8032   A point in the execution path of a piece of software that, when
8033   reached by this execution, can emit an <<def-event,event>>.
8034
8035 instrumentation point name::
8036   See _<<def-event-name,event name>>_.
8037
8038 `java.util.logging`::
8039   The
8040   https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities]
8041   of the Java platform.
8042
8043 log4j::
8044   A http://logging.apache.org/log4j/1.2/[logging library] for Java
8045   developed by the Apache Software Foundation.
8046
8047 log level::
8048   Level of severity of a log statement or user space
8049   <<def-instrumentation-point,instrumentation point>>.
8050
8051 [[def-lttng]]LTTng::
8052   The _Linux Trace Toolkit: next generation_ project.
8053
8054 <<lttng-cli,cmd:lttng>>::
8055   A command-line tool provided by the <<def-lttng-tools,LTTng-tools>>
8056   project which you can use to send and receive control messages to and
8057   from a <<def-session-daemon,session daemon>>.
8058
8059 cmd:lttng-consumerd::
8060   The name of the <<def-consumer-daemon,consumer daemon>> program.
8061
8062 cmd:lttng-crash::
8063   A utility provided by the <<def-lttng-tools,LTTng-tools>> project
8064   which can convert <<def-ring-buffer,ring buffer>> files (usually
8065   <<persistent-memory-file-systems,saved on a persistent memory file
8066   system>>) to <<def-trace,trace>> files.
8067 +
8068 See man:lttng-crash(1).
8069
8070 LTTng Documentation::
8071   This document.
8072
8073 <<lttng-live,LTTng live>>::
8074   A communication protocol between the <<lttng-relayd,relay daemon>> and
8075   live viewers which makes it possible to see <<def-event-record,event
8076   records>> ``live'', as they are received by the
8077   <<def-relay-daemon,relay daemon>>.
8078
8079 <<lttng-modules,LTTng-modules>>::
8080   The https://github.com/lttng/lttng-modules[LTTng-modules] project,
8081   which contains the Linux kernel modules to make the Linux kernel
8082   <<def-instrumentation-point,instrumentation points>> available for
8083   <<def-lttng,LTTng>> tracing.
8084
8085 cmd:lttng-relayd::
8086   The name of the <<def-relay-daemon,relay daemon>> program.
8087
8088 cmd:lttng-sessiond::
8089   The name of the <<def-session-daemon,session daemon>> program.
8090
8091 [[def-lttng-tools]]LTTng-tools::
8092   The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
8093   contains the various programs and libraries used to
8094   <<controlling-tracing,control tracing>>.
8095
8096 [[def-lttng-ust]]<<lttng-ust,LTTng-UST>>::
8097   The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
8098   contains libraries to instrument
8099   <<def-user-application,user applications>>.
8100
8101 <<lttng-ust-agents,LTTng-UST Java agent>>::
8102   A Java package provided by the <<def-lttng-ust,LTTng-UST>> project to
8103   allow the LTTng instrumentation of `java.util.logging` and Apache
8104   log4j{nbsp}1.2 logging statements.
8105
8106 <<lttng-ust-agents,LTTng-UST Python agent>>::
8107   A Python package provided by the <<def-lttng-ust,LTTng-UST>> project
8108   to allow the <<def-lttng,LTTng>> instrumentation of Python logging
8109   statements.
8110
8111 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
8112   The <<def-event-record-loss-mode,event record loss mode>> in which new
8113   <<def-event-record,event records>> _overwrite_ older event records
8114   when there's no <<def-sub-buffer,sub-buffer>> space left to store
8115   them.
8116
8117 <<channel-buffering-schemes,per-process buffering>>::
8118   A <<def-buffering-scheme,buffering scheme>> in which each instrumented
8119   process has its own <<def-sub-buffer,sub-buffers>> for a given user
8120   space <<def-channel,channel>>.
8121
8122 <<channel-buffering-schemes,per-user buffering>>::
8123   A <<def-buffering-scheme,buffering scheme>> in which all the processes
8124   of a Unix user share the same <<def-sub-buffer,sub-buffers>> for a
8125   given user space <<def-channel,channel>>.
8126
8127 [[def-proc-attr]]process attribute::
8128   In the <<pid-tracking,process attribute tracking>> context:
8129 +
8130 * A process ID.
8131 * A virtual process ID.
8132 * A Unix user ID.
8133 * A virtual Unix user ID.
8134 * A Unix group ID.
8135 * A virtual Unix group ID.
8136
8137 [[def-relay-daemon]]<<lttng-relayd,relay daemon>>::
8138   A process which is responsible for receiving the <<def-trace,trace>>
8139   data which a distant <<def-consumer-daemon,consumer daemon>> sends.
8140
8141 [[def-ring-buffer]]ring buffer::
8142   A set of <<def-sub-buffer,sub-buffers>>.
8143
8144 rotation::
8145   See _<<def-tracing-session-rotation,tracing session rotation>>_.
8146
8147 [[def-session-daemon]]<<lttng-sessiond,session daemon>>::
8148   A process which receives control commands from you and orchestrates
8149   the <<def-tracer,tracers>> and various <<def-lttng,LTTng>> daemons.
8150
8151 <<taking-a-snapshot,snapshot>>::
8152   A copy of the current data of all the <<def-sub-buffer,sub-buffers>>
8153   of a given <<def-tracing-session,tracing session>>, saved as
8154   <<def-trace,trace>> files.
8155
8156 [[def-sub-buffer]]sub-buffer::
8157   One part of an <<def-lttng,LTTng>> <<def-ring-buffer,ring buffer>>
8158   which contains <<def-event-record,event records>>.
8159
8160 timestamp::
8161   The time information attached to an <<def-event,event>> when it is
8162   emitted.
8163
8164 [[def-trace]]trace (_noun_)::
8165   A set of:
8166 +
8167 * One http://diamon.org/ctf/[CTF] metadata stream file.
8168 * One or more CTF data stream files which are the concatenations of one
8169   or more flushed <<def-sub-buffer,sub-buffers>>.
8170
8171 [[def-trace-verb]]trace (_verb_)::
8172   The action of recording the <<def-event,events>> emitted by an
8173   application or by a system, or to initiate such recording by
8174   controlling a <<def-tracer,tracer>>.
8175
8176 [[def-trace-chunk]]trace chunk::
8177   A self-contained <<def-trace,trace>> which is part of a
8178   <<def-tracing-session,tracing session>>. Each
8179   <<def-tracing-session-rotation, tracing session rotation>> produces a
8180   <<def-trace-chunk-archive,trace chunk archive>>.
8181
8182 [[def-trace-chunk-archive]]trace chunk archive::
8183   The result of a <<def-tracing-session-rotation, tracing session rotation>>.
8184 +
8185 <<def-lttng,LTTng>> doesn't manage any trace chunk archive, even if its
8186 containing <<def-tracing-session,tracing session>> is still active: you
8187 are free to read it, modify it, move it, or remove it.
8188
8189 Trace Compass::
8190   The http://tracecompass.org[Trace Compass] project and application.
8191
8192 [[def-tracepoint]]tracepoint::
8193   An instrumentation point using the tracepoint mechanism of the Linux
8194   kernel or of <<def-lttng-ust,LTTng-UST>>.
8195
8196 tracepoint definition::
8197   The definition of a single <<def-tracepoint,tracepoint>>.
8198
8199 tracepoint name::
8200   The name of a <<def-tracepoint,tracepoint>>.
8201
8202 [[def-tracepoint-provider]]tracepoint provider::
8203   A set of functions providing <<def-tracepoint,tracepoints>> to an
8204   instrumented <<def-user-application,user application>>.
8205 +
8206 Not to be confused with a <<def-tracepoint-provider-package,tracepoint
8207 provider package>>: many tracepoint providers can exist within a
8208 tracepoint provider package.
8209
8210 [[def-tracepoint-provider-package]]tracepoint provider package::
8211   One or more <<def-tracepoint-provider,tracepoint providers>> compiled
8212   as an https://en.wikipedia.org/wiki/Object_file[object file] or as a
8213   link:https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries[shared
8214   library].
8215
8216 [[def-tracer]]tracer::
8217   A software which records emitted <<def-event,events>>.
8218
8219 <<domain,tracing domain>>::
8220   A namespace for <<def-event,event>> sources.
8221
8222 <<tracing-group,tracing group>>::
8223   The Unix group in which a Unix user can be to be allowed to
8224   <<def-trace-verb,trace>> the Linux kernel.
8225
8226 [[def-tracing-session]]<<tracing-session,tracing session>>::
8227   A stateful dialogue between you and a <<lttng-sessiond,session daemon>>.
8228
8229 [[def-tracing-session-rotation]]<<session-rotation,tracing session rotation>>::
8230   The action of archiving the
8231   <<def-current-trace-chunk,current trace chunk>> of a
8232   <<def-tracing-session,tracing session>>.
8233
8234 tracked <<def-proc-attr,process attribute>>::
8235   A process attribute which is part of an <<def-incl-set,inclusion
8236   set>>.
8237
8238 untracked process attribute::
8239   A process attribute which isn't part of an <<def-incl-set,inclusion
8240   set>>.
8241
8242 [[def-user-application]]user application::
8243   An application running in user space, as opposed to a Linux kernel
8244   module, for example.