2.8/lttng-docs-2.8.txt

   1 The LTTng Documentation
   2 =======================
   3 Philippe Proulx <pproulx@efficios.com>
   4 v2.8, 24 July 2017
   5
   6
   7 include::../common/copyright.txt[]
   8
   9
  10 include::../common/warning-not-maintained.txt[]
  11
  12
  13 include::../common/welcome.txt[]
  14
  15
  16 include::../common/audience.txt[]
  17
  18
  19 [[chapters]]
  20 === What's in this documentation?
  21
  22 The LTTng Documentation is divided into the following sections:
  23
  24 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
  25   rudiments of software tracing and the rationale behind the
  26   LTTng project.
  27 +
  28 You can skip this section if you’re familiar with software tracing and
  29 with the LTTng project.
  30
  31 * **<<installing-lttng,Installation>>** describes the steps to
  32   install the LTTng packages on common Linux distributions and from
  33   their sources.
  34 +
  35 You can skip this section if you already properly installed LTTng on
  36 your target system.
  37
  38 * **<<getting-started,Quick start>>** is a concise guide to
  39   getting started quickly with LTTng kernel and user space tracing.
  40 +
  41 We recommend this section if you're new to LTTng or to software tracing
  42 in general.
  43 +
  44 You can skip this section if you're not new to LTTng.
  45
  46 * **<<core-concepts,Core concepts>>** explains the concepts at
  47   the heart of LTTng.
  48 +
  49 It's a good idea to become familiar with the core concepts
  50 before attempting to use the toolkit.
  51
  52 * **<<plumbing,Components of LTTng>>** describes the various components
  53   of the LTTng machinery, like the daemons, the libraries, and the
  54   command-line interface.
  55 * **<<instrumenting,Instrumentation>>** shows different ways to
  56   instrument user applications and the Linux kernel.
  57 +
  58 Instrumenting source code is essential to provide a meaningful
  59 source of events.
  60 +
  61 You can skip this section if you do not have a programming background.
  62
  63 * **<<controlling-tracing,Tracing control>>** is divided into topics
  64   which demonstrate how to use the vast array of features that
  65   LTTng{nbsp}{revision} offers.
  66 * **<<reference,Reference>>** contains reference tables.
  67 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
  68   to LTTng or to the field of software tracing.
  69
  70
  71 include::../common/convention.txt[]
  72
  73
  74 include::../common/acknowledgements.txt[]
  75
  76
  77 [[whats-new]]
  78 == What's new in LTTng {revision}?
  79
  80 LTTng{nbsp}{revision} bears the name _Isseki Nicho_. The result of a
  81 collaboration between http://www.dieuduciel.com/[Dieu du Ciel!] and
  82 Nagano-based Shiga Kogen,
  83 https://www.beeradvocate.com/beer/profile/1141/53111/[_**Isseki
  84 Nicho**_] is a strong Imperial Dark Saison offering a rich roasted malt
  85 flavor combined with a complex fruity finish typical of Saison yeasts.
  86
  87 New features and changes in LTTng{nbsp}{revision}:
  88
  89 * **Tracing control**:
  90 ** You can attach <<java-application-context,Java application-specific
  91    context fields>> to a <<channel,channel>> with the
  92    man:lttng-add-context(1) command:
  93 +
  94 --
  95 [role="term"]
  96 ----
  97 $ lttng add-context --jul --type='$app.retriever:cur_msg_id'
  98 ----
  99 --
 100 +
 101 Here, `$app` is the prefix of all application-specific context fields,
 102 `retriever` names a _context information retriever_ defined at the
 103 application level, and `cur_msg_id` names a context field read from this
 104 retriever.
 105 +
 106 Both the `java.util.logging` and Apache log4j <<domain,tracing domains>>
 107 are supported.
 108
 109 ** You can use Java application-specific <<adding-context,context>>
 110    fields in the <<enabling-disabling-events,filter expression>> of an
 111    <<event,event rule>>:
 112 +
 113 --
 114 [role="term"]
 115 ----
 116 $ lttng enable-event --log4j my_logger \
 117                    --filter='$app.retriever:cur_msg_id == 23'
 118 ----
 119 --
 120
 121 ** New `lttng status` command which is the equivalent of +lttng list
 122    __CUR__+, where +__CUR__+ is the name of the current
 123    <<tracing-session,tracing session>>.
 124 +
 125 See man:lttng-status(1).
 126
 127 ** New `lttng metadata regenerate` command to
 128    <<metadata-regenerate,regenerate the metadata file of an LTTng
 129    trace>> at any moment. This command is meant to be used to resample
 130    the wall time following a major
 131    https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
 132    so that a system which boots with an incorrect wall time can be
 133    traced before its wall time is NTP-corrected.
 134 +
 135 See man:lttng-metadata(1).
 136
 137 ** New command-line interface warnings when <<event,event records>> or
 138    whole sub-buffers are
 139    <<channel-overwrite-mode-vs-discard-mode,lost>>. The warning messages
 140    are printed when a <<tracing-session,tracing session>> is
 141    <<basic-tracing-session-control,stopped>> (man:lttng-stop(1)
 142    command).
 143
 144 * **User space tracing**:
 145 ** Shared object base address dump in order to map <<event,event
 146    records>> to original source location (file and line number).
 147 +
 148 If you attach the `ip` and `vpid` <<adding-context,context fields>> to a
 149 user space <<channel,channel>> and if you use the
 150 <<liblttng-ust-dl,path:{liblttng-ust-dl.so} helper>>, you can retrieve
 151 the source location where a given event record was generated.
 152 +
 153 The http://diamon.org/babeltrace/[Babeltrace] trace viewer supports this
 154 state dump and those context fields since version 1.4 to print the
 155 source location of a given event record. http://tracecompass.org/[Trace
 156 Compass] also supports this since version 2.0.
 157
 158 ** A <<java-application,Java application>> which uses
 159    `java.util.logging` now adds an LTTng-UST log handler to the desired
 160    JUL loggers.
 161 +
 162 The previous workflow was to initialize the LTTng-UST Java agent
 163 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
 164 an LTTng-UST log handler to the root loggers.
 165
 166 ** A <<java-application,Java application>> which uses Apache log4j now
 167    adds an LTTng-UST log appender to the desired log4j loggers.
 168 +
 169 The previous workflow was to initialize the LTTng-UST Java agent
 170 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
 171 an LTTng-UST appender to the root loggers.
 172
 173 ** Any <<java-application,Java application>> can provide
 174    <<java-application-context,dynamic context fields>> while running
 175    thanks to a new API provided by the <<lttng-ust-agents,LTTng-UST Java
 176    agent>>. You can require LTTng to record specific context fields in
 177    event records, and you can use them in the filter expression of
 178    <<event,event rules>>.
 179
 180 * **Linux kernel tracing**:
 181 ** The LTTng kernel modules can now be built into a Linux kernel image,
 182    that is, not as loadable modules.
 183 +
 184 Follow the project's
 185 https://github.com/lttng/lttng-modules/blob/stable-{revision}/README.md#kernel-built-in-support[`README.md`]
 186 file to learn how.
 187
 188 ** New instrumentation:
 189 *** ARM64 architecture support.
 190 *** x86 page faults.
 191 *** x86 `irq_vectors`.
 192 ** New <<adding-context,context fields>>:
 193 *** `interruptible`
 194 *** `preemptible`
 195 *** `need_reschedule`
 196 *** `migratable` (specific to RT-Preempt)
 197 ** Clock source plugin support for advanced cases where a custom source
 198    of time is needed to timestamp LTTng event records.
 199 +
 200 See https://github.com/lttng/lttng-modules/blob/stable-{revision}/lttng-clock.h[`lttng-clock.h`]
 201 for an overview of the small API.
 202
 203 * **Documentation**:
 204 ** The link:/man[man pages] of the man:lttng(1) command-line tool are
 205    split into one man page per command (à la Git), for example:
 206 +
 207 --
 208 [role="term"]
 209 ----
 210 $ man lttng-enable-event
 211 ----
 212 --
 213 +
 214 You can also use the `--help` option of any man:lttng(1) command to
 215 open its man page.
 216 +
 217 The content and formatting of all the LTTng man pages has improved
 218 dramatically.
 219
 220
 221 [[nuts-and-bolts]]
 222 == Nuts and bolts
 223
 224 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
 225 generation_ is a modern toolkit for tracing Linux systems and
 226 applications. So your first question might be:
 227 **what is tracing?**
 228
 229
 230 [[what-is-tracing]]
 231 === What is tracing?
 232
 233 As the history of software engineering progressed and led to what
 234 we now take for granted--complex, numerous and
 235 interdependent software applications running in parallel on
 236 sophisticated operating systems like Linux--the authors of such
 237 components, software developers, began feeling a natural
 238 urge to have tools that would ensure the robustness and good performance
 239 of their masterpieces.
 240
 241 One major achievement in this field is, inarguably, the
 242 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
 243 an essential tool for developers to find and fix bugs. But even the best
 244 debugger won't help make your software run faster, and nowadays, faster
 245 software means either more work done by the same hardware, or cheaper
 246 hardware for the same work.
 247
 248 A _profiler_ is often the tool of choice to identify performance
 249 bottlenecks. Profiling is suitable to identify _where_ performance is
 250 lost in a given software. The profiler outputs a profile, a statistical
 251 summary of observed events, which you may use to discover which
 252 functions took the most time to execute. However, a profiler won't
 253 report _why_ some identified functions are the bottleneck. Bottlenecks
 254 might only occur when specific conditions are met, conditions that are
 255 sometimes impossible to capture by a statistical profiler, or impossible
 256 to reproduce with an application altered by the overhead of an
 257 event-based profiler. For a thorough investigation of software
 258 performance issues, a history of execution is essential, with the
 259 recorded values of variables and context fields you choose, and
 260 with as little influence as possible on the instrumented software. This
 261 is where tracing comes in handy.
 262
 263 _Tracing_ is a technique used to understand what goes on in a running
 264 software system. The software used for tracing is called a _tracer_,
 265 which is conceptually similar to a tape recorder. When recording,
 266 specific instrumentation points placed in the software source code
 267 generate events that are saved on a giant tape: a _trace_ file. You
 268 can trace user applications and the operating system at the same time,
 269 opening the possibility of resolving a wide range of problems that would
 270 otherwise be extremely challenging.
 271
 272 Tracing is often compared to _logging_. However, tracers and loggers are
 273 two different tools, serving two different purposes. Tracers are
 274 designed to record much lower-level events that occur much more
 275 frequently than log messages, often in the range of thousands per
 276 second, with very little execution overhead. Logging is more appropriate
 277 for a very high-level analysis of less frequent events: user accesses,
 278 exceptional conditions (errors and warnings, for example), database
 279 transactions, instant messaging communications, and such. Simply put,
 280 logging is one of the many use cases that can be satisfied with tracing.
 281
 282 The list of recorded events inside a trace file can be read manually
 283 like a log file for the maximum level of detail, but it is generally
 284 much more interesting to perform application-specific analyses to
 285 produce reduced statistics and graphs that are useful to resolve a
 286 given problem. Trace viewers and analyzers are specialized tools
 287 designed to do this.
 288
 289 In the end, this is what LTTng is: a powerful, open source set of
 290 tools to trace the Linux kernel and user applications at the same time.
 291 LTTng is composed of several components actively maintained and
 292 developed by its link:/community/#where[community].
 293
 294
 295 [[lttng-alternatives]]
 296 === Alternatives to noch:{LTTng}
 297
 298 Excluding proprietary solutions, a few competing software tracers
 299 exist for Linux:
 300
 301 * https://github.com/dtrace4linux/linux[dtrace4linux] is a port of
 302   Sun Microsystems's DTrace to Linux. The cmd:dtrace tool interprets
 303   user scripts and is responsible for loading code into the
 304   Linux kernel for further execution and collecting the outputted data.
 305 * https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF] is a
 306   subsystem in the Linux kernel in which a virtual machine can execute
 307   programs passed from the user space to the kernel. You can attach
 308   such programs to tracepoints and KProbes thanks to a system call, and
 309   they can output data to the user space when executed thanks to
 310   different mechanisms (pipe, VM register values, and eBPF maps, to name
 311   a few).
 312 * https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]
 313   is the de facto function tracer of the Linux kernel. Its user
 314   interface is a set of special files in sysfs.
 315 * https://perf.wiki.kernel.org/[perf] is
 316   a performance analyzing tool for Linux which supports hardware
 317   performance counters, tracepoints, as well as other counters and
 318   types of probes. perf's controlling utility is the cmd:perf command
 319   line/curses tool.
 320 * http://linux.die.net/man/1/strace[strace]
 321   is a command-line utility which records system calls made by a
 322   user process, as well as signal deliveries and changes of process
 323   state. strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace]
 324   to fulfill its function.
 325 * http://www.sysdig.org/[sysdig], like SystemTap, uses scripts to
 326   analyze Linux kernel events. You write scripts, or _chisels_ in
 327   sysdig's jargon, in Lua and sysdig executes them while the system is
 328   being traced or afterwards. sysdig's interface is the cmd:sysdig
 329   command-line tool as well as the curses-based cmd:csysdig tool.
 330 * https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and
 331   user space tracer which uses custom user scripts to produce plain text
 332   traces. SystemTap converts the scripts to the C language, and then
 333   compiles them as Linux kernel modules which are loaded to produce
 334   trace data. SystemTap's primary user interface is the cmd:stap
 335   command-line tool.
 336
 337 The main distinctive features of LTTng is that it produces correlated
 338 kernel and user space traces, as well as doing so with the lowest
 339 overhead amongst other solutions. It produces trace files in the
 340 http://diamon.org/ctf[CTF] format, a file format optimized
 341 for the production and analyses of multi-gigabyte data.
 342
 343 LTTng is the result of more than 10 years of active open source
 344 development by a community of passionate developers.
 345 LTTng{nbsp}{revision} is currently available on major desktop and server
 346 Linux distributions.
 347
 348 The main interface for tracing control is a single command-line tool
 349 named cmd:lttng. The latter can create several tracing sessions, enable
 350 and disable events on the fly, filter events efficiently with custom
 351 user expressions, start and stop tracing, and much more. LTTng can
 352 record the traces on the file system or send them over the network, and
 353 keep them totally or partially. You can view the traces once tracing
 354 becomes inactive or in real-time.
 355
 356 <<installing-lttng,Install LTTng now>> and
 357 <<getting-started,start tracing>>!
 358
 359
 360 [[installing-lttng]]
 361 == Installation
 362
 363 include::../common/warning-no-installation.txt[]
 364
 365 **LTTng** is a set of software <<plumbing,components>> which interact to
 366 <<instrumenting,instrument>> the Linux kernel and user applications, and
 367 to <<controlling-tracing,control tracing>> (start and stop
 368 tracing, enable and disable event rules, and the rest). Those
 369 components are bundled into the following packages:
 370
 371 * **LTTng-tools**: Libraries and command-line interface to
 372   control tracing.
 373 * **LTTng-modules**: Linux kernel modules to instrument and
 374   trace the kernel.
 375 * **LTTng-UST**: Libraries and Java/Python packages to instrument and
 376   trace user applications.
 377
 378 Most distributions mark the LTTng-modules and LTTng-UST packages as
 379 optional when installing LTTng-tools (which is always required). Note
 380 that:
 381
 382 * You only need to install LTTng-modules if you intend to trace the
 383   Linux kernel.
 384 * You only need to install LTTng-UST if you intend to trace user
 385   applications.
 386
 387
 388 [[building-from-source]]
 389 === Build from source
 390
 391 To build and install LTTng{nbsp}{revision} from source:
 392
 393 . Using your distribution's package manager, or from source, install
 394   the following dependencies of LTTng-tools and LTTng-UST:
 395 +
 396 --
 397 * https://sourceforge.net/projects/libuuid/[libuuid]
 398 * http://directory.fsf.org/wiki/Popt[popt]
 399 * http://liburcu.org/[Userspace RCU]
 400 * http://www.xmlsoft.org/[libxml2]
 401 --
 402
 403 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
 404 +
 405 --
 406 [role="term"]
 407 ----
 408 $ cd $(mktemp -d) &&
 409 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
 410 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
 411 cd lttng-modules-2.8.* &&
 412 make &&
 413 sudo make modules_install &&
 414 sudo depmod -a
 415 ----
 416 --
 417
 418 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
 419 +
 420 --
 421 [role="term"]
 422 ----
 423 $ cd $(mktemp -d) &&
 424 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
 425 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
 426 cd lttng-ust-2.8.* &&
 427 ./configure &&
 428 make &&
 429 sudo make install &&
 430 sudo ldconfig
 431 ----
 432 --
 433 +
 434 --
 435 [IMPORTANT]
 436 .Java and Python application tracing
 437 ====
 438 If you need to instrument and trace <<java-application,Java
 439 applications>>, pass the `--enable-java-agent-jul`,
 440 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
 441 `configure` script, depending on which Java logging framework you use.
 442
 443 If you need to instrument and trace <<python-application,Python
 444 applications>>, pass the `--enable-python-agent` option to the
 445 `configure` script. You can set the `PYTHON` environment variable to the
 446 path to the Python interpreter for which to install the LTTng-UST Python
 447 agent package.
 448 ====
 449 --
 450 +
 451 --
 452 [NOTE]
 453 ====
 454 By default, LTTng-UST libraries are installed to
 455 dir:{/usr/local/lib}, which is the de facto directory in which to
 456 keep self-compiled and third-party libraries.
 457
 458 When <<building-tracepoint-providers-and-user-application,linking an
 459 instrumented user application with `liblttng-ust`>>:
 460
 461 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
 462   variable.
 463 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
 464   man:gcc(1), man:g++(1), or man:clang(1).
 465 ====
 466 --
 467
 468 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
 469 +
 470 --
 471 [role="term"]
 472 ----
 473 $ cd $(mktemp -d) &&
 474 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
 475 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
 476 cd lttng-tools-2.8.* &&
 477 ./configure &&
 478 make &&
 479 sudo make install &&
 480 sudo ldconfig
 481 ----
 482 --
 483
 484 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
 485 previous steps automatically for a given version of LTTng and confine
 486 the installed files in a specific directory. This can be useful to test
 487 LTTng without installing it on your system.
 488
 489
 490 [[getting-started]]
 491 == Quick start
 492
 493 This is a short guide to get started quickly with LTTng kernel and user
 494 space tracing.
 495
 496 Before you follow this guide, make sure to <<installing-lttng,install>>
 497 LTTng.
 498
 499 This tutorial walks you through the steps to:
 500
 501 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
 502 . <<tracing-your-own-user-application,Trace a user application>> written
 503   in C.
 504 . <<viewing-and-analyzing-your-traces,View and analyze the
 505   recorded events>>.
 506
 507
 508 [[tracing-the-linux-kernel]]
 509 === Trace the Linux kernel
 510
 511 The following command lines start with the `#` prompt because you need
 512 root privileges to trace the Linux kernel. You can also trace the kernel
 513 as a regular user if your Unix user is a member of the
 514 <<tracing-group,tracing group>>.
 515
 516 . Create a <<tracing-session,tracing session>> which writes its traces
 517   to dir:{/tmp/my-kernel-trace}:
 518 +
 519 --
 520 [role="term"]
 521 ----
 522 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
 523 ----
 524 --
 525
 526 . List the available kernel tracepoints and system calls:
 527 +
 528 --
 529 [role="term"]
 530 ----
 531 # lttng list --kernel
 532 # lttng list --kernel --syscall
 533 ----
 534 --
 535
 536 . Create <<event,event rules>> which match the desired instrumentation
 537   point names, for example the `sched_switch` and `sched_process_fork`
 538   tracepoints, and the man:open(2) and man:close(2) system calls:
 539 +
 540 --
 541 [role="term"]
 542 ----
 543 # lttng enable-event --kernel sched_switch,sched_process_fork
 544 # lttng enable-event --kernel --syscall open,close
 545 ----
 546 --
 547 +
 548 You can also create an event rule which matches _all_ the Linux kernel
 549 tracepoints (this will generate a lot of data when tracing):
 550 +
 551 --
 552 [role="term"]
 553 ----
 554 # lttng enable-event --kernel --all
 555 ----
 556 --
 557
 558 . <<basic-tracing-session-control,Start tracing>>:
 559 +
 560 --
 561 [role="term"]
 562 ----
 563 # lttng start
 564 ----
 565 --
 566
 567 . Do some operation on your system for a few seconds. For example,
 568   load a website, or list the files of a directory.
 569 . <<basic-tracing-session-control,Stop tracing>> and destroy the
 570   tracing session:
 571 +
 572 --
 573 [role="term"]
 574 ----
 575 sudo lttng stop
 576 sudo lttng destroy
 577 ----
 578 --
 579 +
 580 The man:lttng-destroy(1) command does not destroy the trace data; it
 581 only destroys the state of the tracing session.
 582
 583 . For the sake of this example, make the recorded trace accessible to
 584   the non-root users:
 585 +
 586 --
 587 [role="term"]
 588 ----
 589 sudo chown -R $(whoami) /tmp/my-kernel-trace
 590 ----
 591 --
 592
 593 See <<viewing-and-analyzing-your-traces,View and analyze the
 594 recorded events>> to view the recorded events.
 595
 596
 597 [[tracing-your-own-user-application]]
 598 === Trace a user application
 599
 600 This section steps you through a simple example to trace a
 601 _Hello world_ program written in C.
 602
 603 To create the traceable user application:
 604
 605 . Create the tracepoint provider header file, which defines the
 606   tracepoints and the events they can generate:
 607 +
 608 --
 609 [source,c]
 610 .path:{hello-tp.h}
 611 ----
 612 #undef TRACEPOINT_PROVIDER
 613 #define TRACEPOINT_PROVIDER hello_world
 614
 615 #undef TRACEPOINT_INCLUDE
 616 #define TRACEPOINT_INCLUDE "./hello-tp.h"
 617
 618 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
 619 #define _HELLO_TP_H
 620
 621 #include <lttng/tracepoint.h>
 622
 623 TRACEPOINT_EVENT(
 624     hello_world,
 625     my_first_tracepoint,
 626     TP_ARGS(
 627         int, my_integer_arg,
 628         char*, my_string_arg
 629     ),
 630     TP_FIELDS(
 631         ctf_string(my_string_field, my_string_arg)
 632         ctf_integer(int, my_integer_field, my_integer_arg)
 633     )
 634 )
 635
 636 #endif /* _HELLO_TP_H */
 637
 638 #include <lttng/tracepoint-event.h>
 639 ----
 640 --
 641
 642 . Create the tracepoint provider package source file:
 643 +
 644 --
 645 [source,c]
 646 .path:{hello-tp.c}
 647 ----
 648 #define TRACEPOINT_CREATE_PROBES
 649 #define TRACEPOINT_DEFINE
 650
 651 #include "hello-tp.h"
 652 ----
 653 --
 654
 655 . Build the tracepoint provider package:
 656 +
 657 --
 658 [role="term"]
 659 ----
 660 $ gcc -c -I. hello-tp.c
 661 ----
 662 --
 663
 664 . Create the _Hello World_ application source file:
 665 +
 666 --
 667 [source,c]
 668 .path:{hello.c}
 669 ----
 670 #include <stdio.h>
 671 #include "hello-tp.h"
 672
 673 int main(int argc, char *argv[])
 674 {
 675     int x;
 676
 677     puts("Hello, World!\nPress Enter to continue...");
 678
 679     /*
 680      * The following getchar() call is only placed here for the purpose
 681      * of this demonstration, to pause the application in order for
 682      * you to have time to list its tracepoints. It is not
 683      * needed otherwise.
 684      */
 685     getchar();
 686
 687     /*
 688      * A tracepoint() call.
 689      *
 690      * Arguments, as defined in hello-tp.h:
 691      *
 692      * 1. Tracepoint provider name   (required)
 693      * 2. Tracepoint name            (required)
 694      * 3. my_integer_arg             (first user-defined argument)
 695      * 4. my_string_arg              (second user-defined argument)
 696      *
 697      * Notice the tracepoint provider and tracepoint names are
 698      * NOT strings: they are in fact parts of variables that the
 699      * macros in hello-tp.h create.
 700      */
 701     tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
 702
 703     for (x = 0; x < argc; ++x) {
 704         tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
 705     }
 706
 707     puts("Quitting now!");
 708     tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
 709
 710     return 0;
 711 }
 712 ----
 713 --
 714
 715 . Build the application:
 716 +
 717 --
 718 [role="term"]
 719 ----
 720 $ gcc -c hello.c
 721 ----
 722 --
 723
 724 . Link the application with the tracepoint provider package,
 725   `liblttng-ust`, and `libdl`:
 726 +
 727 --
 728 [role="term"]
 729 ----
 730 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
 731 ----
 732 --
 733
 734 Here's the whole build process:
 735
 736 [role="img-100"]
 737 .User space tracing tutorial's build steps.
 738 image::ust-flow.png[]
 739
 740 To trace the user application:
 741
 742 . Run the application with a few arguments:
 743 +
 744 --
 745 [role="term"]
 746 ----
 747 $ ./hello world and beyond
 748 ----
 749 --
 750 +
 751 You see:
 752 +
 753 --
 754 ----
 755 Hello, World!
 756 Press Enter to continue...
 757 ----
 758 --
 759
 760 . Start an LTTng <<lttng-sessiond,session daemon>>:
 761 +
 762 --
 763 [role="term"]
 764 ----
 765 $ lttng-sessiond --daemonize
 766 ----
 767 --
 768 +
 769 Note that a session daemon might already be running, for example as
 770 a service that the distribution's service manager started.
 771
 772 . List the available user space tracepoints:
 773 +
 774 --
 775 [role="term"]
 776 ----
 777 $ lttng list --userspace
 778 ----
 779 --
 780 +
 781 You see the `hello_world:my_first_tracepoint` tracepoint listed
 782 under the `./hello` process.
 783
 784 . Create a <<tracing-session,tracing session>>:
 785 +
 786 --
 787 [role="term"]
 788 ----
 789 $ lttng create my-user-space-session
 790 ----
 791 --
 792
 793 . Create an <<event,event rule>> which matches the
 794   `hello_world:my_first_tracepoint` event name:
 795 +
 796 --
 797 [role="term"]
 798 ----
 799 $ lttng enable-event --userspace hello_world:my_first_tracepoint
 800 ----
 801 --
 802
 803 . <<basic-tracing-session-control,Start tracing>>:
 804 +
 805 --
 806 [role="term"]
 807 ----
 808 $ lttng start
 809 ----
 810 --
 811
 812 . Go back to the running `hello` application and press Enter. The
 813   program executes all `tracepoint()` instrumentation points and exits.
 814 . <<basic-tracing-session-control,Stop tracing>> and destroy the
 815   tracing session:
 816 +
 817 --
 818 [role="term"]
 819 ----
 820 $ lttng stop
 821 $ lttng destroy
 822 ----
 823 --
 824 +
 825 The man:lttng-destroy(1) command does not destroy the trace data; it
 826 only destroys the state of the tracing session.
 827
 828 By default, LTTng saves the traces in
 829 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
 830 where +__name__+ is the tracing session name. The
 831 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
 832
 833 See <<viewing-and-analyzing-your-traces,View and analyze the
 834 recorded events>> to view the recorded events.
 835
 836
 837 [[viewing-and-analyzing-your-traces]]
 838 === View and analyze the recorded events
 839
 840 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
 841 kernel>> and <<tracing-your-own-user-application,Trace a user
 842 application>> tutorials, you can inspect the recorded events.
 843
 844 Many tools are available to read LTTng traces:
 845
 846 * **cmd:babeltrace** is a command-line utility which converts trace
 847   formats; it supports the format that LTTng produces, CTF, as well as a
 848   basic text output which can be ++grep++ed. The cmd:babeltrace command
 849   is part of the http://diamon.org/babeltrace[Babeltrace] project.
 850 * Babeltrace also includes
 851   **https://www.python.org/[Python] bindings** so
 852   that you can easily open and read an LTTng trace with your own script,
 853   benefiting from the power of Python.
 854 * http://tracecompass.org/[**Trace Compass**]
 855   is a graphical user interface for viewing and analyzing any type of
 856   logs or traces, including LTTng's.
 857 * https://github.com/lttng/lttng-analyses[**LTTng analyses**] is a
 858   project which includes many high-level analyses of LTTng kernel
 859   traces, like scheduling statistics, interrupt frequency distribution,
 860   top CPU usage, and more.
 861
 862 NOTE: This section assumes that the traces recorded during the previous
 863 tutorials were saved to their default location, in the
 864 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
 865 environment variable defaults to `$HOME` if not set.
 866
 867
 868 [[viewing-and-analyzing-your-traces-bt]]
 869 ==== Use the cmd:babeltrace command-line tool
 870
 871 The simplest way to list all the recorded events of a trace is to pass
 872 its path to cmd:babeltrace with no options:
 873
 874 [role="term"]
 875 ----
 876 $ babeltrace ~/lttng-traces/my-user-space-session*
 877 ----
 878
 879 cmd:babeltrace finds all traces recursively within the given path and
 880 prints all their events, merging them in chronological order.
 881
 882 You can pipe the output of cmd:babeltrace into a tool like man:grep(1) for
 883 further filtering:
 884
 885 [role="term"]
 886 ----
 887 $ babeltrace /tmp/my-kernel-trace | grep _switch
 888 ----
 889
 890 You can pipe the output of cmd:babeltrace into a tool like man:wc(1) to
 891 count the recorded events:
 892
 893 [role="term"]
 894 ----
 895 $ babeltrace /tmp/my-kernel-trace | grep _open | wc --lines
 896 ----
 897
 898
 899 [[viewing-and-analyzing-your-traces-bt-python]]
 900 ==== Use the Babeltrace Python bindings
 901
 902 The <<viewing-and-analyzing-your-traces-bt,text output of cmd:babeltrace>>
 903 is useful to isolate events by simple matching using man:grep(1) and
 904 similar utilities. However, more elaborate filters, such as keeping only
 905 event records with a field value falling within a specific range, are
 906 not trivial to write using a shell. Moreover, reductions and even the
 907 most basic computations involving multiple event records are virtually
 908 impossible to implement.
 909
 910 Fortunately, Babeltrace ships with Python 3 bindings which makes it easy
 911 to read the event records of an LTTng trace sequentially and compute the
 912 desired information.
 913
 914 The following script accepts an LTTng Linux kernel trace path as its
 915 first argument and prints the short names of the top 5 running processes
 916 on CPU 0 during the whole trace:
 917
 918 [source,python]
 919 .path:{top5proc.py}
 920 ----
 921 from collections import Counter
 922 import babeltrace
 923 import sys
 924
 925
 926 def top5proc():
 927     if len(sys.argv) != 2:
 928         msg = 'Usage: python3 {} TRACEPATH'.format(sys.argv[0])
 929         print(msg, file=sys.stderr)
 930         return False
 931
 932     # A trace collection contains one or more traces
 933     col = babeltrace.TraceCollection()
 934
 935     # Add the trace provided by the user (LTTng traces always have
 936     # the 'ctf' format)
 937     if col.add_trace(sys.argv[1], 'ctf') is None:
 938         raise RuntimeError('Cannot add trace')
 939
 940     # This counter dict contains execution times:
 941     #
 942     #   task command name -> total execution time (ns)
 943     exec_times = Counter()
 944
 945     # This contains the last `sched_switch` timestamp
 946     last_ts = None
 947
 948     # Iterate on events
 949     for event in col.events:
 950         # Keep only `sched_switch` events
 951         if event.name != 'sched_switch':
 952             continue
 953
 954         # Keep only events which happened on CPU 0
 955         if event['cpu_id'] != 0:
 956             continue
 957
 958         # Event timestamp
 959         cur_ts = event.timestamp
 960
 961         if last_ts is None:
 962             # We start here
 963             last_ts = cur_ts
 964
 965         # Previous task command (short) name
 966         prev_comm = event['prev_comm']
 967
 968         # Initialize entry in our dict if not yet done
 969         if prev_comm not in exec_times:
 970             exec_times[prev_comm] = 0
 971
 972         # Compute previous command execution time
 973         diff = cur_ts - last_ts
 974
 975         # Update execution time of this command
 976         exec_times[prev_comm] += diff
 977
 978         # Update last timestamp
 979         last_ts = cur_ts
 980
 981     # Display top 5
 982     for name, ns in exec_times.most_common(5):
 983         s = ns / 1000000000
 984         print('{:20}{} s'.format(name, s))
 985
 986     return True
 987
 988
 989 if __name__ == '__main__':
 990     sys.exit(0 if top5proc() else 1)
 991 ----
 992
 993 Run this script:
 994
 995 [role="term"]
 996 ----
 997 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
 998 ----
 999
1000 Output example:
1001
1002 ----
1003 swapper/0           48.607245889 s
1004 chromium            7.192738188 s
1005 pavucontrol         0.709894415 s
1006 Compositor          0.660867933 s
1007 Xorg.bin            0.616753786 s
1008 ----
1009
1010 Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we
1011 weren't using the CPU that much when tracing, its first position in the
1012 list makes sense.
1013
1014
1015 [[core-concepts]]
1016 == [[understanding-lttng]]Core concepts
1017
1018 From a user's perspective, the LTTng system is built on a few concepts,
1019 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1020 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1021 Understanding how those objects relate to eachother is key in mastering
1022 the toolkit.
1023
1024 The core concepts are:
1025
1026 * <<tracing-session,Tracing session>>
1027 * <<domain,Tracing domain>>
1028 * <<channel,Channel and ring buffer>>
1029 * <<"event","Instrumentation point, event rule, event, and event record">>
1030
1031
1032 [[tracing-session]]
1033 === Tracing session
1034
1035 A _tracing session_ is a stateful dialogue between you and
1036 a <<lttng-sessiond,session daemon>>. You can
1037 <<creating-destroying-tracing-sessions,create a new tracing
1038 session>> with the `lttng create` command.
1039
1040 Anything that you do when you control LTTng tracers happens within a
1041 tracing session. In particular, a tracing session:
1042
1043 * Has its own name.
1044 * Has its own set of trace files.
1045 * Has its own state of activity (started or stopped).
1046 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1047   snapshot, or live).
1048 * Has its own <<channel,channels>> which have their own
1049   <<event,event rules>>.
1050
1051 [role="img-100"]
1052 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1053 image::concepts.png[]
1054
1055 Those attributes and objects are completely isolated between different
1056 tracing sessions.
1057
1058 A tracing session is analogous to a cash machine session:
1059 the operations you do on the banking system through the cash machine do
1060 not alter the data of other users of the same system. In the case of
1061 the cash machine, a session lasts as long as your bank card is inside.
1062 In the case of LTTng, a tracing session lasts from the `lttng create`
1063 command to the `lttng destroy` command.
1064
1065 [role="img-100"]
1066 .Each Unix user has its own set of tracing sessions.
1067 image::many-sessions.png[]
1068
1069
1070 [[tracing-session-mode]]
1071 ==== Tracing session mode
1072
1073 LTTng can send the generated trace data to different locations. The
1074 _tracing session mode_ dictates where to send it. The following modes
1075 are available in LTTng{nbsp}{revision}:
1076
1077 Local mode::
1078   LTTng writes the traces to the file system of the machine being traced
1079   (target system).
1080
1081 Network streaming mode::
1082   LTTng sends the traces over the network to a
1083   <<lttng-relayd,relay daemon>> running on a remote system.
1084
1085 Snapshot mode::
1086   LTTng does not write the traces by default. Instead, you can request
1087   LTTng to <<taking-a-snapshot,take a snapshot>>, that is, a copy of the
1088   current tracing buffers, and to write it to the target's file system
1089   or to send it over the network to a <<lttng-relayd,relay daemon>>
1090   running on a remote system.
1091
1092 Live mode::
1093   This mode is similar to the network streaming mode, but a live
1094   trace viewer can connect to the distant relay daemon to
1095   <<lttng-live,view event records as LTTng generates them>> by
1096   the tracers.
1097
1098
1099 [[domain]]
1100 === Tracing domain
1101
1102 A _tracing domain_ is a namespace for event sources. A tracing domain
1103 has its own properties and features.
1104
1105 There are currently five available tracing domains:
1106
1107 * Linux kernel
1108 * User space
1109 * `java.util.logging` (JUL)
1110 * log4j
1111 * Python
1112
1113 You must specify a tracing domain when using some commands to avoid
1114 ambiguity. For example, since all the domains support named tracepoints
1115 as event sources (instrumentation points that you manually insert in the
1116 source code), you need to specify a tracing domain when
1117 <<enabling-disabling-events,creating an event rule>> because all the
1118 tracing domains could have tracepoints with the same names.
1119
1120 Some features are reserved to specific tracing domains. Dynamic function
1121 entry and return instrumentation points, for example, are currently only
1122 supported in the Linux kernel tracing domain, but support for other
1123 tracing domains could be added in the future.
1124
1125 You can create <<channel,channels>> in the Linux kernel and user space
1126 tracing domains. The other tracing domains have a single default
1127 channel.
1128
1129
1130 [[channel]]
1131 === Channel and ring buffer
1132
1133 A _channel_ is an object which is responsible for a set of ring buffers.
1134 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1135 tracer emits an event, it can record it to one or more
1136 sub-buffers. The attributes of a channel determine what to do when
1137 there's no space left for a new event record because all sub-buffers
1138 are full, where to send a full sub-buffer, and other behaviours.
1139
1140 A channel is always associated to a <<domain,tracing domain>>. The
1141 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1142 a default channel which you cannot configure.
1143
1144 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1145 an event, it records it to the sub-buffers of all
1146 the enabled channels with a satisfied event rule, as long as those
1147 channels are part of active <<tracing-session,tracing sessions>>.
1148
1149
1150 [[channel-buffering-schemes]]
1151 ==== Per-user vs. per-process buffering schemes
1152
1153 A channel has at least one ring buffer _per CPU_. LTTng always
1154 records an event to the ring buffer associated to the CPU on which it
1155 occurred.
1156
1157 Two _buffering schemes_ are available when you
1158 <<enabling-disabling-channels,create a channel>> in the
1159 user space <<domain,tracing domain>>:
1160
1161 Per-user buffering::
1162   Allocate one set of ring buffers--one per CPU--shared by all the
1163   instrumented processes of each Unix user.
1164 +
1165 --
1166 [role="img-100"]
1167 .Per-user buffering scheme.
1168 image::per-user-buffering.png[]
1169 --
1170
1171 Per-process buffering::
1172   Allocate one set of ring buffers--one per CPU--for each
1173   instrumented process.
1174 +
1175 --
1176 [role="img-100"]
1177 .Per-process buffering scheme.
1178 image::per-process-buffering.png[]
1179 --
1180 +
1181 The per-process buffering scheme tends to consume more memory than the
1182 per-user option because systems generally have more instrumented
1183 processes than Unix users running instrumented processes. However, the
1184 per-process buffering scheme ensures that one process having a high
1185 event throughput won't fill all the shared sub-buffers of the same
1186 user, only its own.
1187
1188 The Linux kernel tracing domain has only one available buffering scheme
1189 which is to allocate a single set of ring buffers for the whole system.
1190 This scheme is similar to the per-user option, but with a single, global
1191 user "running" the kernel.
1192
1193
1194 [[channel-overwrite-mode-vs-discard-mode]]
1195 ==== Overwrite vs. discard event loss modes
1196
1197 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1198 arc in the following animation) of a specific channel's ring buffer.
1199 When there's no space left in a sub-buffer, the tracer marks it as
1200 consumable (red) and another, empty sub-buffer starts receiving the
1201 following event records. A <<lttng-consumerd,consumer daemon>>
1202 eventually consumes the marked sub-buffer (returns to white).
1203
1204 [NOTE]
1205 [role="docsvg-channel-subbuf-anim"]
1206 ====
1207 {note-no-anim}
1208 ====
1209
1210 In an ideal world, sub-buffers are consumed faster than they are filled,
1211 as is the case in the previous animation. In the real world,
1212 however, all sub-buffers can be full at some point, leaving no space to
1213 record the following events.
1214
1215 By design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer is
1216 available, it is acceptable to lose event records when the alternative
1217 would be to cause substantial delays in the instrumented application's
1218 execution. LTTng privileges performance over integrity; it aims at
1219 perturbing the traced system as little as possible in order to make
1220 tracing of subtle race conditions and rare interrupt cascades possible.
1221
1222 When it comes to losing event records because no empty sub-buffer is
1223 available, the channel's _event loss mode_ determines what to do. The
1224 available event loss modes are:
1225
1226 Discard mode::
1227   Drop the newest event records until a the tracer
1228   releases a sub-buffer.
1229
1230 Overwrite mode::
1231   Clear the sub-buffer containing the oldest event records and start
1232   writing the newest event records there.
1233 +
1234 This mode is sometimes called _flight recorder mode_ because it's
1235 similar to a
1236 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1237 always keep a fixed amount of the latest data.
1238
1239 Which mechanism you should choose depends on your context: prioritize
1240 the newest or the oldest event records in the ring buffer?
1241
1242 Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
1243 as soon as a there's no space left for a new event record, whereas in
1244 discard mode, the tracer only discards the event record that doesn't
1245 fit.
1246
1247 In discard mode, LTTng increments a count of lost event records when an
1248 event record is lost and saves this count to the trace. In overwrite
1249 mode, since LTTng 2.8, LTTng increments a count of lost sub-buffers when
1250 a sub-buffer is lost and saves this count to the trace. In this mode,
1251 the exact number of lost event records in those lost sub-buffers is not
1252 saved to the trace. Trace analyses can use the trace's saved discarded
1253 event record and sub-buffer counts to decide whether or not to perform
1254 the analyses even if trace data is known to be missing.
1255
1256 There are a few ways to decrease your probability of losing event
1257 records.
1258 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1259 how you can fine-une the sub-buffer count and size of a channel to
1260 virtually stop losing event records, though at the cost of greater
1261 memory usage.
1262
1263
1264 [[channel-subbuf-size-vs-subbuf-count]]
1265 ==== Sub-buffer count and size
1266
1267 When you <<enabling-disabling-channels,create a channel>>, you can
1268 set its number of sub-buffers and their size.
1269
1270 Note that there is noticeable CPU overhead introduced when
1271 switching sub-buffers (marking a full one as consumable and switching
1272 to an empty one for the following events to be recorded). Knowing this,
1273 the following list presents a few practical situations along with how
1274 to configure the sub-buffer count and size for them:
1275
1276 * **High event throughput**: In general, prefer bigger sub-buffers to
1277   lower the risk of losing event records.
1278 +
1279 Having bigger sub-buffers also ensures a lower
1280 <<channel-switch-timer,sub-buffer switching frequency>>.
1281 +
1282 The number of sub-buffers is only meaningful if you create the channel
1283 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1284 other sub-buffers are left unaltered.
1285
1286 * **Low event throughput**: In general, prefer smaller sub-buffers
1287   since the risk of losing event records is low.
1288 +
1289 Because events occur less frequently, the sub-buffer switching frequency
1290 should remain low and thus the tracer's overhead should not be a
1291 problem.
1292
1293 * **Low memory system**: If your target system has a low memory
1294   limit, prefer fewer first, then smaller sub-buffers.
1295 +
1296 Even if the system is limited in memory, you want to keep the
1297 sub-buffers as big as possible to avoid a high sub-buffer switching
1298 frequency.
1299
1300 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1301 which means event data is very compact. For example, the average
1302 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1303 sub-buffer size of 1{nbsp}MiB is considered big.
1304
1305 The previous situations highlight the major trade-off between a few big
1306 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1307 frequency vs. how much data is lost in overwrite mode. Assuming a
1308 constant event throughput and using the overwrite mode, the two
1309 following configurations have the same ring buffer total size:
1310
1311 [NOTE]
1312 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1313 ====
1314 {note-no-anim}
1315 ====
1316
1317 * **2 sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1318   switching frequency, but if a sub-buffer overwrite happens, half of
1319   the event records so far (4{nbsp}MiB) are definitely lost.
1320 * **8 sub-buffers of 1{nbsp}MiB each**: Expect 4{nbsp}times the tracer's
1321   overhead as the previous configuration, but if a sub-buffer
1322   overwrite happens, only the eighth of event records so far are
1323   definitely lost.
1324
1325 In discard mode, the sub-buffers count parameter is pointless: use two
1326 sub-buffers and set their size according to the requirements of your
1327 situation.
1328
1329
1330 [[channel-switch-timer]]
1331 ==== Switch timer period
1332
1333 The _switch timer period_ is an important configurable attribute of
1334 a channel to ensure periodic sub-buffer flushing.
1335
1336 When the _switch timer_ expires, a sub-buffer switch happens. You can
1337 set the switch timer period attribute when you
1338 <<enabling-disabling-channels,create a channel>> to ensure that event
1339 data is consumed and committed to trace files or to a distant relay
1340 daemon periodically in case of a low event throughput.
1341
1342 [NOTE]
1343 [role="docsvg-channel-switch-timer"]
1344 ====
1345 {note-no-anim}
1346 ====
1347
1348 This attribute is also convenient when you use big sub-buffers to cope
1349 with a sporadic high event throughput, even if the throughput is
1350 normally low.
1351
1352
1353 [[channel-read-timer]]
1354 ==== Read timer period
1355
1356 By default, the LTTng tracers use a notification mechanism to signal a
1357 full sub-buffer so that a consumer daemon can consume it. When such
1358 notifications must be avoided, for example in real-time applications,
1359 you can use the channel's _read timer_ instead. When the read timer
1360 fires, the <<lttng-consumerd,consumer daemon>> checks for full,
1361 consumable sub-buffers.
1362
1363
1364 [[tracefile-rotation]]
1365 ==== Trace file count and size
1366
1367 By default, trace files can grow as large as needed. You can set the
1368 maximum size of each trace file that a channel writes when you
1369 <<enabling-disabling-channels,create a channel>>. When the size of
1370 a trace file reaches the channel's fixed maximum size, LTTng creates
1371 another file to contain the next event records. LTTng appends a file
1372 count to each trace file name in this case.
1373
1374 If you set the trace file size attribute when you create a channel, the
1375 maximum number of trace files that LTTng creates is _unlimited_ by
1376 default. To limit them, you can also set a maximum number of trace
1377 files. When the number of trace files reaches the channel's fixed
1378 maximum count, the oldest trace file is overwritten. This mechanism is
1379 called _trace file rotation_.
1380
1381
1382 [[event]]
1383 === Instrumentation point, event rule, event, and event record
1384
1385 An _event rule_ is a set of conditions which must be **all** satisfied
1386 for LTTng to record an occuring event.
1387
1388 You set the conditions when you <<enabling-disabling-events,create
1389 an event rule>>.
1390
1391 You always attach an event rule to <<channel,channel>> when you create
1392 it.
1393
1394 When an event passes the conditions of an event rule, LTTng records it
1395 in one of the attached channel's sub-buffers.
1396
1397 The available conditions, as of LTTng{nbsp}{revision}, are:
1398
1399 * The event rule _is enabled_.
1400 * The instrumentation point's type _is{nbsp}T_.
1401 * The instrumentation point's name (sometimes called _event name_)
1402   _matches{nbsp}N_, but _is not{nbsp}E_.
1403 * The instrumentation point's log level _is as severe as{nbsp}L_, or
1404   _is exactly{nbsp}L_.
1405 * The fields of the event's payload _satisfy_ a filter
1406   expression{nbsp}__F__.
1407
1408 As you can see, all the conditions but the dynamic filter are related to
1409 the event rule's status or to the instrumentation point, not to the
1410 occurring events. This is why, without a filter, checking if an event
1411 passes an event rule is not a dynamic task: when you create or modify an
1412 event rule, all the tracers of its tracing domain enable or disable the
1413 instrumentation points themselves once. This is possible because the
1414 attributes of an instrumentation point (type, name, and log level) are
1415 defined statically. In other words, without a dynamic filter, the tracer
1416 _does not evaluate_ the arguments of an instrumentation point unless it
1417 matches an enabled event rule.
1418
1419 Note that, for LTTng to record an event, the <<channel,channel>> to
1420 which a matching event rule is attached must also be enabled, and the
1421 tracing session owning this channel must be active.
1422
1423 [role="img-100"]
1424 .Logical path from an instrumentation point to an event record.
1425 image::event-rule.png[]
1426
1427 .Event, event record, or event rule?
1428 ****
1429 With so many similar terms, it's easy to get confused.
1430
1431 An **event** is the consequence of the execution of an _instrumentation
1432 point_, like a tracepoint that you manually place in some source code,
1433 or a Linux kernel KProbe. An event is said to _occur_ at a specific
1434 time. Different actions can be taken upon the occurrence of an event,
1435 like record the event's payload to a buffer.
1436
1437 An **event record** is the representation of an event in a sub-buffer. A
1438 tracer is responsible for capturing the payload of an event, current
1439 context variables, the event's ID, and the event's timestamp. LTTng
1440 can append this sub-buffer to a trace file.
1441
1442 An **event rule** is a set of conditions which must all be satisfied for
1443 LTTng to record an occuring event. Events still occur without
1444 satisfying event rules, but LTTng does not record them.
1445 ****
1446
1447
1448 [[plumbing]]
1449 == Components of noch:{LTTng}
1450
1451 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1452 to call LTTng a simple _tool_ since it is composed of multiple
1453 interacting components. This section describes those components,
1454 explains their respective roles, and shows how they connect together to
1455 form the LTTng ecosystem.
1456
1457 The following diagram shows how the most important components of LTTng
1458 interact with user applications, the Linux kernel, and you:
1459
1460 [role="img-100"]
1461 .Control and trace data paths between LTTng components.
1462 image::plumbing.png[]
1463
1464 The LTTng project incorporates:
1465
1466 * **LTTng-tools**: Libraries and command-line interface to
1467   control tracing sessions.
1468 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1469 ** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1470 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1471 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1472 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1473 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1474   applications.
1475 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1476    headers to instrument and trace any native user application.
1477 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1478 *** `liblttng-ust-libc-wrapper`
1479 *** `liblttng-ust-pthread-wrapper`
1480 *** `liblttng-ust-cyg-profile`
1481 *** `liblttng-ust-cyg-profile-fast`
1482 *** `liblttng-ust-dl`
1483 ** User space tracepoint provider source files generator command-line
1484    tool (man:lttng-gen-tp(1)).
1485 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1486    Java applications using `java.util.logging` or
1487    Apache log4j 1.2 logging.
1488 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1489    Python applications using the standard `logging` package.
1490 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1491   the kernel.
1492 ** LTTng kernel tracer module.
1493 ** Tracing ring buffer kernel modules.
1494 ** Probe kernel modules.
1495 ** LTTng logger kernel module.
1496
1497
1498 [[lttng-cli]]
1499 === Tracing control command-line interface
1500
1501 [role="img-100"]
1502 .The tracing control command-line interface.
1503 image::plumbing-lttng-cli.png[]
1504
1505 The _man:lttng(1) command-line tool_ is the standard user interface to
1506 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1507 is part of LTTng-tools.
1508
1509 The cmd:lttng tool is linked with
1510 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1511 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1512
1513 The cmd:lttng tool has a Git-like interface:
1514
1515 [role="term"]
1516 ----
1517 $ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1518 ----
1519
1520 The <<controlling-tracing,Tracing control>> section explores the
1521 available features of LTTng using the cmd:lttng tool.
1522
1523
1524 [[liblttng-ctl-lttng]]
1525 === Tracing control library
1526
1527 [role="img-100"]
1528 .The tracing control library.
1529 image::plumbing-liblttng-ctl.png[]
1530
1531 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1532 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1533 underlying protocol's details. `liblttng-ctl` is part of LTTng-tools.
1534
1535 The <<lttng-cli,cmd:lttng command-line tool>>
1536 is linked with `liblttng-ctl`.
1537
1538 You can use `liblttng-ctl` in C or $$C++$$ source code by including its
1539 "master" header:
1540
1541 [source,c]
1542 ----
1543 #include <lttng/lttng.h>
1544 ----
1545
1546 Some objects are referenced by name (C string), such as tracing
1547 sessions, but most of them require to create a handle first using
1548 `lttng_create_handle()`.
1549
1550 The best available developer documentation for `liblttng-ctl` is, as of
1551 LTTng{nbsp}{revision}, its installed header files. Every function and
1552 structure is thoroughly documented.
1553
1554
1555 [[lttng-ust]]
1556 === User space tracing library
1557
1558 [role="img-100"]
1559 .The user space tracing library.
1560 image::plumbing-liblttng-ust.png[]
1561
1562 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1563 is the LTTng user space tracer. It receives commands from a
1564 <<lttng-sessiond,session daemon>>, for example to
1565 enable and disable specific instrumentation points, and writes event
1566 records to ring buffers shared with a
1567 <<lttng-consumerd,consumer daemon>>.
1568 `liblttng-ust` is part of LTTng-UST.
1569
1570 Public C header files are installed beside `liblttng-ust` to
1571 instrument any <<c-application,C or $$C++$$ application>>.
1572
1573 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1574 packages, use their own library providing tracepoints which is
1575 linked with `liblttng-ust`.
1576
1577 An application or library does not have to initialize `liblttng-ust`
1578 manually: its constructor does the necessary tasks to properly register
1579 to a session daemon. The initialization phase also enables the
1580 instrumentation points matching the <<event,event rules>> that you
1581 already created.
1582
1583
1584 [[lttng-ust-agents]]
1585 === User space tracing agents
1586
1587 [role="img-100"]
1588 .The user space tracing agents.
1589 image::plumbing-lttng-ust-agents.png[]
1590
1591 The _LTTng-UST Java and Python agents_ are regular Java and Python
1592 packages which add LTTng tracing capabilities to the
1593 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1594
1595 In the case of Java, the
1596 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1597 core logging facilities] and
1598 https://logging.apache.org/log4j/1.2/[Apache log4j 1.2] are supported.
1599 Note that Apache Log4{nbsp}2 is not supported.
1600
1601 In the case of Python, the standard
1602 https://docs.python.org/3/library/logging.html[`logging`] package
1603 is supported. Both Python 2 and Python 3 modules can import the
1604 LTTng-UST Python agent package.
1605
1606 The applications using the LTTng-UST agents are in the
1607 `java.util.logging` (JUL),
1608 log4j, and Python <<domain,tracing domains>>.
1609
1610 Both agents use the same mechanism to trace the log statements. When an
1611 agent is initialized, it creates a log handler that attaches to the root
1612 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1613 When the application executes a log statement, it is passed to the
1614 agent's log handler by the root logger. The agent's log handler calls a
1615 native function in a tracepoint provider package shared library linked
1616 with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
1617 other fields, like its logger name and its log level. This native
1618 function contains a user space instrumentation point, hence tracing the
1619 log statement.
1620
1621 The log level condition of an
1622 <<event,event rule>> is considered when tracing
1623 a Java or a Python application, and it's compatible with the standard
1624 JUL, log4j, and Python log levels.
1625
1626
1627 [[lttng-modules]]
1628 === LTTng kernel modules
1629
1630 [role="img-100"]
1631 .The LTTng kernel modules.
1632 image::plumbing-lttng-modules.png[]
1633
1634 The _LTTng kernel modules_ are a set of Linux kernel modules
1635 which implement the kernel tracer of the LTTng project. The LTTng
1636 kernel modules are part of LTTng-modules.
1637
1638 The LTTng kernel modules include:
1639
1640 * A set of _probe_ modules.
1641 +
1642 Each module attaches to a specific subsystem
1643 of the Linux kernel using its tracepoint instrument points. There are
1644 also modules to attach to the entry and return points of the Linux
1645 system call functions.
1646
1647 * _Ring buffer_ modules.
1648 +
1649 A ring buffer implementation is provided as kernel modules. The LTTng
1650 kernel tracer writes to the ring buffer; a
1651 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1652
1653 * The _LTTng kernel tracer_ module.
1654 * The _LTTng logger_ module.
1655 +
1656 The LTTng logger module implements the special path:{/proc/lttng-logger}
1657 file so that any executable can generate LTTng events by opening and
1658 writing to this file.
1659 +
1660 See <<proc-lttng-logger-abi,LTTng logger>>.
1661
1662 Generally, you do not have to load the LTTng kernel modules manually
1663 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
1664 daemon>> loads the necessary modules when starting. If you have extra
1665 probe modules, you can specify to load them to the session daemon on
1666 the command line.
1667
1668 The LTTng kernel modules are installed in
1669 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
1670 the kernel release (see `uname --kernel-release`).
1671
1672
1673 [[lttng-sessiond]]
1674 === Session daemon
1675
1676 [role="img-100"]
1677 .The session daemon.
1678 image::plumbing-sessiond.png[]
1679
1680 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
1681 managing tracing sessions and for controlling the various components of
1682 LTTng. The session daemon is part of LTTng-tools.
1683
1684 The session daemon sends control requests to and receives control
1685 responses from:
1686
1687 * The <<lttng-ust,user space tracing library>>.
1688 +
1689 Any instance of the user space tracing library first registers to
1690 a session daemon. Then, the session daemon can send requests to
1691 this instance, such as:
1692 +
1693 --
1694 ** Get the list of tracepoints.
1695 ** Share an <<event,event rule>> so that the user space tracing library
1696    can enable or disable tracepoints. Amongst the possible conditions
1697    of an event rule is a filter expression which `liblttng-ust` evalutes
1698    when an event occurs.
1699 ** Share <<channel,channel>> attributes and ring buffer locations.
1700 --
1701 +
1702 The session daemon and the user space tracing library use a Unix
1703 domain socket for their communication.
1704
1705 * The <<lttng-ust-agents,user space tracing agents>>.
1706 +
1707 Any instance of a user space tracing agent first registers to
1708 a session daemon. Then, the session daemon can send requests to
1709 this instance, such as:
1710 +
1711 --
1712 ** Get the list of loggers.
1713 ** Enable or disable a specific logger.
1714 --
1715 +
1716 The session daemon and the user space tracing agent use a TCP connection
1717 for their communication.
1718
1719 * The <<lttng-modules,LTTng kernel tracer>>.
1720 * The <<lttng-consumerd,consumer daemon>>.
1721 +
1722 The session daemon sends requests to the consumer daemon to instruct
1723 it where to send the trace data streams, amongst other information.
1724
1725 * The <<lttng-relayd,relay daemon>>.
1726
1727 The session daemon receives commands from the
1728 <<liblttng-ctl-lttng,tracing control library>>.
1729
1730 The root session daemon loads the appropriate
1731 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
1732 a <<lttng-consumerd,consumer daemon>> as soon as you create
1733 an <<event,event rule>>.
1734
1735 The session daemon does not send and receive trace data: this is the
1736 role of the <<lttng-consumerd,consumer daemon>> and
1737 <<lttng-relayd,relay daemon>>. It does, however, generate the
1738 http://diamon.org/ctf/[CTF] metadata stream.
1739
1740 Each Unix user can have its own session daemon instance. The
1741 tracing sessions managed by different session daemons are completely
1742 independent.
1743
1744 The root user's session daemon is the only one which is
1745 allowed to control the LTTng kernel tracer, and its spawned consumer
1746 daemon is the only one which is allowed to consume trace data from the
1747 LTTng kernel tracer. Note, however, that any Unix user which is a member
1748 of the <<tracing-group,tracing group>> is allowed
1749 to create <<channel,channels>> in the
1750 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
1751 kernel.
1752
1753 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
1754 session daemon when using its `create` command if none is currently
1755 running. You can also start the session daemon manually.
1756
1757
1758 [[lttng-consumerd]]
1759 === Consumer daemon
1760
1761 [role="img-100"]
1762 .The consumer daemon.
1763 image::plumbing-consumerd.png[]
1764
1765 The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
1766 ring buffers with user applications or with the LTTng kernel modules to
1767 collect trace data and send it to some location (on disk or to a
1768 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
1769 is part of LTTng-tools.
1770
1771 You do not start a consumer daemon manually: a consumer daemon is always
1772 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
1773 <<event,event rule>>, that is, before you start tracing. When you kill
1774 its owner session daemon, the consumer daemon also exits because it is
1775 the session daemon's child process. Command-line options of
1776 man:lttng-sessiond(8) target the consumer daemon process.
1777
1778 There are up to two running consumer daemons per Unix user, whereas only
1779 one session daemon can run per user. This is because each process can be
1780 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
1781 and 64-bit processes, it is more efficient to have separate
1782 corresponding 32-bit and 64-bit consumer daemons. The root user is an
1783 exception: it can have up to _three_ running consumer daemons: 32-bit
1784 and 64-bit instances for its user applications, and one more
1785 reserved for collecting kernel trace data.
1786
1787
1788 [[lttng-relayd]]
1789 === Relay daemon
1790
1791 [role="img-100"]
1792 .The relay daemon.
1793 image::plumbing-relayd.png[]
1794
1795 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
1796 between remote session and consumer daemons, local trace files, and a
1797 remote live trace viewer. The relay daemon is part of LTTng-tools.
1798
1799 The main purpose of the relay daemon is to implement a receiver of
1800 <<sending-trace-data-over-the-network,trace data over the network>>.
1801 This is useful when the target system does not have much file system
1802 space to record trace files locally.
1803
1804 The relay daemon is also a server to which a
1805 <<lttng-live,live trace viewer>> can
1806 connect. The live trace viewer sends requests to the relay daemon to
1807 receive trace data as the target system emits events. The
1808 communication protocol is named _LTTng live_; it is used over TCP
1809 connections.
1810
1811 Note that you can start the relay daemon on the target system directly.
1812 This is the setup of choice when the use case is to view events as
1813 the target system emits them without the need of a remote system.
1814
1815
1816 [[instrumenting]]
1817 == [[using-lttng]]Instrumentation
1818
1819 There are many examples of tracing and monitoring in our everyday life:
1820
1821 * You have access to real-time and historical weather reports and
1822   forecasts thanks to weather stations installed around the country.
1823 * You know your heart is safe thanks to an electrocardiogram.
1824 * You make sure not to drive your car too fast and to have enough fuel
1825   to reach your destination thanks to gauges visible on your dashboard.
1826
1827 All the previous examples have something in common: they rely on
1828 **instruments**. Without the electrodes attached to the surface of your
1829 body's skin, cardiac monitoring is futile.
1830
1831 LTTng, as a tracer, is no different from those real life examples. If
1832 you're about to trace a software system or, in other words, record its
1833 history of execution, you better have **instrumentation points** in the
1834 subject you're tracing, that is, the actual software.
1835
1836 Various ways were developed to instrument a piece of software for LTTng
1837 tracing. The most straightforward one is to manually place
1838 instrumentation points, called _tracepoints_, in the software's source
1839 code. It is also possible to add instrumentation points dynamically in
1840 the Linux kernel <<domain,tracing domain>>.
1841
1842 If you're only interested in tracing the Linux kernel, your
1843 instrumentation needs are probably already covered by LTTng's built-in
1844 <<lttng-modules,Linux kernel tracepoints>>. You may also wish to trace a
1845 user application which is already instrumented for LTTng tracing.
1846 In such cases, you can skip this whole section and read the topics of
1847 the <<controlling-tracing,Tracing control>> section.
1848
1849 Many methods are available to instrument a piece of software for LTTng
1850 tracing. They are:
1851
1852 * <<c-application,User space instrumentation for C and $$C++$$
1853   applications>>.
1854 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
1855 * <<java-application,User space Java agent>>.
1856 * <<python-application,User space Python agent>>.
1857 * <<proc-lttng-logger-abi,LTTng logger>>.
1858 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
1859
1860
1861 [[c-application]]
1862 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
1863
1864 The procedure to instrument a C or $$C++$$ user application with
1865 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
1866
1867 . <<tracepoint-provider,Create the source files of a tracepoint provider
1868   package>>.
1869 . <<probing-the-application-source-code,Add tracepoints to
1870   the application's source code>>.
1871 . <<building-tracepoint-providers-and-user-application,Build and link
1872   a tracepoint provider package and the user application>>.
1873
1874 If you need quick, man:printf(3)-like instrumentation, you can skip
1875 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
1876 instead.
1877
1878 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
1879 instrument a user application with `liblttng-ust`.
1880
1881
1882 [[tracepoint-provider]]
1883 ==== Create the source files of a tracepoint provider package
1884
1885 A _tracepoint provider_ is a set of compiled functions which provide
1886 **tracepoints** to an application, the type of instrumentation point
1887 supported by LTTng-UST. Those functions can emit events with
1888 user-defined fields and serialize those events as event records to one
1889 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
1890 macro, which you <<probing-the-application-source-code,insert in a user
1891 application's source code>>, calls those functions.
1892
1893 A _tracepoint provider package_ is an object file (`.o`) or a shared
1894 library (`.so`) which contains one or more tracepoint providers.
1895 Its source files are:
1896
1897 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
1898 * A <<tpp-source,tracepoint provider package source>> (`.c`).
1899
1900 A tracepoint provider package is dynamically linked with `liblttng-ust`,
1901 the LTTng user space tracer, at run time.
1902
1903 [role="img-100"]
1904 .User application linked with `liblttng-ust` and containing a tracepoint provider.
1905 image::ust-app.png[]
1906
1907 NOTE: If you need quick, man:printf(3)-like instrumentation, you can
1908 skip creating and using a tracepoint provider and use
1909 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
1910
1911
1912 [[tpp-header]]
1913 ===== Create a tracepoint provider header file template
1914
1915 A _tracepoint provider header file_ contains the tracepoint
1916 definitions of a tracepoint provider.
1917
1918 To create a tracepoint provider header file:
1919
1920 . Start from this template:
1921 +
1922 --
1923 [source,c]
1924 .Tracepoint provider header file template (`.h` file extension).
1925 ----
1926 #undef TRACEPOINT_PROVIDER
1927 #define TRACEPOINT_PROVIDER provider_name
1928
1929 #undef TRACEPOINT_INCLUDE
1930 #define TRACEPOINT_INCLUDE "./tp.h"
1931
1932 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1933 #define _TP_H
1934
1935 #include <lttng/tracepoint.h>
1936
1937 /*
1938  * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
1939  * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
1940  */
1941
1942 #endif /* _TP_H */
1943
1944 #include <lttng/tracepoint-event.h>
1945 ----
1946 --
1947
1948 .  Replace:
1949 +
1950 * `provider_name` with the name of your tracepoint provider.
1951 * `"tp.h"` with the name of your tracepoint provider header file.
1952
1953 . Below the `#include <lttng/tracepoint.h>` line, put your
1954   <<defining-tracepoints,tracepoint definitions>>.
1955
1956 Your tracepoint provider name must be unique amongst all the possible
1957 tracepoint provider names used on the same target system. We
1958 suggest to include the name of your project or company in the name,
1959 for example, `org_lttng_my_project_tpp`.
1960
1961 TIP: [[lttng-gen-tp]]You can use the man:lttng-gen-tp(1) tool to create
1962 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
1963 write are the <<defining-tracepoints,tracepoint definitions>>.
1964
1965
1966 [[defining-tracepoints]]
1967 ===== Create a tracepoint definition
1968
1969 A _tracepoint definition_ defines, for a given tracepoint:
1970
1971 * Its **input arguments**. They are the macro parameters that the
1972   `tracepoint()` macro accepts for this particular tracepoint
1973   in the user application's source code.
1974 * Its **output event fields**. They are the sources of event fields
1975   that form the payload of any event that the execution of the
1976   `tracepoint()` macro emits for this particular tracepoint.
1977
1978 You can create a tracepoint definition by using the
1979 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
1980 line in the
1981 <<tpp-header,tracepoint provider header file template>>.
1982
1983 The syntax of the `TRACEPOINT_EVENT()` macro is:
1984
1985 [source,c]
1986 .`TRACEPOINT_EVENT()` macro syntax.
1987 ----
1988 TRACEPOINT_EVENT(
1989     /* Tracepoint provider name */
1990     provider_name,
1991
1992     /* Tracepoint name */
1993     tracepoint_name,
1994
1995     /* Input arguments */
1996     TP_ARGS(
1997         arguments
1998     ),
1999
2000     /* Output event fields */
2001     TP_FIELDS(
2002         fields
2003     )
2004 )
2005 ----
2006
2007 Replace:
2008
2009 * `provider_name` with your tracepoint provider name.
2010 * `tracepoint_name` with your tracepoint name.
2011 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2012 * `fields` with the <<tpp-def-output-fields,output event field>>
2013   definitions.
2014
2015 This tracepoint emits events named `provider_name:tracepoint_name`.
2016
2017 [IMPORTANT]
2018 .Event name's length limitation
2019 ====
2020 The concatenation of the tracepoint provider name and the
2021 tracepoint name must not exceed **254 characters**. If it does, the
2022 instrumented application compiles and runs, but LTTng throws multiple
2023 warnings and you could experience serious issues.
2024 ====
2025
2026 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2027
2028 [source,c]
2029 .`TP_ARGS()` macro syntax.
2030 ----
2031 TP_ARGS(
2032     type, arg_name
2033 )
2034 ----
2035
2036 Replace:
2037
2038 * `type` with the C type of the argument.
2039 * `arg_name` with the argument name.
2040
2041 You can repeat `type` and `arg_name` up to 10 times to have
2042 more than one argument.
2043
2044 .`TP_ARGS()` usage with three arguments.
2045 ====
2046 [source,c]
2047 ----
2048 TP_ARGS(
2049     int, count,
2050     float, ratio,
2051     const char*, query
2052 )
2053 ----
2054 ====
2055
2056 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2057 tracepoint definition with no input arguments.
2058
2059 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2060 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2061 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2062 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2063 one event field.
2064
2065 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2066 C expression that the tracer evalutes at the `tracepoint()` macro site
2067 in the application's source code. This expression provides a field's
2068 source of data. The argument expression can include input argument names
2069 listed in the `TP_ARGS()` macro.
2070
2071 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2072 must be unique within a given tracepoint definition.
2073
2074 Here's a complete tracepoint definition example:
2075
2076 .Tracepoint definition.
2077 ====
2078 The following tracepoint definition defines a tracepoint which takes
2079 three input arguments and has four output event fields.
2080
2081 [source,c]
2082 ----
2083 #include "my-custom-structure.h"
2084
2085 TRACEPOINT_EVENT(
2086     my_provider,
2087     my_tracepoint,
2088     TP_ARGS(
2089         const struct my_custom_structure*, my_custom_structure,
2090         float, ratio,
2091         const char*, query
2092     ),
2093     TP_FIELDS(
2094         ctf_string(query_field, query)
2095         ctf_float(double, ratio_field, ratio)
2096         ctf_integer(int, recv_size, my_custom_structure->recv_size)
2097         ctf_integer(int, send_size, my_custom_structure->send_size)
2098     )
2099 )
2100 ----
2101
2102 You can refer to this tracepoint definition with the `tracepoint()`
2103 macro in your application's source code like this:
2104
2105 [source,c]
2106 ----
2107 tracepoint(my_provider, my_tracepoint,
2108            my_structure, some_ratio, the_query);
2109 ----
2110 ====
2111
2112 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2113 if they satisfy an enabled <<event,event rule>>.
2114
2115
2116 [[using-tracepoint-classes]]
2117 ===== Use a tracepoint class
2118
2119 A _tracepoint class_ is a class of tracepoints which share the same
2120 output event field definitions. A _tracepoint instance_ is one
2121 instance of such a defined tracepoint class, with its own tracepoint
2122 name.
2123
2124 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2125 shorthand which defines both a tracepoint class and a tracepoint
2126 instance at the same time.
2127
2128 When you build a tracepoint provider package, the C or $$C++$$ compiler
2129 creates one serialization function for each **tracepoint class**. A
2130 serialization function is responsible for serializing the event fields
2131 of a tracepoint to a sub-buffer when tracing.
2132
2133 For various performance reasons, when your situation requires multiple
2134 tracepoint definitions with different names, but with the same event
2135 fields, we recommend that you manually create a tracepoint class
2136 and instantiate as many tracepoint instances as needed. One positive
2137 effect of such a design, amongst other advantages, is that all
2138 tracepoint instances of the same tracepoint class reuse the same
2139 serialization function, thus reducing
2140 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2141
2142 .Use a tracepoint class and tracepoint instances.
2143 ====
2144 Consider the following three tracepoint definitions:
2145
2146 [source,c]
2147 ----
2148 TRACEPOINT_EVENT(
2149     my_app,
2150     get_account,
2151     TP_ARGS(
2152         int, userid,
2153         size_t, len
2154     ),
2155     TP_FIELDS(
2156         ctf_integer(int, userid, userid)
2157         ctf_integer(size_t, len, len)
2158     )
2159 )
2160
2161 TRACEPOINT_EVENT(
2162     my_app,
2163     get_settings,
2164     TP_ARGS(
2165         int, userid,
2166         size_t, len
2167     ),
2168     TP_FIELDS(
2169         ctf_integer(int, userid, userid)
2170         ctf_integer(size_t, len, len)
2171     )
2172 )
2173
2174 TRACEPOINT_EVENT(
2175     my_app,
2176     get_transaction,
2177     TP_ARGS(
2178         int, userid,
2179         size_t, len
2180     ),
2181     TP_FIELDS(
2182         ctf_integer(int, userid, userid)
2183         ctf_integer(size_t, len, len)
2184     )
2185 )
2186 ----
2187
2188 In this case, we create three tracepoint classes, with one implicit
2189 tracepoint instance for each of them: `get_account`, `get_settings`, and
2190 `get_transaction`. However, they all share the same event field names
2191 and types. Hence three identical, yet independent serialization
2192 functions are created when you build the tracepoint provider package.
2193
2194 A better design choice is to define a single tracepoint class and three
2195 tracepoint instances:
2196
2197 [source,c]
2198 ----
2199 /* The tracepoint class */
2200 TRACEPOINT_EVENT_CLASS(
2201     /* Tracepoint provider name */
2202     my_app,
2203
2204     /* Tracepoint class name */
2205     my_class,
2206
2207     /* Input arguments */
2208     TP_ARGS(
2209         int, userid,
2210         size_t, len
2211     ),
2212
2213     /* Output event fields */
2214     TP_FIELDS(
2215         ctf_integer(int, userid, userid)
2216         ctf_integer(size_t, len, len)
2217     )
2218 )
2219
2220 /* The tracepoint instances */
2221 TRACEPOINT_EVENT_INSTANCE(
2222     /* Tracepoint provider name */
2223     my_app,
2224
2225     /* Tracepoint class name */
2226     my_class,
2227
2228     /* Tracepoint name */
2229     get_account,
2230
2231     /* Input arguments */
2232     TP_ARGS(
2233         int, userid,
2234         size_t, len
2235     )
2236 )
2237 TRACEPOINT_EVENT_INSTANCE(
2238     my_app,
2239     my_class,
2240     get_settings,
2241     TP_ARGS(
2242         int, userid,
2243         size_t, len
2244     )
2245 )
2246 TRACEPOINT_EVENT_INSTANCE(
2247     my_app,
2248     my_class,
2249     get_transaction,
2250     TP_ARGS(
2251         int, userid,
2252         size_t, len
2253     )
2254 )
2255 ----
2256 ====
2257
2258
2259 [[assigning-log-levels]]
2260 ===== Assign a log level to a tracepoint definition
2261
2262 You can assign an optional _log level_ to a
2263 <<defining-tracepoints,tracepoint definition>>.
2264
2265 Assigning different levels of severity to tracepoint definitions can
2266 be useful: when you <<enabling-disabling-events,create an event rule>>,
2267 you can target tracepoints having a log level as severe as a specific
2268 value.
2269
2270 The concept of LTTng-UST log levels is similar to the levels found
2271 in typical logging frameworks:
2272
2273 * In a logging framework, the log level is given by the function
2274   or method name you use at the log statement site: `debug()`,
2275   `info()`, `warn()`, `error()`, and so on.
2276 * In LTTng-UST, you statically assign the log level to a tracepoint
2277   definition; any `tracepoint()` macro invocation which refers to
2278   this definition has this log level.
2279
2280 You can assign a log level to a tracepoint definition with the
2281 `TRACEPOINT_LOGLEVEL()` macro. You must use this macro _after_ the
2282 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2283 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2284 tracepoint.
2285
2286 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2287
2288 [source,c]
2289 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2290 ----
2291 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2292 ----
2293
2294 Replace:
2295
2296 * `provider_name` with the tracepoint provider name.
2297 * `tracepoint_name` with the tracepoint name.
2298 * `log_level` with the log level to assign to the tracepoint
2299   definition named `tracepoint_name` in the `provider_name`
2300   tracepoint provider.
2301 +
2302 See man:lttng-ust(3) for a list of available log level names.
2303
2304 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2305 ====
2306 [source,c]
2307 ----
2308 /* Tracepoint definition */
2309 TRACEPOINT_EVENT(
2310     my_app,
2311     get_transaction,
2312     TP_ARGS(
2313         int, userid,
2314         size_t, len
2315     ),
2316     TP_FIELDS(
2317         ctf_integer(int, userid, userid)
2318         ctf_integer(size_t, len, len)
2319     )
2320 )
2321
2322 /* Log level assignment */
2323 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2324 ----
2325 ====
2326
2327
2328 [[tpp-source]]
2329 ===== Create a tracepoint provider package source file
2330
2331 A _tracepoint provider package source file_ is a C source file which
2332 includes a <<tpp-header,tracepoint provider header file>> to expand its
2333 macros into event serialization and other functions.
2334
2335 You can always use the following tracepoint provider package source
2336 file template:
2337
2338 [source,c]
2339 .Tracepoint provider package source file template.
2340 ----
2341 #define TRACEPOINT_CREATE_PROBES
2342
2343 #include "tp.h"
2344 ----
2345
2346 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2347 header file>> name. You may also include more than one tracepoint
2348 provider header file here to create a tracepoint provider package
2349 holding more than one tracepoint providers.
2350
2351
2352 [[probing-the-application-source-code]]
2353 ==== Add tracepoints to an application's source code
2354
2355 Once you <<tpp-header,create a tracepoint provider header file>>, you
2356 can use the `tracepoint()` macro in your application's
2357 source code to insert the tracepoints that this header
2358 <<defining-tracepoints,defines>>.
2359
2360 The `tracepoint()` macro takes at least two parameters: the tracepoint
2361 provider name and the tracepoint name. The corresponding tracepoint
2362 definition defines the other parameters.
2363
2364 .`tracepoint()` usage.
2365 ====
2366 The following <<defining-tracepoints,tracepoint definition>> defines a
2367 tracepoint which takes two input arguments and has two output event
2368 fields.
2369
2370 [source,c]
2371 .Tracepoint provider header file.
2372 ----
2373 #include "my-custom-structure.h"
2374
2375 TRACEPOINT_EVENT(
2376     my_provider,
2377     my_tracepoint,
2378     TP_ARGS(
2379         int, argc,
2380         const char*, cmd_name
2381     ),
2382     TP_FIELDS(
2383         ctf_string(cmd_name, cmd_name)
2384         ctf_integer(int, number_of_args, argc)
2385     )
2386 )
2387 ----
2388
2389 You can refer to this tracepoint definition with the `tracepoint()`
2390 macro in your application's source code like this:
2391
2392 [source,c]
2393 .Application's source file.
2394 ----
2395 #include "tp.h"
2396
2397 int main(int argc, char* argv[])
2398 {
2399     tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2400
2401     return 0;
2402 }
2403 ----
2404
2405 Note how the application's source code includes
2406 the tracepoint provider header file containing the tracepoint
2407 definitions to use, path:{tp.h}.
2408 ====
2409
2410 .`tracepoint()` usage with a complex tracepoint definition.
2411 ====
2412 Consider this complex tracepoint definition, where multiple event
2413 fields refer to the same input arguments in their argument expression
2414 parameter:
2415
2416 [source,c]
2417 .Tracepoint provider header file.
2418 ----
2419 /* For `struct stat` */
2420 #include <sys/types.h>
2421 #include <sys/stat.h>
2422 #include <unistd.h>
2423
2424 TRACEPOINT_EVENT(
2425     my_provider,
2426     my_tracepoint,
2427     TP_ARGS(
2428         int, my_int_arg,
2429         char*, my_str_arg,
2430         struct stat*, st
2431     ),
2432     TP_FIELDS(
2433         ctf_integer(int, my_constant_field, 23 + 17)
2434         ctf_integer(int, my_int_arg_field, my_int_arg)
2435         ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2436         ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2437                                      my_str_arg[2] + my_str_arg[3])
2438         ctf_string(my_str_arg_field, my_str_arg)
2439         ctf_integer_hex(off_t, size_field, st->st_size)
2440         ctf_float(double, size_dbl_field, (double) st->st_size)
2441         ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2442                           size_t, strlen(my_str_arg) / 2)
2443     )
2444 )
2445 ----
2446
2447 You can refer to this tracepoint definition with the `tracepoint()`
2448 macro in your application's source code like this:
2449
2450 [source,c]
2451 .Application's source file.
2452 ----
2453 #define TRACEPOINT_DEFINE
2454 #include "tp.h"
2455
2456 int main(void)
2457 {
2458     struct stat s;
2459
2460     stat("/etc/fstab", &s);
2461     tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2462
2463     return 0;
2464 }
2465 ----
2466
2467 If you look at the event record that LTTng writes when tracing this
2468 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2469 it should look like this:
2470
2471 .Event record fields
2472 |====
2473 |Field's name            |Field's value
2474 |`my_constant_field`     |40
2475 |`my_int_arg_field`      |23
2476 |`my_int_arg_field2`     |529
2477 |`sum4_field`            |389
2478 |`my_str_arg_field`      |`Hello, World!`
2479 |`size_field`            |0x12d
2480 |`size_dbl_field`        |301.0
2481 |`half_my_str_arg_field` |`Hello,`
2482 |====
2483 ====
2484
2485 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2486 compute--they use the call stack, for example. To avoid this
2487 computation when the tracepoint is disabled, you can use the
2488 `tracepoint_enabled()` and `do_tracepoint()` macros.
2489
2490 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2491 is:
2492
2493 [source,c]
2494 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2495 ----
2496 tracepoint_enabled(provider_name, tracepoint_name)
2497 do_tracepoint(provider_name, tracepoint_name, ...)
2498 ----
2499
2500 Replace:
2501
2502 * `provider_name` with the tracepoint provider name.
2503 * `tracepoint_name` with the tracepoint name.
2504
2505 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2506 `tracepoint_name` from the provider named `provider_name` is enabled
2507 **at run time**.
2508
2509 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2510 if the tracepoint is enabled. Using `tracepoint()` with
2511 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2512 the `tracepoint_enabled()` check, thus a race condition is
2513 possible in this situation:
2514
2515 [source,c]
2516 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2517 ----
2518 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2519     stuff = prepare_stuff();
2520 }
2521
2522 tracepoint(my_provider, my_tracepoint, stuff);
2523 ----
2524
2525 If the tracepoint is enabled after the condition, then `stuff` is not
2526 prepared: the emitted event will either contain wrong data, or the whole
2527 application could crash (segmentation fault, for example).
2528
2529 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2530 `STAP_PROBEV()` call. If you need it, you must emit
2531 this call yourself.
2532
2533
2534 [[building-tracepoint-providers-and-user-application]]
2535 ==== Build and link a tracepoint provider package and an application
2536
2537 Once you have one or more <<tpp-header,tracepoint provider header
2538 files>> and a <<tpp-source,tracepoint provider package source file>>,
2539 you can create the tracepoint provider package by compiling its source
2540 file. From here, multiple build and run scenarios are possible. The
2541 following table shows common application and library configurations
2542 along with the required command lines to achieve them.
2543
2544 In the following diagrams, we use the following file names:
2545
2546 `app`::
2547   Executable application.
2548
2549 `app.o`::
2550   Application's object file.
2551
2552 `tpp.o`::
2553   Tracepoint provider package object file.
2554
2555 `tpp.a`::
2556   Tracepoint provider package archive file.
2557
2558 `libtpp.so`::
2559   Tracepoint provider package shared object file.
2560
2561 `emon.o`::
2562   User library object file.
2563
2564 `libemon.so`::
2565   User library shared object file.
2566
2567 We use the following symbols in the diagrams of table below:
2568
2569 [role="img-100"]
2570 .Symbols used in the build scenario diagrams.
2571 image::ust-sit-symbols.png[]
2572
2573 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2574 variable in the following instructions.
2575
2576 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2577 .Common tracepoint provider package scenarios.
2578 |====
2579 |Scenario |Instructions
2580
2581 |
2582 The instrumented application is statically linked with
2583 the tracepoint provider package object.
2584
2585 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2586
2587 |
2588 include::../common/ust-sit-step-tp-o.txt[]
2589
2590 To build the instrumented application:
2591
2592 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2593 +
2594 --
2595 [source,c]
2596 ----
2597 #define TRACEPOINT_DEFINE
2598 ----
2599 --
2600
2601 . Compile the application source file:
2602 +
2603 --
2604 [role="term"]
2605 ----
2606 $ gcc -c app.c
2607 ----
2608 --
2609
2610 . Build the application:
2611 +
2612 --
2613 [role="term"]
2614 ----
2615 $ gcc -o app app.o tpp.o -llttng-ust -ldl
2616 ----
2617 --
2618
2619 To run the instrumented application:
2620
2621 * Start the application:
2622 +
2623 --
2624 [role="term"]
2625 ----
2626 $ ./app
2627 ----
2628 --
2629
2630 |
2631 The instrumented application is statically linked with the
2632 tracepoint provider package archive file.
2633
2634 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2635
2636 |
2637 To create the tracepoint provider package archive file:
2638
2639 . Compile the <<tpp-source,tracepoint provider package source file>>:
2640 +
2641 --
2642 [role="term"]
2643 ----
2644 $ gcc -I. -c tpp.c
2645 ----
2646 --
2647
2648 . Create the tracepoint provider package archive file:
2649 +
2650 --
2651 [role="term"]
2652 ----
2653 $ ar rcs tpp.a tpp.o
2654 ----
2655 --
2656
2657 To build the instrumented application:
2658
2659 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2660 +
2661 --
2662 [source,c]
2663 ----
2664 #define TRACEPOINT_DEFINE
2665 ----
2666 --
2667
2668 . Compile the application source file:
2669 +
2670 --
2671 [role="term"]
2672 ----
2673 $ gcc -c app.c
2674 ----
2675 --
2676
2677 . Build the application:
2678 +
2679 --
2680 [role="term"]
2681 ----
2682 $ gcc -o app app.o tpp.a -llttng-ust -ldl
2683 ----
2684 --
2685
2686 To run the instrumented application:
2687
2688 * Start the application:
2689 +
2690 --
2691 [role="term"]
2692 ----
2693 $ ./app
2694 ----
2695 --
2696
2697 |
2698 The instrumented application is linked with the tracepoint provider
2699 package shared object.
2700
2701 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
2702
2703 |
2704 include::../common/ust-sit-step-tp-so.txt[]
2705
2706 To build the instrumented application:
2707
2708 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2709 +
2710 --
2711 [source,c]
2712 ----
2713 #define TRACEPOINT_DEFINE
2714 ----
2715 --
2716
2717 . Compile the application source file:
2718 +
2719 --
2720 [role="term"]
2721 ----
2722 $ gcc -c app.c
2723 ----
2724 --
2725
2726 . Build the application:
2727 +
2728 --
2729 [role="term"]
2730 ----
2731 $ gcc -o app app.o -ldl -L. -ltpp
2732 ----
2733 --
2734
2735 To run the instrumented application:
2736
2737 * Start the application:
2738 +
2739 --
2740 [role="term"]
2741 ----
2742 $ ./app
2743 ----
2744 --
2745
2746 |
2747 The tracepoint provider package shared object is preloaded before the
2748 instrumented application starts.
2749
2750 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
2751
2752 |
2753 include::../common/ust-sit-step-tp-so.txt[]
2754
2755 To build the instrumented application:
2756
2757 . In path:{app.c}, before including path:{tpp.h}, add the
2758   following lines:
2759 +
2760 --
2761 [source,c]
2762 ----
2763 #define TRACEPOINT_DEFINE
2764 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2765 ----
2766 --
2767
2768 . Compile the application source file:
2769 +
2770 --
2771 [role="term"]
2772 ----
2773 $ gcc -c app.c
2774 ----
2775 --
2776
2777 . Build the application:
2778 +
2779 --
2780 [role="term"]
2781 ----
2782 $ gcc -o app app.o -ldl
2783 ----
2784 --
2785
2786 To run the instrumented application with tracing support:
2787
2788 * Preload the tracepoint provider package shared object and
2789   start the application:
2790 +
2791 --
2792 [role="term"]
2793 ----
2794 $ LD_PRELOAD=./libtpp.so ./app
2795 ----
2796 --
2797
2798 To run the instrumented application without tracing support:
2799
2800 * Start the application:
2801 +
2802 --
2803 [role="term"]
2804 ----
2805 $ ./app
2806 ----
2807 --
2808
2809 |
2810 The instrumented application dynamically loads the tracepoint provider
2811 package shared object.
2812
2813 See the <<dlclose-warning,warning about `dlclose()`>>.
2814
2815 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
2816
2817 |
2818 include::../common/ust-sit-step-tp-so.txt[]
2819
2820 To build the instrumented application:
2821
2822 . In path:{app.c}, before including path:{tpp.h}, add the
2823   following lines:
2824 +
2825 --
2826 [source,c]
2827 ----
2828 #define TRACEPOINT_DEFINE
2829 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2830 ----
2831 --
2832
2833 . Compile the application source file:
2834 +
2835 --
2836 [role="term"]
2837 ----
2838 $ gcc -c app.c
2839 ----
2840 --
2841
2842 . Build the application:
2843 +
2844 --
2845 [role="term"]
2846 ----
2847 $ gcc -o app app.o -ldl
2848 ----
2849 --
2850
2851 To run the instrumented application:
2852
2853 * Start the application:
2854 +
2855 --
2856 [role="term"]
2857 ----
2858 $ ./app
2859 ----
2860 --
2861
2862 |
2863 The application is linked with the instrumented user library.
2864
2865 The instrumented user library is statically linked with the tracepoint
2866 provider package object file.
2867
2868 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
2869
2870 |
2871 include::../common/ust-sit-step-tp-o-fpic.txt[]
2872
2873 To build the instrumented user library:
2874
2875 . In path:{emon.c}, before including path:{tpp.h}, add the
2876   following line:
2877 +
2878 --
2879 [source,c]
2880 ----
2881 #define TRACEPOINT_DEFINE
2882 ----
2883 --
2884
2885 . Compile the user library source file:
2886 +
2887 --
2888 [role="term"]
2889 ----
2890 $ gcc -I. -fpic -c emon.c
2891 ----
2892 --
2893
2894 . Build the user library shared object:
2895 +
2896 --
2897 [role="term"]
2898 ----
2899 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
2900 ----
2901 --
2902
2903 To build the application:
2904
2905 . Compile the application source file:
2906 +
2907 --
2908 [role="term"]
2909 ----
2910 $ gcc -c app.c
2911 ----
2912 --
2913
2914 . Build the application:
2915 +
2916 --
2917 [role="term"]
2918 ----
2919 $ gcc -o app app.o -L. -lemon
2920 ----
2921 --
2922
2923 To run the application:
2924
2925 * Start the application:
2926 +
2927 --
2928 [role="term"]
2929 ----
2930 $ ./app
2931 ----
2932 --
2933
2934 |
2935 The application is linked with the instrumented user library.
2936
2937 The instrumented user library is linked with the tracepoint provider
2938 package shared object.
2939
2940 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
2941
2942 |
2943 include::../common/ust-sit-step-tp-so.txt[]
2944
2945 To build the instrumented user library:
2946
2947 . In path:{emon.c}, before including path:{tpp.h}, add the
2948   following line:
2949 +
2950 --
2951 [source,c]
2952 ----
2953 #define TRACEPOINT_DEFINE
2954 ----
2955 --
2956
2957 . Compile the user library source file:
2958 +
2959 --
2960 [role="term"]
2961 ----
2962 $ gcc -I. -fpic -c emon.c
2963 ----
2964 --
2965
2966 . Build the user library shared object:
2967 +
2968 --
2969 [role="term"]
2970 ----
2971 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
2972 ----
2973 --
2974
2975 To build the application:
2976
2977 . Compile the application source file:
2978 +
2979 --
2980 [role="term"]
2981 ----
2982 $ gcc -c app.c
2983 ----
2984 --
2985
2986 . Build the application:
2987 +
2988 --
2989 [role="term"]
2990 ----
2991 $ gcc -o app app.o -L. -lemon
2992 ----
2993 --
2994
2995 To run the application:
2996
2997 * Start the application:
2998 +
2999 --
3000 [role="term"]
3001 ----
3002 $ ./app
3003 ----
3004 --
3005
3006 |
3007 The tracepoint provider package shared object is preloaded before the
3008 application starts.
3009
3010 The application is linked with the instrumented user library.
3011
3012 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3013
3014 |
3015 include::../common/ust-sit-step-tp-so.txt[]
3016
3017 To build the instrumented user library:
3018
3019 . In path:{emon.c}, before including path:{tpp.h}, add the
3020   following lines:
3021 +
3022 --
3023 [source,c]
3024 ----
3025 #define TRACEPOINT_DEFINE
3026 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3027 ----
3028 --
3029
3030 . Compile the user library source file:
3031 +
3032 --
3033 [role="term"]
3034 ----
3035 $ gcc -I. -fpic -c emon.c
3036 ----
3037 --
3038
3039 . Build the user library shared object:
3040 +
3041 --
3042 [role="term"]
3043 ----
3044 $ gcc -shared -o libemon.so emon.o -ldl
3045 ----
3046 --
3047
3048 To build the application:
3049
3050 . Compile the application source file:
3051 +
3052 --
3053 [role="term"]
3054 ----
3055 $ gcc -c app.c
3056 ----
3057 --
3058
3059 . Build the application:
3060 +
3061 --
3062 [role="term"]
3063 ----
3064 $ gcc -o app app.o -L. -lemon
3065 ----
3066 --
3067
3068 To run the application with tracing support:
3069
3070 * Preload the tracepoint provider package shared object and
3071   start the application:
3072 +
3073 --
3074 [role="term"]
3075 ----
3076 $ LD_PRELOAD=./libtpp.so ./app
3077 ----
3078 --
3079
3080 To run the application without tracing support:
3081
3082 * Start the application:
3083 +
3084 --
3085 [role="term"]
3086 ----
3087 $ ./app
3088 ----
3089 --
3090
3091 |
3092 The application is linked with the instrumented user library.
3093
3094 The instrumented user library dynamically loads the tracepoint provider
3095 package shared object.
3096
3097 See the <<dlclose-warning,warning about `dlclose()`>>.
3098
3099 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3100
3101 |
3102 include::../common/ust-sit-step-tp-so.txt[]
3103
3104 To build the instrumented user library:
3105
3106 . In path:{emon.c}, before including path:{tpp.h}, add the
3107   following lines:
3108 +
3109 --
3110 [source,c]
3111 ----
3112 #define TRACEPOINT_DEFINE
3113 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3114 ----
3115 --
3116
3117 . Compile the user library source file:
3118 +
3119 --
3120 [role="term"]
3121 ----
3122 $ gcc -I. -fpic -c emon.c
3123 ----
3124 --
3125
3126 . Build the user library shared object:
3127 +
3128 --
3129 [role="term"]
3130 ----
3131 $ gcc -shared -o libemon.so emon.o -ldl
3132 ----
3133 --
3134
3135 To build the application:
3136
3137 . Compile the application source file:
3138 +
3139 --
3140 [role="term"]
3141 ----
3142 $ gcc -c app.c
3143 ----
3144 --
3145
3146 . Build the application:
3147 +
3148 --
3149 [role="term"]
3150 ----
3151 $ gcc -o app app.o -L. -lemon
3152 ----
3153 --
3154
3155 To run the application:
3156
3157 * Start the application:
3158 +
3159 --
3160 [role="term"]
3161 ----
3162 $ ./app
3163 ----
3164 --
3165
3166 |
3167 The application dynamically loads the instrumented user library.
3168
3169 The instrumented user library is linked with the tracepoint provider
3170 package shared object.
3171
3172 See the <<dlclose-warning,warning about `dlclose()`>>.
3173
3174 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3175
3176 |
3177 include::../common/ust-sit-step-tp-so.txt[]
3178
3179 To build the instrumented user library:
3180
3181 . In path:{emon.c}, before including path:{tpp.h}, add the
3182   following line:
3183 +
3184 --
3185 [source,c]
3186 ----
3187 #define TRACEPOINT_DEFINE
3188 ----
3189 --
3190
3191 . Compile the user library source file:
3192 +
3193 --
3194 [role="term"]
3195 ----
3196 $ gcc -I. -fpic -c emon.c
3197 ----
3198 --
3199
3200 . Build the user library shared object:
3201 +
3202 --
3203 [role="term"]
3204 ----
3205 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3206 ----
3207 --
3208
3209 To build the application:
3210
3211 . Compile the application source file:
3212 +
3213 --
3214 [role="term"]
3215 ----
3216 $ gcc -c app.c
3217 ----
3218 --
3219
3220 . Build the application:
3221 +
3222 --
3223 [role="term"]
3224 ----
3225 $ gcc -o app app.o -ldl -L. -lemon
3226 ----
3227 --
3228
3229 To run the application:
3230
3231 * Start the application:
3232 +
3233 --
3234 [role="term"]
3235 ----
3236 $ ./app
3237 ----
3238 --
3239
3240 |
3241 The application dynamically loads the instrumented user library.
3242
3243 The instrumented user library dynamically loads the tracepoint provider
3244 package shared object.
3245
3246 See the <<dlclose-warning,warning about `dlclose()`>>.
3247
3248 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3249
3250 |
3251 include::../common/ust-sit-step-tp-so.txt[]
3252
3253 To build the instrumented user library:
3254
3255 . In path:{emon.c}, before including path:{tpp.h}, add the
3256   following lines:
3257 +
3258 --
3259 [source,c]
3260 ----
3261 #define TRACEPOINT_DEFINE
3262 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3263 ----
3264 --
3265
3266 . Compile the user library source file:
3267 +
3268 --
3269 [role="term"]
3270 ----
3271 $ gcc -I. -fpic -c emon.c
3272 ----
3273 --
3274
3275 . Build the user library shared object:
3276 +
3277 --
3278 [role="term"]
3279 ----
3280 $ gcc -shared -o libemon.so emon.o -ldl
3281 ----
3282 --
3283
3284 To build the application:
3285
3286 . Compile the application source file:
3287 +
3288 --
3289 [role="term"]
3290 ----
3291 $ gcc -c app.c
3292 ----
3293 --
3294
3295 . Build the application:
3296 +
3297 --
3298 [role="term"]
3299 ----
3300 $ gcc -o app app.o -ldl -L. -lemon
3301 ----
3302 --
3303
3304 To run the application:
3305
3306 * Start the application:
3307 +
3308 --
3309 [role="term"]
3310 ----
3311 $ ./app
3312 ----
3313 --
3314
3315 |
3316 The tracepoint provider package shared object is preloaded before the
3317 application starts.
3318
3319 The application dynamically loads the instrumented user library.
3320
3321 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3322
3323 |
3324 include::../common/ust-sit-step-tp-so.txt[]
3325
3326 To build the instrumented user library:
3327
3328 . In path:{emon.c}, before including path:{tpp.h}, add the
3329   following lines:
3330 +
3331 --
3332 [source,c]
3333 ----
3334 #define TRACEPOINT_DEFINE
3335 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3336 ----
3337 --
3338
3339 . Compile the user library source file:
3340 +
3341 --
3342 [role="term"]
3343 ----
3344 $ gcc -I. -fpic -c emon.c
3345 ----
3346 --
3347
3348 . Build the user library shared object:
3349 +
3350 --
3351 [role="term"]
3352 ----
3353 $ gcc -shared -o libemon.so emon.o -ldl
3354 ----
3355 --
3356
3357 To build the application:
3358
3359 . Compile the application source file:
3360 +
3361 --
3362 [role="term"]
3363 ----
3364 $ gcc -c app.c
3365 ----
3366 --
3367
3368 . Build the application:
3369 +
3370 --
3371 [role="term"]
3372 ----
3373 $ gcc -o app app.o -L. -lemon
3374 ----
3375 --
3376
3377 To run the application with tracing support:
3378
3379 * Preload the tracepoint provider package shared object and
3380   start the application:
3381 +
3382 --
3383 [role="term"]
3384 ----
3385 $ LD_PRELOAD=./libtpp.so ./app
3386 ----
3387 --
3388
3389 To run the application without tracing support:
3390
3391 * Start the application:
3392 +
3393 --
3394 [role="term"]
3395 ----
3396 $ ./app
3397 ----
3398 --
3399
3400 |
3401 The application is statically linked with the tracepoint provider
3402 package object file.
3403
3404 The application is linked with the instrumented user library.
3405
3406 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3407
3408 |
3409 include::../common/ust-sit-step-tp-o.txt[]
3410
3411 To build the instrumented user library:
3412
3413 . In path:{emon.c}, before including path:{tpp.h}, add the
3414   following line:
3415 +
3416 --
3417 [source,c]
3418 ----
3419 #define TRACEPOINT_DEFINE
3420 ----
3421 --
3422
3423 . Compile the user library source file:
3424 +
3425 --
3426 [role="term"]
3427 ----
3428 $ gcc -I. -fpic -c emon.c
3429 ----
3430 --
3431
3432 . Build the user library shared object:
3433 +
3434 --
3435 [role="term"]
3436 ----
3437 $ gcc -shared -o libemon.so emon.o
3438 ----
3439 --
3440
3441 To build the application:
3442
3443 . Compile the application source file:
3444 +
3445 --
3446 [role="term"]
3447 ----
3448 $ gcc -c app.c
3449 ----
3450 --
3451
3452 . Build the application:
3453 +
3454 --
3455 [role="term"]
3456 ----
3457 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3458 ----
3459 --
3460
3461 To run the instrumented application:
3462
3463 * Start the application:
3464 +
3465 --
3466 [role="term"]
3467 ----
3468 $ ./app
3469 ----
3470 --
3471
3472 |
3473 The application is statically linked with the tracepoint provider
3474 package object file.
3475
3476 The application dynamically loads the instrumented user library.
3477
3478 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3479
3480 |
3481 include::../common/ust-sit-step-tp-o.txt[]
3482
3483 To build the application:
3484
3485 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3486 +
3487 --
3488 [source,c]
3489 ----
3490 #define TRACEPOINT_DEFINE
3491 ----
3492 --
3493
3494 . Compile the application source file:
3495 +
3496 --
3497 [role="term"]
3498 ----
3499 $ gcc -c app.c
3500 ----
3501 --
3502
3503 . Build the application:
3504 +
3505 --
3506 [role="term"]
3507 ----
3508 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3509     -llttng-ust -ldl
3510 ----
3511 --
3512 +
3513 The `--export-dynamic` option passed to the linker is necessary for the
3514 dynamically loaded library to ``see'' the tracepoint symbols defined in
3515 the application.
3516
3517 To build the instrumented user library:
3518
3519 . Compile the user library source file:
3520 +
3521 --
3522 [role="term"]
3523 ----
3524 $ gcc -I. -fpic -c emon.c
3525 ----
3526 --
3527
3528 . Build the user library shared object:
3529 +
3530 --
3531 [role="term"]
3532 ----
3533 $ gcc -shared -o libemon.so emon.o
3534 ----
3535 --
3536
3537 To run the application:
3538
3539 * Start the application:
3540 +
3541 --
3542 [role="term"]
3543 ----
3544 $ ./app
3545 ----
3546 --
3547 |====
3548
3549 [[dlclose-warning]]
3550 [IMPORTANT]
3551 .Do not use man:dlclose(3) on a tracepoint provider package
3552 ====
3553 Never use man:dlclose(3) on any shared object which:
3554
3555 * Is linked with, statically or dynamically, a tracepoint provider
3556   package.
3557 * Calls man:dlopen(3) itself to dynamically open a tracepoint provider
3558   package shared object.
3559
3560 This is currently considered **unsafe** due to a lack of reference
3561 counting from LTTng-UST to the shared object.
3562
3563 A known workaround (available since glibc 2.2) is to use the
3564 `RTLD_NODELETE` flag when calling man:dlopen(3) initially. This has the
3565 effect of not unloading the loaded shared object, even if man:dlclose(3)
3566 is called.
3567
3568 You can also preload the tracepoint provider package shared object with
3569 the env:LD_PRELOAD environment variable to overcome this limitation.
3570 ====
3571
3572
3573 [[using-lttng-ust-with-daemons]]
3574 ===== Use noch:{LTTng-UST} with daemons
3575
3576 If your instrumented application calls man:fork(2), man:clone(2),
3577 or BSD's man:rfork(2), without a following man:exec(3)-family
3578 system call, you must preload the path:{liblttng-ust-fork.so} shared
3579 object when starting the application.
3580
3581 [role="term"]
3582 ----
3583 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3584 ----
3585
3586 If your tracepoint provider package is
3587 a shared library which you also preload, you must put both
3588 shared objects in env:LD_PRELOAD:
3589
3590 [role="term"]
3591 ----
3592 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3593 ----
3594
3595
3596 [[lttng-ust-pkg-config]]
3597 ===== Use noch:{pkg-config}
3598
3599 On some distributions, LTTng-UST ships with a
3600 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3601 metadata file. If this is your case, then you can use cmd:pkg-config to
3602 build an application on the command line:
3603
3604 [role="term"]
3605 ----
3606 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3607 ----
3608
3609
3610 [[instrumenting-32-bit-app-on-64-bit-system]]
3611 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3612
3613 In order to trace a 32-bit application running on a 64-bit system,
3614 LTTng must use a dedicated 32-bit
3615 <<lttng-consumerd,consumer daemon>>.
3616
3617 The following steps show how to build and install a 32-bit consumer
3618 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3619 build and install the 32-bit LTTng-UST libraries, and how to build and
3620 link an instrumented 32-bit application in that context.
3621
3622 To build a 32-bit instrumented application for a 64-bit target system,
3623 assuming you have a fresh target system with no installed Userspace RCU
3624 or LTTng packages:
3625
3626 . Download, build, and install a 32-bit version of Userspace RCU:
3627 +
3628 --
3629 [role="term"]
3630 ----
3631 $ cd $(mktemp -d) &&
3632 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3633 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3634 cd userspace-rcu-0.9.* &&
3635 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3636 make &&
3637 sudo make install &&
3638 sudo ldconfig
3639 ----
3640 --
3641
3642 . Using your distribution's package manager, or from source, install
3643   the following 32-bit versions of the following dependencies of
3644   LTTng-tools and LTTng-UST:
3645 +
3646 --
3647 * https://sourceforge.net/projects/libuuid/[libuuid]
3648 * http://directory.fsf.org/wiki/Popt[popt]
3649 * http://www.xmlsoft.org/[libxml2]
3650 --
3651
3652 . Download, build, and install a 32-bit version of the latest
3653   LTTng-UST{nbsp}{revision}:
3654 +
3655 --
3656 [role="term"]
3657 ----
3658 $ cd $(mktemp -d) &&
3659 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
3660 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
3661 cd lttng-ust-2.8.* &&
3662 ./configure --libdir=/usr/local/lib32 \
3663             CFLAGS=-m32 CXXFLAGS=-m32 \
3664             LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
3665 make &&
3666 sudo make install &&
3667 sudo ldconfig
3668 ----
3669 --
3670 +
3671 [NOTE]
3672 ====
3673 Depending on your distribution,
3674 32-bit libraries could be installed at a different location than
3675 `/usr/lib32`. For example, Debian is known to install
3676 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
3677
3678 In this case, make sure to set `LDFLAGS` to all the
3679 relevant 32-bit library paths, for example:
3680
3681 [role="term"]
3682 ----
3683 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
3684 ----
3685 ====
3686
3687 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
3688   the 32-bit consumer daemon:
3689 +
3690 --
3691 [role="term"]
3692 ----
3693 $ cd $(mktemp -d) &&
3694 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
3695 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
3696 cd lttng-tools-2.8.* &&
3697 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3698             LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
3699             --disable-bin-lttng --disable-bin-lttng-crash \
3700             --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
3701 make &&
3702 cd src/bin/lttng-consumerd &&
3703 sudo make install &&
3704 sudo ldconfig
3705 ----
3706 --
3707
3708 . From your distribution or from source,
3709   <<installing-lttng,install>> the 64-bit versions of
3710   LTTng-UST and Userspace RCU.
3711 . Download, build, and install the 64-bit version of the
3712   latest LTTng-tools{nbsp}{revision}:
3713 +
3714 --
3715 [role="term"]
3716 ----
3717 $ cd $(mktemp -d) &&
3718 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
3719 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
3720 cd lttng-tools-2.8.* &&
3721 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
3722             --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
3723 make &&
3724 sudo make install &&
3725 sudo ldconfig
3726 ----
3727 --
3728
3729 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
3730   when linking your 32-bit application:
3731 +
3732 ----
3733 -m32 -L/usr/lib32 -L/usr/local/lib32 \
3734 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
3735 ----
3736 +
3737 For example, let's rebuild the quick start example in
3738 <<tracing-your-own-user-application,Trace a user application>> as an
3739 instrumented 32-bit application:
3740 +
3741 --
3742 [role="term"]
3743 ----
3744 $ gcc -m32 -c -I. hello-tp.c
3745 $ gcc -m32 -c hello.c
3746 $ gcc -m32 -o hello hello.o hello-tp.o \
3747     -L/usr/lib32 -L/usr/local/lib32 \
3748     -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
3749     -llttng-ust -ldl
3750 ----
3751 --
3752
3753 No special action is required to execute the 32-bit application and
3754 to trace it: use the command-line man:lttng(1) tool as usual.
3755
3756
3757 [role="since-2.5"]
3758 [[tracef]]
3759 ==== Use `tracef()`
3760
3761 man:tracef(3) is a small LTTng-UST API designed for quick,
3762 man:printf(3)-like instrumentation without the burden of
3763 <<tracepoint-provider,creating>> and
3764 <<building-tracepoint-providers-and-user-application,building>>
3765 a tracepoint provider package.
3766
3767 To use `tracef()` in your application:
3768
3769 . In the C or C++ source files where you need to use `tracef()`,
3770   include `<lttng/tracef.h>`:
3771 +
3772 --
3773 [source,c]
3774 ----
3775 #include <lttng/tracef.h>
3776 ----
3777 --
3778
3779 . In the application's source code, use `tracef()` like you would use
3780   man:printf(3):
3781 +
3782 --
3783 [source,c]
3784 ----
3785     /* ... */
3786
3787     tracef("my message: %d (%s)", my_integer, my_string);
3788
3789     /* ... */
3790 ----
3791 --
3792
3793 . Link your application with `liblttng-ust`:
3794 +
3795 --
3796 [role="term"]
3797 ----
3798 $ gcc -o app app.c -llttng-ust
3799 ----
3800 --
3801
3802 To trace the events that `tracef()` calls emit:
3803
3804 * <<enabling-disabling-events,Create an event rule>> which matches the
3805   `lttng_ust_tracef:*` event name:
3806 +
3807 --
3808 [role="term"]
3809 ----
3810 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
3811 ----
3812 --
3813
3814 [IMPORTANT]
3815 .Limitations of `tracef()`
3816 ====
3817 The `tracef()` utility function was developed to make user space tracing
3818 super simple, albeit with notable disadvantages compared to
3819 <<defining-tracepoints,user-defined tracepoints>>:
3820
3821 * All the emitted events have the same tracepoint provider and
3822   tracepoint names, respectively `lttng_ust_tracef` and `event`.
3823 * There is no static type checking.
3824 * The only event record field you actually get, named `msg`, is a string
3825   potentially containing the values you passed to `tracef()`
3826   using your own format string. This also means that you cannot filter
3827   events with a custom expression at run time because there are no
3828   isolated fields.
3829 * Since `tracef()` uses the C standard library's man:vasprintf(3)
3830   function behind the scenes to format the strings at run time, its
3831   expected performance is lower than with user-defined tracepoints,
3832   which do not require a conversion to a string.
3833
3834 Taking this into consideration, `tracef()` is useful for some quick
3835 prototyping and debugging, but you should not consider it for any
3836 permanent and serious applicative instrumentation.
3837 ====
3838
3839
3840 [role="since-2.7"]
3841 [[tracelog]]
3842 ==== Use `tracelog()`
3843
3844 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
3845 the difference that it accepts an additional log level parameter.
3846
3847 The goal of `tracelog()` is to ease the migration from logging to
3848 tracing.
3849
3850 To use `tracelog()` in your application:
3851
3852 . In the C or C++ source files where you need to use `tracelog()`,
3853   include `<lttng/tracelog.h>`:
3854 +
3855 --
3856 [source,c]
3857 ----
3858 #include <lttng/tracelog.h>
3859 ----
3860 --
3861
3862 . In the application's source code, use `tracelog()` like you would use
3863   man:printf(3), except for the first parameter which is the log
3864   level:
3865 +
3866 --
3867 [source,c]
3868 ----
3869     /* ... */
3870
3871     tracelog(TRACE_WARNING, "my message: %d (%s)",
3872              my_integer, my_string);
3873
3874     /* ... */
3875 ----
3876 --
3877 +
3878 See man:lttng-ust(3) for a list of available log level names.
3879
3880 . Link your application with `liblttng-ust`:
3881 +
3882 --
3883 [role="term"]
3884 ----
3885 $ gcc -o app app.c -llttng-ust
3886 ----
3887 --
3888
3889 To trace the events that `tracelog()` calls emit with a log level
3890 _as severe as_ a specific log level:
3891
3892 * <<enabling-disabling-events,Create an event rule>> which matches the
3893   `lttng_ust_tracelog:*` event name and a minimum level
3894   of severity:
3895 +
3896 --
3897 [role="term"]
3898 ----
3899 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
3900                    --loglevel=TRACE_WARNING
3901 ----
3902 --
3903
3904 To trace the events that `tracelog()` calls emit with a
3905 _specific log level_:
3906
3907 * Create an event rule which matches the `lttng_ust_tracelog:*`
3908   event name and a specific log level:
3909 +
3910 --
3911 [role="term"]
3912 ----
3913 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
3914                    --loglevel-only=TRACE_INFO
3915 ----
3916 --
3917
3918
3919 [[prebuilt-ust-helpers]]
3920 === Prebuilt user space tracing helpers
3921
3922 The LTTng-UST package provides a few helpers in the form of preloadable
3923 shared objects which automatically instrument system functions and
3924 calls.
3925
3926 The helper shared objects are normally found in dir:{/usr/lib}. If you
3927 built LTTng-UST <<building-from-source,from source>>, they are probably
3928 located in dir:{/usr/local/lib}.
3929
3930 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
3931 are:
3932
3933 path:{liblttng-ust-libc-wrapper.so}::
3934 path:{liblttng-ust-pthread-wrapper.so}::
3935   <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
3936   memory and POSIX threads function tracing>>.
3937
3938 path:{liblttng-ust-cyg-profile.so}::
3939 path:{liblttng-ust-cyg-profile-fast.so}::
3940   <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
3941
3942 path:{liblttng-ust-dl.so}::
3943   <<liblttng-ust-dl,Dynamic linker tracing>>.
3944
3945 To use a user space tracing helper with any user application:
3946
3947 * Preload the helper shared object when you start the application:
3948 +
3949 --
3950 [role="term"]
3951 ----
3952 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
3953 ----
3954 --
3955 +
3956 You can preload more than one helper:
3957 +
3958 --
3959 [role="term"]
3960 ----
3961 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
3962 ----
3963 --
3964
3965
3966 [role="since-2.3"]
3967 [[liblttng-ust-libc-pthread-wrapper]]
3968 ==== Instrument C standard library memory and POSIX threads functions
3969
3970 The path:{liblttng-ust-libc-wrapper.so} and
3971 path:{liblttng-ust-pthread-wrapper.so} helpers
3972 add instrumentation to some C standard library and POSIX
3973 threads functions.
3974
3975 [role="growable"]
3976 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
3977 |====
3978 |TP provider name |TP name |Instrumented function
3979
3980 .6+|`lttng_ust_libc` |`malloc`         |man:malloc(3)
3981                      |`calloc`         |man:calloc(3)
3982                      |`realloc`        |man:realloc(3)
3983                      |`free`           |man:free(3)
3984                      |`memalign`       |man:memalign(3)
3985                      |`posix_memalign` |man:posix_memalign(3)
3986 |====
3987
3988 [role="growable"]
3989 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
3990 |====
3991 |TP provider name |TP name |Instrumented function
3992
3993 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
3994                         |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
3995                         |`pthread_mutex_trylock`  |man:pthread_mutex_trylock(3p)
3996                         |`pthread_mutex_unlock`   |man:pthread_mutex_unlock(3p)
3997 |====
3998
3999 When you preload the shared object, it replaces the functions listed
4000 in the previous tables by wrappers which contain tracepoints and call
4001 the replaced functions.
4002
4003
4004 [[liblttng-ust-cyg-profile]]
4005 ==== Instrument function entry and exit
4006
4007 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4008 to the entry and exit points of functions.
4009
4010 man:gcc(1) and man:clang(1) have an option named
4011 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4012 which generates instrumentation calls for entry and exit to functions.
4013 The LTTng-UST function tracing helpers,
4014 path:{liblttng-ust-cyg-profile.so} and
4015 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4016 to add tracepoints to the two generated functions (which contain
4017 `cyg_profile` in their names, hence the helper's name).
4018
4019 To use the LTTng-UST function tracing helper, the source files to
4020 instrument must be built using the `-finstrument-functions` compiler
4021 flag.
4022
4023 There are two versions of the LTTng-UST function tracing helper:
4024
4025 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4026   that you should only use when it can be _guaranteed_ that the
4027   complete event stream is recorded without any lost event record.
4028   Any kind of duplicate information is left out.
4029 +
4030 Assuming no event record is lost, having only the function addresses on
4031 entry is enough to create a call graph, since an event record always
4032 contains the ID of the CPU that generated it.
4033 +
4034 You can use a tool like man:addr2line(1) to convert function addresses
4035 back to source file names and line numbers.
4036
4037 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4038 which also works in use cases where event records might get discarded or
4039 not recorded from application startup.
4040 In these cases, the trace analyzer needs more information to be
4041 able to reconstruct the program flow.
4042
4043 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4044 points of this helper.
4045
4046 All the tracepoints that this helper provides have the
4047 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4048
4049 TIP: It's sometimes a good idea to limit the number of source files that
4050 you compile with the `-finstrument-functions` option to prevent LTTng
4051 from writing an excessive amount of trace data at run time. When using
4052 man:gcc(1), you can use the
4053 `-finstrument-functions-exclude-function-list` option to avoid
4054 instrument entries and exits of specific function names.
4055
4056
4057 [role="since-2.4"]
4058 [[liblttng-ust-dl]]
4059 ==== Instrument the dynamic linker
4060
4061 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4062 man:dlopen(3) and man:dlclose(3) function calls.
4063
4064 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4065 of this helper.
4066
4067
4068 [role="since-2.4"]
4069 [[java-application]]
4070 === User space Java agent
4071
4072 You can instrument any Java application which uses one of the following
4073 logging frameworks:
4074
4075 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4076   (JUL) core logging facilities.
4077 * http://logging.apache.org/log4j/1.2/[**Apache log4j 1.2**], since
4078   LTTng 2.6. Note that Apache Log4j{nbsp}2 is not supported.
4079
4080 [role="img-100"]
4081 .LTTng-UST Java agent imported by a Java application.
4082 image::java-app.png[]
4083
4084 Note that the methods described below are new in LTTng{nbsp}2.8.
4085 Previous LTTng versions use another technique.
4086
4087 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4088 and https://ci.lttng.org/[continuous integration], thus this version is
4089 directly supported. However, the LTTng-UST Java agent is also tested
4090 with OpenJDK{nbsp}7.
4091
4092
4093 [role="since-2.8"]
4094 [[jul]]
4095 ==== Use the LTTng-UST Java agent for `java.util.logging`
4096
4097 To use the LTTng-UST Java agent in a Java application which uses
4098 `java.util.logging` (JUL):
4099
4100 . In the Java application's source code, import the LTTng-UST
4101   log handler package for `java.util.logging`:
4102 +
4103 --
4104 [source,java]
4105 ----
4106 import org.lttng.ust.agent.jul.LttngLogHandler;
4107 ----
4108 --
4109
4110 . Create an LTTng-UST JUL log handler:
4111 +
4112 --
4113 [source,java]
4114 ----
4115 Handler lttngUstLogHandler = new LttngLogHandler();
4116 ----
4117 --
4118
4119 . Add this handler to the JUL loggers which should emit LTTng events:
4120 +
4121 --
4122 [source,java]
4123 ----
4124 Logger myLogger = Logger.getLogger("some-logger");
4125
4126 myLogger.addHandler(lttngUstLogHandler);
4127 ----
4128 --
4129
4130 . Use `java.util.logging` log statements and configuration as usual.
4131   The loggers with an attached LTTng-UST log handler can emit
4132   LTTng events.
4133
4134 . Before exiting the application, remove the LTTng-UST log handler from
4135   the loggers attached to it and call its `close()` method:
4136 +
4137 --
4138 [source,java]
4139 ----
4140 myLogger.removeHandler(lttngUstLogHandler);
4141 lttngUstLogHandler.close();
4142 ----
4143 --
4144 +
4145 This is not strictly necessary, but it is recommended for a clean
4146 disposal of the handler's resources.
4147
4148 . Include the LTTng-UST Java agent's common and JUL-specific JAR files,
4149   path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4150   in the
4151   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4152   path] when you build the Java application.
4153 +
4154 The JAR files are typically located in dir:{/usr/share/java}.
4155 +
4156 IMPORTANT: The LTTng-UST Java agent must be
4157 <<installing-lttng,installed>> for the logging framework your
4158 application uses.
4159
4160 .Use the LTTng-UST Java agent for `java.util.logging`.
4161 ====
4162 [source,java]
4163 .path:{Test.java}
4164 ----
4165 import java.io.IOException;
4166 import java.util.logging.Handler;
4167 import java.util.logging.Logger;
4168 import org.lttng.ust.agent.jul.LttngLogHandler;
4169
4170 public class Test
4171 {
4172     private static final int answer = 42;
4173
4174     public static void main(String[] argv) throws Exception
4175     {
4176         // Create a logger
4177         Logger logger = Logger.getLogger("jello");
4178
4179         // Create an LTTng-UST log handler
4180         Handler lttngUstLogHandler = new LttngLogHandler();
4181
4182         // Add the LTTng-UST log handler to our logger
4183         logger.addHandler(lttngUstLogHandler);
4184
4185         // Log at will!
4186         logger.info("some info");
4187         logger.warning("some warning");
4188         Thread.sleep(500);
4189         logger.finer("finer information; the answer is " + answer);
4190         Thread.sleep(123);
4191         logger.severe("error!");
4192
4193         // Not mandatory, but cleaner
4194         logger.removeHandler(lttngUstLogHandler);
4195         lttngUstLogHandler.close();
4196     }
4197 }
4198 ----
4199
4200 Build this example:
4201
4202 [role="term"]
4203 ----
4204 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4205 ----
4206
4207 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4208 <<enabling-disabling-events,create an event rule>> matching the
4209 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4210
4211 [role="term"]
4212 ----
4213 $ lttng create
4214 $ lttng enable-event --jul jello
4215 $ lttng start
4216 ----
4217
4218 Run the compiled class:
4219
4220 [role="term"]
4221 ----
4222 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4223 ----
4224
4225 <<basic-tracing-session-control,Stop tracing>> and inspect the
4226 recorded events:
4227
4228 [role="term"]
4229 ----
4230 $ lttng stop
4231 $ lttng view
4232 ----
4233 ====
4234
4235 In the resulting trace, an <<event,event record>> generated by a Java
4236 application using `java.util.logging` is named `lttng_jul:event` and
4237 has the following fields:
4238
4239 `msg`::
4240   Log record's message.
4241
4242 `logger_name`::
4243   Logger name.
4244
4245 `class_name`::
4246   Name of the class in which the log statement was executed.
4247
4248 `method_name`::
4249   Name of the method in which the log statement was executed.
4250
4251 `long_millis`::
4252   Logging time (timestamp in milliseconds).
4253
4254 `int_loglevel`::
4255   Log level integer value.
4256
4257 `int_threadid`::
4258   ID of the thread in which the log statement was executed.
4259
4260 You can use the opt:lttng-enable-event(1):--loglevel or
4261 opt:lttng-enable-event(1):--loglevel-only option of the
4262 man:lttng-enable-event(1) command to target a range of JUL log levels
4263 or a specific JUL log level.
4264
4265
4266 [role="since-2.8"]
4267 [[log4j]]
4268 ==== Use the LTTng-UST Java agent for Apache log4j
4269
4270 To use the LTTng-UST Java agent in a Java application which uses
4271 Apache log4j 1.2:
4272
4273 . In the Java application's source code, import the LTTng-UST
4274   log appender package for Apache log4j:
4275 +
4276 --
4277 [source,java]
4278 ----
4279 import org.lttng.ust.agent.log4j.LttngLogAppender;
4280 ----
4281 --
4282
4283 . Create an LTTng-UST log4j log appender:
4284 +
4285 --
4286 [source,java]
4287 ----
4288 Appender lttngUstLogAppender = new LttngLogAppender();
4289 ----
4290 --
4291
4292 . Add this appender to the log4j loggers which should emit LTTng events:
4293 +
4294 --
4295 [source,java]
4296 ----
4297 Logger myLogger = Logger.getLogger("some-logger");
4298
4299 myLogger.addAppender(lttngUstLogAppender);
4300 ----
4301 --
4302
4303 . Use Apache log4j log statements and configuration as usual. The
4304   loggers with an attached LTTng-UST log appender can emit LTTng events.
4305
4306 . Before exiting the application, remove the LTTng-UST log appender from
4307   the loggers attached to it and call its `close()` method:
4308 +
4309 --
4310 [source,java]
4311 ----
4312 myLogger.removeAppender(lttngUstLogAppender);
4313 lttngUstLogAppender.close();
4314 ----
4315 --
4316 +
4317 This is not strictly necessary, but it is recommended for a clean
4318 disposal of the appender's resources.
4319
4320 . Include the LTTng-UST Java agent's common and log4j-specific JAR
4321   files, path:{lttng-ust-agent-common.jar} and
4322   path:{lttng-ust-agent-log4j.jar}, in the
4323   https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4324   path] when you build the Java application.
4325 +
4326 The JAR files are typically located in dir:{/usr/share/java}.
4327 +
4328 IMPORTANT: The LTTng-UST Java agent must be
4329 <<installing-lttng,installed>> for the logging framework your
4330 application uses.
4331
4332 .Use the LTTng-UST Java agent for Apache log4j.
4333 ====
4334 [source,java]
4335 .path:{Test.java}
4336 ----
4337 import org.apache.log4j.Appender;
4338 import org.apache.log4j.Logger;
4339 import org.lttng.ust.agent.log4j.LttngLogAppender;
4340
4341 public class Test
4342 {
4343     private static final int answer = 42;
4344
4345     public static void main(String[] argv) throws Exception
4346     {
4347         // Create a logger
4348         Logger logger = Logger.getLogger("jello");
4349
4350         // Create an LTTng-UST log appender
4351         Appender lttngUstLogAppender = new LttngLogAppender();
4352
4353         // Add the LTTng-UST log appender to our logger
4354         logger.addAppender(lttngUstLogAppender);
4355
4356         // Log at will!
4357         logger.info("some info");
4358         logger.warn("some warning");
4359         Thread.sleep(500);
4360         logger.debug("debug information; the answer is " + answer);
4361         Thread.sleep(123);
4362         logger.fatal("error!");
4363
4364         // Not mandatory, but cleaner
4365         logger.removeAppender(lttngUstLogAppender);
4366         lttngUstLogAppender.close();
4367     }
4368 }
4369
4370 ----
4371
4372 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4373 file):
4374
4375 [role="term"]
4376 ----
4377 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4378 ----
4379
4380 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4381 <<enabling-disabling-events,create an event rule>> matching the
4382 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4383
4384 [role="term"]
4385 ----
4386 $ lttng create
4387 $ lttng enable-event --log4j jello
4388 $ lttng start
4389 ----
4390
4391 Run the compiled class:
4392
4393 [role="term"]
4394 ----
4395 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4396 ----
4397
4398 <<basic-tracing-session-control,Stop tracing>> and inspect the
4399 recorded events:
4400
4401 [role="term"]
4402 ----
4403 $ lttng stop
4404 $ lttng view
4405 ----
4406 ====
4407
4408 In the resulting trace, an <<event,event record>> generated by a Java
4409 application using log4j is named `lttng_log4j:event` and
4410 has the following fields:
4411
4412 `msg`::
4413   Log record's message.
4414
4415 `logger_name`::
4416   Logger name.
4417
4418 `class_name`::
4419   Name of the class in which the log statement was executed.
4420
4421 `method_name`::
4422   Name of the method in which the log statement was executed.
4423
4424 `filename`::
4425   Name of the file in which the executed log statement is located.
4426
4427 `line_number`::
4428   Line number at which the log statement was executed.
4429
4430 `timestamp`::
4431   Logging timestamp.
4432
4433 `int_loglevel`::
4434   Log level integer value.
4435
4436 `thread_name`::
4437   Name of the Java thread in which the log statement was executed.
4438
4439 You can use the opt:lttng-enable-event(1):--loglevel or
4440 opt:lttng-enable-event(1):--loglevel-only option of the
4441 man:lttng-enable-event(1) command to target a range of Apache log4j log levels
4442 or a specific log4j log level.
4443
4444
4445 [role="since-2.8"]
4446 [[java-application-context]]
4447 ==== Provide application-specific context fields in a Java application
4448
4449 A Java application-specific context field is a piece of state provided
4450 by the application which <<adding-context,you can add>>, using the
4451 man:lttng-add-context(1) command, to each <<event,event record>>
4452 produced by the log statements of this application.
4453
4454 For example, a given object might have a current request ID variable.
4455 You can create a context information retriever for this object and
4456 assign a name to this current request ID. You can then, using the
4457 man:lttng-add-context(1) command, add this context field by name to
4458 the JUL or log4j <<channel,channel>>.
4459
4460 To provide application-specific context fields in a Java application:
4461
4462 . In the Java application's source code, import the LTTng-UST
4463   Java agent context classes and interfaces:
4464 +
4465 --
4466 [source,java]
4467 ----
4468 import org.lttng.ust.agent.context.ContextInfoManager;
4469 import org.lttng.ust.agent.context.IContextInfoRetriever;
4470 ----
4471 --
4472
4473 . Create a context information retriever class, that is, a class which
4474   implements the `IContextInfoRetriever` interface:
4475 +
4476 --
4477 [source,java]
4478 ----
4479 class MyContextInfoRetriever implements IContextInfoRetriever
4480 {
4481     @Override
4482     public Object retrieveContextInfo(String key)
4483     {
4484         if (key.equals("intCtx")) {
4485             return (short) 17;
4486         } else if (key.equals("strContext")) {
4487             return "context value!";
4488         } else {
4489             return null;
4490         }
4491     }
4492 }
4493 ----
4494 --
4495 +
4496 This `retrieveContextInfo()` method is the only member of the
4497 `IContextInfoRetriever` interface. Its role is to return the current
4498 value of a state by name to create a context field. The names of the
4499 context fields and which state variables they return depends on your
4500 specific scenario.
4501 +
4502 All primitive types and objects are supported as context fields.
4503 When `retrieveContextInfo()` returns an object, the context field
4504 serializer calls its `toString()` method to add a string field to
4505 event records. The method can also return `null`, which means that
4506 no context field is available for the required name.
4507
4508 . Register an instance of your context information retriever class to
4509   the context information manager singleton:
4510 +
4511 --
4512 [source,java]
4513 ----
4514 IContextInfoRetriever cir = new MyContextInfoRetriever();
4515 ContextInfoManager cim = ContextInfoManager.getInstance();
4516 cim.registerContextInfoRetriever("retrieverName", cir);
4517 ----
4518 --
4519
4520 . Before exiting the application, remove your context information
4521   retriever from the context information manager singleton:
4522 +
4523 --
4524 [source,java]
4525 ----
4526 ContextInfoManager cim = ContextInfoManager.getInstance();
4527 cim.unregisterContextInfoRetriever("retrieverName");
4528 ----
4529 --
4530 +
4531 This is not strictly necessary, but it is recommended for a clean
4532 disposal of some manager's resources.
4533
4534 . Build your Java application with LTTng-UST Java agent support as
4535   usual, following the procedure for either the <<jul,JUL>> or
4536   <<log4j,Apache log4j>> framework.
4537
4538
4539 .Provide application-specific context fields in a Java application.
4540 ====
4541 [source,java]
4542 .path:{Test.java}
4543 ----
4544 import java.util.logging.Handler;
4545 import java.util.logging.Logger;
4546 import org.lttng.ust.agent.jul.LttngLogHandler;
4547 import org.lttng.ust.agent.context.ContextInfoManager;
4548 import org.lttng.ust.agent.context.IContextInfoRetriever;
4549
4550 public class Test
4551 {
4552     // Our context information retriever class
4553     private static class MyContextInfoRetriever
4554     implements IContextInfoRetriever
4555     {
4556         @Override
4557         public Object retrieveContextInfo(String key) {
4558             if (key.equals("intCtx")) {
4559                 return (short) 17;
4560             } else if (key.equals("strContext")) {
4561                 return "context value!";
4562             } else {
4563                 return null;
4564             }
4565         }
4566     }
4567
4568     private static final int answer = 42;
4569
4570     public static void main(String args[]) throws Exception
4571     {
4572         // Get the context information manager instance
4573         ContextInfoManager cim = ContextInfoManager.getInstance();
4574
4575         // Create and register our context information retriever
4576         IContextInfoRetriever cir = new MyContextInfoRetriever();
4577         cim.registerContextInfoRetriever("myRetriever", cir);
4578
4579         // Create a logger
4580         Logger logger = Logger.getLogger("jello");
4581
4582         // Create an LTTng-UST log handler
4583         Handler lttngUstLogHandler = new LttngLogHandler();
4584
4585         // Add the LTTng-UST log handler to our logger
4586         logger.addHandler(lttngUstLogHandler);
4587
4588         // Log at will!
4589         logger.info("some info");
4590         logger.warning("some warning");
4591         Thread.sleep(500);
4592         logger.finer("finer information; the answer is " + answer);
4593         Thread.sleep(123);
4594         logger.severe("error!");
4595
4596         // Not mandatory, but cleaner
4597         logger.removeHandler(lttngUstLogHandler);
4598         lttngUstLogHandler.close();
4599         cim.unregisterContextInfoRetriever("myRetriever");
4600     }
4601 }
4602 ----
4603
4604 Build this example:
4605
4606 [role="term"]
4607 ----
4608 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4609 ----
4610
4611 <<creating-destroying-tracing-sessions,Create a tracing session>>
4612 and <<enabling-disabling-events,create an event rule>> matching the
4613 `jello` JUL logger:
4614
4615 [role="term"]
4616 ----
4617 $ lttng create
4618 $ lttng enable-event --jul jello
4619 ----
4620
4621 <<adding-context,Add the application-specific context fields>> to the
4622 JUL channel:
4623
4624 [role="term"]
4625 ----
4626 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
4627 $ lttng add-context --jul --type='$app.myRetriever:strContext'
4628 ----
4629
4630 <<basic-tracing-session-control,Start tracing>>:
4631
4632 [role="term"]
4633 ----
4634 $ lttng start
4635 ----
4636
4637 Run the compiled class:
4638
4639 [role="term"]
4640 ----
4641 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4642 ----
4643
4644 <<basic-tracing-session-control,Stop tracing>> and inspect the
4645 recorded events:
4646
4647 [role="term"]
4648 ----
4649 $ lttng stop
4650 $ lttng view
4651 ----
4652 ====
4653
4654
4655 [role="since-2.7"]
4656 [[python-application]]
4657 === User space Python agent
4658
4659 You can instrument a Python 2 or Python 3 application which uses the
4660 standard https://docs.python.org/3/library/logging.html[`logging`]
4661 package.
4662
4663 Each log statement emits an LTTng event once the
4664 application module imports the
4665 <<lttng-ust-agents,LTTng-UST Python agent>> package.
4666
4667 [role="img-100"]
4668 .A Python application importing the LTTng-UST Python agent.
4669 image::python-app.png[]
4670
4671 To use the LTTng-UST Python agent:
4672
4673 . In the Python application's source code, import the LTTng-UST Python
4674   agent:
4675 +
4676 --
4677 [source,python]
4678 ----
4679 import lttngust
4680 ----
4681 --
4682 +
4683 The LTTng-UST Python agent automatically adds its logging handler to the
4684 root logger at import time.
4685 +
4686 Any log statement that the application executes before this import does
4687 not emit an LTTng event.
4688 +
4689 IMPORTANT: The LTTng-UST Python agent must be
4690 <<installing-lttng,installed>>.
4691
4692 . Use log statements and logging configuration as usual.
4693   Since the LTTng-UST Python agent adds a handler to the _root_
4694   logger, you can trace any log statement from any logger.
4695
4696 .Use the LTTng-UST Python agent.
4697 ====
4698 [source,python]
4699 .path:{test.py}
4700 ----
4701 import lttngust
4702 import logging
4703 import time
4704
4705
4706 def example():
4707     logging.basicConfig()
4708     logger = logging.getLogger('my-logger')
4709
4710     while True:
4711         logger.debug('debug message')
4712         logger.info('info message')
4713         logger.warn('warn message')
4714         logger.error('error message')
4715         logger.critical('critical message')
4716         time.sleep(1)
4717
4718
4719 if __name__ == '__main__':
4720     example()
4721 ----
4722
4723 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
4724 logging handler which prints to the standard error stream, is not
4725 strictly required for LTTng-UST tracing to work, but in versions of
4726 Python preceding 3.2, you could see a warning message which indicates
4727 that no handler exists for the logger `my-logger`.
4728
4729 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4730 <<enabling-disabling-events,create an event rule>> matching the
4731 `my-logger` Python logger, and <<basic-tracing-session-control,start
4732 tracing>>:
4733
4734 [role="term"]
4735 ----
4736 $ lttng create
4737 $ lttng enable-event --python my-logger
4738 $ lttng start
4739 ----
4740
4741 Run the Python script:
4742
4743 [role="term"]
4744 ----
4745 $ python test.py
4746 ----
4747
4748 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4749 events:
4750
4751 [role="term"]
4752 ----
4753 $ lttng stop
4754 $ lttng view
4755 ----
4756 ====
4757
4758 In the resulting trace, an <<event,event record>> generated by a Python
4759 application is named `lttng_python:event` and has the following fields:
4760
4761 `asctime`::
4762   Logging time (string).
4763
4764 `msg`::
4765   Log record's message.
4766
4767 `logger_name`::
4768   Logger name.
4769
4770 `funcName`::
4771   Name of the function in which the log statement was executed.
4772
4773 `lineno`::
4774   Line number at which the log statement was executed.
4775
4776 `int_loglevel`::
4777   Log level integer value.
4778
4779 `thread`::
4780   ID of the Python thread in which the log statement was executed.
4781
4782 `threadName`::
4783   Name of the Python thread in which the log statement was executed.
4784
4785 You can use the opt:lttng-enable-event(1):--loglevel or
4786 opt:lttng-enable-event(1):--loglevel-only option of the
4787 man:lttng-enable-event(1) command to target a range of Python log levels
4788 or a specific Python log level.
4789
4790 When an application imports the LTTng-UST Python agent, the agent tries
4791 to register to a <<lttng-sessiond,session daemon>>. Note that you must
4792 <<start-sessiond,start the session daemon>> _before_ you run the Python
4793 application. If a session daemon is found, the agent tries to register
4794 to it during 5{nbsp}seconds, after which the application continues
4795 without LTTng tracing support. You can override this timeout value with
4796 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
4797 (milliseconds).
4798
4799 If the session daemon stops while a Python application with an imported
4800 LTTng-UST Python agent runs, the agent retries to connect and to
4801 register to a session daemon every 3{nbsp}seconds. You can override this
4802 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
4803 variable.
4804
4805
4806 [role="since-2.5"]
4807 [[proc-lttng-logger-abi]]
4808 === LTTng logger
4809
4810 The `lttng-tracer` Linux kernel module, part of
4811 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger file
4812 path:{/proc/lttng-logger} when it's loaded. Any application can write
4813 text data to this file to emit an LTTng event.
4814
4815 [role="img-100"]
4816 .An application writes to the LTTng logger file to emit an LTTng event.
4817 image::lttng-logger.png[]
4818
4819 The LTTng logger is the quickest method--not the most efficient,
4820 however--to add instrumentation to an application. It is designed
4821 mostly to instrument shell scripts:
4822
4823 [role="term"]
4824 ----
4825 $ echo "Some message, some $variable" > /proc/lttng-logger
4826 ----
4827
4828 Any event that the LTTng logger emits is named `lttng_logger` and
4829 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
4830 other instrumentation points in the kernel tracing domain, **any Unix
4831 user** can <<enabling-disabling-events,create an event rule>> which
4832 matches its event name, not only the root user or users in the
4833 <<tracing-group,tracing group>>.
4834
4835 To use the LTTng logger:
4836
4837 * From any application, write text data to the path:{/proc/lttng-logger}
4838   file.
4839
4840 The `msg` field of `lttng_logger` event records contains the
4841 recorded message.
4842
4843 NOTE: The maximum message length of an LTTng logger event is
4844 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
4845 than one event to contain the remaining data.
4846
4847 You should not use the LTTng logger to trace a user application which
4848 can be instrumented in a more efficient way, namely:
4849
4850 * <<c-application,C and $$C++$$ applications>>.
4851 * <<java-application,Java applications>>.
4852 * <<python-application,Python applications>>.
4853
4854 .Use the LTTng logger.
4855 ====
4856 [source,bash]
4857 .path:{test.bash}
4858 ----
4859 echo 'Hello, World!' > /proc/lttng-logger
4860 sleep 2
4861 df --human-readable --print-type / > /proc/lttng-logger
4862 ----
4863
4864 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4865 <<enabling-disabling-events,create an event rule>> matching the
4866 `lttng_logger` Linux kernel tracepoint, and
4867 <<basic-tracing-session-control,start tracing>>:
4868
4869 [role="term"]
4870 ----
4871 $ lttng create
4872 $ lttng enable-event --kernel lttng_logger
4873 $ lttng start
4874 ----
4875
4876 Run the Bash script:
4877
4878 [role="term"]
4879 ----
4880 $ bash test.bash
4881 ----
4882
4883 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4884 events:
4885
4886 [role="term"]
4887 ----
4888 $ lttng stop
4889 $ lttng view
4890 ----
4891 ====
4892
4893
4894 [[instrumenting-linux-kernel]]
4895 === LTTng kernel tracepoints
4896
4897 NOTE: This section shows how to _add_ instrumentation points to the
4898 Linux kernel. The kernel's subsystems are already thoroughly
4899 instrumented at strategic places for LTTng when you
4900 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
4901 package.
4902
4903 ////
4904 There are two methods to instrument the Linux kernel:
4905
4906 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
4907   tracepoint which uses the `TRACE_EVENT()` API.
4908 +
4909 Choose this if you want to instrumentation a Linux kernel tree with an
4910 instrumentation point compatible with ftrace, perf, and SystemTap.
4911
4912 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
4913   instrument an out-of-tree kernel module.
4914 +
4915 Choose this if you don't need ftrace, perf, or SystemTap support.
4916 ////
4917
4918
4919 [[linux-add-lttng-layer]]
4920 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
4921
4922 This section shows how to add an LTTng layer to existing ftrace
4923 instrumentation using the `TRACE_EVENT()` API.
4924
4925 This section does not document the `TRACE_EVENT()` macro. You can
4926 read the following articles to learn more about this API:
4927
4928 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part 1)]
4929 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part 2)]
4930 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part 3)]
4931
4932 The following procedure assumes that your ftrace tracepoints are
4933 correctly defined in their own header and that they are created in
4934 one source file using the `CREATE_TRACE_POINTS` definition.
4935
4936 To add an LTTng layer over an existing ftrace tracepoint:
4937
4938 . Make sure the following kernel configuration options are
4939   enabled:
4940 +
4941 --
4942 * `CONFIG_MODULES`
4943 * `CONFIG_KALLSYMS`
4944 * `CONFIG_HIGH_RES_TIMERS`
4945 * `CONFIG_TRACEPOINTS`
4946 --
4947
4948 . Build the Linux source tree with your custom ftrace tracepoints.
4949 . Boot the resulting Linux image on your target system.
4950 +
4951 Confirm that the tracepoints exist by looking for their names in the
4952 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
4953 is your subsystem's name.
4954
4955 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
4956 +
4957 --
4958 [role="term"]
4959 ----
4960 $ cd $(mktemp -d) &&
4961 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
4962 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
4963 cd lttng-modules-2.8.*
4964 ----
4965 --
4966
4967 . In dir:{instrumentation/events/lttng-module}, relative to the root
4968   of the LTTng-modules source tree, create a header file named
4969   +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
4970   LTTng-modules tracepoint definitions using the LTTng-modules
4971   macros in it.
4972 +
4973 Start with this template:
4974 +
4975 --
4976 [source,c]
4977 .path:{instrumentation/events/lttng-module/my_subsys.h}
4978 ----
4979 #undef TRACE_SYSTEM
4980 #define TRACE_SYSTEM my_subsys
4981
4982 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
4983 #define _LTTNG_MY_SUBSYS_H
4984
4985 #include "../../../probes/lttng-tracepoint-event.h"
4986 #include <linux/tracepoint.h>
4987
4988 LTTNG_TRACEPOINT_EVENT(
4989     /*
4990      * Format is identical to TRACE_EVENT()'s version for the three
4991      * following macro parameters:
4992      */
4993     my_subsys_my_event,
4994     TP_PROTO(int my_int, const char *my_string),
4995     TP_ARGS(my_int, my_string),
4996
4997     /* LTTng-modules specific macros */
4998     TP_FIELDS(
4999         ctf_integer(int, my_int_field, my_int)
5000         ctf_string(my_bar_field, my_bar)
5001     )
5002 )
5003
5004 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5005
5006 #include "../../../probes/define_trace.h"
5007 ----
5008 --
5009 +
5010 The entries in the `TP_FIELDS()` section are the list of fields for the
5011 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5012 ftrace's `TRACE_EVENT()` macro.
5013 +
5014 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5015 complete description of the available `ctf_*()` macros.
5016
5017 . Create the LTTng-modules probe's kernel module C source file,
5018   +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5019   subsystem name:
5020 +
5021 --
5022 [source,c]
5023 .path:{probes/lttng-probe-my-subsys.c}
5024 ----
5025 #include <linux/module.h>
5026 #include "../lttng-tracer.h"
5027
5028 /*
5029  * Build-time verification of mismatch between mainline
5030  * TRACE_EVENT() arguments and the LTTng-modules adaptation
5031  * layer LTTNG_TRACEPOINT_EVENT() arguments.
5032  */
5033 #include <trace/events/my_subsys.h>
5034
5035 /* Create LTTng tracepoint probes */
5036 #define LTTNG_PACKAGE_BUILD
5037 #define CREATE_TRACE_POINTS
5038 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5039
5040 #include "../instrumentation/events/lttng-module/my_subsys.h"
5041
5042 MODULE_LICENSE("GPL and additional rights");
5043 MODULE_AUTHOR("Your name <your-email>");
5044 MODULE_DESCRIPTION("LTTng my_subsys probes");
5045 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5046     __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5047     __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5048     LTTNG_MODULES_EXTRAVERSION);
5049 ----
5050 --
5051
5052 . Edit path:{probes/KBuild} and add your new kernel module object
5053   next to the existing ones:
5054 +
5055 --
5056 [source,make]
5057 .path:{probes/KBuild}
5058 ----
5059 # ...
5060
5061 obj-m += lttng-probe-module.o
5062 obj-m += lttng-probe-power.o
5063
5064 obj-m += lttng-probe-my-subsys.o
5065
5066 # ...
5067 ----
5068 --
5069
5070 . Build and install the LTTng kernel modules:
5071 +
5072 --
5073 [role="term"]
5074 ----
5075 $ make KERNELDIR=/path/to/linux
5076 # make modules_install && depmod -a
5077 ----
5078 --
5079 +
5080 Replace `/path/to/linux` with the path to the Linux source tree where
5081 you defined and used tracepoints with ftrace's `TRACE_EVENT()` macro.
5082
5083 Note that you can also use the
5084 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5085 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5086 C code that need to be executed before the event fields are recorded.
5087
5088 The best way to learn how to use the previous LTTng-modules macros is to
5089 inspect the existing LTTng-modules tracepoint definitions in the
5090 dir:{instrumentation/events/lttng-module} header files. Compare them
5091 with the Linux kernel mainline versions in the
5092 dir:{include/trace/events} directory of the Linux source tree.
5093
5094
5095 [role="since-2.7"]
5096 [[lttng-tracepoint-event-code]]
5097 ===== Use custom C code to access the data for tracepoint fields
5098
5099 Although we recommended to always use the
5100 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5101 the arguments and fields of an LTTng-modules tracepoint when possible,
5102 sometimes you need a more complex process to access the data that the
5103 tracer records as event record fields. In other words, you need local
5104 variables and multiple C{nbsp}statements instead of simple
5105 argument-based expressions that you pass to the
5106 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5107
5108 You can use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5109 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5110 a block of C{nbsp}code to be executed before LTTng records the fields.
5111 The structure of this macro is:
5112
5113 [source,c]
5114 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5115 ----
5116 LTTNG_TRACEPOINT_EVENT_CODE(
5117     /*
5118      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5119      * version for the following three macro parameters:
5120      */
5121     my_subsys_my_event,
5122     TP_PROTO(int my_int, const char *my_string),
5123     TP_ARGS(my_int, my_string),
5124
5125     /* Declarations of custom local variables */
5126     TP_locvar(
5127         int a = 0;
5128         unsigned long b = 0;
5129         const char *name = "(undefined)";
5130         struct my_struct *my_struct;
5131     ),
5132
5133     /*
5134      * Custom code which uses both tracepoint arguments
5135      * (in TP_ARGS()) and local variables (in TP_locvar()).
5136      *
5137      * Local variables are actually members of a structure pointed
5138      * to by the special variable tp_locvar.
5139      */
5140     TP_code(
5141         if (my_int) {
5142             tp_locvar->a = my_int + 17;
5143             tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5144             tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5145             tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5146             put_my_struct(tp_locvar->my_struct);
5147
5148             if (tp_locvar->b) {
5149                 tp_locvar->a = 1;
5150             }
5151         }
5152     ),
5153
5154     /*
5155      * Format identical to the LTTNG_TRACEPOINT_EVENT()
5156      * version for this, except that tp_locvar members can be
5157      * used in the argument expression parameters of
5158      * the ctf_*() macros.
5159      */
5160     TP_FIELDS(
5161         ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5162         ctf_integer(int, my_struct_a, tp_locvar->a)
5163         ctf_string(my_string_field, my_string)
5164         ctf_string(my_struct_name, tp_locvar->name)
5165     )
5166 )
5167 ----
5168
5169 IMPORTANT: The C code defined in `TP_code()` must not have any side
5170 effects when executed. In particular, the code must not allocate
5171 memory or get resources without deallocating this memory or putting
5172 those resources afterwards.
5173
5174
5175 [[instrumenting-linux-kernel-tracing]]
5176 ==== Load and unload a custom probe kernel module
5177
5178 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5179 kernel module>> in the kernel before it can emit LTTng events.
5180
5181 To load the default probe kernel modules and a custom probe kernel
5182 module:
5183
5184 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5185   probe modules to load when starting a root <<lttng-sessiond,session
5186   daemon>>:
5187 +
5188 --
5189 .Load the `my_subsys`, `usb`, and the default probe modules.
5190 ====
5191 [role="term"]
5192 ----
5193 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5194 ----
5195 ====
5196 --
5197 +
5198 You only need to pass the subsystem name, not the whole kernel module
5199 name.
5200
5201 To load _only_ a given custom probe kernel module:
5202
5203 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5204   modules to load when starting a root session daemon:
5205 +
5206 --
5207 .Load only the `my_subsys` and `usb` probe modules.
5208 ====
5209 [role="term"]
5210 ----
5211 # lttng-sessiond --kmod-probes=my_subsys,usb
5212 ----
5213 ====
5214 --
5215
5216 To confirm that a probe module is loaded:
5217
5218 * Use man:lsmod(8):
5219 +
5220 --
5221 [role="term"]
5222 ----
5223 $ lsmod | grep lttng_probe_usb
5224 ----
5225 --
5226
5227 To unload the loaded probe modules:
5228
5229 * Kill the session daemon with `SIGTERM`:
5230 +
5231 --
5232 [role="term"]
5233 ----
5234 # pkill lttng-sessiond
5235 ----
5236 --
5237 +
5238 You can also use man:modprobe(8)'s `--remove` option if the session
5239 daemon terminates abnormally.
5240
5241
5242 [[controlling-tracing]]
5243 == Tracing control
5244
5245 Once an application or a Linux kernel is
5246 <<instrumenting,instrumented>> for LTTng tracing,
5247 you can _trace_ it.
5248
5249 This section is divided in topics on how to use the various
5250 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5251 command-line tool>>, to _control_ the LTTng daemons and tracers.
5252
5253 NOTE: In the following subsections, we refer to an man:lttng(1) command
5254 using its man page name. For example, instead of _Run the `create`
5255 command to..._, we use _Run the man:lttng-create(1) command to..._.
5256
5257
5258 [[start-sessiond]]
5259 === Start a session daemon
5260
5261 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5262 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5263 command-line tool.
5264
5265 You will see the following error when you run a command while no session
5266 daemon is running:
5267
5268 ----
5269 Error: No session daemon is available
5270 ----
5271
5272 The only command that automatically runs a session daemon is
5273 man:lttng-create(1), which you use to
5274 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5275 this is most of the time the first operation that you do, sometimes it's
5276 not. Some examples are:
5277
5278 * <<list-instrumentation-points,List the available instrumentation points>>.
5279 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5280
5281 [[tracing-group]] Each Unix user must have its own running session
5282 daemon to trace user applications. The session daemon that the root user
5283 starts is the only one allowed to control the LTTng kernel tracer. Users
5284 that are part of the _tracing group_ can control the root session
5285 daemon. The default tracing group name is `tracing`; you can set it to
5286 something else with the opt:lttng-sessiond(8):--group option when you
5287 start the root session daemon.
5288
5289 To start a user session daemon:
5290
5291 * Run man:lttng-sessiond(8):
5292 +
5293 --
5294 [role="term"]
5295 ----
5296 $ lttng-sessiond --daemonize
5297 ----
5298 --
5299
5300 To start the root session daemon:
5301
5302 * Run man:lttng-sessiond(8) as the root user:
5303 +
5304 --
5305 [role="term"]
5306 ----
5307 # lttng-sessiond --daemonize
5308 ----
5309 --
5310
5311 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5312 start the session daemon in foreground.
5313
5314 To stop a session daemon, use man:kill(1) on its process ID (standard
5315 `TERM` signal).
5316
5317 Note that some Linux distributions could manage the LTTng session daemon
5318 as a service. In this case, you should use the service manager to
5319 start, restart, and stop session daemons.
5320
5321
5322 [[creating-destroying-tracing-sessions]]
5323 === Create and destroy a tracing session
5324
5325 Almost all the LTTng control operations happen in the scope of
5326 a <<tracing-session,tracing session>>, which is the dialogue between the
5327 <<lttng-sessiond,session daemon>> and you.
5328
5329 To create a tracing session with a generated name:
5330
5331 * Use the man:lttng-create(1) command:
5332 +
5333 --
5334 [role="term"]
5335 ----
5336 $ lttng create
5337 ----
5338 --
5339
5340 The created tracing session's name is `auto` followed by the
5341 creation date.
5342
5343 To create a tracing session with a specific name:
5344
5345 * Use the optional argument of the man:lttng-create(1) command:
5346 +
5347 --
5348 [role="term"]
5349 ----
5350 $ lttng create my-session
5351 ----
5352 --
5353 +
5354 Replace `my-session` with the specific tracing session name.
5355
5356 LTTng appends the creation date to the created tracing session's name.
5357
5358 LTTng writes the traces of a tracing session in
5359 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5360 name of the tracing session. Note that the env:LTTNG_HOME environment
5361 variable defaults to `$HOME` if not set.
5362
5363 To output LTTng traces to a non-default location:
5364
5365 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5366 +
5367 --
5368 [role="term"]
5369 ----
5370 $ lttng create my-session --output=/tmp/some-directory
5371 ----
5372 --
5373
5374 You may create as many tracing sessions as you wish.
5375
5376 To list all the existing tracing sessions for your Unix user:
5377
5378 * Use the man:lttng-list(1) command:
5379 +
5380 --
5381 [role="term"]
5382 ----
5383 $ lttng list
5384 ----
5385 --
5386
5387 When you create a tracing session, it is set as the _current tracing
5388 session_. The following man:lttng(1) commands operate on the current
5389 tracing session when you don't specify one:
5390
5391 [role="list-3-cols"]
5392 * `add-context`
5393 * `destroy`
5394 * `disable-channel`
5395 * `disable-event`
5396 * `enable-channel`
5397 * `enable-event`
5398 * `load`
5399 * `save`
5400 * `snapshot`
5401 * `start`
5402 * `stop`
5403 * `track`
5404 * `untrack`
5405 * `view`
5406
5407 To change the current tracing session:
5408
5409 * Use the man:lttng-set-session(1) command:
5410 +
5411 --
5412 [role="term"]
5413 ----
5414 $ lttng set-session new-session
5415 ----
5416 --
5417 +
5418 Replace `new-session` by the name of the new current tracing session.
5419
5420 When you are done tracing in a given tracing session, you can destroy
5421 it. This operation frees the resources taken by the tracing session
5422 to destroy; it does not destroy the trace data that LTTng wrote for
5423 this tracing session.
5424
5425 To destroy the current tracing session:
5426
5427 * Use the man:lttng-destroy(1) command:
5428 +
5429 --
5430 [role="term"]
5431 ----
5432 $ lttng destroy
5433 ----
5434 --
5435
5436
5437 [[list-instrumentation-points]]
5438 === List the available instrumentation points
5439
5440 The <<lttng-sessiond,session daemon>> can query the running instrumented
5441 user applications and the Linux kernel to get a list of available
5442 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5443 they are tracepoints and system calls. For the user space tracing
5444 domain, they are tracepoints. For the other tracing domains, they are
5445 logger names.
5446
5447 To list the available instrumentation points:
5448
5449 * Use the man:lttng-list(1) command with the requested tracing domain's
5450   option amongst:
5451 +
5452 --
5453 * opt:lttng-list(1):--kernel: Linux kernel tracepoints (your Unix user
5454   must be a root user, or it must be a member of the
5455   <<tracing-group,tracing group>>).
5456 * opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall: Linux
5457   kernel system calls (your Unix user must be a root user, or it must be
5458   a member of the tracing group).
5459 * opt:lttng-list(1):--userspace: user space tracepoints.
5460 * opt:lttng-list(1):--jul: `java.util.logging` loggers.
5461 * opt:lttng-list(1):--log4j: Apache log4j loggers.
5462 * opt:lttng-list(1):--python: Python loggers.
5463 --
5464
5465 .List the available user space tracepoints.
5466 ====
5467 [role="term"]
5468 ----
5469 $ lttng list --userspace
5470 ----
5471 ====
5472
5473 .List the available Linux kernel system call tracepoints.
5474 ====
5475 [role="term"]
5476 ----
5477 $ lttng list --kernel --syscall
5478 ----
5479 ====
5480
5481
5482 [[enabling-disabling-events]]
5483 === Create and enable an event rule
5484
5485 Once you <<creating-destroying-tracing-sessions,create a tracing
5486 session>>, you can create <<event,event rules>> with the
5487 man:lttng-enable-event(1) command.
5488
5489 You specify each condition with a command-line option. The available
5490 condition options are shown in the following table.
5491
5492 [role="growable",cols="asciidoc,asciidoc,default"]
5493 .Condition command-line options for the man:lttng-enable-event(1) command.
5494 |====
5495 |Option |Description |Applicable tracing domains
5496
5497 |
5498 One of:
5499
5500 . `--syscall`
5501 . +--probe=__ADDR__+
5502 . +--function=__ADDR__+
5503
5504 |
5505 Instead of using the default _tracepoint_ instrumentation type, use:
5506
5507 . A Linux system call.
5508 . A Linux https://lwn.net/Articles/132196/[KProbe] (symbol or address).
5509 . The entry and return points of a Linux function (symbol or address).
5510
5511 |Linux kernel.
5512
5513 |First positional argument.
5514
5515 |
5516 Tracepoint or system call name. In the case of a Linux KProbe or
5517 function, this is a custom name given to the event rule. With the
5518 JUL, log4j, and Python domains, this is a logger name.
5519
5520 With a tracepoint, logger, or system call name, the last character
5521 can be `*` to match anything that remains.
5522
5523 |All.
5524
5525 |
5526 One of:
5527
5528 . +--loglevel=__LEVEL__+
5529 . +--loglevel-only=__LEVEL__+
5530
5531 |
5532 . Match only tracepoints or log statements with a logging level at
5533   least as severe as +__LEVEL__+.
5534 . Match only tracepoints or log statements with a logging level
5535   equal to +__LEVEL__+.
5536
5537 See man:lttng-enable-event(1) for the list of available logging level
5538 names.
5539
5540 |User space, JUL, log4j, and Python.
5541
5542 |+--exclude=__EXCLUSIONS__+
5543
5544 |
5545 When you use a `*` character at the end of the tracepoint or logger
5546 name (first positional argument), exclude the specific names in the
5547 comma-delimited list +__EXCLUSIONS__+.
5548
5549 |
5550 User space, JUL, log4j, and Python.
5551
5552 |+--filter=__EXPR__+
5553
5554 |
5555 Match only events which satisfy the expression +__EXPR__+.
5556
5557 See man:lttng-enable-event(1) to learn more about the syntax of a
5558 filter expression.
5559
5560 |All.
5561
5562 |====
5563
5564 You attach an event rule to a <<channel,channel>> on creation. If you do
5565 not specify the channel with the opt:lttng-enable-event(1):--channel
5566 option, and if the event rule to create is the first in its
5567 <<domain,tracing domain>> for a given tracing session, then LTTng
5568 creates a _default channel_ for you. This default channel is reused in
5569 subsequent invocations of the man:lttng-enable-event(1) command for the
5570 same tracing domain.
5571
5572 An event rule is always enabled at creation time.
5573
5574 The following examples show how you can combine the previous
5575 command-line options to create simple to more complex event rules.
5576
5577 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5578 ====
5579 [role="term"]
5580 ----
5581 $ lttng enable-event --kernel sched_switch
5582 ----
5583 ====
5584
5585 .Create an event rule matching four Linux kernel system calls (default channel).
5586 ====
5587 [role="term"]
5588 ----
5589 $ lttng enable-event --kernel --syscall open,write,read,close
5590 ----
5591 ====
5592
5593 .Create event rules matching tracepoints with filter expressions (default channel).
5594 ====
5595 [role="term"]
5596 ----
5597 $ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5598 ----
5599
5600 [role="term"]
5601 ----
5602 $ lttng enable-event --kernel --all \
5603                    --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5604 ----
5605
5606 [role="term"]
5607 ----
5608 $ lttng enable-event --jul my_logger \
5609                    --filter='$app.retriever:cur_msg_id > 3'
5610 ----
5611
5612 IMPORTANT: Make sure to always quote the filter string when you
5613 use man:lttng(1) from a shell.
5614 ====
5615
5616 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5617 ====
5618 [role="term"]
5619 ----
5620 $ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5621 ----
5622
5623 IMPORTANT: Make sure to always quote the wildcard character when you
5624 use man:lttng(1) from a shell.
5625 ====
5626
5627 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5628 ====
5629 [role="term"]
5630 ----
5631 $ lttng enable-event --python my-app.'*' \
5632                    --exclude='my-app.module,my-app.hello'
5633 ----
5634 ====
5635
5636 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5637 ====
5638 [role="term"]
5639 ----
5640 $ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5641 ----
5642 ====
5643
5644 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5645 ====
5646 [role="term"]
5647 ----
5648 $ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5649 ----
5650 ====
5651
5652 The event rules of a given channel form a whitelist: as soon as an
5653 emitted event passes one of them, LTTng can record the event. For
5654 example, an event named `my_app:my_tracepoint` emitted from a user space
5655 tracepoint with a `TRACE_ERROR` log level passes both of the following
5656 rules:
5657
5658 [role="term"]
5659 ----
5660 $ lttng enable-event --userspace my_app:my_tracepoint
5661 $ lttng enable-event --userspace my_app:my_tracepoint \
5662                    --loglevel=TRACE_INFO
5663 ----
5664
5665 The second event rule is redundant: the first one includes
5666 the second one.
5667
5668
5669 [[disable-event-rule]]
5670 === Disable an event rule
5671
5672 To disable an event rule that you <<enabling-disabling-events,created>>
5673 previously, use the man:lttng-disable-event(1) command. This command
5674 disables _all_ the event rules (of a given tracing domain and channel)
5675 which match an instrumentation point. The other conditions are not
5676 supported as of LTTng{nbsp}{revision}.
5677
5678 The LTTng tracer does not record an emitted event which passes
5679 a _disabled_ event rule.
5680
5681 .Disable an event rule matching a Python logger (default channel).
5682 ====
5683 [role="term"]
5684 ----
5685 $ lttng disable-event --python my-logger
5686 ----
5687 ====
5688
5689 .Disable an event rule matching all `java.util.logging` loggers (default channel).
5690 ====
5691 [role="term"]
5692 ----
5693 $ lttng disable-event --jul '*'
5694 ----
5695 ====
5696
5697 .Disable _all_ the event rules of the default channel.
5698 ====
5699 The opt:lttng-disable-event(1):--all-events option is not, like the
5700 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
5701 equivalent of the event name `*` (wildcard): it disables _all_ the event
5702 rules of a given channel.
5703
5704 [role="term"]
5705 ----
5706 $ lttng disable-event --jul --all-events
5707 ----
5708 ====
5709
5710 NOTE: You cannot delete an event rule once you create it.
5711
5712
5713 [[status]]
5714 === Get the status of a tracing session
5715
5716 To get the status of the current tracing session, that is, its
5717 parameters, its channels, event rules, and their attributes:
5718
5719 * Use the man:lttng-status(1) command:
5720 +
5721 --
5722 [role="term"]
5723 ----
5724 $ lttng status
5725 ----
5726 --
5727 +
5728
5729 To get the status of any tracing session:
5730
5731 * Use the man:lttng-list(1) command with the tracing session's name:
5732 +
5733 --
5734 [role="term"]
5735 ----
5736 $ lttng list my-session
5737 ----
5738 --
5739 +
5740 Replace `my-session` with the desired tracing session's name.
5741
5742
5743 [[basic-tracing-session-control]]
5744 === Start and stop a tracing session
5745
5746 Once you <<creating-destroying-tracing-sessions,create a tracing
5747 session>> and
5748 <<enabling-disabling-events,create one or more event rules>>,
5749 you can start and stop the tracers for this tracing session.
5750
5751 To start tracing in the current tracing session:
5752
5753 * Use the man:lttng-start(1) command:
5754 +
5755 --
5756 [role="term"]
5757 ----
5758 $ lttng start
5759 ----
5760 --
5761
5762 LTTng is very flexible: you can launch user applications before
5763 or after the you start the tracers. The tracers only record the events
5764 if they pass enabled event rules and if they occur while the tracers are
5765 started.
5766
5767 To stop tracing in the current tracing session:
5768
5769 * Use the man:lttng-stop(1) command:
5770 +
5771 --
5772 [role="term"]
5773 ----
5774 $ lttng stop
5775 ----
5776 --
5777 +
5778 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
5779 records>> or lost sub-buffers since the last time you ran
5780 man:lttng-start(1), warnings are printed when you run the
5781 man:lttng-stop(1) command.
5782
5783
5784 [[enabling-disabling-channels]]
5785 === Create a channel
5786
5787 Once you create a tracing session, you can create a <<channel,channel>>
5788 with the man:lttng-enable-channel(1) command.
5789
5790 Note that LTTng automatically creates a default channel when, for a
5791 given <<domain,tracing domain>>, no channels exist and you
5792 <<enabling-disabling-events,create>> the first event rule. This default
5793 channel is named `channel0` and its attributes are set to reasonable
5794 values. Therefore, you only need to create a channel when you need
5795 non-default attributes.
5796
5797 You specify each non-default channel attribute with a command-line
5798 option when you use the man:lttng-enable-channel(1) command. The
5799 available command-line options are:
5800
5801 [role="growable",cols="asciidoc,asciidoc"]
5802 .Command-line options for the man:lttng-enable-channel(1) command.
5803 |====
5804 |Option |Description
5805
5806 |`--overwrite`
5807
5808 |
5809 Use the _overwrite_
5810 <<channel-overwrite-mode-vs-discard-mode,event loss mode>> instead of
5811 the default _discard_ mode.
5812
5813 |`--buffers-pid` (user space tracing domain only)
5814
5815 |
5816 Use the per-process <<channel-buffering-schemes,buffering scheme>>
5817 instead of the default per-user buffering scheme.
5818
5819 |+--subbuf-size=__SIZE__+
5820
5821 |
5822 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
5823 either for each Unix user (default), or for each instrumented process.
5824
5825 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
5826
5827 |+--num-subbuf=__COUNT__+
5828
5829 |
5830 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
5831 for each Unix user (default), or for each instrumented process.
5832
5833 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
5834
5835 |+--tracefile-size=__SIZE__+
5836
5837 |
5838 Set the maximum size of each trace file that this channel writes within
5839 a stream to +__SIZE__+ bytes instead of no maximum.
5840
5841 See <<tracefile-rotation,Trace file count and size>>.
5842
5843 |+--tracefile-count=__COUNT__+
5844
5845 |
5846 Limit the number of trace files that this channel creates to
5847 +__COUNT__+ channels instead of no limit.
5848
5849 See <<tracefile-rotation,Trace file count and size>>.
5850
5851 |+--switch-timer=__PERIODUS__+
5852
5853 |
5854 Set the <<channel-switch-timer,switch timer period>>
5855 to +__PERIODUS__+{nbsp}µs.
5856
5857 |+--read-timer=__PERIODUS__+
5858
5859 |
5860 Set the <<channel-read-timer,read timer period>>
5861 to +__PERIODUS__+{nbsp}µs.
5862
5863 |+--output=__TYPE__+ (Linux kernel tracing domain only)
5864
5865 |
5866 Set the channel's output type to +__TYPE__+, either `mmap` or `splice`.
5867
5868 |====
5869
5870 You can only create a channel in the Linux kernel and user space
5871 <<domain,tracing domains>>: other tracing domains have their own channel
5872 created on the fly when <<enabling-disabling-events,creating event
5873 rules>>.
5874
5875 [IMPORTANT]
5876 ====
5877 Because of a current LTTng limitation, you must create all channels
5878 _before_ you <<basic-tracing-session-control,start tracing>> in a given
5879 tracing session, that is, before the first time you run
5880 man:lttng-start(1).
5881
5882 Since LTTng automatically creates a default channel when you use the
5883 man:lttng-enable-event(1) command with a specific tracing domain, you
5884 cannot, for example, create a Linux kernel event rule, start tracing,
5885 and then create a user space event rule, because no user space channel
5886 exists yet and it's too late to create one.
5887
5888 For this reason, make sure to configure your channels properly
5889 before starting the tracers for the first time!
5890 ====
5891
5892 The following examples show how you can combine the previous
5893 command-line options to create simple to more complex channels.
5894
5895 .Create a Linux kernel channel with default attributes.
5896 ====
5897 [role="term"]
5898 ----
5899 $ lttng enable-channel --kernel my-channel
5900 ----
5901 ====
5902
5903 .Create a user space channel with 4 sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
5904 ====
5905 [role="term"]
5906 ----
5907 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
5908                      --buffers-pid my-channel
5909 ----
5910 ====
5911
5912 .Create a Linux kernel channel which rotates 8 trace files of 4{nbsp}MiB each for each stream
5913 ====
5914 [role="term"]
5915 ----
5916 $ lttng enable-channel --kernel --tracefile-count=8 \
5917                      --tracefile-size=4194304 my-channel
5918 ----
5919 ====
5920
5921 .Create a user space channel in overwrite (or _flight recorder_) mode.
5922 ====
5923 [role="term"]
5924 ----
5925 $ lttng enable-channel --userspace --overwrite my-channel
5926 ----
5927 ====
5928
5929 You can <<enabling-disabling-events,create>> the same event rule in
5930 two different channels:
5931
5932 [role="term"]
5933 ----
5934 $ lttng enable-event --userspace --channel=my-channel app:tp
5935 $ lttng enable-event --userspace --channel=other-channel app:tp
5936 ----
5937
5938 If both channels are enabled, when a tracepoint named `app:tp` is
5939 reached, LTTng records two events, one for each channel.
5940
5941
5942 [[disable-channel]]
5943 === Disable a channel
5944
5945 To disable a specific channel that you <<enabling-disabling-channels,created>>
5946 previously, use the man:lttng-disable-channel(1) command.
5947
5948 .Disable a specific Linux kernel channel.
5949 ====
5950 [role="term"]
5951 ----
5952 $ lttng disable-channel --kernel my-channel
5953 ----
5954 ====
5955
5956 The state of a channel precedes the individual states of event rules
5957 attached to it: event rules which belong to a disabled channel, even if
5958 they are enabled, are also considered disabled.
5959
5960
5961 [[adding-context]]
5962 === Add context fields to a channel
5963
5964 Event record fields in trace files provide important information about
5965 events that occured previously, but sometimes some external context may
5966 help you solve a problem faster. Examples of context fields are:
5967
5968 * The **process ID**, **thread ID**, **process name**, and
5969   **process priority** of the thread in which the event occurs.
5970 * The **hostname** of the system on which the event occurs.
5971 * The current values of many possible **performance counters** using
5972   perf, for example:
5973 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
5974 ** Cache misses.
5975 ** Branch instructions, misses, and loads.
5976 ** CPU faults.
5977 * Any context defined at the application level (supported for the
5978   JUL and log4j <<domain,tracing domains>>).
5979
5980 To get the full list of available context fields, see
5981 `lttng add-context --list`. Some context fields are reserved for a
5982 specific <<domain,tracing domain>> (Linux kernel or user space).
5983
5984 You add context fields to <<channel,channels>>. All the events
5985 that a channel with added context fields records contain those fields.
5986
5987 To add context fields to one or all the channels of a given tracing
5988 session:
5989
5990 * Use the man:lttng-add-context(1) command.
5991
5992 .Add context fields to all the channels of the current tracing session.
5993 ====
5994 The following command line adds the virtual process identifier and
5995 the per-thread CPU cycles count fields to all the user space channels
5996 of the current tracing session.
5997
5998 [role="term"]
5999 ----
6000 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6001 ----
6002 ====
6003
6004 .Add a context field to a specific channel.
6005 ====
6006 The following command line adds the thread identifier context field
6007 to the Linux kernel channel named `my-channel` in the current
6008 tracing session.
6009
6010 [role="term"]
6011 ----
6012 $ lttng add-context --kernel --channel=my-channel --type=tid
6013 ----
6014 ====
6015
6016 .Add an application-specific context field to a specific channel.
6017 ====
6018 The following command line adds the `cur_msg_id` context field of the
6019 `retriever` context retriever for all the instrumented
6020 <<java-application,Java applications>> recording <<event,event records>>
6021 in the channel named `my-channel`:
6022
6023 [role="term"]
6024 ----
6025 $ lttng add-context --kernel --channel=my-channel \
6026                   --type='$app:retriever:cur_msg_id'
6027 ----
6028
6029 IMPORTANT: Make sure to always quote the `$` character when you
6030 use man:lttng-add-context(1) from a shell.
6031 ====
6032
6033 NOTE: You cannot remove context fields from a channel once you add it.
6034
6035
6036 [role="since-2.7"]
6037 [[pid-tracking]]
6038 === Track process IDs
6039
6040 It's often useful to allow only specific process IDs (PIDs) to emit
6041 events. For example, you may wish to record all the system calls made by
6042 a given process (à la http://linux.die.net/man/1/strace[strace]).
6043
6044 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6045 purpose. Both commands operate on a whitelist of process IDs. You _add_
6046 entries to this whitelist with the man:lttng-track(1) command and remove
6047 entries with the man:lttng-untrack(1) command. Any process which has one
6048 of the PIDs in the whitelist is allowed to emit LTTng events which pass
6049 an enabled <<event,event rule>>.
6050
6051 NOTE: The PID tracker tracks the _numeric process IDs_. Should a
6052 process with a given tracked ID exit and another process be given this
6053 ID, then the latter would also be allowed to emit events.
6054
6055 .Track and untrack process IDs.
6056 ====
6057 For the sake of the following example, assume the target system has 16
6058 possible PIDs.
6059
6060 When you
6061 <<creating-destroying-tracing-sessions,create a tracing session>>,
6062 the whitelist contains all the possible PIDs:
6063
6064 [role="img-100"]
6065 .All PIDs are tracked.
6066 image::track-all.png[]
6067
6068 When the whitelist is full and you use the man:lttng-track(1) command to
6069 specify some PIDs to track, LTTng first clears the whitelist, then it
6070 tracks the specific PIDs. After:
6071
6072 [role="term"]
6073 ----
6074 $ lttng track --pid=3,4,7,10,13
6075 ----
6076
6077 the whitelist is:
6078
6079 [role="img-100"]
6080 .PIDs 3, 4, 7, 10, and 13 are tracked.
6081 image::track-3-4-7-10-13.png[]
6082
6083 You can add more PIDs to the whitelist afterwards:
6084
6085 [role="term"]
6086 ----
6087 $ lttng track --pid=1,15,16
6088 ----
6089
6090 The result is:
6091
6092 [role="img-100"]
6093 .PIDs 1, 15, and 16 are added to the whitelist.
6094 image::track-1-3-4-7-10-13-15-16.png[]
6095
6096 The man:lttng-untrack(1) command removes entries from the PID tracker's
6097 whitelist. Given the previous example, the following command:
6098
6099 [role="term"]
6100 ----
6101 $ lttng untrack --pid=3,7,10,13
6102 ----
6103
6104 leads to this whitelist:
6105
6106 [role="img-100"]
6107 .PIDs 3, 7, 10, and 13 are removed from the whitelist.
6108 image::track-1-4-15-16.png[]
6109
6110 LTTng can track all possible PIDs again using the
6111 opt:lttng-track(1):--all option:
6112
6113 [role="term"]
6114 ----
6115 $ lttng track --pid --all
6116 ----
6117
6118 The result is, again:
6119
6120 [role="img-100"]
6121 .All PIDs are tracked.
6122 image::track-all.png[]
6123 ====
6124
6125 .Track only specific PIDs
6126 ====
6127 A very typical use case with PID tracking is to start with an empty
6128 whitelist, then <<basic-tracing-session-control,start the tracers>>, and
6129 then add PIDs manually while tracers are active. You can accomplish this
6130 by using the opt:lttng-untrack(1):--all option of the
6131 man:lttng-untrack(1) command to clear the whitelist after you
6132 <<creating-destroying-tracing-sessions,create a tracing session>>:
6133
6134 [role="term"]
6135 ----
6136 $ lttng untrack --pid --all
6137 ----
6138
6139 gives:
6140
6141 [role="img-100"]
6142 .No PIDs are tracked.
6143 image::untrack-all.png[]
6144
6145 If you trace with this whitelist configuration, the tracer records no
6146 events for this <<domain,tracing domain>> because no processes are
6147 tracked. You can use the man:lttng-track(1) command as usual to track
6148 specific PIDs, for example:
6149
6150 [role="term"]
6151 ----
6152 $ lttng track --pid=6,11
6153 ----
6154
6155 Result:
6156
6157 [role="img-100"]
6158 .PIDs 6 and 11 are tracked.
6159 image::track-6-11.png[]
6160 ====
6161
6162
6163 [role="since-2.5"]
6164 [[saving-loading-tracing-session]]
6165 === Save and load tracing session configurations
6166
6167 Configuring a <<tracing-session,tracing session>> can be long. Some of
6168 the tasks involved are:
6169
6170 * <<enabling-disabling-channels,Create channels>> with
6171   specific attributes.
6172 * <<adding-context,Add context fields>> to specific channels.
6173 * <<enabling-disabling-events,Create event rules>> with specific log
6174   level and filter conditions.
6175
6176 If you use LTTng to solve real world problems, chances are you have to
6177 record events using the same tracing session setup over and over,
6178 modifying a few variables each time in your instrumented program
6179 or environment. To avoid constant tracing session reconfiguration,
6180 the man:lttng(1) command-line tool can save and load tracing session
6181 configurations to/from XML files.
6182
6183 To save a given tracing session configuration:
6184
6185 * Use the man:lttng-save(1) command:
6186 +
6187 --
6188 [role="term"]
6189 ----
6190 $ lttng save my-session
6191 ----
6192 --
6193 +
6194 Replace `my-session` with the name of the tracing session to save.
6195
6196 LTTng saves tracing session configurations to
6197 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6198 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6199 the opt:lttng-save(1):--output-path option to change this destination
6200 directory.
6201
6202 LTTng saves all configuration parameters, for example:
6203
6204 * The tracing session name.
6205 * The trace data output path.
6206 * The channels with their state and all their attributes.
6207 * The context fields you added to channels.
6208 * The event rules with their state, log level and filter conditions.
6209
6210 To load a tracing session:
6211
6212 * Use the man:lttng-load(1) command:
6213 +
6214 --
6215 [role="term"]
6216 ----
6217 $ lttng load my-session
6218 ----
6219 --
6220 +
6221 Replace `my-session` with the name of the tracing session to load.
6222
6223 When LTTng loads a configuration, it restores your saved tracing session
6224 as if you just configured it manually.
6225
6226 See man:lttng(1) for the complete list of command-line options. You
6227 can also save and load all many sessions at a time, and decide in which
6228 directory to output the XML files.
6229
6230
6231 [[sending-trace-data-over-the-network]]
6232 === Send trace data over the network
6233
6234 LTTng can send the recorded trace data to a remote system over the
6235 network instead of writing it to the local file system.
6236
6237 To send the trace data over the network:
6238
6239 . On the _remote_ system (which can also be the target system),
6240   start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6241 +
6242 --
6243 [role="term"]
6244 ----
6245 $ lttng-relayd
6246 ----
6247 --
6248
6249 . On the _target_ system, create a tracing session configured to
6250   send trace data over the network:
6251 +
6252 --
6253 [role="term"]
6254 ----
6255 $ lttng create my-session --set-url=net://remote-system
6256 ----
6257 --
6258 +
6259 Replace `remote-system` by the host name or IP address of the
6260 remote system. See man:lttng-create(1) for the exact URL format.
6261
6262 . On the target system, use the man:lttng(1) command-line tool as usual.
6263   When tracing is active, the target's consumer daemon sends sub-buffers
6264   to the relay daemon running on the remote system instead of flushing
6265   them to the local file system. The relay daemon writes the received
6266   packets to the local file system.
6267
6268 The relay daemon writes trace files to
6269 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6270 +__hostname__+ is the host name of the target system and +__session__+
6271 is the tracing session name. Note that the env:LTTNG_HOME environment
6272 variable defaults to `$HOME` if not set. Use the
6273 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6274 trace files to another base directory.
6275
6276
6277 [role="since-2.4"]
6278 [[lttng-live]]
6279 === View events as LTTng emits them (noch:{LTTng} live)
6280
6281 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6282 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6283 display events as LTTng emits them on the target system while tracing is
6284 active.
6285
6286 The relay daemon creates a _tee_: it forwards the trace data to both
6287 the local file system and to connected live viewers:
6288
6289 [role="img-90"]
6290 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6291 image::live.png[]
6292
6293 To use LTTng live:
6294
6295 . On the _target system_, create a <<tracing-session,tracing session>>
6296   in _live mode_:
6297 +
6298 --
6299 [role="term"]
6300 ----
6301 $ lttng create my-session --live
6302 ----
6303 --
6304 +
6305 This spawns a local relay daemon.
6306
6307 . Start the live viewer and configure it to connect to the relay
6308   daemon. For example, with http://diamon.org/babeltrace[Babeltrace]:
6309 +
6310 --
6311 [role="term"]
6312 ----
6313 $ babeltrace --input-format=lttng-live
6314            net://localhost/host/hostname/my-session
6315 ----
6316 --
6317 +
6318 Replace:
6319 +
6320 --
6321 * `hostname` with the host name of the target system.
6322 * `my-session` with the name of the tracing session to view.
6323 --
6324
6325 . Configure the tracing session as usual with the man:lttng(1)
6326   command-line tool, and <<basic-tracing-session-control,start tracing>>.
6327
6328 You can list the available live tracing sessions with Babeltrace:
6329
6330 [role="term"]
6331 ----
6332 $ babeltrace --input-format=lttng-live net://localhost
6333 ----
6334
6335 You can start the relay daemon on another system. In this case, you need
6336 to specify the relay daemon's URL when you create the tracing session
6337 with the opt:lttng-create(1):--set-url option. You also need to replace
6338 `localhost` in the procedure above with the host name of the system on
6339 which the relay daemon is running.
6340
6341 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6342 command-line options.
6343
6344
6345 [role="since-2.3"]
6346 [[taking-a-snapshot]]
6347 === Take a snapshot of the current sub-buffers of a tracing session
6348
6349 The normal behavior of LTTng is to append full sub-buffers to growing
6350 trace data files. This is ideal to keep a full history of the events
6351 that occurred on the target system, but it can
6352 represent too much data in some situations. For example, you may wish
6353 to trace your application continuously until some critical situation
6354 happens, in which case you only need the latest few recorded
6355 events to perform the desired analysis, not multi-gigabyte trace files.
6356
6357 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6358 current sub-buffers of a given <<tracing-session,tracing session>>.
6359 LTTng can write the snapshot to the local file system or send it over
6360 the network.
6361
6362 To take a snapshot:
6363
6364 . Create a tracing session in _snapshot mode_:
6365 +
6366 --
6367 [role="term"]
6368 ----
6369 $ lttng create my-session --snapshot
6370 ----
6371 --
6372 +
6373 The <<channel-overwrite-mode-vs-discard-mode,event loss mode>> of
6374 <<channel,channels>> created in this mode is automatically set to
6375 _overwrite_ (flight recorder mode).
6376
6377 . Configure the tracing session as usual with the man:lttng(1)
6378   command-line tool, and <<basic-tracing-session-control,start tracing>>.
6379
6380 . **Optional**: When you need to take a snapshot,
6381   <<basic-tracing-session-control,stop tracing>>.
6382 +
6383 You can take a snapshot when the tracers are active, but if you stop
6384 them first, you are sure that the data in the sub-buffers does not
6385 change before you actually take the snapshot.
6386
6387 . Take a snapshot:
6388 +
6389 --
6390 [role="term"]
6391 ----
6392 $ lttng snapshot record --name=my-first-snapshot
6393 ----
6394 --
6395 +
6396 LTTng writes the current sub-buffers of all the current tracing
6397 session's channels to trace files on the local file system. Those trace
6398 files have `my-first-snapshot` in their name.
6399
6400 There is no difference between the format of a normal trace file and the
6401 format of a snapshot: viewers of LTTng traces also support LTTng
6402 snapshots.
6403
6404 By default, LTTng writes snapshot files to the path shown by
6405 `lttng snapshot list-output`. You can change this path or decide to send
6406 snapshots over the network using either:
6407
6408 . An output path or URL that you specify when you create the
6409   tracing session.
6410 . An snapshot output path or URL that you add using
6411   `lttng snapshot add-output`
6412 . An output path or URL that you provide directly to the
6413   `lttng snapshot record` command.
6414
6415 Method 3 overrides method 2, which overrides method 1. When you
6416 specify a URL, a relay daemon must listen on a remote system (see
6417 <<sending-trace-data-over-the-network,Send trace data over the network>>).
6418
6419
6420 [role="since-2.6"]
6421 [[mi]]
6422 === Use the machine interface
6423
6424 With any command of the man:lttng(1) command-line tool, you can set the
6425 opt:lttng(1):--mi option to `xml` (before the command name) to get an
6426 XML machine interface output, for example:
6427
6428 [role="term"]
6429 ----
6430 $ lttng --mi=xml enable-event --kernel --syscall open
6431 ----
6432
6433 A schema definition (XSD) is
6434 https://github.com/lttng/lttng-tools/blob/stable-2.8/src/common/mi-lttng-3.0.xsd[available]
6435 to ease the integration with external tools as much as possible.
6436
6437
6438 [role="since-2.8"]
6439 [[metadata-regenerate]]
6440 === Regenerate the metadata of an LTTng trace
6441
6442 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
6443 data stream files and a metadata file. This metadata file contains,
6444 amongst other things, information about the offset of the clock sources
6445 used to timestamp <<event,event records>> when tracing.
6446
6447 If, once a <<tracing-session,tracing session>> is
6448 <<basic-tracing-session-control,started>>, a major
6449 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
6450 happens, the trace's clock offset also needs to be updated. You
6451 can use the man:lttng-metadata(1) command to do so.
6452
6453 The main use case of this command is to allow a system to boot with
6454 an incorrect wall time and trace it with LTTng before its wall time
6455 is corrected. Once the system is known to be in a state where its
6456 wall time is correct, it can run `lttng metadata regenerate`.
6457
6458 To regenerate the metadata of an LTTng trace:
6459
6460 * Use the `regenerate` action of the man:lttng-metadata(1) command:
6461 +
6462 --
6463 [role="term"]
6464 ----
6465 $ lttng metadata regenerate
6466 ----
6467 --
6468
6469 [IMPORTANT]
6470 ====
6471 `lttng metadata regenerate` has the following limitations:
6472
6473 * Tracing session <<creating-destroying-tracing-sessions,created>>
6474   in non-live mode.
6475 * User space <<channel,channels>>, if any, using
6476   <<channel-buffering-schemes,per-user buffering>>.
6477 ====
6478
6479
6480 [role="since-2.7"]
6481 [[persistent-memory-file-systems]]
6482 === Record trace data on persistent memory file systems
6483
6484 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
6485 (NVRAM) is random-access memory that retains its information when power
6486 is turned off (non-volatile). Systems with such memory can store data
6487 structures in RAM and retrieve them after a reboot, without flushing
6488 to typical _storage_.
6489
6490 Linux supports NVRAM file systems thanks to either
6491 http://pramfs.sourceforge.net/[PRAMFS] or
6492 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
6493 (requires Linux 4.1+).
6494
6495 This section does not describe how to operate such file systems;
6496 we assume that you have a working persistent memory file system.
6497
6498 When you create a <<tracing-session,tracing session>>, you can specify
6499 the path of the shared memory holding the sub-buffers. If you specify a
6500 location on an NVRAM file system, then you can retrieve the latest
6501 recorded trace data when the system reboots after a crash.
6502
6503 To record trace data on a persistent memory file system and retrieve the
6504 trace data after a system crash:
6505
6506 . Create a tracing session with a sub-buffer shared memory path located
6507   on an NVRAM file system:
6508 +
6509 --
6510 [role="term"]
6511 ----
6512 $ lttng create my-session --shm-path=/path/to/shm
6513 ----
6514 --
6515
6516 . Configure the tracing session as usual with the man:lttng(1)
6517   command-line tool, and <<basic-tracing-session-control,start tracing>>.
6518
6519 . After a system crash, use the man:lttng-crash(1) command-line tool to
6520   view the trace data recorded on the NVRAM file system:
6521 +
6522 --
6523 [role="term"]
6524 ----
6525 $ lttng-crash /path/to/shm
6526 ----
6527 --
6528
6529 The binary layout of the ring buffer files is not exactly the same as
6530 the trace files layout. This is why you need to use man:lttng-crash(1)
6531 instead of your preferred trace viewer directly.
6532
6533 To convert the ring buffer files to LTTng trace files:
6534
6535 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
6536 +
6537 --
6538 [role="term"]
6539 ----
6540 $ lttng-crash --extract=/path/to/trace /path/to/shm
6541 ----
6542 --
6543
6544
6545 [[reference]]
6546 == Reference
6547
6548 [[lttng-modules-ref]]
6549 === noch:{LTTng-modules}
6550
6551 [role="since-2.7"]
6552 [[lttng-modules-tp-fields]]
6553 ==== Tracepoint fields macros (for `TP_FIELDS()`)
6554
6555 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
6556 tracepoint fields, which must be listed within `TP_FIELDS()` in
6557 `LTTNG_TRACEPOINT_EVENT()`, are:
6558
6559 [role="func-desc growable",cols="asciidoc,asciidoc"]
6560 .Available macros to define LTTng-modules tracepoint fields
6561 |====
6562 |Macro |Description and parameters
6563
6564 |
6565 +ctf_integer(__t__, __n__, __e__)+
6566
6567 +ctf_integer_nowrite(__t__, __n__, __e__)+
6568
6569 +ctf_user_integer(__t__, __n__, __e__)+
6570
6571 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
6572 |
6573 Standard integer, displayed in base 10.
6574
6575 +__t__+::
6576   Integer C type (`int`, `long`, `size_t`, ...).
6577
6578 +__n__+::
6579   Field name.
6580
6581 +__e__+::
6582   Argument expression.
6583
6584 |
6585 +ctf_integer_hex(__t__, __n__, __e__)+
6586
6587 +ctf_user_integer_hex(__t__, __n__, __e__)+
6588 |
6589 Standard integer, displayed in base 16.
6590
6591 +__t__+::
6592   Integer C type.
6593
6594 +__n__+::
6595   Field name.
6596
6597 +__e__+::
6598   Argument expression.
6599
6600 |+ctf_integer_oct(__t__, __n__, __e__)+
6601 |
6602 Standard integer, displayed in base 8.
6603
6604 +__t__+::
6605   Integer C type.
6606
6607 +__n__+::
6608   Field name.
6609
6610 +__e__+::
6611   Argument expression.
6612
6613 |
6614 +ctf_integer_network(__t__, __n__, __e__)+
6615
6616 +ctf_user_integer_network(__t__, __n__, __e__)+
6617 |
6618 Integer in network byte order (big-endian), displayed in base 10.
6619
6620 +__t__+::
6621   Integer C type.
6622
6623 +__n__+::
6624   Field name.
6625
6626 +__e__+::
6627   Argument expression.
6628
6629 |
6630 +ctf_integer_network_hex(__t__, __n__, __e__)+
6631
6632 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
6633 |
6634 Integer in network byte order, displayed in base 16.
6635
6636 +__t__+::
6637   Integer C type.
6638
6639 +__n__+::
6640   Field name.
6641
6642 +__e__+::
6643   Argument expression.
6644
6645 |
6646 +ctf_string(__n__, __e__)+
6647
6648 +ctf_string_nowrite(__n__, __e__)+
6649
6650 +ctf_user_string(__n__, __e__)+
6651
6652 +ctf_user_string_nowrite(__n__, __e__)+
6653 |
6654 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
6655
6656 +__n__+::
6657   Field name.
6658
6659 +__e__+::
6660   Argument expression.
6661
6662 |
6663 +ctf_array(__t__, __n__, __e__, __s__)+
6664
6665 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
6666
6667 +ctf_user_array(__t__, __n__, __e__, __s__)+
6668
6669 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
6670 |
6671 Statically-sized array of integers.
6672
6673 +__t__+::
6674   Array element C type.
6675
6676 +__n__+::
6677   Field name.
6678
6679 +__e__+::
6680   Argument expression.
6681
6682 +__s__+::
6683   Number of elements.
6684
6685 |
6686 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
6687
6688 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6689
6690 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
6691
6692 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6693 |
6694 Statically-sized array of bits.
6695
6696 The type of +__e__+ must be an integer type. +__s__+ is the number
6697 of elements of such type in +__e__+, not the number of bits.
6698
6699 +__t__+::
6700   Array element C type.
6701
6702 +__n__+::
6703   Field name.
6704
6705 +__e__+::
6706   Argument expression.
6707
6708 +__s__+::
6709   Number of elements.
6710
6711 |
6712 +ctf_array_text(__t__, __n__, __e__, __s__)+
6713
6714 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
6715
6716 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
6717
6718 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
6719 |
6720 Statically-sized array, printed as text.
6721
6722 The string does not need to be null-terminated.
6723
6724 +__t__+::
6725   Array element C type (always `char`).
6726
6727 +__n__+::
6728   Field name.
6729
6730 +__e__+::
6731   Argument expression.
6732
6733 +__s__+::
6734   Number of elements.
6735
6736 |
6737 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
6738
6739 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6740
6741 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
6742
6743 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6744 |
6745 Dynamically-sized array of integers.
6746
6747 The type of +__E__+ must be unsigned.
6748
6749 +__t__+::
6750   Array element C type.
6751
6752 +__n__+::
6753   Field name.
6754
6755 +__e__+::
6756   Argument expression.
6757
6758 +__T__+::
6759   Length expression C type.
6760
6761 +__E__+::
6762   Length expression.
6763
6764 |
6765 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
6766
6767 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
6768 |
6769 Dynamically-sized array of integers, displayed in base 16.
6770
6771 The type of +__E__+ must be unsigned.
6772
6773 +__t__+::
6774   Array element C type.
6775
6776 +__n__+::
6777   Field name.
6778
6779 +__e__+::
6780   Argument expression.
6781
6782 +__T__+::
6783   Length expression C type.
6784
6785 +__E__+::
6786   Length expression.
6787
6788 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
6789 |
6790 Dynamically-sized array of integers in network byte order (big-endian),
6791 displayed in base 10.
6792
6793 The type of +__E__+ must be unsigned.
6794
6795 +__t__+::
6796   Array element C type.
6797
6798 +__n__+::
6799   Field name.
6800
6801 +__e__+::
6802   Argument expression.
6803
6804 +__T__+::
6805   Length expression C type.
6806
6807 +__E__+::
6808   Length expression.
6809
6810 |
6811 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
6812
6813 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
6814
6815 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
6816
6817 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
6818 |
6819 Dynamically-sized array of bits.
6820
6821 The type of +__e__+ must be an integer type. +__s__+ is the number
6822 of elements of such type in +__e__+, not the number of bits.
6823
6824 The type of +__E__+ must be unsigned.
6825
6826 +__t__+::
6827   Array element C type.
6828
6829 +__n__+::
6830   Field name.
6831
6832 +__e__+::
6833   Argument expression.
6834
6835 +__T__+::
6836   Length expression C type.
6837
6838 +__E__+::
6839   Length expression.
6840
6841 |
6842 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
6843
6844 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
6845
6846 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
6847
6848 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
6849 |
6850 Dynamically-sized array, displayed as text.
6851
6852 The string does not need to be null-terminated.
6853
6854 The type of +__E__+ must be unsigned.
6855
6856 The behaviour is undefined if +__e__+ is `NULL`.
6857
6858 +__t__+::
6859   Sequence element C type (always `char`).
6860
6861 +__n__+::
6862   Field name.
6863
6864 +__e__+::
6865   Argument expression.
6866
6867 +__T__+::
6868   Length expression C type.
6869
6870 +__E__+::
6871   Length expression.
6872 |====
6873
6874 Use the `_user` versions when the argument expression, `e`, is
6875 a user space address. In the cases of `ctf_user_integer*()` and
6876 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
6877 be addressable.
6878
6879 The `_nowrite` versions omit themselves from the session trace, but are
6880 otherwise identical. This means the `_nowrite` fields won't be written
6881 in the recorded trace. Their primary purpose is to make some
6882 of the event context available to the
6883 <<enabling-disabling-events,event filters>> without having to
6884 commit the data to sub-buffers.
6885
6886
6887 [[glossary]]
6888 == Glossary
6889
6890 Terms related to LTTng and to tracing in general:
6891
6892 Babeltrace::
6893   The http://diamon.org/babeltrace[Babeltrace] project, which includes
6894   the cmd:babeltrace command, some libraries, and Python bindings.
6895
6896 <<channel-buffering-schemes,buffering scheme>>::
6897   A layout of sub-buffers applied to a given channel.
6898
6899 <<channel,channel>>::
6900   An entity which is responsible for a set of ring buffers.
6901 +
6902 <<event,Event rules>> are always attached to a specific channel.
6903
6904 clock::
6905   A reference of time for a tracer.
6906
6907 <<lttng-consumerd,consumer daemon>>::
6908   A process which is responsible for consuming the full sub-buffers
6909   and write them to a file system or send them over the network.
6910
6911 <<channel-overwrite-mode-vs-discard-mode,discard mode>>:: The event loss
6912   mode in which the tracer _discards_ new event records when there's no
6913   sub-buffer space left to store them.
6914
6915 event::
6916   The consequence of the execution of an instrumentation
6917   point, like a tracepoint that you manually place in some source code,
6918   or a Linux kernel KProbe.
6919 +
6920 An event is said to _occur_ at a specific time. Different actions can
6921 be taken upon the occurrence of an event, like record the event's payload
6922 to a sub-buffer.
6923
6924 <<channel-overwrite-mode-vs-discard-mode,event loss mode>>::
6925   The mechanism by which event records of a given channel are lost
6926   (not recorded) when there is no sub-buffer space left to store them.
6927
6928 [[def-event-name]]event name::
6929   The name of an event, which is also the name of the event record.
6930   This is also called the _instrumentation point name_.
6931
6932 event record::
6933   A record, in a trace, of the payload of an event which occured.
6934
6935 <<event,event rule>>::
6936   Set of conditions which must be satisfied for one or more occuring
6937   events to be recorded.
6938
6939 `java.util.logging`::
6940   Java platform's
6941   https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities].
6942
6943 <<instrumenting,instrumentation>>::
6944   The use of LTTng probes to make a piece of software traceable.
6945
6946 instrumentation point::
6947   A point in the execution path of a piece of software that, when
6948   reached by this execution, can emit an event.
6949
6950 instrumentation point name::
6951   See _<<def-event-name,event name>>_.
6952
6953 log4j::
6954   A http://logging.apache.org/log4j/1.2/[logging library] for Java
6955   developed by the Apache Software Foundation.
6956
6957 log level::
6958   Level of severity of a log statement or user space
6959   instrumentation point.
6960
6961 LTTng::
6962   The _Linux Trace Toolkit: next generation_ project.
6963
6964 <<lttng-cli,cmd:lttng>>::
6965   A command-line tool provided by the LTTng-tools project which you
6966   can use to send and receive control messages to and from a
6967   session daemon.
6968
6969 LTTng analyses::
6970   The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
6971   which is a set of analyzing programs that are used to obtain a
6972   higher level view of an LTTng trace.
6973
6974 cmd:lttng-consumerd::
6975   The name of the consumer daemon program.
6976
6977 cmd:lttng-crash::
6978   A utility provided by the LTTng-tools project which can convert
6979   ring buffer files (usually
6980   <<persistent-memory-file-systems,saved on a persistent memory file system>>)
6981   to trace files.
6982
6983 LTTng Documentation::
6984   This document.
6985
6986 <<lttng-live,LTTng live>>::
6987   A communication protocol between the relay daemon and live viewers
6988   which makes it possible to see events "live", as they are received by
6989   the relay daemon.
6990
6991 <<lttng-modules,LTTng-modules>>::
6992   The https://github.com/lttng/lttng-modules[LTTng-modules] project,
6993   which contains the Linux kernel modules to make the Linux kernel
6994   instrumentation points available for LTTng tracing.
6995
6996 cmd:lttng-relayd::
6997   The name of the relay daemon program.
6998
6999 cmd:lttng-sessiond::
7000   The name of the session daemon program.
7001
7002 LTTng-tools::
7003   The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
7004   contains the various programs and libraries used to
7005   <<controlling-tracing,control tracing>>.
7006
7007 <<lttng-ust,LTTng-UST>>::
7008   The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
7009   contains libraries to instrument user applications.
7010
7011 <<lttng-ust-agents,LTTng-UST Java agent>>::
7012   A Java package provided by the LTTng-UST project to allow the
7013   LTTng instrumentation of `java.util.logging` and Apache log4j 1.2
7014   logging statements.
7015
7016 <<lttng-ust-agents,LTTng-UST Python agent>>::
7017   A Python package provided by the LTTng-UST project to allow the
7018   LTTng instrumentation of Python logging statements.
7019
7020 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
7021   The event loss mode in which new event records overwrite older
7022   event records when there's no sub-buffer space left to store them.
7023
7024 <<channel-buffering-schemes,per-process buffering>>::
7025   A buffering scheme in which each instrumented process has its own
7026   sub-buffers for a given user space channel.
7027
7028 <<channel-buffering-schemes,per-user buffering>>::
7029   A buffering scheme in which all the processes of a Unix user share the
7030   same sub-buffer for a given user space channel.
7031
7032 <<lttng-relayd,relay daemon>>::
7033   A process which is responsible for receiving the trace data sent by
7034   a distant consumer daemon.
7035
7036 ring buffer::
7037   A set of sub-buffers.
7038
7039 <<lttng-sessiond,session daemon>>::
7040   A process which receives control commands from you and orchestrates
7041   the tracers and various LTTng daemons.
7042
7043 <<taking-a-snapshot,snapshot>>::
7044   A copy of the current data of all the sub-buffers of a given tracing
7045   session, saved as trace files.
7046
7047 sub-buffer::
7048   One part of an LTTng ring buffer which contains event records.
7049
7050 timestamp::
7051   The time information attached to an event when it is emitted.
7052
7053 trace (_noun_)::
7054   A set of files which are the concatenations of one or more
7055   flushed sub-buffers.
7056
7057 trace (_verb_)::
7058   The action of recording the events emitted by an application
7059   or by a system, or to initiate such recording by controlling
7060   a tracer.
7061
7062 Trace Compass::
7063   The http://tracecompass.org[Trace Compass] project and application.
7064
7065 tracepoint::
7066   An instrumentation point using the tracepoint mechanism of the Linux
7067   kernel or of LTTng-UST.
7068
7069 tracepoint definition::
7070   The definition of a single tracepoint.
7071
7072 tracepoint name::
7073   The name of a tracepoint.
7074
7075 tracepoint provider::
7076   A set of functions providing tracepoints to an instrumented user
7077   application.
7078 +
7079 Not to be confused with a _tracepoint provider package_: many tracepoint
7080 providers can exist within a tracepoint provider package.
7081
7082 tracepoint provider package::
7083   One or more tracepoint providers compiled as an object file or as
7084   a shared library.
7085
7086 tracer::
7087   A software which records emitted events.
7088
7089 <<domain,tracing domain>>::
7090   A namespace for event sources.
7091
7092 <<tracing-group,tracing group>>::
7093   The Unix group in which a Unix user can be to be allowed to trace the
7094   Linux kernel.
7095
7096 <<tracing-session,tracing session>>::
7097   A stateful dialogue between you and a <<lttng-sessiond,session
7098   daemon>>.
7099
7100 user application::
7101   An application running in user space, as opposed to a Linux kernel
7102   module, for example.