| 1 | lttng-concepts(7) |
| 2 | ================= |
| 3 | :revdate: 3 May 2021 |
| 4 | :sect-event-rule: INSTRUMENTATION POINT, EVENT RULE, AND EVENT |
| 5 | :sect-session: TRACING SESSION |
| 6 | :sect-domain: TRACING DOMAIN |
| 7 | :sect-channel: CHANNEL AND RING BUFFER |
| 8 | :sect-recording-event-rule: RECORDING EVENT RULE AND EVENT RECORD |
| 9 | |
| 10 | |
| 11 | NAME |
| 12 | ---- |
| 13 | lttng-concepts - LTTng concepts |
| 14 | |
| 15 | |
| 16 | DESCRIPTION |
| 17 | ----------- |
| 18 | This manual page documents the concepts of LTTng. |
| 19 | |
| 20 | Many other LTTng manual pages refer to this one so that you can |
| 21 | understand what are the various LTTng objects and how they relate to |
| 22 | each other. |
| 23 | |
| 24 | The concepts of LTTng{nbsp}{lttng_version} are: |
| 25 | |
| 26 | * Instrumentation point, event rule, and event |
| 27 | * Trigger |
| 28 | * Tracing session |
| 29 | * Tracing domain |
| 30 | * Channel and ring buffer |
| 31 | * Recording event rule and event record |
| 32 | |
| 33 | |
| 34 | [[event-rule]] |
| 35 | {sect-event-rule} |
| 36 | ----------------- |
| 37 | An _instrumentation point_ is a point, within a piece of software, |
| 38 | which, when executed, creates an LTTng _event_. |
| 39 | |
| 40 | LTTng offers various types of instrumentation; see the |
| 41 | <<inst-point-types,Instrumentation point types>> section below to learn |
| 42 | about them. |
| 43 | |
| 44 | An _event rule_ is a set of conditions to match a set of events. |
| 45 | |
| 46 | When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is |
| 47 | said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies *all* the |
| 48 | conditions of{nbsp}__ER__. This concept is similar to a regular |
| 49 | expression which matches a set of strings. |
| 50 | |
| 51 | When an event rule matches an event, LTTng _emits_ the event, therefore |
| 52 | attempting to execute one or more actions. |
| 53 | |
| 54 | [IMPORTANT] |
| 55 | ==== |
| 56 | The event creation and emission processes are documentation concepts to |
| 57 | help understand the journey from an instrumentation point to the |
| 58 | execution of actions. |
| 59 | |
| 60 | The actual creation of an event can be costly because LTTng needs to |
| 61 | evaluate the arguments of the instrumentation point. |
| 62 | |
| 63 | In practice, LTTng implements various optimizations for the Linux kernel |
| 64 | and user space tracing domains (see the <<domain,{sect-domain}>> section |
| 65 | below) to avoid actually creating an event |
| 66 | when the tracer knows, thanks to properties which are independent from |
| 67 | the event payload and current context, that it would never emit such an |
| 68 | event. Those properties are: |
| 69 | |
| 70 | * The instrumentation point type (see the |
| 71 | <<inst-point-types,Instrumentation point types>> section below). |
| 72 | |
| 73 | * The instrumentation point name. |
| 74 | |
| 75 | * The instrumentation point log level. |
| 76 | |
| 77 | * For a recording event rule (see the |
| 78 | <<recording-event-rule,{sect-recording-event-rule}>> section |
| 79 | below): |
| 80 | ** The status of the rule itself. |
| 81 | ** The status of the channel (see the <<channel,{sect-channel}>> section |
| 82 | below). |
| 83 | ** The activity of the tracing session (started or stopped; see |
| 84 | the <<session,{sect-session}>> section below). |
| 85 | ** Whether or not the process for which LTTng would create the event is |
| 86 | allowed to record events (see man:lttng-track(1)). |
| 87 | |
| 88 | In other words: if, for a given instrumentation point{nbsp}__IP__, the |
| 89 | LTTng tracer knows that it would never emit an event, |
| 90 | executing{nbsp}__IP__ represents a simple boolean variable check and, |
| 91 | for a Linux kernel recording event rule, a few process attribute checks. |
| 92 | ==== |
| 93 | |
| 94 | As of LTTng{nbsp}{lttng_version}, there are two places where you can |
| 95 | find an event rule: |
| 96 | |
| 97 | Recording event rule:: |
| 98 | A specific type of event rule of which the action is to record the |
| 99 | matched event as an event record. |
| 100 | + |
| 101 | See the <<recording-event-rule,{sect-recording-event-rule}>> section |
| 102 | below. |
| 103 | + |
| 104 | Create or enable a recording event rule with the |
| 105 | man:lttng-enable-event(1) command. |
| 106 | + |
| 107 | List the recording event rules of a specific tracing session |
| 108 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) |
| 109 | commands. |
| 110 | |
| 111 | ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13):: |
| 112 | When the event rule of the trigger condition matches an event, LTTng |
| 113 | can execute user-defined actions such as sending an LTTng |
| 114 | notification, starting a tracing session, and more. |
| 115 | + |
| 116 | See man:lttng-add-trigger(1) and man:lttng-event-rule(7). |
| 117 | |
| 118 | For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy *all* the |
| 119 | basic conditions of an event rule{nbsp}__ER__, that is: |
| 120 | |
| 121 | * The instrumentation point from which LTTng creates{nbsp}__E__ has a |
| 122 | specific type. |
| 123 | + |
| 124 | See the <<inst-point-types,Instrumentation point types>> section below. |
| 125 | |
| 126 | * A pattern matches the name of{nbsp}__E__ while another pattern |
| 127 | doesn't. |
| 128 | |
| 129 | * The log level of the instrumentation point from which LTTng |
| 130 | creates{nbsp}__E__ is at least as severe as some value, or is exactly |
| 131 | some value. |
| 132 | |
| 133 | * The fields of the payload of{nbsp}__E__ and the current context fields |
| 134 | satisfy a filter expression. |
| 135 | |
| 136 | A recording event rule has additional, implicit conditions to satisfy. |
| 137 | See the <<recording-event-rule,{sect-recording-event-rule}>> section |
| 138 | below to learn more. |
| 139 | |
| 140 | |
| 141 | [[inst-point-types]] |
| 142 | Instrumentation point types |
| 143 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 144 | As of LTTng{nbsp}{lttng_version}, the available instrumentation point |
| 145 | types are, depending on the tracing domain (see the |
| 146 | <<domain,{sect-domain}>> section below): |
| 147 | |
| 148 | Linux kernel:: |
| 149 | LTTng tracepoint::: |
| 150 | A statically defined point in the source code of the kernel |
| 151 | image or of a kernel module using the LTTng-modules macros. |
| 152 | + |
| 153 | List the available Linux kernel tracepoints with `lttng list --kernel`. |
| 154 | See man:lttng-list(1) to learn more. |
| 155 | |
| 156 | Linux kernel system call::: |
| 157 | Entry, exit, or both of a Linux kernel system call. |
| 158 | + |
| 159 | List the available Linux kernel system call instrumentation points with |
| 160 | `lttng list --kernel --syscall`. See man:lttng-list(1) to learn more. |
| 161 | |
| 162 | Linux kprobe::: |
| 163 | A single probe dynamically placed in the compiled kernel code. |
| 164 | + |
| 165 | When you create such an instrumentation point, you set its memory |
| 166 | address or symbol name. |
| 167 | |
| 168 | Linux user space probe::: |
| 169 | A single probe dynamically placed at the entry of a compiled |
| 170 | user space application/library function through the kernel. |
| 171 | + |
| 172 | When you create such an instrumentation point, you set: |
| 173 | + |
| 174 | -- |
| 175 | With the ELF method:: |
| 176 | Its application/library path and its symbol name. |
| 177 | |
| 178 | With the USDT method:: |
| 179 | Its application/library path, its provider name, and its probe name. |
| 180 | + |
| 181 | ``USDT'' stands for SystemTap User-level Statically Defined Tracing, |
| 182 | a DTrace-style marker. |
| 183 | -- |
| 184 | + |
| 185 | As of LTTng{nbsp}{lttng_version}, LTTng only supports USDT probes which |
| 186 | are :not: reference-counted. |
| 187 | |
| 188 | Linux kretprobe::: |
| 189 | Entry, exit, or both of a Linux kernel function. |
| 190 | + |
| 191 | When you create such an instrumentation point, you set the memory |
| 192 | address or symbol name of its function. |
| 193 | |
| 194 | User space:: |
| 195 | LTTng tracepoint::: |
| 196 | A statically defined point in the source code of a C/$$C++$$ |
| 197 | application/library using the LTTng-UST macros. |
| 198 | + |
| 199 | List the available Linux kernel tracepoints with |
| 200 | `lttng list --userspace`. See man:lttng-list(1) to learn more. |
| 201 | |
| 202 | `java.util.logging`, Apache log4j, and Python:: |
| 203 | Java or Python logging statement::: |
| 204 | A method call on a Java or Python logger attached to an |
| 205 | LTTng-UST handler. |
| 206 | + |
| 207 | List the available Java and Python loggers with `lttng list --jul`, |
| 208 | `lttng list --log4j`, and `lttng list --python`. See man:lttng-list(1) |
| 209 | to learn more. |
| 210 | |
| 211 | |
| 212 | [[trigger]] |
| 213 | TRIGGER |
| 214 | ------- |
| 215 | A _trigger_ associates a condition to one or more actions. |
| 216 | |
| 217 | When the condition of a trigger is satisfied, LTTng attempts to execute |
| 218 | its actions. |
| 219 | |
| 220 | As of LTTng{nbsp}{lttng_version}, the available trigger conditions and |
| 221 | actions are: |
| 222 | |
| 223 | Conditions:: |
| 224 | + |
| 225 | * The consumed buffer size of a given tracing |
| 226 | session (see the <<session,{sect-session}>> section below) |
| 227 | becomes greater than some value. |
| 228 | |
| 229 | * The buffer usage of a given channel (see the |
| 230 | <<channel,{sect-channel}>> section below) becomes greater than some |
| 231 | value. |
| 232 | |
| 233 | * The buffer usage of a given channel becomes less than some value. |
| 234 | |
| 235 | * There's an ongoing tracing session rotation (see the |
| 236 | <<rotation,Tracing session rotation>> section below). |
| 237 | |
| 238 | * A tracing session rotation becomes completed. |
| 239 | |
| 240 | * An event rule matches an event. |
| 241 | + |
| 242 | As of LTTng{nbsp}{lttng_version}, this is the only available condition |
| 243 | when you add a trigger with the man:lttng-add-trigger(1) command. The |
| 244 | other ones are available through the liblttng-ctl C{nbsp}API. |
| 245 | |
| 246 | Actions:: |
| 247 | + |
| 248 | * Send a notification to a user application. |
| 249 | * Start a given tracing session, like man:lttng-start(1) would do. |
| 250 | * Stop a given tracing session, like man:lttng-stop(1) would do. |
| 251 | * Archive the current trace chunk of a given tracing session (rotate), |
| 252 | like man:lttng-rotate(1) would do. |
| 253 | * Take a snapshot of a given tracing session, like man:lttng-snapshot(1) |
| 254 | would do. |
| 255 | |
| 256 | A trigger belongs to a session daemon (see man:lttng-sessiond(8)), not |
| 257 | to a specific tracing session. For a given session daemon, each Unix |
| 258 | user has its own, private triggers. Note, however, that the `root` Unix |
| 259 | user may, for the root session daemon: |
| 260 | |
| 261 | * Add a trigger as another Unix user. |
| 262 | |
| 263 | * List all the triggers, regardless of their owner. |
| 264 | |
| 265 | * Remove a trigger which belongs to another Unix user. |
| 266 | |
| 267 | For a given session daemon and Unix user, a trigger has a unique name. |
| 268 | |
| 269 | Add a trigger to a session daemon with the man:lttng-add-trigger(1) |
| 270 | command. |
| 271 | |
| 272 | List the triggers of your Unix user (or of all users if your |
| 273 | Unix user is `root`) with the man:lttng-list-triggers(1) command. |
| 274 | |
| 275 | Remove a trigger with the man:lttng-remove-trigger(1) command. |
| 276 | |
| 277 | |
| 278 | [[session]] |
| 279 | {sect-session} |
| 280 | -------------- |
| 281 | A _tracing session_ is a stateful dialogue between you and a session |
| 282 | daemon (see man:lttng-sessiond(8)) for everything related to event |
| 283 | recording. |
| 284 | |
| 285 | Everything that you do when you control LTTng tracers to record events |
| 286 | happens within a tracing session. In particular, a tracing session: |
| 287 | |
| 288 | * Has its own name, unique for a given session daemon. |
| 289 | |
| 290 | * Has its own set of trace files, if any. |
| 291 | |
| 292 | * Has its own state of activity (started or stopped). |
| 293 | + |
| 294 | An active tracing session is an implicit recording event rule condition |
| 295 | (see the <<recording-event-rule,{sect-recording-event-rule}>> section |
| 296 | below). |
| 297 | |
| 298 | * Has its own mode (local, network streaming, snapshot, or live). |
| 299 | + |
| 300 | See the <<session-modes,Tracing session modes>> section below to learn |
| 301 | more. |
| 302 | |
| 303 | * Has its own channels (see the <<channel,{sect-channel}>> section |
| 304 | below) to which are attached their own recording event rules. |
| 305 | |
| 306 | * Has its own process attribute inclusion sets (see man:lttng-track(1)). |
| 307 | |
| 308 | Those attributes and objects are completely isolated between different |
| 309 | tracing sessions. |
| 310 | |
| 311 | A tracing session is like an ATM session: the operations you do on the |
| 312 | banking system through the ATM don't alter the data of other users of |
| 313 | the same system. In the case of the ATM, a session lasts as long as your |
| 314 | bank card is inside. In the case of LTTng, a tracing session lasts from |
| 315 | the man:lttng-create(1) command to the man:lttng-destroy(1) command. |
| 316 | |
| 317 | A tracing session belongs to a session daemon (see |
| 318 | man:lttng-sessiond(8)). For a given session daemon, each Unix user has |
| 319 | its own, private tracing sessions. Note, however, that the `root` Unix |
| 320 | user may operate on or destroy another user's tracing session. |
| 321 | |
| 322 | Create a tracing session with the man:lttng-create(1) command. |
| 323 | |
| 324 | List the tracing sessions of the connected session daemon with |
| 325 | the man:lttng-list(1) command. |
| 326 | |
| 327 | Start and stop a tracing session with the man:lttng-start(1) and |
| 328 | man:lttng-stop(1) commands. |
| 329 | |
| 330 | Save and load a tracing session with the man:lttng-save(1) and |
| 331 | man:lttng-load(1) commands. |
| 332 | |
| 333 | Archive the current trace chunk of (rotate) a tracing session with the |
| 334 | man:lttng-rotate(1) command. |
| 335 | |
| 336 | Destroy a tracing session with the man:lttng-destroy(1) command. |
| 337 | |
| 338 | |
| 339 | Current tracing session |
| 340 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 341 | When you run the man:lttng-create(1) command, LTTng creates the |
| 342 | `$LTTNG_HOME/.lttngrc` file if it doesn't exist (`$LTTNG_HOME` defaults |
| 343 | to `$HOME`). |
| 344 | |
| 345 | `$LTTNG_HOME/.lttngrc` contains the name of the _current tracing |
| 346 | session_. |
| 347 | |
| 348 | When you create a new tracing session with the `create` command, LTTng |
| 349 | updates the current tracing session. |
| 350 | |
| 351 | The following man:lttng(1) commands select the current tracing session |
| 352 | if you don't specify one: |
| 353 | |
| 354 | * man:lttng-add-context(1) |
| 355 | * man:lttng-clear(1) |
| 356 | * man:lttng-destroy(1) |
| 357 | * man:lttng-disable-channel(1) |
| 358 | * man:lttng-disable-event(1) |
| 359 | * man:lttng-disable-rotation(1) |
| 360 | * man:lttng-enable-channel(1) |
| 361 | * man:lttng-enable-event(1) |
| 362 | * man:lttng-enable-rotation(1) |
| 363 | * man:lttng-regenerate(1) |
| 364 | * man:lttng-rotate(1) |
| 365 | * man:lttng-save(1) |
| 366 | * man:lttng-snapshot(1) |
| 367 | * man:lttng-start(1) |
| 368 | * man:lttng-status(1) |
| 369 | * man:lttng-stop(1) |
| 370 | * man:lttng-track(1) |
| 371 | * man:lttng-untrack(1) |
| 372 | * man:lttng-view(1) |
| 373 | |
| 374 | Set the current tracing session manually with the |
| 375 | man:lttng-set-session(1) command, without having to edit the `.lttngrc` |
| 376 | file. |
| 377 | |
| 378 | |
| 379 | [[session-modes]] |
| 380 | Tracing session modes |
| 381 | ~~~~~~~~~~~~~~~~~~~~~ |
| 382 | LTTng offers four tracing session modes: |
| 383 | |
| 384 | Local mode:: |
| 385 | Write the trace data to the local file system. |
| 386 | |
| 387 | Network streaming mode:: |
| 388 | Send the trace data over the network to a listening relay daemon |
| 389 | (see man:lttng-relayd(8)). |
| 390 | |
| 391 | Snapshot mode:: |
| 392 | Only write the trace data to the local file system or send it to a |
| 393 | listening relay daemon (man:lttng-relayd(8)) when LTTng takes a |
| 394 | snapshot. |
| 395 | + |
| 396 | LTTng forces all the channels (see the <<channel,{sect-channel}>> |
| 397 | section below) to be created to be configured to be snapshot-ready. |
| 398 | + |
| 399 | LTTng takes a snapshot of such a tracing session when: |
| 400 | + |
| 401 | -- |
| 402 | * You run the man:lttng-snapshot(1) command. |
| 403 | |
| 404 | * LTTng executes a `snapshot-session` trigger action (see the |
| 405 | <<trigger,TRIGGER>> section above). |
| 406 | -- |
| 407 | |
| 408 | Live mode:: |
| 409 | Send the trace data over the network to a listening relay daemon |
| 410 | (see man:lttng-relayd(8)) for live reading. |
| 411 | + |
| 412 | An LTTng live reader (for example, man:babeltrace2(1)) can connect to |
| 413 | the same relay daemon to receive trace data while the tracing session is |
| 414 | active. |
| 415 | |
| 416 | |
| 417 | [[rotation]] |
| 418 | Tracing session rotation |
| 419 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 420 | A _tracing session rotation_ is the action of archiving the current |
| 421 | trace chunk of the tracing session to the file system. |
| 422 | |
| 423 | Once LTTng archives a trace chunk, it does :not: manage it anymore: you |
| 424 | can read it, modify it, move it, or remove it. |
| 425 | |
| 426 | An _archived trace chunk_ is a collection of metadata and data stream |
| 427 | files which form a self-contained LTTng trace. See the |
| 428 | <<trace-chunk-naming,Trace chunk naming>> section below to learn how |
| 429 | LTTng names a trace chunk archive directory. |
| 430 | |
| 431 | The _current trace chunk_ of a given tracing session includes: |
| 432 | |
| 433 | * The stream files which LTTng already wrote to the file system, and |
| 434 | which are not part of a previously archived trace chunk, since the |
| 435 | most recent event amongst: |
| 436 | |
| 437 | ** The first time the tracing session was started, either with the |
| 438 | man:lttng-start(1) command or with a `start-session` trigger action |
| 439 | (see the <<trigger,TRIGGER>> section above). |
| 440 | |
| 441 | ** The last rotation, performed with: |
| 442 | |
| 443 | *** An man:lttng-rotate(1) command. |
| 444 | |
| 445 | *** A rotation schedule previously set with |
| 446 | man:lttng-enable-rotation(1). |
| 447 | |
| 448 | *** An executed `rotate-session` trigger action (see the |
| 449 | <<trigger,TRIGGER>> section above). |
| 450 | |
| 451 | * The content of all the non-flushed sub-buffers of the channels of the |
| 452 | tracing session. |
| 453 | |
| 454 | |
| 455 | [[trace-chunk-naming]] |
| 456 | Trace chunk archive naming |
| 457 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 458 | A trace chunk archive is a subdirectory of the `archives` subdirectory |
| 459 | within the output directory of a tracing session (see the |
| 460 | nloption:--output option of the man:lttng-create(1) command and |
| 461 | of man:lttng-relayd(8)). |
| 462 | |
| 463 | A trace chunk archive contains, through tracing domain and possibly |
| 464 | UID/PID subdirectories, metadata and data stream files. |
| 465 | |
| 466 | A trace chunk archive is, at the same time: |
| 467 | |
| 468 | * A self-contained LTTng trace. |
| 469 | |
| 470 | * A member of a set of trace chunk archives which form the complete |
| 471 | trace of a tracing session. |
| 472 | |
| 473 | In other words, an LTTng trace reader can read both the tracing |
| 474 | session output directory (all the trace chunk archives), or a |
| 475 | single trace chunk archive. |
| 476 | |
| 477 | When LTTng performs a tracing session rotation, it names the resulting |
| 478 | trace chunk archive as such, relative to the output directory of the |
| 479 | tracing session: |
| 480 | |
| 481 | [verse] |
| 482 | archives/__BEGIN__-__END__-__ID__ |
| 483 | |
| 484 | __BEGIN__:: |
| 485 | Date and time of the beginning of the trace chunk archive with |
| 486 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where |
| 487 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the |
| 488 | time zone offset from UTC. |
| 489 | + |
| 490 | Example: `20171119T152407-0500` |
| 491 | |
| 492 | __END__:: |
| 493 | Date and time of the end of the trace chunk archive with |
| 494 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where |
| 495 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the |
| 496 | time zone offset from UTC. |
| 497 | + |
| 498 | Example: `20180118T152407+0930` |
| 499 | |
| 500 | __ID__:: |
| 501 | Unique numeric identifier of the trace chunk within its tracing |
| 502 | session. |
| 503 | |
| 504 | Trace chunk archive name example: |
| 505 | |
| 506 | ---- |
| 507 | archives/20171119T152407-0500-20171119T151422-0500-3 |
| 508 | ---- |
| 509 | |
| 510 | |
| 511 | [[domain]] |
| 512 | {sect-domain} |
| 513 | ------------- |
| 514 | A _tracing domain_ identifies a type of LTTng tracer. |
| 515 | |
| 516 | A tracing domain has its own properties and features. |
| 517 | |
| 518 | There are currently five available tracing domains: |
| 519 | |
| 520 | [options="header"] |
| 521 | |=== |
| 522 | |Tracing domain |``Event rule matches'' trigger condition option |Option for other CLI commands |
| 523 | |
| 524 | |Linux kernel |
| 525 | |nloption:--domain=++kernel++ |
| 526 | |nloption:--kernel |
| 527 | |
| 528 | |User space |
| 529 | |nloption:--domain=++user++ |
| 530 | |nloption:--userspace |
| 531 | |
| 532 | |`java.util.logging` (JUL) |
| 533 | |nloption:--domain=++jul++ |
| 534 | |nloption:--jul |
| 535 | |
| 536 | |Apache log4j |
| 537 | |nloption:--domain=++log4j++ |
| 538 | |nloption:--log4j |
| 539 | |
| 540 | |Python |
| 541 | |nloption:--domain=++python++ |
| 542 | |nloption:--python |
| 543 | |=== |
| 544 | |
| 545 | You must specify a tracing domain to target a type of LTTng tracer when |
| 546 | using some man:lttng(1) commands to avoid ambiguity. For example, |
| 547 | because the Linux kernel and user space tracing domains support named |
| 548 | tracepoints as instrumentation points (see the |
| 549 | <<"event-rule","{sect-event-rule}">> section above), you need to specify |
| 550 | a tracing domain when you create an event rule because both tracing |
| 551 | domains could have tracepoints sharing the same name. |
| 552 | |
| 553 | You can create channels (see the <<channel,{sect-channel}>> section |
| 554 | below) in the Linux kernel and user space tracing domains. The other |
| 555 | tracing domains have a single, default channel. |
| 556 | |
| 557 | |
| 558 | [[channel]] |
| 559 | {sect-channel} |
| 560 | -------------- |
| 561 | A _channel_ is an object which is responsible for a set of ring buffers. |
| 562 | |
| 563 | Each ring buffer is divided into multiple _sub-buffers_. When a |
| 564 | recording event rule (see the |
| 565 | <<recording-event-rule,{sect-recording-event-rule} section below) |
| 566 | matches an event, LTTng can record it to one or more sub-buffers of one |
| 567 | or more channels. |
| 568 | |
| 569 | When you create a channel with the man:lttng-enable-channel(1) command, |
| 570 | you set its final attributes, that is: |
| 571 | |
| 572 | * Its buffering scheme. |
| 573 | + |
| 574 | See the <<channel-buf-scheme,Buffering scheme>> section below. |
| 575 | |
| 576 | * What to do when there's no |
| 577 | space left for a new event record because all sub-buffers are full. |
| 578 | + |
| 579 | See the <<channel-er-loss-mode,Event record loss mode>> section below. |
| 580 | |
| 581 | * The size of each ring buffer and how many sub-buffers a ring buffer |
| 582 | has. |
| 583 | + |
| 584 | See the <<channel-sub-buf-size-count,Sub-buffer size and count>> section |
| 585 | below. |
| 586 | |
| 587 | * The size of each trace file LTTng writes for this channel and the |
| 588 | maximum count of trace files. |
| 589 | + |
| 590 | See the <<channel-max-trace-file-size-count,Maximum trace file size and |
| 591 | count>> section below. |
| 592 | |
| 593 | * The periods of its read, switch, and monitor timers. |
| 594 | + |
| 595 | See the <<channel-timers,Timers>> section below. |
| 596 | |
| 597 | * For a Linux kernel channel: its output type (man:mmap(2) or |
| 598 | man:splice(2)). |
| 599 | + |
| 600 | See the nloption:--output option of the man:lttng-enable-channel(1) |
| 601 | command. |
| 602 | |
| 603 | * For a user space channel: the value of its blocking timeout. |
| 604 | + |
| 605 | See the nloption:--blocking-timeout option of the |
| 606 | man:lttng-enable-channel(1) command. |
| 607 | |
| 608 | Note that the man:lttng-enable-event(1) command can automatically create |
| 609 | a default channel with sane defaults when no channel exists for the |
| 610 | provided tracing domain. |
| 611 | |
| 612 | A channel is always associated to a tracing domain (see the |
| 613 | <<domain,{sect-domain}>> section below). The `java.util.logging` (JUL), |
| 614 | log4j, and Python tracing domains each have a default channel which you |
| 615 | can't configure. |
| 616 | |
| 617 | A channel owns recording event rules. |
| 618 | |
| 619 | List the channels of a given tracing session with the |
| 620 | man:lttng-list(1) and man:lttng-status(1) commands. |
| 621 | |
| 622 | Disable an enabled channel with the man:lttng-disable-channel(1) |
| 623 | command. |
| 624 | |
| 625 | |
| 626 | [[channel-buf-scheme]] |
| 627 | Buffering scheme |
| 628 | ~~~~~~~~~~~~~~~~ |
| 629 | A channel has at least one ring buffer per CPU. LTTng always records an |
| 630 | event to the ring buffer dedicated to the CPU which emits it. |
| 631 | |
| 632 | The buffering scheme of a user space channel determines what has its own |
| 633 | set of per-CPU ring buffers: |
| 634 | |
| 635 | Per-user buffering (nloption:--buffers-uid option of the man:lttng-enable-channel(1) command):: |
| 636 | Allocate one set of ring buffers (one per CPU) shared by all the |
| 637 | instrumented processes of: |
| 638 | If your Unix user is `root`::: |
| 639 | Each Unix user. |
| 640 | Otherwise::: |
| 641 | Your Unix user. |
| 642 | |
| 643 | Per-process buffering (nloption:--buffers-pid option of the man:lttng-enable-channel(1) command):: |
| 644 | Allocate one set of ring buffers (one per CPU) for each instrumented |
| 645 | process of: |
| 646 | If your Unix user is `root`::: |
| 647 | All Unix users. |
| 648 | Otherwise::: |
| 649 | Your Unix user. |
| 650 | |
| 651 | The per-process buffering scheme tends to consume more memory than the |
| 652 | per-user option because systems generally have more instrumented |
| 653 | processes than Unix users running instrumented processes. However, the |
| 654 | per-process buffering scheme ensures that one process having a high |
| 655 | event throughput won't fill all the shared sub-buffers of the same Unix |
| 656 | user, only its own. |
| 657 | |
| 658 | The buffering scheme of a Linux kernel channel is always to allocate a |
| 659 | single set of ring buffers for the whole system. This scheme is similar |
| 660 | to the per-user option, but with a single, global user ``running'' the |
| 661 | kernel. |
| 662 | |
| 663 | |
| 664 | [[channel-er-loss-mode]] |
| 665 | Event record loss mode |
| 666 | ~~~~~~~~~~~~~~~~~~~~~~ |
| 667 | When LTTng emits an event, LTTng can record it to a specific, available |
| 668 | sub-buffer within the ring buffers of specific channels. When there's no |
| 669 | space left in a sub-buffer, the tracer marks it as consumable and |
| 670 | another, available sub-buffer starts receiving the following event |
| 671 | records. An LTTng consumer daemon eventually consumes the marked |
| 672 | sub-buffer, which returns to the available state. |
| 673 | |
| 674 | In an ideal world, sub-buffers are consumed faster than they are filled. |
| 675 | In the real world, however, all sub-buffers can be full at some point, |
| 676 | leaving no space to record the following events. |
| 677 | |
| 678 | By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when |
| 679 | there's no available sub-buffer to record an event, it's acceptable to |
| 680 | lose event records when the alternative would be to cause substantial |
| 681 | delays in the execution of the instrumented application. LTTng |
| 682 | privileges performance over integrity; it aims at perturbing the |
| 683 | instrumented application as little as possible in order to make the |
| 684 | detection of subtle race conditions and rare interrupt cascades |
| 685 | possible. |
| 686 | |
| 687 | Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports |
| 688 | a _blocking mode_. See the nloption:--blocking-timeout of the |
| 689 | man:lttng-enable-channel(1) command to learn how to use the blocking |
| 690 | mode. |
| 691 | |
| 692 | When it comes to losing event records because there's no available |
| 693 | sub-buffer, or because the blocking timeout of the channel is |
| 694 | reached, the _event record loss mode_ of the channel determines what to |
| 695 | do. The available event record loss modes are: |
| 696 | |
| 697 | Discard mode:: |
| 698 | Drop the newest event records until a sub-buffer becomes available. |
| 699 | + |
| 700 | This is the only available mode when you specify a blocking timeout. |
| 701 | + |
| 702 | With this mode, LTTng increments a count of lost event records when an |
| 703 | event record is lost and saves this count to the trace. A trace reader |
| 704 | can use the saved discarded event record count of the trace to decide |
| 705 | whether or not to perform some analysis even if trace data is known to |
| 706 | be missing. |
| 707 | |
| 708 | Overwrite mode:: |
| 709 | Clear the sub-buffer containing the oldest event records and start |
| 710 | writing the newest event records there. |
| 711 | + |
| 712 | This mode is sometimes called _flight recorder mode_ because it's |
| 713 | similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight |
| 714 | recorder]: always keep a fixed amount of the latest data. It's also |
| 715 | similar to the roll mode of an oscilloscope. |
| 716 | + |
| 717 | Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer |
| 718 | its sequence number within its data stream. With a local, network |
| 719 | streaming, or live tracing session (see the <<session-modes,Tracing |
| 720 | session modes>> section above), a trace reader can use such sequence |
| 721 | numbers to report lost packets. A trace reader can use the saved |
| 722 | discarded sub-buffer (packet) count of the trace to decide whether or |
| 723 | not to perform some analysis even if trace data is known to be missing. |
| 724 | + |
| 725 | With this mode, LTTng doesn't write to the trace the exact number of |
| 726 | lost event records in the lost sub-buffers. |
| 727 | |
| 728 | Which mechanism you should choose depends on your context: prioritize |
| 729 | the newest or the oldest event records in the ring buffer? |
| 730 | |
| 731 | Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_ |
| 732 | as soon as a there's no space left for a new event record, whereas in |
| 733 | discard mode, the tracer only discards the event record that doesn't |
| 734 | fit. |
| 735 | |
| 736 | Set the event record loss mode of a channel with the nloption:--discard |
| 737 | and nloption:--overwrite options of the man:lttng-enable-channel(1) |
| 738 | command. |
| 739 | |
| 740 | There are a few ways to decrease your probability of losing event |
| 741 | records. The <<channel-sub-buf-size-count,Sub-buffer size and count>> |
| 742 | section below shows how to fine-tune the sub-buffer size and count of a |
| 743 | channel to virtually stop losing event records, though at the cost of |
| 744 | greater memory usage. |
| 745 | |
| 746 | |
| 747 | [[channel-sub-buf-size-count]] |
| 748 | Sub-buffer size and count |
| 749 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 750 | A channel has one or more ring buffer for each CPU of the target system. |
| 751 | |
| 752 | See the <<channel-buf-scheme,Buffering scheme>> section above to learn |
| 753 | how many ring buffers of a given channel are dedicated to each CPU |
| 754 | depending on its buffering scheme. |
| 755 | |
| 756 | Set the size of each sub-buffer the ring buffers of a channel contain |
| 757 | with the nloption:--subbuf-size option of the |
| 758 | man:lttng-enable-channel(1) command. |
| 759 | |
| 760 | Set the number of sub-buffers each ring buffer of a channel contains |
| 761 | with the nloption:--num-subbuf option of the man:lttng-enable-channel(1) |
| 762 | command. |
| 763 | |
| 764 | Note that LTTng switching the current sub-buffer of a ring buffer |
| 765 | (marking a full one as consumable and switching to an available one for |
| 766 | LTTng to record the next events) introduces noticeable CPU overhead. |
| 767 | Knowing this, the following list presents a few practical situations |
| 768 | along with how to configure the sub-buffer size and count for them: |
| 769 | |
| 770 | High event throughput:: |
| 771 | In general, prefer large sub-buffers to lower the risk of losing |
| 772 | event records. |
| 773 | + |
| 774 | Having larger sub-buffers also ensures a lower sub-buffer switching |
| 775 | frequency (see the <<channel-timers,Timers>> section below). |
| 776 | + |
| 777 | The sub-buffer count is only meaningful if you create the channel in |
| 778 | overwrite mode (see the <<channel-er-loss-mode,Event record loss mode>> |
| 779 | section above): in this case, if LTTng overwrites a sub-buffer, then the |
| 780 | other sub-buffers are left unaltered. |
| 781 | |
| 782 | Low event throughput:: |
| 783 | In general, prefer smaller sub-buffers since the risk of losing |
| 784 | event records is low. |
| 785 | + |
| 786 | Because LTTng emits events less frequently, the sub-buffer switching |
| 787 | frequency should remain low and therefore the overhead of the tracer |
| 788 | shouldn't be a problem. |
| 789 | |
| 790 | Low memory system:: |
| 791 | If your target system has a low memory limit, prefer fewer first, |
| 792 | then smaller sub-buffers. |
| 793 | + |
| 794 | Even if the system is limited in memory, you want to keep the |
| 795 | sub-buffers as large as possible to avoid a high sub-buffer switching |
| 796 | frequency. |
| 797 | |
| 798 | Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format, |
| 799 | which means event record data is very compact. For example, the average |
| 800 | LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a |
| 801 | sub-buffer size of 1{nbsp}MiB is considered large. |
| 802 | |
| 803 | The previous scenarios highlight the major trade-off between a few large |
| 804 | sub-buffers and more, smaller sub-buffers: sub-buffer switching |
| 805 | frequency vs. how many event records are lost in overwrite mode. |
| 806 | Assuming a constant event throughput and using the overwrite mode, the |
| 807 | two following configurations have the same ring buffer total size: |
| 808 | |
| 809 | Two sub-buffers of 4{nbsp}MiB each:: |
| 810 | Expect a very low sub-buffer switching frequency, but if LTTng |
| 811 | ever needs to overwrite a sub-buffer, half of the event records so |
| 812 | far (4{nbsp}MiB) are definitely lost. |
| 813 | |
| 814 | Eight sub-buffers of 1{nbsp}MiB each:: |
| 815 | Expect four times the tracer overhead of the configuration above, |
| 816 | but if LTTng needs to overwrite a sub-buffer, only the eighth of |
| 817 | event records so far (1{nbsp}MiB) are definitely lost. |
| 818 | |
| 819 | In discard mode, the sub-buffer count parameter is pointless: use two |
| 820 | sub-buffers and set their size according to your requirements. |
| 821 | |
| 822 | |
| 823 | [[channel-max-trace-file-size-count]] |
| 824 | Maximum trace file size and count |
| 825 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 826 | By default, trace files can grow as large as needed. |
| 827 | |
| 828 | Set the maximum size of each trace file that LTTng writes of a given |
| 829 | channel with the nloption:--tracefile-size option of the man:lttng-enable-channel(1) |
| 830 | command. |
| 831 | |
| 832 | When the size of a trace file reaches the fixed maximum size of the |
| 833 | channel, LTTng creates another file to contain the next event records. |
| 834 | LTTng appends a file count to each trace file name in this case. |
| 835 | |
| 836 | If you set the trace file size attribute when you create a channel, the |
| 837 | maximum number of trace files that LTTng creates is _unlimited_ by |
| 838 | default. To limit them, use the nloption:--tracefile-count option of |
| 839 | man:lttng-enable-channel(1). When the number of trace files reaches the |
| 840 | fixed maximum count of the channel, LTTng overwrites the oldest trace |
| 841 | file. This mechanism is called _trace file rotation_. |
| 842 | |
| 843 | [IMPORTANT] |
| 844 | ==== |
| 845 | Even if you don't limit the trace file count, always assume that LTTng |
| 846 | manages all the trace files of the tracing session. |
| 847 | |
| 848 | In other words, there's no safe way to know if LTTng still holds a given |
| 849 | trace file open with the trace file rotation feature. |
| 850 | |
| 851 | The only way to obtain an unmanaged, self-contained LTTng trace before |
| 852 | you destroy the tracing session is with the tracing session rotation |
| 853 | feature (see the <<rotation,Tracing session rotation>> section above), |
| 854 | which is available since LTTng{nbsp}2.11. |
| 855 | ==== |
| 856 | |
| 857 | |
| 858 | [[channel-timers]] |
| 859 | Timers |
| 860 | ~~~~~~ |
| 861 | Each channel can have up to three optional timers: |
| 862 | |
| 863 | Switch timer:: |
| 864 | When this timer expires, a sub-buffer switch happens: for each ring |
| 865 | buffer of the channel, LTTng marks the current sub-buffer as |
| 866 | consumable and switches to an available one to record the next |
| 867 | events. |
| 868 | + |
| 869 | A switch timer is useful to ensure that LTTng consumes and commits trace |
| 870 | data to trace files or to a distant relay daemon (man:lttng-relayd(8)) |
| 871 | periodically in case of a low event throughput. |
| 872 | + |
| 873 | Such a timer is also convenient when you use large sub-buffers (see the |
| 874 | <<channel-sub-buf-size-count,Sub-buffer size and count>> section above) |
| 875 | to cope with a sporadic high event throughput, even if the throughput is |
| 876 | otherwise low. |
| 877 | + |
| 878 | Set the period of the switch timer of a channel, or disable the timer |
| 879 | altogether, with the nloption:--switch-timer option of the |
| 880 | man:lttng-enable-channel(1) command. |
| 881 | |
| 882 | Read timer:: |
| 883 | When this timer expires, LTTng checks for full, consumable |
| 884 | sub-buffers. |
| 885 | + |
| 886 | By default, the LTTng tracers use an asynchronous message mechanism to |
| 887 | signal a full sub-buffer so that a consumer daemon can consume it. |
| 888 | + |
| 889 | When such messages must be avoided, for example in real-time |
| 890 | applications, use this timer instead. |
| 891 | + |
| 892 | Set the period of the read timer of a channel, or disable the timer |
| 893 | altogether, with the nloption:--read-timer option of the |
| 894 | man:lttng-enable-channel(1) command. |
| 895 | |
| 896 | Monitor timer:: |
| 897 | When this timer expires, the consumer daemon samples some channel |
| 898 | statistics to evaluate the following trigger conditions: |
| 899 | + |
| 900 | -- |
| 901 | . The consumed buffer size of a given tracing session becomes greater |
| 902 | than some value. |
| 903 | . The buffer usage of a given channel becomes greater than some value. |
| 904 | . The buffer usage of a given channel becomes less than some value. |
| 905 | -- |
| 906 | + |
| 907 | If you disable the monitor timer of a channel{nbsp}__C__: |
| 908 | + |
| 909 | -- |
| 910 | * The consumed buffer size value of the tracing session of{nbsp}__C__ |
| 911 | could be wrong for trigger condition type{nbsp}1: the consumed buffer |
| 912 | size of{nbsp}__C__ won't be part of the grand total. |
| 913 | |
| 914 | * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3) |
| 915 | for{nbsp}__C__ will never be satisfied. |
| 916 | -- |
| 917 | + |
| 918 | See the <<trigger,TRIGGER>> section above to learn more about triggers. |
| 919 | + |
| 920 | Set the period of the monitor timer of a channel, or disable the timer |
| 921 | altogether, with the nloption:--monitor-timer option of the |
| 922 | man:lttng-enable-channel(1) command. |
| 923 | |
| 924 | |
| 925 | [[recording-event-rule]] |
| 926 | {sect-recording-event-rule} |
| 927 | --------------------------- |
| 928 | A _recording event rule_ is a specific type of event rule (see the |
| 929 | <<"event-rule","{sect-event-rule}">> section above) of which the action is |
| 930 | to serialize and record the matched event as an _event record_. |
| 931 | |
| 932 | Set the explicit conditions of a recording event rule when you create it |
| 933 | with the man:lttng-enable-event(1) command. A recording event rule also |
| 934 | has the following implicit conditions: |
| 935 | |
| 936 | * The recording event rule itself is enabled. |
| 937 | + |
| 938 | A recording event rule is enabled on creation. |
| 939 | |
| 940 | * The channel to which the recording event rule is attached is enabled. |
| 941 | + |
| 942 | A channel is enabled on creation. |
| 943 | + |
| 944 | See the <<channel,{sect-channel}>> section above. |
| 945 | |
| 946 | * The tracing session of the recording event rule is active (started). |
| 947 | + |
| 948 | A tracing session is inactive (stopped) on creation. |
| 949 | + |
| 950 | See the <<session,{sect-session}>> section above. |
| 951 | |
| 952 | * The process for which LTTng creates an event to match is allowed to |
| 953 | record events. |
| 954 | + |
| 955 | All processes are allowed to record events on tracing session |
| 956 | creation. |
| 957 | + |
| 958 | Use the man:lttng-track(1) and man:lttng-untrack(1) commands to select |
| 959 | which processes are allowed to record events based on specific process |
| 960 | attributes. |
| 961 | |
| 962 | You always attach a recording event rule to a channel, which belongs to |
| 963 | a tracing session, when you create it. |
| 964 | |
| 965 | When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__, |
| 966 | LTTng attempts to serialize and record{nbsp}__E__ to one of the |
| 967 | available sub-buffers of the channel to which{nbsp}__E__ is attached. |
| 968 | |
| 969 | When multiple matching recording event rules are attached to the same |
| 970 | channel, LTTng attempts to serialize and record the matched event |
| 971 | _once_. In the following example, the second recording event rule is |
| 972 | redundant when both are enabled: |
| 973 | |
| 974 | [role="term"] |
| 975 | ---- |
| 976 | $ lttng enable-event --userspace hello:world |
| 977 | $ lttng enable-event --userspace hello:world --loglevel=INFO |
| 978 | ---- |
| 979 | |
| 980 | List the recording event rules of a specific tracing session |
| 981 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) |
| 982 | commands. |
| 983 | |
| 984 | Disable a recording event rule with the man:lttng-disable-event(1) |
| 985 | command. |
| 986 | |
| 987 | As of LTTng{nbsp}{lttng_version}, you cannot remove a recording event |
| 988 | rule: it exists as long as its tracing session exists. |
| 989 | |
| 990 | |
| 991 | include::common-footer.txt[] |
| 992 | |
| 993 | |
| 994 | SEE ALSO |
| 995 | -------- |
| 996 | man:lttng(1), |
| 997 | man:lttng-relayd(8), |
| 998 | man:lttng-sessiond(8) |