Commit | Line | Data |
---|---|---|
26f0c779 PP |
1 | lttng-concepts(7) |
2 | ================= | |
a55e08e0 | 3 | :revdate: 18 May 2021 |
26f0c779 PP |
4 | :sect-event-rule: INSTRUMENTATION POINT, EVENT RULE, AND EVENT |
5 | :sect-session: TRACING SESSION | |
6 | :sect-domain: TRACING DOMAIN | |
7 | :sect-channel: CHANNEL AND RING BUFFER | |
8 | :sect-recording-event-rule: RECORDING EVENT RULE AND EVENT RECORD | |
9 | ||
10 | ||
11 | NAME | |
12 | ---- | |
13 | lttng-concepts - LTTng concepts | |
14 | ||
15 | ||
16 | DESCRIPTION | |
17 | ----------- | |
18 | This manual page documents the concepts of LTTng. | |
19 | ||
20 | Many other LTTng manual pages refer to this one so that you can | |
21 | understand what are the various LTTng objects and how they relate to | |
22 | each other. | |
23 | ||
24 | The concepts of LTTng{nbsp}{lttng_version} are: | |
25 | ||
26 | * Instrumentation point, event rule, and event | |
27 | * Trigger | |
28 | * Tracing session | |
29 | * Tracing domain | |
30 | * Channel and ring buffer | |
31 | * Recording event rule and event record | |
32 | ||
33 | ||
34 | [[event-rule]] | |
35 | {sect-event-rule} | |
36 | ----------------- | |
37 | An _instrumentation point_ is a point, within a piece of software, | |
38 | which, when executed, creates an LTTng _event_. | |
39 | ||
40 | LTTng offers various types of instrumentation; see the | |
da39b67c PP |
41 | ``<<inst-point-types,Instrumentation point types>>'' section below to |
42 | learn about them. | |
26f0c779 PP |
43 | |
44 | An _event rule_ is a set of conditions to match a set of events. | |
45 | ||
46 | When LTTng creates an event{nbsp}__E__, an event rule{nbsp}__ER__ is | |
47 | said to __match__{nbsp}__E__ when{nbsp}__E__ satisfies *all* the | |
48 | conditions of{nbsp}__ER__. This concept is similar to a regular | |
49 | expression which matches a set of strings. | |
50 | ||
51 | When an event rule matches an event, LTTng _emits_ the event, therefore | |
52 | attempting to execute one or more actions. | |
53 | ||
54 | [IMPORTANT] | |
55 | ==== | |
56 | The event creation and emission processes are documentation concepts to | |
57 | help understand the journey from an instrumentation point to the | |
58 | execution of actions. | |
59 | ||
60 | The actual creation of an event can be costly because LTTng needs to | |
101b92da | 61 | evaluate the arguments of the instrumentation point. |
26f0c779 PP |
62 | |
63 | In practice, LTTng implements various optimizations for the Linux kernel | |
da39b67c PP |
64 | and user space tracing domains (see the ``<<domain,{sect-domain}>>'' |
65 | section below) to avoid actually creating an event when the tracer | |
66 | knows, thanks to properties which are independent from the event payload | |
67 | and current context, that it would never emit such an event. Those | |
68 | properties are: | |
26f0c779 PP |
69 | |
70 | * The instrumentation point type (see the | |
da39b67c | 71 | ``<<inst-point-types,Instrumentation point types>>'' section below). |
26f0c779 PP |
72 | |
73 | * The instrumentation point name. | |
74 | ||
75 | * The instrumentation point log level. | |
76 | ||
77 | * For a recording event rule (see the | |
da39b67c | 78 | ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
26f0c779 PP |
79 | below): |
80 | ** The status of the rule itself. | |
da39b67c PP |
81 | ** The status of the channel (see the ``<<channel,{sect-channel}>>'' |
82 | section below). | |
26f0c779 | 83 | ** The activity of the tracing session (started or stopped; see |
da39b67c | 84 | the ``<<session,{sect-session}>>'' section below). |
26f0c779 PP |
85 | ** Whether or not the process for which LTTng would create the event is |
86 | allowed to record events (see man:lttng-track(1)). | |
87 | ||
88 | In other words: if, for a given instrumentation point{nbsp}__IP__, the | |
89 | LTTng tracer knows that it would never emit an event, | |
90 | executing{nbsp}__IP__ represents a simple boolean variable check and, | |
91 | for a Linux kernel recording event rule, a few process attribute checks. | |
92 | ==== | |
93 | ||
94 | As of LTTng{nbsp}{lttng_version}, there are two places where you can | |
95 | find an event rule: | |
96 | ||
97 | Recording event rule:: | |
98 | A specific type of event rule of which the action is to record the | |
99 | matched event as an event record. | |
100 | + | |
da39b67c | 101 | See the ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
26f0c779 PP |
102 | below. |
103 | + | |
104 | Create or enable a recording event rule with the | |
105 | man:lttng-enable-event(1) command. | |
106 | + | |
107 | List the recording event rules of a specific tracing session | |
108 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) | |
109 | commands. | |
110 | ||
111 | ``Event rule matches'' <<trigger,trigger>> condition (since LTTng{nbsp}2.13):: | |
112 | When the event rule of the trigger condition matches an event, LTTng | |
113 | can execute user-defined actions such as sending an LTTng | |
114 | notification, starting a tracing session, and more. | |
115 | + | |
116 | See man:lttng-add-trigger(1) and man:lttng-event-rule(7). | |
117 | ||
118 | For LTTng to emit an event{nbsp}__E__,{nbsp}__E__ must satisfy *all* the | |
119 | basic conditions of an event rule{nbsp}__ER__, that is: | |
120 | ||
121 | * The instrumentation point from which LTTng creates{nbsp}__E__ has a | |
122 | specific type. | |
123 | + | |
da39b67c PP |
124 | See the ``<<inst-point-types,Instrumentation point types>>'' section |
125 | below. | |
26f0c779 PP |
126 | |
127 | * A pattern matches the name of{nbsp}__E__ while another pattern | |
128 | doesn't. | |
129 | ||
130 | * The log level of the instrumentation point from which LTTng | |
131 | creates{nbsp}__E__ is at least as severe as some value, or is exactly | |
132 | some value. | |
133 | ||
134 | * The fields of the payload of{nbsp}__E__ and the current context fields | |
135 | satisfy a filter expression. | |
136 | ||
137 | A recording event rule has additional, implicit conditions to satisfy. | |
da39b67c | 138 | See the ``<<recording-event-rule,{sect-recording-event-rule}>>'' section |
26f0c779 PP |
139 | below to learn more. |
140 | ||
141 | ||
142 | [[inst-point-types]] | |
143 | Instrumentation point types | |
144 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
145 | As of LTTng{nbsp}{lttng_version}, the available instrumentation point | |
146 | types are, depending on the tracing domain (see the | |
da39b67c | 147 | ``<<domain,{sect-domain}>>'' section below): |
26f0c779 PP |
148 | |
149 | Linux kernel:: | |
150 | LTTng tracepoint::: | |
151 | A statically defined point in the source code of the kernel | |
152 | image or of a kernel module using the LTTng-modules macros. | |
153 | + | |
154 | List the available Linux kernel tracepoints with `lttng list --kernel`. | |
155 | See man:lttng-list(1) to learn more. | |
156 | ||
157 | Linux kernel system call::: | |
158 | Entry, exit, or both of a Linux kernel system call. | |
159 | + | |
160 | List the available Linux kernel system call instrumentation points with | |
161 | `lttng list --kernel --syscall`. See man:lttng-list(1) to learn more. | |
162 | ||
163 | Linux kprobe::: | |
164 | A single probe dynamically placed in the compiled kernel code. | |
165 | + | |
166 | When you create such an instrumentation point, you set its memory | |
167 | address or symbol name. | |
168 | ||
169 | Linux user space probe::: | |
170 | A single probe dynamically placed at the entry of a compiled | |
171 | user space application/library function through the kernel. | |
172 | + | |
173 | When you create such an instrumentation point, you set: | |
174 | + | |
175 | -- | |
176 | With the ELF method:: | |
177 | Its application/library path and its symbol name. | |
178 | ||
179 | With the USDT method:: | |
180 | Its application/library path, its provider name, and its probe name. | |
181 | + | |
182 | ``USDT'' stands for SystemTap User-level Statically Defined Tracing, | |
183 | a DTrace-style marker. | |
184 | -- | |
185 | + | |
186 | As of LTTng{nbsp}{lttng_version}, LTTng only supports USDT probes which | |
187 | are :not: reference-counted. | |
188 | ||
189 | Linux kretprobe::: | |
190 | Entry, exit, or both of a Linux kernel function. | |
191 | + | |
192 | When you create such an instrumentation point, you set the memory | |
193 | address or symbol name of its function. | |
194 | ||
195 | User space:: | |
196 | LTTng tracepoint::: | |
197 | A statically defined point in the source code of a C/$$C++$$ | |
198 | application/library using the LTTng-UST macros. | |
199 | + | |
200 | List the available Linux kernel tracepoints with | |
201 | `lttng list --userspace`. See man:lttng-list(1) to learn more. | |
202 | ||
203 | `java.util.logging`, Apache log4j, and Python:: | |
204 | Java or Python logging statement::: | |
205 | A method call on a Java or Python logger attached to an | |
206 | LTTng-UST handler. | |
207 | + | |
208 | List the available Java and Python loggers with `lttng list --jul`, | |
209 | `lttng list --log4j`, and `lttng list --python`. See man:lttng-list(1) | |
210 | to learn more. | |
211 | ||
212 | ||
213 | [[trigger]] | |
214 | TRIGGER | |
215 | ------- | |
216 | A _trigger_ associates a condition to one or more actions. | |
217 | ||
218 | When the condition of a trigger is satisfied, LTTng attempts to execute | |
219 | its actions. | |
220 | ||
221 | As of LTTng{nbsp}{lttng_version}, the available trigger conditions and | |
222 | actions are: | |
223 | ||
224 | Conditions:: | |
225 | + | |
226 | * The consumed buffer size of a given tracing | |
da39b67c | 227 | session (see the ``<<session,{sect-session}>>'' section below) |
26f0c779 PP |
228 | becomes greater than some value. |
229 | ||
230 | * The buffer usage of a given channel (see the | |
da39b67c PP |
231 | ``<<channel,{sect-channel}>>'' section below) becomes greater than |
232 | some value. | |
26f0c779 PP |
233 | |
234 | * The buffer usage of a given channel becomes less than some value. | |
235 | ||
236 | * There's an ongoing tracing session rotation (see the | |
da39b67c | 237 | ``<<rotation,Tracing session rotation>>'' section below). |
26f0c779 PP |
238 | |
239 | * A tracing session rotation becomes completed. | |
240 | ||
241 | * An event rule matches an event. | |
242 | + | |
243 | As of LTTng{nbsp}{lttng_version}, this is the only available condition | |
244 | when you add a trigger with the man:lttng-add-trigger(1) command. The | |
245 | other ones are available through the liblttng-ctl C{nbsp}API. | |
246 | ||
247 | Actions:: | |
248 | + | |
249 | * Send a notification to a user application. | |
250 | * Start a given tracing session, like man:lttng-start(1) would do. | |
251 | * Stop a given tracing session, like man:lttng-stop(1) would do. | |
252 | * Archive the current trace chunk of a given tracing session (rotate), | |
253 | like man:lttng-rotate(1) would do. | |
254 | * Take a snapshot of a given tracing session, like man:lttng-snapshot(1) | |
255 | would do. | |
256 | ||
257 | A trigger belongs to a session daemon (see man:lttng-sessiond(8)), not | |
258 | to a specific tracing session. For a given session daemon, each Unix | |
259 | user has its own, private triggers. Note, however, that the `root` Unix | |
260 | user may, for the root session daemon: | |
261 | ||
262 | * Add a trigger as another Unix user. | |
263 | ||
264 | * List all the triggers, regardless of their owner. | |
265 | ||
266 | * Remove a trigger which belongs to another Unix user. | |
267 | ||
268 | For a given session daemon and Unix user, a trigger has a unique name. | |
269 | ||
270 | Add a trigger to a session daemon with the man:lttng-add-trigger(1) | |
271 | command. | |
272 | ||
273 | List the triggers of your Unix user (or of all users if your | |
274 | Unix user is `root`) with the man:lttng-list-triggers(1) command. | |
275 | ||
276 | Remove a trigger with the man:lttng-remove-trigger(1) command. | |
277 | ||
278 | ||
279 | [[session]] | |
280 | {sect-session} | |
281 | -------------- | |
282 | A _tracing session_ is a stateful dialogue between you and a session | |
283 | daemon (see man:lttng-sessiond(8)) for everything related to event | |
284 | recording. | |
285 | ||
286 | Everything that you do when you control LTTng tracers to record events | |
287 | happens within a tracing session. In particular, a tracing session: | |
288 | ||
289 | * Has its own name, unique for a given session daemon. | |
290 | ||
291 | * Has its own set of trace files, if any. | |
292 | ||
293 | * Has its own state of activity (started or stopped). | |
294 | + | |
295 | An active tracing session is an implicit recording event rule condition | |
da39b67c PP |
296 | (see the ``<<recording-event-rule,{sect-recording-event-rule}>>'' |
297 | section below). | |
26f0c779 PP |
298 | |
299 | * Has its own mode (local, network streaming, snapshot, or live). | |
300 | + | |
da39b67c PP |
301 | See the ``<<session-modes,Tracing session modes>>'' section below to |
302 | learn more. | |
26f0c779 | 303 | |
da39b67c | 304 | * Has its own channels (see the ``<<channel,{sect-channel}>>'' section |
26f0c779 PP |
305 | below) to which are attached their own recording event rules. |
306 | ||
307 | * Has its own process attribute inclusion sets (see man:lttng-track(1)). | |
308 | ||
309 | Those attributes and objects are completely isolated between different | |
310 | tracing sessions. | |
311 | ||
312 | A tracing session is like an ATM session: the operations you do on the | |
313 | banking system through the ATM don't alter the data of other users of | |
314 | the same system. In the case of the ATM, a session lasts as long as your | |
315 | bank card is inside. In the case of LTTng, a tracing session lasts from | |
316 | the man:lttng-create(1) command to the man:lttng-destroy(1) command. | |
317 | ||
318 | A tracing session belongs to a session daemon (see | |
319 | man:lttng-sessiond(8)). For a given session daemon, each Unix user has | |
320 | its own, private tracing sessions. Note, however, that the `root` Unix | |
321 | user may operate on or destroy another user's tracing session. | |
322 | ||
323 | Create a tracing session with the man:lttng-create(1) command. | |
324 | ||
325 | List the tracing sessions of the connected session daemon with | |
326 | the man:lttng-list(1) command. | |
327 | ||
328 | Start and stop a tracing session with the man:lttng-start(1) and | |
329 | man:lttng-stop(1) commands. | |
330 | ||
331 | Save and load a tracing session with the man:lttng-save(1) and | |
332 | man:lttng-load(1) commands. | |
333 | ||
334 | Archive the current trace chunk of (rotate) a tracing session with the | |
335 | man:lttng-rotate(1) command. | |
336 | ||
337 | Destroy a tracing session with the man:lttng-destroy(1) command. | |
338 | ||
339 | ||
340 | Current tracing session | |
341 | ~~~~~~~~~~~~~~~~~~~~~~~ | |
342 | When you run the man:lttng-create(1) command, LTTng creates the | |
343 | `$LTTNG_HOME/.lttngrc` file if it doesn't exist (`$LTTNG_HOME` defaults | |
344 | to `$HOME`). | |
345 | ||
346 | `$LTTNG_HOME/.lttngrc` contains the name of the _current tracing | |
347 | session_. | |
348 | ||
349 | When you create a new tracing session with the `create` command, LTTng | |
350 | updates the current tracing session. | |
351 | ||
352 | The following man:lttng(1) commands select the current tracing session | |
353 | if you don't specify one: | |
354 | ||
355 | * man:lttng-add-context(1) | |
356 | * man:lttng-clear(1) | |
357 | * man:lttng-destroy(1) | |
358 | * man:lttng-disable-channel(1) | |
359 | * man:lttng-disable-event(1) | |
360 | * man:lttng-disable-rotation(1) | |
361 | * man:lttng-enable-channel(1) | |
362 | * man:lttng-enable-event(1) | |
363 | * man:lttng-enable-rotation(1) | |
364 | * man:lttng-regenerate(1) | |
365 | * man:lttng-rotate(1) | |
366 | * man:lttng-save(1) | |
367 | * man:lttng-snapshot(1) | |
368 | * man:lttng-start(1) | |
369 | * man:lttng-status(1) | |
370 | * man:lttng-stop(1) | |
371 | * man:lttng-track(1) | |
372 | * man:lttng-untrack(1) | |
373 | * man:lttng-view(1) | |
374 | ||
375 | Set the current tracing session manually with the | |
376 | man:lttng-set-session(1) command, without having to edit the `.lttngrc` | |
377 | file. | |
378 | ||
379 | ||
380 | [[session-modes]] | |
381 | Tracing session modes | |
382 | ~~~~~~~~~~~~~~~~~~~~~ | |
383 | LTTng offers four tracing session modes: | |
384 | ||
385 | Local mode:: | |
386 | Write the trace data to the local file system. | |
387 | ||
388 | Network streaming mode:: | |
389 | Send the trace data over the network to a listening relay daemon | |
390 | (see man:lttng-relayd(8)). | |
391 | ||
392 | Snapshot mode:: | |
393 | Only write the trace data to the local file system or send it to a | |
394 | listening relay daemon (man:lttng-relayd(8)) when LTTng takes a | |
395 | snapshot. | |
396 | + | |
da39b67c | 397 | LTTng forces all the channels (see the ``<<channel,{sect-channel}>>'' |
26f0c779 PP |
398 | section below) to be created to be configured to be snapshot-ready. |
399 | + | |
400 | LTTng takes a snapshot of such a tracing session when: | |
401 | + | |
402 | -- | |
403 | * You run the man:lttng-snapshot(1) command. | |
404 | ||
405 | * LTTng executes a `snapshot-session` trigger action (see the | |
da39b67c | 406 | ``<<trigger,TRIGGER>>'' section above). |
26f0c779 PP |
407 | -- |
408 | ||
409 | Live mode:: | |
410 | Send the trace data over the network to a listening relay daemon | |
411 | (see man:lttng-relayd(8)) for live reading. | |
412 | + | |
413 | An LTTng live reader (for example, man:babeltrace2(1)) can connect to | |
414 | the same relay daemon to receive trace data while the tracing session is | |
415 | active. | |
416 | ||
417 | ||
418 | [[rotation]] | |
419 | Tracing session rotation | |
420 | ~~~~~~~~~~~~~~~~~~~~~~~~ | |
421 | A _tracing session rotation_ is the action of archiving the current | |
422 | trace chunk of the tracing session to the file system. | |
423 | ||
424 | Once LTTng archives a trace chunk, it does :not: manage it anymore: you | |
425 | can read it, modify it, move it, or remove it. | |
426 | ||
427 | An _archived trace chunk_ is a collection of metadata and data stream | |
428 | files which form a self-contained LTTng trace. See the | |
da39b67c | 429 | ``<<trace-chunk-naming,Trace chunk naming>>'' section below to learn how |
26f0c779 PP |
430 | LTTng names a trace chunk archive directory. |
431 | ||
432 | The _current trace chunk_ of a given tracing session includes: | |
433 | ||
434 | * The stream files which LTTng already wrote to the file system, and | |
435 | which are not part of a previously archived trace chunk, since the | |
436 | most recent event amongst: | |
437 | ||
438 | ** The first time the tracing session was started, either with the | |
439 | man:lttng-start(1) command or with a `start-session` trigger action | |
da39b67c | 440 | (see the ``<<trigger,TRIGGER>>'' section above). |
26f0c779 PP |
441 | |
442 | ** The last rotation, performed with: | |
443 | ||
444 | *** An man:lttng-rotate(1) command. | |
445 | ||
446 | *** A rotation schedule previously set with | |
447 | man:lttng-enable-rotation(1). | |
448 | ||
449 | *** An executed `rotate-session` trigger action (see the | |
da39b67c | 450 | ``<<trigger,TRIGGER>>'' section above). |
26f0c779 PP |
451 | |
452 | * The content of all the non-flushed sub-buffers of the channels of the | |
453 | tracing session. | |
454 | ||
455 | ||
456 | [[trace-chunk-naming]] | |
457 | Trace chunk archive naming | |
458 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
459 | A trace chunk archive is a subdirectory of the `archives` subdirectory | |
460 | within the output directory of a tracing session (see the | |
461 | nloption:--output option of the man:lttng-create(1) command and | |
462 | of man:lttng-relayd(8)). | |
463 | ||
464 | A trace chunk archive contains, through tracing domain and possibly | |
465 | UID/PID subdirectories, metadata and data stream files. | |
466 | ||
467 | A trace chunk archive is, at the same time: | |
468 | ||
469 | * A self-contained LTTng trace. | |
470 | ||
471 | * A member of a set of trace chunk archives which form the complete | |
472 | trace of a tracing session. | |
473 | ||
474 | In other words, an LTTng trace reader can read both the tracing | |
475 | session output directory (all the trace chunk archives), or a | |
476 | single trace chunk archive. | |
477 | ||
478 | When LTTng performs a tracing session rotation, it names the resulting | |
479 | trace chunk archive as such, relative to the output directory of the | |
480 | tracing session: | |
481 | ||
482 | [verse] | |
483 | archives/__BEGIN__-__END__-__ID__ | |
484 | ||
485 | __BEGIN__:: | |
486 | Date and time of the beginning of the trace chunk archive with | |
487 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where | |
488 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the | |
489 | time zone offset from UTC. | |
490 | + | |
491 | Example: `20171119T152407-0500` | |
492 | ||
493 | __END__:: | |
494 | Date and time of the end of the trace chunk archive with | |
495 | the ISO{nbsp}8601-compatible __YYYYmmddTHHMMSS±HHMM__ form, where | |
496 | __YYYYmmdd__ is the date and __HHMMSS±HHMM__ is the time with the | |
497 | time zone offset from UTC. | |
498 | + | |
499 | Example: `20180118T152407+0930` | |
500 | ||
501 | __ID__:: | |
502 | Unique numeric identifier of the trace chunk within its tracing | |
503 | session. | |
504 | ||
505 | Trace chunk archive name example: | |
506 | ||
507 | ---- | |
508 | archives/20171119T152407-0500-20171119T151422-0500-3 | |
509 | ---- | |
510 | ||
511 | ||
512 | [[domain]] | |
513 | {sect-domain} | |
514 | ------------- | |
515 | A _tracing domain_ identifies a type of LTTng tracer. | |
516 | ||
517 | A tracing domain has its own properties and features. | |
518 | ||
519 | There are currently five available tracing domains: | |
520 | ||
521 | [options="header"] | |
522 | |=== | |
523 | |Tracing domain |``Event rule matches'' trigger condition option |Option for other CLI commands | |
524 | ||
525 | |Linux kernel | |
a55e08e0 | 526 | |nloption:--type option starts with `kernel:` |
26f0c779 PP |
527 | |nloption:--kernel |
528 | ||
529 | |User space | |
a55e08e0 | 530 | |nloption:--type option starts with `user:` |
26f0c779 PP |
531 | |nloption:--userspace |
532 | ||
533 | |`java.util.logging` (JUL) | |
a55e08e0 | 534 | |nloption:--type option starts with `jul:` |
26f0c779 PP |
535 | |nloption:--jul |
536 | ||
537 | |Apache log4j | |
a55e08e0 | 538 | |nloption:--type option starts with `log4j:` |
26f0c779 PP |
539 | |nloption:--log4j |
540 | ||
541 | |Python | |
a55e08e0 | 542 | |nloption:--type option starts with `python:` |
26f0c779 PP |
543 | |nloption:--python |
544 | |=== | |
545 | ||
546 | You must specify a tracing domain to target a type of LTTng tracer when | |
7013e167 PP |
547 | using some man:lttng(1) commands to avoid ambiguity. For example, |
548 | because the Linux kernel and user space tracing domains support named | |
549 | tracepoints as instrumentation points (see the | |
da39b67c PP |
550 | ``<<"event-rule","{sect-event-rule}">>'' section above), you need to |
551 | specify a tracing domain when you create an event rule because both | |
552 | tracing domains could have tracepoints sharing the same name. | |
26f0c779 | 553 | |
da39b67c | 554 | You can create channels (see the ``<<channel,{sect-channel}>>'' section |
26f0c779 PP |
555 | below) in the Linux kernel and user space tracing domains. The other |
556 | tracing domains have a single, default channel. | |
557 | ||
558 | ||
559 | [[channel]] | |
560 | {sect-channel} | |
561 | -------------- | |
562 | A _channel_ is an object which is responsible for a set of ring buffers. | |
563 | ||
564 | Each ring buffer is divided into multiple _sub-buffers_. When a | |
565 | recording event rule (see the | |
da39b67c | 566 | ``<<recording-event-rule,{sect-recording-event-rule}>>'' section below) |
26f0c779 PP |
567 | matches an event, LTTng can record it to one or more sub-buffers of one |
568 | or more channels. | |
569 | ||
570 | When you create a channel with the man:lttng-enable-channel(1) command, | |
571 | you set its final attributes, that is: | |
572 | ||
573 | * Its buffering scheme. | |
574 | + | |
da39b67c | 575 | See the ``<<channel-buf-scheme,Buffering scheme>>'' section below. |
26f0c779 PP |
576 | |
577 | * What to do when there's no | |
578 | space left for a new event record because all sub-buffers are full. | |
579 | + | |
da39b67c PP |
580 | See the ``<<channel-er-loss-mode,Event record loss mode>>'' section |
581 | below. | |
26f0c779 PP |
582 | |
583 | * The size of each ring buffer and how many sub-buffers a ring buffer | |
584 | has. | |
585 | + | |
da39b67c PP |
586 | See the ``<<channel-sub-buf-size-count,Sub-buffer size and count>>'' |
587 | section below. | |
26f0c779 PP |
588 | |
589 | * The size of each trace file LTTng writes for this channel and the | |
590 | maximum count of trace files. | |
591 | + | |
da39b67c PP |
592 | See the ``<<channel-max-trace-file-size-count,Maximum trace file size |
593 | and count>>'' section below. | |
26f0c779 PP |
594 | |
595 | * The periods of its read, switch, and monitor timers. | |
596 | + | |
da39b67c | 597 | See the ``<<channel-timers,Timers>>'' section below. |
26f0c779 PP |
598 | |
599 | * For a Linux kernel channel: its output type (man:mmap(2) or | |
600 | man:splice(2)). | |
601 | + | |
602 | See the nloption:--output option of the man:lttng-enable-channel(1) | |
603 | command. | |
604 | ||
605 | * For a user space channel: the value of its blocking timeout. | |
606 | + | |
607 | See the nloption:--blocking-timeout option of the | |
608 | man:lttng-enable-channel(1) command. | |
609 | ||
610 | Note that the man:lttng-enable-event(1) command can automatically create | |
611 | a default channel with sane defaults when no channel exists for the | |
612 | provided tracing domain. | |
613 | ||
614 | A channel is always associated to a tracing domain (see the | |
da39b67c PP |
615 | ``<<domain,{sect-domain}>>'' section below). The `java.util.logging` |
616 | (JUL), log4j, and Python tracing domains each have a default channel | |
617 | which you can't configure. | |
26f0c779 PP |
618 | |
619 | A channel owns recording event rules. | |
620 | ||
621 | List the channels of a given tracing session with the | |
622 | man:lttng-list(1) and man:lttng-status(1) commands. | |
623 | ||
624 | Disable an enabled channel with the man:lttng-disable-channel(1) | |
625 | command. | |
626 | ||
627 | ||
628 | [[channel-buf-scheme]] | |
629 | Buffering scheme | |
630 | ~~~~~~~~~~~~~~~~ | |
631 | A channel has at least one ring buffer per CPU. LTTng always records an | |
632 | event to the ring buffer dedicated to the CPU which emits it. | |
633 | ||
634 | The buffering scheme of a user space channel determines what has its own | |
635 | set of per-CPU ring buffers: | |
636 | ||
637 | Per-user buffering (nloption:--buffers-uid option of the man:lttng-enable-channel(1) command):: | |
638 | Allocate one set of ring buffers (one per CPU) shared by all the | |
639 | instrumented processes of: | |
640 | If your Unix user is `root`::: | |
641 | Each Unix user. | |
642 | Otherwise::: | |
643 | Your Unix user. | |
644 | ||
645 | Per-process buffering (nloption:--buffers-pid option of the man:lttng-enable-channel(1) command):: | |
646 | Allocate one set of ring buffers (one per CPU) for each instrumented | |
647 | process of: | |
648 | If your Unix user is `root`::: | |
649 | All Unix users. | |
650 | Otherwise::: | |
651 | Your Unix user. | |
652 | ||
653 | The per-process buffering scheme tends to consume more memory than the | |
654 | per-user option because systems generally have more instrumented | |
655 | processes than Unix users running instrumented processes. However, the | |
656 | per-process buffering scheme ensures that one process having a high | |
657 | event throughput won't fill all the shared sub-buffers of the same Unix | |
658 | user, only its own. | |
659 | ||
660 | The buffering scheme of a Linux kernel channel is always to allocate a | |
661 | single set of ring buffers for the whole system. This scheme is similar | |
662 | to the per-user option, but with a single, global user ``running'' the | |
663 | kernel. | |
664 | ||
665 | ||
666 | [[channel-er-loss-mode]] | |
667 | Event record loss mode | |
668 | ~~~~~~~~~~~~~~~~~~~~~~ | |
669 | When LTTng emits an event, LTTng can record it to a specific, available | |
670 | sub-buffer within the ring buffers of specific channels. When there's no | |
671 | space left in a sub-buffer, the tracer marks it as consumable and | |
672 | another, available sub-buffer starts receiving the following event | |
673 | records. An LTTng consumer daemon eventually consumes the marked | |
674 | sub-buffer, which returns to the available state. | |
675 | ||
676 | In an ideal world, sub-buffers are consumed faster than they are filled. | |
677 | In the real world, however, all sub-buffers can be full at some point, | |
678 | leaving no space to record the following events. | |
679 | ||
680 | By default, LTTng-modules and LTTng-UST are _non-blocking_ tracers: when | |
681 | there's no available sub-buffer to record an event, it's acceptable to | |
682 | lose event records when the alternative would be to cause substantial | |
683 | delays in the execution of the instrumented application. LTTng | |
684 | privileges performance over integrity; it aims at perturbing the | |
685 | instrumented application as little as possible in order to make the | |
686 | detection of subtle race conditions and rare interrupt cascades | |
687 | possible. | |
688 | ||
689 | Since LTTng{nbsp}2.10, the LTTng user space tracer, LTTng-UST, supports | |
690 | a _blocking mode_. See the nloption:--blocking-timeout of the | |
691 | man:lttng-enable-channel(1) command to learn how to use the blocking | |
692 | mode. | |
693 | ||
694 | When it comes to losing event records because there's no available | |
695 | sub-buffer, or because the blocking timeout of the channel is | |
696 | reached, the _event record loss mode_ of the channel determines what to | |
697 | do. The available event record loss modes are: | |
698 | ||
699 | Discard mode:: | |
700 | Drop the newest event records until a sub-buffer becomes available. | |
701 | + | |
702 | This is the only available mode when you specify a blocking timeout. | |
703 | + | |
704 | With this mode, LTTng increments a count of lost event records when an | |
705 | event record is lost and saves this count to the trace. A trace reader | |
706 | can use the saved discarded event record count of the trace to decide | |
707 | whether or not to perform some analysis even if trace data is known to | |
708 | be missing. | |
709 | ||
710 | Overwrite mode:: | |
711 | Clear the sub-buffer containing the oldest event records and start | |
712 | writing the newest event records there. | |
713 | + | |
714 | This mode is sometimes called _flight recorder mode_ because it's | |
715 | similar to a https://en.wikipedia.org/wiki/Flight_recorder[flight | |
716 | recorder]: always keep a fixed amount of the latest data. It's also | |
717 | similar to the roll mode of an oscilloscope. | |
718 | + | |
719 | Since LTTng{nbsp}2.8, with this mode, LTTng writes to a given sub-buffer | |
720 | its sequence number within its data stream. With a local, network | |
da39b67c PP |
721 | streaming, or live tracing session (see the ``<<session-modes,Tracing |
722 | session modes>>'' section above), a trace reader can use such sequence | |
26f0c779 PP |
723 | numbers to report lost packets. A trace reader can use the saved |
724 | discarded sub-buffer (packet) count of the trace to decide whether or | |
725 | not to perform some analysis even if trace data is known to be missing. | |
726 | + | |
727 | With this mode, LTTng doesn't write to the trace the exact number of | |
728 | lost event records in the lost sub-buffers. | |
729 | ||
730 | Which mechanism you should choose depends on your context: prioritize | |
731 | the newest or the oldest event records in the ring buffer? | |
732 | ||
733 | Beware that, in overwrite mode, the tracer abandons a _whole sub-buffer_ | |
734 | as soon as a there's no space left for a new event record, whereas in | |
735 | discard mode, the tracer only discards the event record that doesn't | |
736 | fit. | |
737 | ||
738 | Set the event record loss mode of a channel with the nloption:--discard | |
739 | and nloption:--overwrite options of the man:lttng-enable-channel(1) | |
740 | command. | |
741 | ||
742 | There are a few ways to decrease your probability of losing event | |
da39b67c PP |
743 | records. The ``<<channel-sub-buf-size-count,Sub-buffer size and |
744 | count>>'' section below shows how to fine-tune the sub-buffer size and | |
745 | count of a channel to virtually stop losing event records, though at the | |
746 | cost of greater memory usage. | |
26f0c779 PP |
747 | |
748 | ||
749 | [[channel-sub-buf-size-count]] | |
750 | Sub-buffer size and count | |
751 | ~~~~~~~~~~~~~~~~~~~~~~~~~ | |
752 | A channel has one or more ring buffer for each CPU of the target system. | |
753 | ||
da39b67c PP |
754 | See the ``<<channel-buf-scheme,Buffering scheme>>'' section above to |
755 | learn how many ring buffers of a given channel are dedicated to each CPU | |
26f0c779 PP |
756 | depending on its buffering scheme. |
757 | ||
758 | Set the size of each sub-buffer the ring buffers of a channel contain | |
759 | with the nloption:--subbuf-size option of the | |
760 | man:lttng-enable-channel(1) command. | |
761 | ||
762 | Set the number of sub-buffers each ring buffer of a channel contains | |
763 | with the nloption:--num-subbuf option of the man:lttng-enable-channel(1) | |
764 | command. | |
765 | ||
766 | Note that LTTng switching the current sub-buffer of a ring buffer | |
767 | (marking a full one as consumable and switching to an available one for | |
768 | LTTng to record the next events) introduces noticeable CPU overhead. | |
769 | Knowing this, the following list presents a few practical situations | |
770 | along with how to configure the sub-buffer size and count for them: | |
771 | ||
772 | High event throughput:: | |
773 | In general, prefer large sub-buffers to lower the risk of losing | |
774 | event records. | |
775 | + | |
776 | Having larger sub-buffers also ensures a lower sub-buffer switching | |
da39b67c | 777 | frequency (see the ``<<channel-timers,Timers>>'' section below). |
26f0c779 PP |
778 | + |
779 | The sub-buffer count is only meaningful if you create the channel in | |
da39b67c PP |
780 | overwrite mode (see the ``<<channel-er-loss-mode,Event record loss |
781 | mode>>'' section above): in this case, if LTTng overwrites a sub-buffer, | |
782 | then the other sub-buffers are left unaltered. | |
26f0c779 PP |
783 | |
784 | Low event throughput:: | |
785 | In general, prefer smaller sub-buffers since the risk of losing | |
786 | event records is low. | |
787 | + | |
788 | Because LTTng emits events less frequently, the sub-buffer switching | |
789 | frequency should remain low and therefore the overhead of the tracer | |
790 | shouldn't be a problem. | |
791 | ||
792 | Low memory system:: | |
793 | If your target system has a low memory limit, prefer fewer first, | |
794 | then smaller sub-buffers. | |
795 | + | |
796 | Even if the system is limited in memory, you want to keep the | |
797 | sub-buffers as large as possible to avoid a high sub-buffer switching | |
798 | frequency. | |
799 | ||
800 | Note that LTTng uses https://diamon.org/ctf/[CTF] as its trace format, | |
801 | which means event record data is very compact. For example, the average | |
802 | LTTng kernel event record weights about 32{nbsp}bytes. Therefore, a | |
803 | sub-buffer size of 1{nbsp}MiB is considered large. | |
804 | ||
805 | The previous scenarios highlight the major trade-off between a few large | |
806 | sub-buffers and more, smaller sub-buffers: sub-buffer switching | |
807 | frequency vs. how many event records are lost in overwrite mode. | |
808 | Assuming a constant event throughput and using the overwrite mode, the | |
809 | two following configurations have the same ring buffer total size: | |
810 | ||
811 | Two sub-buffers of 4{nbsp}MiB each:: | |
812 | Expect a very low sub-buffer switching frequency, but if LTTng | |
813 | ever needs to overwrite a sub-buffer, half of the event records so | |
814 | far (4{nbsp}MiB) are definitely lost. | |
815 | ||
816 | Eight sub-buffers of 1{nbsp}MiB each:: | |
817 | Expect four times the tracer overhead of the configuration above, | |
818 | but if LTTng needs to overwrite a sub-buffer, only the eighth of | |
819 | event records so far (1{nbsp}MiB) are definitely lost. | |
820 | ||
821 | In discard mode, the sub-buffer count parameter is pointless: use two | |
822 | sub-buffers and set their size according to your requirements. | |
823 | ||
824 | ||
825 | [[channel-max-trace-file-size-count]] | |
826 | Maximum trace file size and count | |
827 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
828 | By default, trace files can grow as large as needed. | |
829 | ||
830 | Set the maximum size of each trace file that LTTng writes of a given | |
831 | channel with the nloption:--tracefile-size option of the man:lttng-enable-channel(1) | |
832 | command. | |
833 | ||
834 | When the size of a trace file reaches the fixed maximum size of the | |
835 | channel, LTTng creates another file to contain the next event records. | |
836 | LTTng appends a file count to each trace file name in this case. | |
837 | ||
838 | If you set the trace file size attribute when you create a channel, the | |
839 | maximum number of trace files that LTTng creates is _unlimited_ by | |
840 | default. To limit them, use the nloption:--tracefile-count option of | |
841 | man:lttng-enable-channel(1). When the number of trace files reaches the | |
842 | fixed maximum count of the channel, LTTng overwrites the oldest trace | |
843 | file. This mechanism is called _trace file rotation_. | |
844 | ||
845 | [IMPORTANT] | |
846 | ==== | |
847 | Even if you don't limit the trace file count, always assume that LTTng | |
848 | manages all the trace files of the tracing session. | |
849 | ||
850 | In other words, there's no safe way to know if LTTng still holds a given | |
851 | trace file open with the trace file rotation feature. | |
852 | ||
853 | The only way to obtain an unmanaged, self-contained LTTng trace before | |
854 | you destroy the tracing session is with the tracing session rotation | |
da39b67c PP |
855 | feature (see the ``<<rotation,Tracing session rotation>>'' section |
856 | above), which is available since LTTng{nbsp}2.11. | |
26f0c779 PP |
857 | ==== |
858 | ||
859 | ||
860 | [[channel-timers]] | |
861 | Timers | |
862 | ~~~~~~ | |
863 | Each channel can have up to three optional timers: | |
864 | ||
865 | Switch timer:: | |
866 | When this timer expires, a sub-buffer switch happens: for each ring | |
867 | buffer of the channel, LTTng marks the current sub-buffer as | |
868 | consumable and switches to an available one to record the next | |
869 | events. | |
870 | + | |
871 | A switch timer is useful to ensure that LTTng consumes and commits trace | |
872 | data to trace files or to a distant relay daemon (man:lttng-relayd(8)) | |
873 | periodically in case of a low event throughput. | |
874 | + | |
875 | Such a timer is also convenient when you use large sub-buffers (see the | |
da39b67c PP |
876 | ``<<channel-sub-buf-size-count,Sub-buffer size and count>>'' section |
877 | above) to cope with a sporadic high event throughput, even if the | |
878 | throughput is otherwise low. | |
26f0c779 PP |
879 | + |
880 | Set the period of the switch timer of a channel, or disable the timer | |
881 | altogether, with the nloption:--switch-timer option of the | |
882 | man:lttng-enable-channel(1) command. | |
883 | ||
884 | Read timer:: | |
885 | When this timer expires, LTTng checks for full, consumable | |
886 | sub-buffers. | |
887 | + | |
888 | By default, the LTTng tracers use an asynchronous message mechanism to | |
889 | signal a full sub-buffer so that a consumer daemon can consume it. | |
890 | + | |
891 | When such messages must be avoided, for example in real-time | |
892 | applications, use this timer instead. | |
893 | + | |
894 | Set the period of the read timer of a channel, or disable the timer | |
895 | altogether, with the nloption:--read-timer option of the | |
896 | man:lttng-enable-channel(1) command. | |
897 | ||
898 | Monitor timer:: | |
899 | When this timer expires, the consumer daemon samples some channel | |
26f0c779 PP |
900 | statistics to evaluate the following trigger conditions: |
901 | + | |
902 | -- | |
903 | . The consumed buffer size of a given tracing session becomes greater | |
904 | than some value. | |
905 | . The buffer usage of a given channel becomes greater than some value. | |
906 | . The buffer usage of a given channel becomes less than some value. | |
907 | -- | |
908 | + | |
909 | If you disable the monitor timer of a channel{nbsp}__C__: | |
910 | + | |
911 | -- | |
912 | * The consumed buffer size value of the tracing session of{nbsp}__C__ | |
913 | could be wrong for trigger condition type{nbsp}1: the consumed buffer | |
914 | size of{nbsp}__C__ won't be part of the grand total. | |
915 | ||
916 | * The buffer usage trigger conditions (types{nbsp}2 and{nbsp}3) | |
917 | for{nbsp}__C__ will never be satisfied. | |
918 | -- | |
919 | + | |
da39b67c PP |
920 | See the ``<<trigger,TRIGGER>>'' section above to learn more about |
921 | triggers. | |
26f0c779 PP |
922 | + |
923 | Set the period of the monitor timer of a channel, or disable the timer | |
924 | altogether, with the nloption:--monitor-timer option of the | |
925 | man:lttng-enable-channel(1) command. | |
926 | ||
927 | ||
928 | [[recording-event-rule]] | |
929 | {sect-recording-event-rule} | |
930 | --------------------------- | |
931 | A _recording event rule_ is a specific type of event rule (see the | |
da39b67c PP |
932 | ``<<"event-rule","{sect-event-rule}">>'' section above) of which the |
933 | action is to serialize and record the matched event as an _event | |
934 | record_. | |
26f0c779 PP |
935 | |
936 | Set the explicit conditions of a recording event rule when you create it | |
937 | with the man:lttng-enable-event(1) command. A recording event rule also | |
938 | has the following implicit conditions: | |
939 | ||
940 | * The recording event rule itself is enabled. | |
941 | + | |
942 | A recording event rule is enabled on creation. | |
943 | ||
944 | * The channel to which the recording event rule is attached is enabled. | |
945 | + | |
946 | A channel is enabled on creation. | |
947 | + | |
da39b67c | 948 | See the ``<<channel,{sect-channel}>>'' section above. |
26f0c779 PP |
949 | |
950 | * The tracing session of the recording event rule is active (started). | |
951 | + | |
952 | A tracing session is inactive (stopped) on creation. | |
953 | + | |
da39b67c | 954 | See the ``<<session,{sect-session}>>'' section above. |
26f0c779 PP |
955 | |
956 | * The process for which LTTng creates an event to match is allowed to | |
957 | record events. | |
958 | + | |
959 | All processes are allowed to record events on tracing session | |
960 | creation. | |
961 | + | |
962 | Use the man:lttng-track(1) and man:lttng-untrack(1) commands to select | |
963 | which processes are allowed to record events based on specific process | |
964 | attributes. | |
965 | ||
966 | You always attach a recording event rule to a channel, which belongs to | |
967 | a tracing session, when you create it. | |
968 | ||
969 | When a recording event rule{nbsp}__ER__ matches an event{nbsp}__E__, | |
970 | LTTng attempts to serialize and record{nbsp}__E__ to one of the | |
971 | available sub-buffers of the channel to which{nbsp}__E__ is attached. | |
972 | ||
973 | When multiple matching recording event rules are attached to the same | |
974 | channel, LTTng attempts to serialize and record the matched event | |
975 | _once_. In the following example, the second recording event rule is | |
976 | redundant when both are enabled: | |
977 | ||
978 | [role="term"] | |
979 | ---- | |
980 | $ lttng enable-event --userspace hello:world | |
4fc37e3e | 981 | $ lttng enable-event --userspace hello:world --loglevel=INFO |
26f0c779 PP |
982 | ---- |
983 | ||
984 | List the recording event rules of a specific tracing session | |
985 | and/or channel with the man:lttng-list(1) and man:lttng-status(1) | |
986 | commands. | |
987 | ||
988 | Disable a recording event rule with the man:lttng-disable-event(1) | |
989 | command. | |
990 | ||
991 | As of LTTng{nbsp}{lttng_version}, you cannot remove a recording event | |
992 | rule: it exists as long as its tracing session exists. | |
993 | ||
994 | ||
995 | include::common-footer.txt[] | |
996 | ||
997 | ||
998 | SEE ALSO | |
999 | -------- | |
1000 | man:lttng(1), | |
1001 | man:lttng-relayd(8), | |
1002 | man:lttng-sessiond(8) |