1 \input texinfo @c -*-texinfo-*-
4 @settitle LTTng Userspace Tracer (UST) Manual
8 This manual is for program, version version.
10 Copyright @copyright{} copyright-owner.
13 Permission is granted to ...
18 @title LTTng Userspace Tracer (UST) Manual
19 @c @subtitle subtitle-if-any
20 @c @subtitle second-subtitle
23 @c The following two commands
24 @c start the copyright page.
26 @c @vskip 0pt plus 1filll
32 @c So the toc is printed at the start.
37 @top LTTng Userspace Tracer
39 This manual is for UST 0.11.
46 * Instrumenting an application::
51 * List of environment variables detected by libust::
53 @c * Copying:: Your rights and freedoms.
62 * Supported platforms::
68 The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to
71 Code may be instrumented with either markers or tracepoints. A highly efficient
72 lockless tracer records these events to a trace buffers. These buffers are reaped
73 by a deamon which writes trace data to disk.
75 High performance is achieved by the use of lockless buffering algorithms, RCU and
76 per-cpu buffers. In addition, special care is taken to minize cache impact.
80 The LTTng Userspace Tracer is intended to be linkable to open source software
81 as well as to proprietary applications. This was accomplished by licensing
82 the code that needs to be linked to the traced program as @acronym{LGPL}.
84 Components licensed as LGPL v2.1:
91 Components licensed as GPL v2:
98 @node Supported platforms
99 @section Supported platforms
101 UST can currently trace applications running on Linux, on the x86-32, x86-64
102 and PowerPC 32 architectures.
105 @chapter Installation
107 The LTTng userspace tracer is a library and a set of userspace tools.
109 The following packages are required:
115 This contains the tracing library, the ust-consumerd daemon, trace control tools
116 and other helper tools.
118 Repository: @url{http://git.dorsal.polymtl.ca}
123 This is the userspace read-copy update library by Mathieu Desnoyers.
125 Available in Debian as package liburcu-dev.
127 Home page: @url{http://lttng.org/urcu}
132 LTTV is a graphical (and text) viewer for LTTng traces.
134 Home page: @url{http://lttng.org}
138 Liburcu should be installed first. UST may then be compiled and installed. LTTV
139 has no dependency on the other packages; it may therefore be installed on a
140 system which does not have UST installed.
142 Refer to the README in each of these packages for installation instructions.
150 First, instrument a program with a marker.
155 #include <ust/marker.h>
157 int main(int argc, char **argv)
162 /* ... set values of v and st ... */
165 ust_marker(myevent, "firstarg %d secondarg %s", v, st);
167 /* a marker without arguments: */
168 ust_marker(myotherevent, MARK_NOARGS);
176 Then compile it in the regular way, linking it with libust. For example:
179 gcc -o foo -lust foo.c
182 Run the program with @command{usttrace}. The @command{usttrace} output says where the trace
189 Finally, open the trace in LTTV.
192 lttv-gui -t /path/to/trace
195 The trace can also be dumped as text in the console:
198 lttv -m textDump -t /path/to/trace
201 @node Instrumenting an application
202 @chapter Instrumenting an application
204 In order to record a trace of events occurring in a application, the
205 application must be instrumented. Instrumentation points resemble function
206 calls. When the program reaches an instrumentation point, an event is
209 There are no limitations on the type of code that may be instrumented.
210 Multi-threaded programs may be instrumented without problem. Signal handlers
211 may be instrumented as well.
213 There are two APIs to instrument programs: markers and tracepoints. Markers are
214 quick to add and are usually used for temporary instrumentation. Tracepoints
215 provide a way to instrument code more cleanly and are suited for permanent
218 In addition to executable programs, shared libraries may also be instrumented
219 with the methods described in this chapter.
229 Adding a marker is simply a matter of inserting one line in the program.
233 #include <ust/marker.h>
235 int main(int argc, char **argv)
240 /* ... set values of v and st ... */
243 ust_marker(myevent, "firstarg %d secondarg %s", v, st);
245 /* another marker without arguments: */
246 ust_marker(myotherevent, MARK_NOARGS);
253 The invocation of the ust_marker() macro requires at least 2 arguments. The
254 first, "myevent", is the name of the event. The second is a format string
255 that announces the names and the types of the event arguments. Its
256 format resembles that of a printf() format string; it is described
257 thoroughly in Appendix x.
259 A given Marker may appear more than once in the same program. Other Markers may
260 have the same name and a different format string, although this might induce
261 some confusion at analysis time.
266 The Tracepoints API uses the Markers, but provides a higher-level abstraction.
267 Whereas the markers API provides limited type checking, the Tracepoints API
268 provides more thorough type checking and discharges from the need to insert
269 format strings directly in the code and to have format strings appear more than
270 once if a given marker is reused.
273 The @command{usttrace} tool always uses the early tracing mode.
276 A function instrumented with a tracepoint looks like this:
287 /* ... set values of v and st ... */
290 trace_myevent(v, st);
295 Another file, here tp.h, contains declarations for the tracepoint.
299 #include <ust/tracepoint.h>
301 DECLARE_TRACEPOINT(myevent, TP_PROTO(int v, char *st),
306 A third file, here tp.c, contains definitions for the tracepoint.
310 #include <ust/marker.h>
313 DEFINE_TRACEPOINT(myevent);
315 void myevent_probe(int v, char *st)
317 ust_marker(myevent, "v %d st %s", v, st);
320 static void __attribute__((constructor)) init()
322 register_trace_myevent(myevent_probe);
327 Here, tp.h and tp.c could contain declarations and definitions for other
328 tracepoints. The constructor would contain other register_* calls.
330 @node Recording a trace
331 @chapter Recording a trace
334 * Using @command{usttrace}::
335 * Setting up the recording manually::
336 * Using early tracing::
338 * Tracing across @code{fork()} and @code{clone()}::
339 * Tracing programs and libraries that were not linked to libust::
342 @node Using @command{usttrace}
343 @section Using @command{usttrace}
345 The simplest way to record a trace is to use the @command{usttrace} script. An
346 example is given in the quickstart above.
348 The @command{usttrace} script automatically:
350 @item creates a daemon
351 @item enables all markers
352 @item runs the command specified on the command line
353 @item after the command ends, prints the location where the trace was saved
356 Each subdirectory of the save location contains the trace of one process that
357 was generated by the command. The name of a subdirectory consists in the the PID
358 of the process, followed by the timestamp of its creation.
360 The save location also contains logs of the tracing.
362 When using @command{usttrace}, the early tracing is always active, which means
363 that the tracing is guaranteed to be started by the time the process enters its
364 @code{main()} function.
366 Several @command{usttrace}'s may be run simultaneously without risk of
367 conflict. This facilitates the use of the tracer by idependent users on a
368 system. Each instance of @command{usttrace} starts its own daemon which
369 collects the events of the processes it creates.
371 @node Setting up the recording manually
372 @section Setting up the recording manually
374 Instead of using @command{usttrace}, a trace may be recorded on an already
377 First the daemon must be started.
381 # Make sure the directory for the communication sockets exists.
382 $ mkdir /tmp/ustsocks
384 # Make sure the directory where ust-consumerd will write the trace exists.
390 # We assume the program we want to trace is already running and that
393 # List the available markers
394 $ ustctl list-markers 1234
395 # A column indicates 0 for an inactive marker and 1 for an active marker.
398 $ ustctl enable-marker 1234 auto ust/mymark
401 $ ustctl create-trace 1234 auto
404 $ ustctl start-trace 1234 auto
409 $ ustctl stop-trace 1234 auto
412 $ ustctl destroy-trace 1234 auto
416 For more information about the manual mode, see the ustctl(1) man page.
418 @node Using early tracing
419 @section Using early tracing
421 Early tracing consists in starting the tracing as early as possible in the
422 program, so no events are lost between program start and the point where the
423 command to start the tracing is given. When using early tracing, it is
424 guaranteed that by the time the traced program enters its @code{main()}
425 function, the tracing will be started.
427 When using @command{usttrace}, the early tracing is always active.
429 When using the manual mode (@command{ustctl}), early tracing is enabled using
430 environment variables. Setting @env{UST_TRACE} to @code{1}, enables early
431 tracing, while setting @env{UST_AUTOPROBE} to @code{1} enables all markers
436 @section Crash recovery
438 When a process being traced crashes, the daemon is able to recover all the
439 events in its buffers that were successfully commited. This is possible because
440 the buffers are in a shared memory segment which remains available to the
441 daemon even after the termination of the traced process.
443 @node Tracing across @code{fork()} and @code{clone()}
444 @section Tracing across @code{fork()} and @code{clone()}
446 Tracing across @code{clone()} when the @code{CLONE_VM} flag is specified is
447 supported without any particular action.
449 When @code{clone()} is called without @code{CLONE_VM} or @code{fork()} is
450 called, a new address space is created and the tracer must be notified to
451 create new buffers for it.
453 This can be done automatically, by @env{LD_PRELOAD}'ing @file{libinterfork.so}.
454 This library intercepts calls to @code{fork()} and informs the tracer it is
455 being called. When using @command{usttrace}, this is accomplied by specifying
456 the @option{-f} command line argument.
458 Alternatively, the program can call @code{ust_before_fork()} before calling
459 @code{fork()} or @code{clone()} with @code{CLONE_VM}. After the call,
460 @code{ust_after_fork_parent()} must be called in the parent process and
461 @code{ust_after_fork_child()} must be called in the child process.
464 @node Tracing programs and libraries that were not linked to libust
465 @section Tracing programs and libraries that were not linked to libust
467 Some programs need to be traced even though they were not linked to libust
468 either because they were not instrumented or because it was not practical.
470 An executable that is not instrumented can still yield interesting traces when
471 at least one of its dynamic libraries is instrumented. It is also possible to
472 trace certain function calls by intercepting them with a specially crafted
473 library that is linked with @env{LD_PRELOAD} at program start.
475 In any case, a program that was not linked to libust at compile time must be
476 linked to it at run time with @env{LD_PRELOAD}. This can be accomplished with
477 @command{usttrace}'s @option{-l} option. It can also be done by setting the
478 @env{LD_PRELOAD} environment variable on the command line. For example:
482 # Run ls with usttrace, LD_PRELOAD'ing libust
483 # (assuming one of the libraries used by ls is instrumented).
486 # Run ls, manually adding the LD_PRELOAD.
487 $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls
498 @chapter Viewing traces
500 Traces may be viewed with LTTV. An example of command for launching LTTV is
501 given in the quickstart.
504 * Viewing multiple traces::
505 * Combined kernel-userspace tracing::
508 @node Viewing multiple traces
509 @section Viewing multiple traces
511 When tracing multi-process applications or several applications simultaneously,
512 more than one trace will be obtained. LTTV can open and display all these
513 traces simultaneously.
515 @node Combined kernel-userspace tracing
516 @section Combined kernel-userspace tracing
518 In addition to multiple userspace traces, LTTV can open a kernel trace recorded
519 with the LTTng kernel tracer. This provides events that enable the rendering of
520 the Control Flow View and the Resource View.
522 When doing so, it is necessary to use the same time source for the kernel
523 tracer as well as the userspace tracer. Currently, the recommended method is to
524 use the timestamp counter for both. The TSC can however only be used on architectures
525 where it is synchronized across cores.
528 @chapter Resource Usage
530 The purpose of this section is to give an overview of the resource usage of libust. For
531 a developer, knowing this can be important: because libust is linked with applications, it
532 needs to share some resources with it. Some applications may make some assumptions that are in
533 conflict with libust's usage of resources.
535 In practice however, libust is designed to be transparent and is compatible
536 with the vast majority of applications. This means no changes are required in
537 the application (or library) being linked to libust.
539 Libust is initialized by a constructor, which by definition runs before the
540 @code{main()} function of the application starts. This constructor creates a
541 thread called the @emph{listener thread}. The listener thread initializes a
542 named socket and waits for connections for ust-consumerd or ustctl.
544 Libust-specific code may:
546 @item use @code{malloc()} and @code{free()}
547 @item map shared memory segment in the process adress space
548 @item intercept some library calls, specifically @code{fork()} and @code{clone()}
549 @item do interprocess communication with the daemon or ustctl
550 @item create and open named sockets
556 @item handle any signal (all signals are blocked in the listener thread)
557 @item change any process-wide setting that could confuse the application
560 @node List of environment variables detected by libust
561 @appendix List of environment variables detected by libust
563 The behavior of tracing can be influenced by setting special environment
564 variables in the environment of the traced application. This section
565 describes these variables.
572 If set to 1, start tracing as soon as the program starts. Tracing is
573 guaranteed to be started by the time the @code{main()} function starts.
578 If set to @code{1}, enable all markers by the time the @code{main()} function starts.
581 @env{UST_AUTOCOLLECT}
583 If set to @code{0}, disable notification of daemon on trace start. Useful for
589 If set to @code{1}, enable overwriting of buffers on overrun.
594 If set, defines the default number of subbuffers per buffer.
597 @env{UST_SUBBUF_SIZE}
599 If set, defines the default size of subbuffers, in bytes.
603 @node GDB integration
604 @appendix GDB integration
606 GDB, the GNU Debugger, can use UST markers as GDB tracepoints (note GDB has its
607 own concept of tracepoint). This feature is called GDB Static Tracepoints. When
608 a GDB tracepoint is hit, GDB collects the marker arguments, as well as the
609 state of the registers. Support for GDB is currently work in progress.