1 \input texinfo @c -*-texinfo-*-
4 @settitle LTTng Userspace Tracer (UST) Manual
8 This manual is for program, version version.
10 Copyright @copyright{} copyright-owner.
13 Permission is granted to ...
18 @title LTTng Userspace Tracer (UST) Manual
19 @c @subtitle subtitle-if-any
20 @c @subtitle second-subtitle
23 @c The following two commands
24 @c start the copyright page.
26 @c @vskip 0pt plus 1filll
32 @c So the toc is printed at the start.
37 @top LTTng Userspace Tracer
39 This manual is for UST 0.1.
46 * Instrumenting an application::
50 @c * Copying:: Your rights and freedoms.
58 * Supported platforms::
63 The LTTng Userspace Tracer is intended to be linkable to open source software
64 as well as to proprietary applications. This was accomplished by licensing
65 the code that needs to be linked to the traced program as @acronym{LGPL}.
67 Components licensed as LGPL v2.1:
74 Components licensed as GPL v2:
81 @node Supported platforms
82 @section Supported platforms
84 UST can currently trace applications running on Linux, on the x86-32 and x86-64 architectures.
89 The LTTng userspace tracer is a library and a set of userspace tools.
91 The following packages are required:
97 This contains the tracing library, the ustd daemon, trace control tools
98 and other helper tools.
100 Repository: http://git.dorsal.polymtl.ca
105 This is a library that contains a userspace port of some kernel APIs.
107 Repository: http://git.dorsal.polymtl.ca
112 This is the userspace read-copy update library by Mathieu Desnoyers.
114 Available in Debian as package liburcu-dev.
116 Home page: http://lttng.org/?q=node/18
121 LTTV is a graphical (and text) viewer for LTTng traces.
123 Home page: http://lttng.org
127 Libkcompat and liburcu should be installed first. UST may then be compiled
128 and installed. LTTV has no dependency on the other packages; it may therefore
129 be installed on a system which does not have UST installed.
131 Refer to the README in each of these packages for installation instructions.
139 First, instrument a program with a marker.
144 #include <ust/marker.h>
146 int main(int argc, char **argv)
151 /* ... set values of v and st ... */
154 trace_mark(ust, myevent, "firstarg %d secondarg %s", v, st);
156 /* a marker without arguments: */
157 trace_mark(ust, myotherevent, MARK_NOARGS);
165 Then compile it in the regular way, linking it with libust. For example:
168 gcc -o foo -lust foo.c
171 Run the program with @command{usttrace}. The @command{usttrace} output says where the trace
178 Finally, open the trace in LTTV.
181 lttv-gui -t /path/to/trace
184 The trace can also be dumped as text in the console:
187 lttv -m textDump -t /path/to/trace
190 @node Instrumenting an application
191 @chapter Instrumenting an application
193 In order to record a trace of events occurring in a application, the
194 application must be instrumented. Instrumentation points resemble function
195 calls. When the program reaches an instrumentation point, an event is
198 There are no limitations on the type of code that may be instrumented.
199 Multi-threaded programs may be instrumented without problem. Signal handlers
200 may be instrumented as well.
202 There are two APIs to instrument programs: markers and tracepoints. Markers are
203 quick to add and are usually used for temporary instrumentation. Tracepoints
204 provide a way to instrument code more cleanly and are suited for permanent
207 In addition to executable programs, shared libraries may also be instrumented
208 with the methods described in this chapter.
218 Adding a marker is simply a matter of insert one line in the program.
222 #include <ust/marker.h>
224 int main(int argc, char **argv)
229 /* ... set values of v and st ... */
232 trace_mark(main, myevent, "firstarg %d secondarg %s", v, st);
234 /* a marker without arguments: */
235 trace_mark(main, myotherevent, MARK_NOARGS);
242 The invocation of the trace_mark() macro requires at least 3 arguments. The
243 first, here "main", is the name of the event category. It is also the name of
244 the channel the event will go in. The second, here "myevent" is the name of the
245 event. The third is a format string that announces the names and the types of
246 the event arguments. Its format resembles that of a printf() format string; it
247 is described thoroughly in Appendix x.
249 A given Marker may appear more than once in the same program. Other Markers may
250 have the same name and a different format string, although this might induce
251 some confusion at analysis time.
256 The Tracepoints API uses the Markers, but provides a higher-level abstraction.
257 Whereas the markers API provides limited type checking, the Tracepoints API
258 provides more thorough type checking and discharges from the need to insert
259 format strings directly in the code and to have format strings appear more than
260 once if a given marker is reused.
262 @quotation Note Although this example uses @emph{mychannel} as the channel, the
263 only channel name currently supported with early tracing is @strong{ust}. The
264 @command{usttrace} tool always uses the early tracing mode. When using manual
265 mode without early tracing, any channel name may be used. @end quotation
267 A function instrumented with a tracepoint looks like this:
278 /* ... set values of v and st ... */
281 trace_mychannel_myevent(v, st);
286 Another file, here tp.h, contains declarations for the tracepoint.
290 #include <ust/tracepoint.h>
292 DECLARE_TRACE(mychannel_myevent, TPPROTO(int v, char *st),
297 A third file, here tp.c, contains definitions for the tracepoint.
301 #include <ust/marker.h>
304 DEFINE_TRACE(mychannel_myevent);
306 void mychannel_myevent_probe(int v, char *st)
308 trace_mark(mychannel, myevent, "v %d st %s", v, st);
311 static void __attribute__((constructor)) init()
313 register_trace_mychannel_myevent(mychannel_myevent_probe);
318 Here, tp.h and tp.c could contain declarations and definitions for other
319 tracepoints. The constructor would contain other register_* calls.
321 @node Recording a trace
322 @chapter Recording a trace
325 * Using @command{usttrace}::
326 * Setting up the recording manually::
327 * Using early tracing::
329 * Tracing across @code{fork()} and @code{clone()}::
330 * Tracing programs and libraries that were not linked to libust::
333 @node Using @command{usttrace}
334 @section Using @command{usttrace}
336 The simplest way to record a trace is to use the @command{usttrace} script. An
337 example is given in the quickstart above.
339 The @command{usttrace} script automatically:
341 @item creates a daemon
342 @item enables all markers
343 @item runs the command specified on the command line
344 @item after the command ends, prints the location where the trace was saved
347 Each subdirectory of the save location contains the trace of one process that
348 was generated by the command. The name of a subdirectory consists in the the PID
349 of the process, followed by the timestamp of its creation.
351 The save location also contains logs of the tracing.
353 When using @command{usttrace}, the early tracing is always active, which means
354 that the tracing is guaranteed to be started by the time the process enters its
357 Several @command{usttrace}'s may be run simultaneously without risk of
358 conflict. This facilitates the use of the tracer by idependent users on a
359 system. Each instance of @command{usttrace} starts its own daemon which
360 collects the events of the processes it creates.
362 @node Setting up the recording manually
363 @section Setting up the recording manually
365 Instead of using @command{usttrace}, a trace may be recorded on an already
368 First the daemon must be started.
372 # Make sure the directory for the communication sockets exists.
373 $ mkdir /tmp/ustsocks
375 # Make sure the directory where ustd will write the trace exists.
381 # We assume the program we want to trace is already running and that
384 # List the available markers
385 $ ustctl --list-markers 1234
386 # A column indicates 0 for an inactive marker and 1 for an active marker.
389 $ ustctl --enable-marker ust/mymark 1234
392 $ ustctl --create-trace 1234
395 $ ustctl --start-trace 1234
400 $ ustctl --stop-trace 1234
403 $ ustctl --destroy-trace 1234
407 @node Using early tracing
408 @section Using early tracing
410 Early tracing consists in starting the tracing as early as possible in the
411 program, so no events are lost between program start and the point where the
412 command to start the tracing is given. When using early tracing, it is
413 guaranteed that by the time the traced program enters its @code{main()}
414 function, the tracing will be started.
416 When using @command{usttrace}, the early tracing is always active.
418 When using the manual mode (@command{ustctl}), early tracing is enabled using
419 environment variables. Setting @env{UST_TRACE} to @code{1}, enables early
420 tracing, while setting @env{UST_AUTOPROBE} to @code{1} enables all markers
425 @section Crash recovery
427 When a process being traced crashes, the daemon is able to recover all the
428 events in its buffers that were successfully commited. This is possible because
429 the buffers are in a shared memory segment which remains available to the
430 daemon even after the termination of the traced process.
432 @node Tracing across @code{fork()} and @code{clone()}
433 @section Tracing across @code{fork()} and @code{clone()}
435 Tracing across @code{clone()} when the @code{CLONE_VM} flag is specified is
436 supported without any particular action.
438 When @code{clone()} is called without @code{CLONE_VM} or @code{fork()} is
439 called, a new address space is created and the tracer must be notified to
440 create new buffers for it.
442 This can be done automatically, by @env{LD_PRELOAD}'ing @file{libinterfork.so}.
443 This library intercepts calls to @code{fork()} and informs the tracer it is
444 being called. When using @command{usttrace}, this is accomplied by specifying
445 the @option{-f} command line argument.
447 Alternatively, the program can call @code{ust_before_fork()} before calling
448 @code{fork()} or @code{clone()} with @code{CLONE_VM}. After the call,
449 @code{ust_after_fork_parent()} must be called in the parent process and
450 @code{ust_after_fork_child()} must be called in the child process.
453 @node Tracing programs and libraries that were not linked to libust
454 @section Tracing programs and libraries that were not linked to libust
456 Some programs need to be traced even though they were not linked to libust
457 either because they were not instrumented or because it was not practical.
459 An executable that is not instrumented can still yield interesting traces when
460 at least one of its dynamic libraries is instrumented. It is also possible to
461 trace certain function calls by intercepting them with a specially crafted
462 library that is linked with @env{LD_PRELOAD} at program start.
464 In any case, a program that was not linked to libust at compile time must be
465 linked to it at run time with @env{LD_PRELOAD}. This can be accomplished with
466 @command{usttrace}'s @option{-l} option. It can also be done by setting the
467 @env{LD_PRELOAD} environment variable on the command line. For example:
471 # Run ls with usttrace, LD_PRELOAD'ing libust
472 # (assuming one of the libraries used by ls is instrumented).
475 # Run ls, manually adding the LD_PRELOAD.
476 $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls
487 @chapter Viewing traces
489 Traces may be viewed with LTTV. An example of command for launching LTTV is
490 given in the quickstart.
493 * Viewing multiple traces::
494 * Combined kernel-userspace tracing::
497 @node Viewing multiple traces
498 @section Viewing multiple traces
500 When tracing multi-process applications or several applications simultaneously,
501 more than one trace will be obtained. LTTV can open and display all these
502 traces simultaneously.
504 @node Combined kernel-userspace tracing
505 @section Combined kernel-userspace tracing
507 In addition to multiple userspace traces, LTTV can open a kernel trace recorded
508 with the LTTng kernel tracer. This provides events that enable the rendering of
509 the Control Flow View and the Resource View.
511 When doing so, it is necessary to use the same time source for the kernel
512 tracer as well as the userspace tracer. Currently, the recommended method is to
513 use the timestamp counter for both. The TSC can however only be used on architectures
514 where it is synchronized across cores.