doc/proposals/0003-network.consumer.txt

   1 RFC - Network consumer
   2
   3 Author: David Goulet <david.goulet@efficios.com>
   4
   5 Contributors:
   6     * Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
   7     * Julien Desfossez <julien.desfossez@efficios.com>
   8
   9 Version:
  10     - v0.1: 16/04/2012
  11         * Initial proposal
  12         - v0.2: 04/05/2012
  13                 * Add snapshot description
  14                 * Propose a new API and lttng cli command
  15         - v0.3: 23/07/2012
  16                 * Remove snapshot and focus only on network consumer for straming.
  17
  18 Introduction
  19 -----------------
  20
  21 This RFC proposes a way for the lttng 2.0 session daemon to handle network
  22 session for streaming purposes and eventually remote control.
  23
  24 The next sections introduce the concept of network session and how it is
  25 envisioned in the lttng 2.0 toolchain.
  26
  27 Please note that this RFC is neither final nor complete without the community
  28 feedbacks. The text below is a proposal.
  29
  30 Network consumer
  31 -----------------
  32
  33 For version 2.1 of lttngt-tools, we propose to add a network consumer which will
  34 be used for streaming.
  35
  36 We allow to pass a full URI to the enable consumer command to override the
  37 current consumer or define a new one.
  38
  39 We should at least support the following protocols:
  40
  41 * net://HOST[:PORT_CTRL[:PORT_DATA]]
  42 * tcp://HOST:PORT
  43 * tcp6://HOST:PORT
  44 * udp://HOST:PORT
  45 * udp6://HOST:PORT
  46 * file://
  47
  48 The net:// URI scheme makes the control and data stream use the default
  49 transport protocol which is TCP. The same remote host is also
  50 used for both. The ports can be specified and if not the defaults are used which
  51 are 5342 for control and 5343 for data.
  52
  53 If URI not recognized, we use the arguments as a file name (same behavior as
  54 using file:///).
  55
  56 The control and data stream are two separate arguments of the API since we allow
  57 the user to control the protocol and path (address). However, for a transfer to
  58 succeed, the lttng-sessiond and the remote end must establish a session for the
  59 control _and_ data path. If one fails to do so, the procedure is aborted. Thus,
  60 a different address for the control path from the data path is allowed but the
  61 user has to make sure that both communication channels end up at the same
  62 physical destination.
  63
  64 Note that the control path is a crucial and high priority channel of
  65 communication so for now we only allow it to use the TCP protocol.
  66
  67 Session with Network Transport
  68 -----------------
  69
  70 In order to tell the session daemon where to send the data for streaming, a
  71 tracing session has to be aware of some information of the remote target.
  72
  73     * Remote end network address (Ex: IP or Hostname)
  74     * Destination control port
  75     * Destination data port
  76
  77 Streaming can be initiated by telling the session daemon that a specific session
  78 is set for network streaming. This will make the session daemon establish a
  79 connection with the remote end.  Once tracing starts, the local consumer will be
  80 made aware of this information and will start sending data following a strict
  81 protocol defined in the streaming RFC written by Julien Desfossez.
  82
  83 Finally, a trace received by a network consumer will have a new "namespace"
  84 prepended to the trace output directory hierarchy: the hostname from _where_ the
  85 trace is coming from.
  86
  87 host01
  88 \-- my_session1
  89     \-- ust
  90         \-- my_app1[...]
  91             \-- trace data...
  92     \-- kernel
  93         \-- trace data...
  94
  95 Client API integration
  96 -----------------
  97
  98 Adding an API call to set attributes such as network information to a session.
  99 Since lttng_create_session only takes a name and a path, a new call is required
 100 to pass this information. The naming convention is NOT final and can be
 101 improved.
 102
 103 struct lttng_handle handle;
 104
 105 enum lttng_dst_type {
 106         LTTNG_DST_IPV4,
 107         LTTNG_DST_IPV6,
 108         LTTNG_DST_HOST,
 109         LTTNG_DST_PATH,
 110 };
 111
 112 enum lttng_uri_type {
 113         LTTNG_URI_HOP,
 114         LTTNG_URI_DST,
 115 };
 116
 117 enum lttng_stream_type {
 118         LTTNG_STREAM_CONTROL,
 119         LTTNG_STREAM_DATA
 120 };
 121
 122 enum lttng_proto_type {
 123         LTTNG_UDP,
 124         LTTNG_TCP,
 125 };
 126
 127 #define LTTNG_NETWORK_PADDING1_LEN      32
 128 #define LTTNG_NETWORK_PADDING2_LEN      128
 129 struct lttng_uri {
 130         enum lttng_dst_type dtype;
 131         enum lttng_uri_type utype;
 132         enum lttng_stream_type stype;
 133         enum lttng_proto proto;
 134         in_port_t port;
 135         char padding[LTTNG_NETWORK_PADDING1_LEN];
 136         char subdir[PATH_MAX];
 137         union {
 138                 char ipv4[INET_ADDRSTRLEN];
 139                 char ipv6[INET6_ADDRSTRLEN];
 140                 char path[PATH_NAME];
 141                 char padding[LTTNG_NETWORK_PADDING2_LEN];
 142         } dst;
 143 };
 144
 145 /* Set URI in the consumer template*/
 146 lttng_set_consumer_uri(handle, struct lttng_uri *u);
 147
 148
 149 /*
 150  * Enable consumer template for the session. Once enabled, no more URI setting
 151  * are possible for that specific consumer.
 152  */
 153 lttng_enable_consumer(handle);
 154
 155 /*
 156  * Disable the consumer means that the consumer will stop consuming but will
 157  * still be exist. Executing the enable_consumer call again will simply re
 158  * enable it.
 159  */
 160 lttng_disable_consumer(handle);
 161
 162 If lttng_set_consumer_uri is executed on a session which already has a network
 163 consumer attached to it, the present consumer is freed and a new template is
 164 added.
 165
 166 We propose to add two commands to the lttng command line actions:
 167
 168 i) lttng enable-consumer [URI] [OPTIONS]
 169         -s SESSION_NAME
 170         -C, --control-uri=[HOP1,]URI
 171         -D, --data-uri=[HOP1,]URI
 172
 173 ii) lttng disable-consumer
 174         -s SESSION_NAME
 175
 176 Each option defining URI(s) can contains a list of hops preceeding the final
 177 destination. However, the proxy feature is still not supported but we prefer to
 178 inform the community of is future existence.
 179
 180 So, the regular chain of command to enable a network consumer would be:
 181
 182 # lttng create session1
 183 // The command sets the destination but uses the default protocols and ports.
 184 # lttng enable-consumer net://192.168.1.10
 185 # lttng enable-event -a -k
 186 # lttng start
 187 (tracing...)
 188 # lttng stop
 189
 190 (This example considers that there is a lttng-relayd on the remote end.)
 191
 192 Session daemon integration
 193 -----------------
 194
 195 As mentioned earlier, the session daemon will be in charge of establishing a
 196 streaming session with the target over the network i.e. creating the control and
 197 data path bidirectional socket. Once done, a network consumer is spawned and
 198 those sockets are passed over.
 199
 200 From there, the session daemon can interact with the consumer by stopping the
 201 network streaming or re-establishing a local trace collection with a non network
 202 consumer.