Commit | Line | Data |
---|---|---|
d589a916 PP |
1 | Userspace RCU Implementation |
2 | ============================ | |
3 | ||
4 | by Mathieu Desnoyers and Paul E. McKenney | |
5 | ||
6 | ||
7 | Building | |
8 | -------- | |
9 | ||
10 | ./bootstrap # skip if using tarball | |
11 | ./configure | |
12 | make | |
13 | make install | |
14 | ldconfig | |
15 | ||
16 | Hints: | |
17 | ||
18 | - Forcing 32-bit build: | |
19 | ||
20 | CFLAGS="-m32 -g -O2" ./configure | |
21 | ||
22 | - Forcing 64-bit build: | |
23 | ||
24 | CFLAGS="-m64 -g -O2" ./configure | |
25 | ||
26 | - Forcing a 32-bit build with 386 backward compatibility: | |
27 | ||
28 | CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu | |
29 | ||
30 | - Forcing a 32-bit build for Sparcv9 (typical for Sparc v9) | |
31 | ||
32 | CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure | |
33 | ||
34 | ||
35 | Architectures supported | |
36 | ----------------------- | |
37 | ||
38 | Currently, the following architectures are supported: | |
39 | ||
40 | - Linux x86 (i386, i486, i586, i686) | |
41 | - x86 64-bit | |
42 | - PowerPC 32/64 | |
43 | - S390, S390x | |
44 | - ARM 32/64 | |
45 | - MIPS | |
46 | - Alpha | |
47 | - ia64 | |
48 | - Sparcv9 32/64 | |
49 | - Tilera | |
50 | - hppa/PA-RISC | |
51 | ||
52 | Tested on Linux, FreeBSD 8.2/8.3/9.0/9.1/10.0 i386/amd64, and Cygwin. | |
53 | Should also work on: | |
54 | ||
55 | - Android | |
56 | - NetBSD 5 | |
57 | - OpenBSD | |
58 | - Darwin | |
59 | ||
60 | (more testing needed before claiming support for these OS). | |
61 | ||
62 | Linux ARM depends on running a Linux kernel 2.6.15 or better, GCC 4.4 or | |
63 | better. | |
64 | ||
65 | The GCC compiler versions 3.3, 3.4, 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5 are | |
66 | supported, with the following exceptions: | |
67 | ||
68 | - GCC 3.3 and 3.4 have a bug that prevents them from generating volatile | |
69 | accesses to offsets in a TLS structure on 32-bit x86. These versions are | |
70 | therefore not compatible with `liburcu` on x86 32-bit | |
71 | (i386, i486, i586, i686). | |
72 | The problem has been reported to the GCC community: | |
73 | http://www.mail-archive.com/gcc-bugs@gcc.gnu.org/msg281255.html | |
74 | - GCC 3.3 cannot match the "xchg" instruction on 32-bit x86 build. | |
75 | See http://kerneltrap.org/node/7507 | |
76 | - Alpha, ia64 and ARM architectures depend on GCC 4.x with atomic builtins | |
77 | support. For ARM this was introduced with GCC 4.4: | |
78 | http://gcc.gnu.org/gcc-4.4/changes.html. | |
79 | ||
80 | Clang version 3.0 (based on LLVM 3.0) is supported. | |
81 | ||
82 | Building on MacOS X (Darwin) requires a work-around for processor | |
83 | detection: | |
84 | ||
85 | - 32-bit: | |
86 | ||
87 | ./configure --build=i686-apple-darwin11 | |
88 | ||
89 | - 64-bit: | |
90 | ||
91 | ./configure --build=x86_64-apple-darwin11 | |
92 | ||
93 | For developers using the Git tree: | |
94 | ||
95 | This source tree is based on the autotools suite from GNU to simplify | |
96 | portability. Here are some things you should have on your system in order to | |
97 | compile the git repository tree : | |
98 | ||
99 | - GNU autotools (automake >=1.10, autoconf >=2.50, autoheader >=2.50) | |
100 | (make sure your system wide `automake` points to a recent version!) | |
101 | - GNU Libtool >=2.2 | |
102 | (for more information, go to http://www.gnu.org/software/autoconf/) | |
103 | ||
104 | If you get the tree from the repository, you will need to use the `bootstrap` | |
105 | script in the root of the tree. It calls all the GNU tools needed to prepare | |
106 | the tree configuration. | |
107 | ||
108 | Test scripts provided in the `tests/` directory of the source tree depend | |
109 | on `bash` and the `seq` program. | |
110 | ||
111 | ||
112 | API | |
113 | --- | |
114 | ||
115 | See the relevant API documentation files in `doc/`. The APIs provided by | |
116 | Userspace RCU are, by prefix: | |
117 | ||
dcb9c05a | 118 | - `rcu_`: Read-Copy Update (see [`doc/rcu-api.md`](doc/rcu-api.md)) |
d589a916 PP |
119 | - `cmm_`: Concurrent Memory Model |
120 | - `caa_`: Concurrent Architecture Abstraction | |
121 | - `cds_`: Concurrent Data Structures | |
dcb9c05a | 122 | (see [`doc/cds-api.md`](doc/cds-api.md)) |
d589a916 | 123 | - `uatomic_`: Userspace Atomic |
dcb9c05a | 124 | (see [`doc/uatomic-api.md`](doc/uatomic-api.md)) |
d589a916 PP |
125 | |
126 | ||
127 | Quick start guide | |
128 | ----------------- | |
129 | ||
130 | ### Usage of all urcu libraries: | |
131 | ||
132 | - Define `_LGPL_SOURCE` (only) if your code is LGPL or GPL compatible | |
133 | before including the `urcu.h` or `urcu-qsbr.h` header. If your application | |
134 | is distributed under another license, function calls will be generated | |
135 | instead of inlines, so your application can link with the library. | |
136 | - Linking with one of the libraries below is always necessary even for | |
137 | LGPL and GPL applications. | |
138 | - Define `URCU_INLINE_SMALL_FUNCTIONS` before including Userspace RCU | |
139 | headers if you want Userspace RCU to inline small functions (10 | |
140 | lines or less) into the application. It can be used by applications | |
141 | distributed under any kind of license, and does *not* make the | |
142 | application a derived work of Userspace RCU. | |
143 | ||
144 | Those small inlined functions are guaranteed to match the library | |
145 | content as long as the library major version is unchanged. | |
146 | Therefore, the application *must* be compiled with headers matching | |
147 | the library major version number. Applications using | |
148 | `URCU_INLINE_SMALL_FUNCTIONS` may be unable to use debugging | |
149 | features of Userspace RCU without being recompiled. | |
150 | ||
151 | ||
152 | ### Usage of `liburcu` | |
153 | ||
154 | 1. `#include <urcu.h>` | |
155 | 2. Link the application with `-lurcu` | |
156 | ||
157 | This is the preferred version of the library, in terms of | |
158 | grace-period detection speed, read-side speed and flexibility. | |
159 | Dynamically detects kernel support for `sys_membarrier()`. Falls back | |
160 | on `urcu-mb` scheme if support is not present, which has slower | |
161 | read-side. | |
162 | ||
163 | ||
164 | ### Usage of `liburcu-qsbr` | |
165 | ||
166 | 1. `#include <urcu-qsbr.h>` | |
167 | 2. Link with `-lurcu-qsbr` | |
168 | ||
169 | The QSBR flavor of RCU needs to have each reader thread executing | |
170 | `rcu_quiescent_state()` periodically to progress. `rcu_thread_online()` | |
171 | and `rcu_thread_offline()` can be used to mark long periods for which | |
172 | the threads are not active. It provides the fastest read-side at the | |
173 | expense of more intrusiveness in the application code. | |
174 | ||
175 | ||
176 | ### Usage of `liburcu-mb` | |
177 | ||
178 | 1. `#include <urcu.h>` | |
179 | 2. Compile any `_LGPL_SOURCE` code using this library with `-DRCU_MB` | |
180 | 3. Link with `-lurcu-mb` | |
181 | ||
182 | This version of the urcu library uses memory barriers on the writer | |
183 | and reader sides. This results in faster grace-period detection, but | |
184 | results in slower reads. | |
185 | ||
186 | ||
187 | ### Usage of `liburcu-signal` | |
188 | ||
189 | 1. `#include <urcu.h>` | |
190 | 2. Compile any `_LGPL_SOURCE` code using this library with `-DRCU_SIGNAL` | |
191 | 3. Link the application with `-lurcu-signal` | |
192 | ||
193 | Version of the library that requires a signal, typically `SIGUSR1`. Can | |
194 | be overridden with `-DSIGRCU` by modifying `Makefile.build.inc`. | |
195 | ||
196 | ||
197 | ### Usage of `liburcu-bp` | |
198 | ||
199 | 1. `#include <urcu-bp.h>` | |
200 | 2. Link with `-lurcu-bp` | |
201 | ||
202 | The BP library flavor stands for "bulletproof". It is specifically | |
203 | designed to help tracing library to hook on applications without | |
204 | requiring to modify these applications. `rcu_init()`, | |
205 | `rcu_register_thread()` and `rcu_unregister_thread()` all become nops. | |
206 | The state is dealt with by the library internally at the expense of | |
207 | read-side and write-side performance. | |
208 | ||
209 | ||
210 | ### Initialization | |
211 | ||
212 | Each thread that has reader critical sections (that uses | |
213 | `rcu_read_lock()`/`rcu_read_unlock()` must first register to the URCU | |
214 | library. This is done by calling `rcu_register_thread()`. Unregistration | |
215 | must be performed before exiting the thread by using | |
216 | `rcu_unregister_thread()`. | |
217 | ||
218 | ||
219 | ### Reading | |
220 | ||
221 | Reader critical sections must be protected by locating them between | |
222 | calls to `rcu_read_lock()` and `rcu_read_unlock()`. Inside that lock, | |
223 | `rcu_dereference()` may be called to read an RCU protected pointer. | |
224 | ||
225 | ||
226 | ### Writing | |
227 | ||
228 | `rcu_assign_pointer()` and `rcu_xchg_pointer()` may be called anywhere. | |
229 | After, `synchronize_rcu()` must be called. When it returns, the old | |
230 | values are not in usage anymore. | |
231 | ||
232 | ||
233 | ### Usage of `liburcu-defer` | |
234 | ||
235 | - Follow instructions for either `liburcu`, `liburcu-qsbr`, | |
236 | `liburcu-mb`, `liburcu-signal`, or `liburcu-bp` above. | |
237 | The `liburcu-defer` functionality is pulled into each of | |
238 | those library modules. | |
239 | - Provides `defer_rcu()` primitive to enqueue delayed callbacks. Queued | |
240 | callbacks are executed in batch periodically after a grace period. | |
241 | Do _not_ use `defer_rcu()` within a read-side critical section, because | |
242 | it may call `synchronize_rcu()` if the thread queue is full. | |
243 | This can lead to deadlock or worse. | |
244 | - Requires that `rcu_defer_barrier()` must be called in library destructor | |
245 | if a library queues callbacks and is expected to be unloaded with | |
246 | `dlclose()`. | |
247 | ||
248 | Its API is currently experimental. It may change in future library releases. | |
249 | ||
250 | ||
251 | ### Usage of `urcu-call-rcu` | |
252 | ||
253 | - Follow instructions for either `liburcu`, `liburcu-qsbr`, | |
254 | `liburcu-mb`, `liburcu-signal`, or `liburcu-bp` above. | |
255 | The `urcu-call-rcu` functionality is pulled into each of | |
256 | those library modules. | |
257 | - Provides the `call_rcu()` primitive to enqueue delayed callbacks | |
258 | in a manner similar to `defer_rcu()`, but without ever delaying | |
259 | for a grace period. On the other hand, `call_rcu()`'s best-case | |
260 | overhead is not quite as good as that of `defer_rcu()`. | |
261 | - Provides `call_rcu()` to allow asynchronous handling of RCU | |
262 | grace periods. A number of additional functions are provided | |
263 | to manage the helper threads used by `call_rcu()`, but reasonable | |
264 | defaults are used if these additional functions are not invoked. | |
dcb9c05a | 265 | See [`doc/rcu-api.md`](doc/rcu-api.md) in userspace-rcu documentation |
d589a916 PP |
266 | for more details. |
267 | ||
268 | ||
269 | ### Being careful with signals | |
270 | ||
271 | The `liburcu` library uses signals internally. The signal handler is | |
272 | registered with the `SA_RESTART` flag. However, these signals may cause | |
273 | some non-restartable system calls to fail with `errno = EINTR`. Care | |
274 | should be taken to restart system calls manually if they fail with this | |
275 | error. A list of non-restartable system calls may be found in | |
276 | `signal(7)`. The `liburcu-mb` and `liburcu-qsbr` versions of the Userspace RCU | |
277 | library do not require any signal. | |
278 | ||
279 | Read-side critical sections are allowed in a signal handler, | |
280 | except those setup with `sigaltstack(2)`, with `liburcu` and | |
281 | `liburcu-mb`. Be careful, however, to disable these signals | |
282 | between thread creation and calls to `rcu_register_thread()`, because a | |
283 | signal handler nesting on an unregistered thread would not be | |
284 | allowed to call `rcu_read_lock()`. | |
285 | ||
286 | Read-side critical sections are _not_ allowed in a signal handler with | |
287 | `liburcu-qsbr`, unless signals are disabled explicitly around each | |
288 | `rcu_quiescent_state()` calls, when threads are put offline and around | |
289 | calls to `synchronize_rcu()`. Even then, we do not recommend it. | |
290 | ||
291 | ||
292 | ### Interaction with mutexes | |
293 | ||
294 | One must be careful to do not cause deadlocks due to interaction of | |
295 | `synchronize_rcu()` and RCU read-side with mutexes. If `synchronize_rcu()` | |
296 | is called with a mutex held, this mutex (or any mutex which has this | |
297 | mutex in its dependency chain) should not be acquired from within a RCU | |
298 | read-side critical section. | |
299 | ||
300 | This is especially important to understand in the context of the | |
301 | QSBR flavor: a registered reader thread being "online" by | |
302 | default should be considered as within a RCU read-side critical | |
303 | section unless explicitly put "offline". Therefore, if | |
304 | `synchronize_rcu()` is called with a mutex held, this mutex, as | |
305 | well as any mutex which has this mutex in its dependency chain | |
306 | should only be taken when the RCU reader thread is "offline" | |
307 | (this can be performed by calling `rcu_thread_offline()`). | |
308 | ||
309 | ||
310 | ### Interaction with `fork()` | |
311 | ||
312 | Special care must be taken for applications performing `fork()` without | |
313 | any following `exec()`. This is caused by the fact that Linux only clones | |
314 | the thread calling `fork()`, and thus never replicates any of the other | |
315 | parent thread into the child process. Most `liburcu` implementations | |
316 | require that all registrations (as reader, `defer_rcu` and `call_rcu` | |
317 | threads) should be released before a `fork()` is performed, except for the | |
318 | rather common scenario where `fork()` is immediately followed by `exec()` in | |
319 | the child process. The only implementation not subject to that rule is | |
320 | `liburcu-bp`, which is designed to handle `fork()` by calling | |
321 | `rcu_bp_before_fork`, `rcu_bp_after_fork_parent` and | |
322 | `rcu_bp_after_fork_child`. | |
323 | ||
324 | Applications that use `call_rcu()` and that `fork()` without | |
325 | doing an immediate `exec()` must take special action. The parent | |
326 | must invoke `call_rcu_before_fork()` before the `fork()` and | |
327 | `call_rcu_after_fork_parent()` after the `fork()`. The child | |
328 | process must invoke `call_rcu_after_fork_child()`. | |
329 | Even though these three APIs are suitable for passing to | |
330 | `pthread_atfork()`, use of `pthread_atfork()` is **STRONGLY | |
331 | DISCOURAGED** for programs calling the glibc memory allocator | |
332 | (`malloc()`, `calloc()`, `free()`, ...) within `call_rcu` callbacks. | |
333 | This is due to limitations in the way glibc memory allocator | |
334 | handles calls to the memory allocator from concurrent threads | |
335 | while the `pthread_atfork()` handlers are executing. | |
336 | ||
337 | Combining e.g.: | |
338 | ||
339 | - call to `free()` from callbacks executed within `call_rcu` worker | |
340 | threads, | |
341 | - executing `call_rcu` atfork handlers within the glibc pthread | |
342 | atfork mechanism, | |
343 | ||
344 | will sometimes trigger interesting process hangs. This usually | |
345 | hangs on a memory allocator lock within glibc. | |
346 | ||
347 | ||
348 | ### Thread Local Storage (TLS) | |
349 | ||
350 | Userspace RCU can fall back on `pthread_getspecific()` to emulate | |
351 | TLS variables on systems where it is not available. This behavior | |
352 | can be forced by specifying `--disable-compiler-tls` as configure | |
353 | argument. | |
354 | ||
355 | ||
356 | ### Usage of `DEBUG_RCU` | |
357 | ||
358 | `DEBUG_RCU` is used to add internal debugging self-checks to the | |
359 | RCU library. This define adds a performance penalty when enabled. | |
360 | Can be enabled by uncommenting the corresponding line in | |
361 | `Makefile.build.inc`. | |
362 | ||
363 | ||
364 | ### Usage of `DEBUG_YIELD` | |
365 | ||
366 | `DEBUG_YIELD` is used to add random delays in the code for testing | |
367 | purposes. | |
368 | ||
369 | ||
370 | ### SMP support | |
371 | ||
372 | By default the library is configured to use synchronization primitives | |
373 | adequate for SMP systems. On uniprocessor systems, support for SMP | |
374 | systems can be disabled with: | |
375 | ||
376 | ./configure --disable-smp-support | |
377 | ||
378 | theoretically yielding slightly better performance. | |
379 | ||
380 | ||
381 | Make targets | |
382 | ------------ | |
383 | ||
384 | In addition to the usual `make check` target, Userspace RCU features | |
385 | `make regtest` and `make bench` targets: | |
386 | ||
387 | - `make check`: short tests, meant to be run when rebuilding or | |
388 | porting Userspace RCU. | |
389 | - `make regtest`: long (many hours) test, meant to be run when | |
390 | modifying Userspace RCU or porting it to a new architecture or | |
391 | operating system. | |
392 | - `make bench`: long (many hours) benchmarks. | |
393 | ||
394 | ||
395 | Contacts | |
396 | -------- | |
397 | ||
398 | You can contact the maintainers on the following mailing list: | |
dcb9c05a | 399 | `lttng-dev@lists.lttng.org`. |