| 1 | Userspace RCU Implementation |
| 2 | by Mathieu Desnoyers and Paul E. McKenney |
| 3 | |
| 4 | BUILDING |
| 5 | -------- |
| 6 | |
| 7 | ./bootstrap (skip if using tarball) |
| 8 | ./configure |
| 9 | make |
| 10 | make install |
| 11 | ldconfig |
| 12 | |
| 13 | Hints: Forcing 32-bit build: |
| 14 | * CFLAGS="-m32 -g -O2" ./configure |
| 15 | |
| 16 | Forcing 64-bit build: |
| 17 | * CFLAGS="-m64 -g -O2" ./configure |
| 18 | |
| 19 | Forcing a 32-bit build with 386 backward compatibility: |
| 20 | * CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu |
| 21 | |
| 22 | Forcing a 32-bit build for Sparcv9 (typical for Sparc v9) |
| 23 | * CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure |
| 24 | |
| 25 | |
| 26 | ARCHITECTURES SUPPORTED |
| 27 | ----------------------- |
| 28 | |
| 29 | Currently, Linux x86 (i386, i486, i586, i686), x86 64-bit, PowerPC 32/64, |
| 30 | S390, S390x, ARM, Alpha, ia64 and Sparcv9 32/64 are supported. Tested on |
| 31 | Linux, FreeBSD 8.2/9.0, and Cygwin. Should also work on: Android, NetBSD 5, |
| 32 | OpenBSD, Darwin (more testing needed before claiming support for these OS). |
| 33 | |
| 34 | Linux ARM depends on running a Linux kernel 2.6.15 or better, GCC 4.4 or |
| 35 | better. |
| 36 | |
| 37 | The gcc compiler versions 3.3, 3.4, 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5 are |
| 38 | supported, with the following exceptions: |
| 39 | |
| 40 | - gcc 3.3 and 3.4 have a bug that prevents them from generating volatile |
| 41 | accesses to offsets in a TLS structure on 32-bit x86. These versions are |
| 42 | therefore not compatible with liburcu on x86 32-bit (i386, i486, i586, i686). |
| 43 | The problem has been reported to the gcc community: |
| 44 | http://www.mail-archive.com/gcc-bugs@gcc.gnu.org/msg281255.html |
| 45 | - gcc 3.3 cannot match the "xchg" instruction on 32-bit x86 build. |
| 46 | See: http://kerneltrap.org/node/7507 |
| 47 | - Alpha, ia64 and ARM architectures depend on gcc 4.x with atomic builtins |
| 48 | support. For ARM this was introduced with gcc 4.4: |
| 49 | http://gcc.gnu.org/gcc-4.4/changes.html |
| 50 | |
| 51 | Clang version 3.0 (based on LLVM 3.0) is supported. |
| 52 | |
| 53 | For developers using the git tree: |
| 54 | |
| 55 | This source tree is based on the autotools suite from GNU to simplify |
| 56 | portability. Here are some things you should have on your system in order to |
| 57 | compile the git repository tree : |
| 58 | |
| 59 | - GNU autotools (automake >=1.10, autoconf >=2.50, autoheader >=2.50) |
| 60 | (make sure your system wide "automake" points to a recent version!) |
| 61 | - GNU Libtool >=2.2 |
| 62 | (for more information, go to http://www.gnu.org/software/autoconf/) |
| 63 | |
| 64 | If you get the tree from the repository, you will need to use the "bootstrap" |
| 65 | script in the root of the tree. It calls all the GNU tools needed to prepare the |
| 66 | tree configuration. |
| 67 | |
| 68 | Test scripts provided in the tests/ directory of the source tree depend |
| 69 | on "bash" and the "seq" program. |
| 70 | |
| 71 | |
| 72 | API |
| 73 | --- |
| 74 | |
| 75 | See the relevant API documentation files in doc/. The APIs provided by |
| 76 | Userspace RCU are, by prefix: |
| 77 | |
| 78 | - rcu_ : Read-Copy Update (see doc/rcu-api.txt) |
| 79 | - cmm_ : Concurrent Memory Model |
| 80 | - caa_ : Concurrent Architecture Abstraction |
| 81 | - cds_ : Concurrent Data Structures (see doc/cds-api.txt) |
| 82 | - uatomic_: Userspace Atomic (see doc/uatomic-api.txt) |
| 83 | |
| 84 | |
| 85 | QUICK START GUIDE |
| 86 | ----------------- |
| 87 | |
| 88 | Usage of all urcu libraries |
| 89 | |
| 90 | * Define _LGPL_SOURCE (only) if your code is LGPL or GPL compatible |
| 91 | before including the urcu.h or urcu-qsbr.h header. If your application |
| 92 | is distributed under another license, function calls will be generated |
| 93 | instead of inlines, so your application can link with the library. |
| 94 | * Linking with one of the libraries below is always necessary even for |
| 95 | LGPL and GPL applications. |
| 96 | |
| 97 | Usage of liburcu |
| 98 | |
| 99 | * #include <urcu.h> |
| 100 | * Link the application with "-lurcu". |
| 101 | * This is the preferred version of the library, in terms of |
| 102 | grace-period detection speed, read-side speed and flexibility. |
| 103 | Dynamically detects kernel support for sys_membarrier(). Falls back |
| 104 | on urcu-mb scheme if support is not present, which has slower |
| 105 | read-side. |
| 106 | |
| 107 | Usage of liburcu-qsbr |
| 108 | |
| 109 | * #include <urcu-qsbr.h> |
| 110 | * Link with "-lurcu-qsbr". |
| 111 | * The QSBR flavor of RCU needs to have each reader thread executing |
| 112 | rcu_quiescent_state() periodically to progress. rcu_thread_online() |
| 113 | and rcu_thread_offline() can be used to mark long periods for which |
| 114 | the threads are not active. It provides the fastest read-side at the |
| 115 | expense of more intrusiveness in the application code. |
| 116 | |
| 117 | Usage of liburcu-mb |
| 118 | |
| 119 | * #include <urcu.h> |
| 120 | * Compile any _LGPL_SOURCE code using this library with "-DRCU_MB". |
| 121 | * Link with "-lurcu-mb". |
| 122 | * This version of the urcu library uses memory barriers on the writer |
| 123 | and reader sides. This results in faster grace-period detection, but |
| 124 | results in slower reads. |
| 125 | |
| 126 | Usage of liburcu-signal |
| 127 | |
| 128 | * #include <urcu.h> |
| 129 | * Compile any _LGPL_SOURCE code using this library with "-DRCU_SIGNAL". |
| 130 | * Link the application with "-lurcu-signal". |
| 131 | * Version of the library that requires a signal, typically SIGUSR1. Can |
| 132 | be overridden with -DSIGRCU by modifying Makefile.build.inc. |
| 133 | |
| 134 | Usage of liburcu-bp |
| 135 | |
| 136 | * #include <urcu-bp.h> |
| 137 | * Link with "-lurcu-bp". |
| 138 | * The BP library flavor stands for "bulletproof". It is specifically |
| 139 | designed to help tracing library to hook on applications without |
| 140 | requiring to modify these applications. rcu_init(), |
| 141 | rcu_register_thread() and rcu_unregister_thread() all become nops. |
| 142 | The state is dealt with by the library internally at the expense of |
| 143 | read-side and write-side performance. |
| 144 | |
| 145 | Initialization |
| 146 | |
| 147 | Each thread that has reader critical sections (that uses |
| 148 | rcu_read_lock()/rcu_read_unlock() must first register to the URCU |
| 149 | library. This is done by calling rcu_register_thread(). Unregistration |
| 150 | must be performed before exiting the thread by using |
| 151 | rcu_unregister_thread(). |
| 152 | |
| 153 | Reading |
| 154 | |
| 155 | Reader critical sections must be protected by locating them between |
| 156 | calls to rcu_read_lock() and rcu_read_unlock(). Inside that lock, |
| 157 | rcu_dereference() may be called to read an RCU protected pointer. |
| 158 | |
| 159 | Writing |
| 160 | |
| 161 | rcu_assign_pointer() and rcu_xchg_pointer() may be called anywhere. |
| 162 | After, synchronize_rcu() must be called. When it returns, the old |
| 163 | values are not in usage anymore. |
| 164 | |
| 165 | Usage of liburcu-defer |
| 166 | |
| 167 | * Follow instructions for either liburcu, liburcu-qsbr, |
| 168 | liburcu-mb, liburcu-signal, or liburcu-bp above. |
| 169 | The liburcu-defer functionality is pulled into each of |
| 170 | those library modules. |
| 171 | * Provides defer_rcu() primitive to enqueue delayed callbacks. Queued |
| 172 | callbacks are executed in batch periodically after a grace period. |
| 173 | Do _not_ use defer_rcu() within a read-side critical section, because |
| 174 | it may call synchronize_rcu() if the thread queue is full. |
| 175 | This can lead to deadlock or worse. |
| 176 | * Requires that rcu_defer_barrier() must be called in library destructor |
| 177 | if a library queues callbacks and is expected to be unloaded with |
| 178 | dlclose(). |
| 179 | * Its API is currently experimental. It may change in future library |
| 180 | releases. |
| 181 | |
| 182 | Usage of urcu-call-rcu |
| 183 | |
| 184 | * Follow instructions for either liburcu, liburcu-qsbr, |
| 185 | liburcu-mb, liburcu-signal, or liburcu-bp above. |
| 186 | The urcu-call-rcu functionality is provided for each of |
| 187 | these library modules. |
| 188 | * Provides the call_rcu() primitive to enqueue delayed callbacks |
| 189 | in a manner similar to defer_rcu(), but without ever delaying |
| 190 | for a grace period. On the other hand, call_rcu()'s best-case |
| 191 | overhead is not quite as good as that of defer_rcu(). |
| 192 | * Provides call_rcu() to allow asynchronous handling of RCU |
| 193 | grace periods. A number of additional functions are provided |
| 194 | to manage the helper threads used by call_rcu(), but reasonable |
| 195 | defaults are used if these additional functions are not invoked. |
| 196 | See rcu-api.txt in userspace-rcu documentation for more details. |
| 197 | |
| 198 | Being careful with signals |
| 199 | |
| 200 | The liburcu library uses signals internally. The signal handler is |
| 201 | registered with the SA_RESTART flag. However, these signals may cause |
| 202 | some non-restartable system calls to fail with errno = EINTR. Care |
| 203 | should be taken to restart system calls manually if they fail with this |
| 204 | error. A list of non-restartable system calls may be found in |
| 205 | signal(7). The liburcu-mb and liburcu-qsbr versions of the Userspace RCU |
| 206 | library do not require any signal. |
| 207 | |
| 208 | Read-side critical sections are allowed in a signal handler, |
| 209 | except those setup with sigaltstack(2), with liburcu and |
| 210 | liburcu-mb. Be careful, however, to disable these signals |
| 211 | between thread creation and calls to rcu_register_thread(), because a |
| 212 | signal handler nesting on an unregistered thread would not be |
| 213 | allowed to call rcu_read_lock(). |
| 214 | |
| 215 | Read-side critical sections are _not_ allowed in a signal handler with |
| 216 | liburcu-qsbr, unless signals are disabled explicitly around each |
| 217 | rcu_quiescent_state() calls, when threads are put offline and around |
| 218 | calls to synchronize_rcu(). Even then, we do not recommend it. |
| 219 | |
| 220 | Interaction with mutexes |
| 221 | |
| 222 | One must be careful to do not cause deadlocks due to interaction of |
| 223 | synchronize_rcu() and RCU read-side with mutexes. If synchronize_rcu() |
| 224 | is called with a mutex held, this mutex (or any mutex which has this |
| 225 | mutex in its dependency chain) should not be acquired from within a RCU |
| 226 | read-side critical section. |
| 227 | |
| 228 | This is especially important to understand in the context of the |
| 229 | QSBR flavor: a registered reader thread being "online" by |
| 230 | default should be considered as within a RCU read-side critical |
| 231 | section unless explicitly put "offline". Therefore, if |
| 232 | synchronize_rcu() is called with a mutex held, this mutex, as |
| 233 | well as any mutex which has this mutex in its dependency chain |
| 234 | should only be taken when the RCU reader thread is "offline" |
| 235 | (this can be performed by calling rcu_thread_offline()). |
| 236 | |
| 237 | Interaction with fork() |
| 238 | |
| 239 | Special care must be taken for applications performing fork() without |
| 240 | any following exec(). This is caused by the fact that Linux only clones |
| 241 | the thread calling fork(), and thus never replicates any of the other |
| 242 | parent thread into the child process. Most liburcu implementations |
| 243 | require that all registrations (as reader, defer_rcu and call_rcu |
| 244 | threads) should be released before a fork() is performed, except for the |
| 245 | rather common scenario where fork() is immediately followed by exec() in |
| 246 | the child process. The only implementation not subject to that rule is |
| 247 | liburcu-bp, which is designed to handle fork() by calling |
| 248 | rcu_bp_before_fork, rcu_bp_after_fork_parent and |
| 249 | rcu_bp_after_fork_child. |
| 250 | |
| 251 | Applications that use call_rcu() and that fork() without |
| 252 | doing an immediate exec() must take special action. The parent |
| 253 | must invoke call_rcu_before_fork() before the fork() and |
| 254 | call_rcu_after_fork_parent() after the fork(). The child |
| 255 | process must invoke call_rcu_after_fork_child(). |
| 256 | Even though these three APIs are suitable for passing to |
| 257 | pthread_atfork(), use of pthread_atfork() is *STRONGLY |
| 258 | DISCOURAGED* for programs calling the glibc memory allocator |
| 259 | (malloc(), calloc(), free(), ...) within call_rcu callbacks. |
| 260 | This is due to limitations in the way glibc memory allocator |
| 261 | handles calls to the memory allocator from concurrent threads |
| 262 | while the pthread_atfork() handlers are executing. |
| 263 | Combining e.g.: |
| 264 | * call to free() from callbacks executed within call_rcu worker |
| 265 | threads, |
| 266 | * executing call_rcu atfork handlers within the glibc pthread |
| 267 | atfork mechanism, |
| 268 | will sometimes trigger interesting process hangs. This usually |
| 269 | hangs on a memory allocator lock within glibc. |
| 270 | |
| 271 | Thread Local Storage (TLS) |
| 272 | |
| 273 | Userspace RCU can fall back on pthread_getspecific() to emulate |
| 274 | TLS variables on systems where it is not available. This behavior |
| 275 | can be forced by specifying --disable-compiler-tls as configure |
| 276 | argument. |
| 277 | |
| 278 | Usage of DEBUG_RCU |
| 279 | |
| 280 | DEBUG_RCU is used to add internal debugging self-checks to the |
| 281 | RCU library. This define adds a performance penalty when enabled. |
| 282 | Can be enabled by uncommenting the corresponding line in |
| 283 | Makefile.build.inc. |
| 284 | |
| 285 | Usage of DEBUG_YIELD |
| 286 | |
| 287 | DEBUG_YIELD is used to add random delays in the code for testing |
| 288 | purposes. |
| 289 | |
| 290 | SMP support |
| 291 | |
| 292 | By default the library is configured to use synchronization primitives |
| 293 | adequate for SMP systems. On uniprocessor systems, support for SMP |
| 294 | systems can be disabled with: |
| 295 | |
| 296 | ./configure --disable-smp-support |
| 297 | |
| 298 | theoretically yielding slightly better performance. |