Commit | Line | Data |
---|---|---|
7ac06cef | 1 | Userspace RCU Implementation |
c97ae6eb | 2 | by Mathieu Desnoyers and Paul E. McKenney |
6991f61a | 3 | |
c97ae6eb PMF |
4 | BUILDING |
5 | -------- | |
6991f61a | 6 | |
48d848c7 PMF |
7 | ./bootstrap (skip if using tarball) |
8 | ./configure | |
c97ae6eb PMF |
9 | make |
10 | make install | |
e197ac6f | 11 | ldconfig |
9ca52251 | 12 | |
7d413817 | 13 | Hints: Forcing 32-bit build: |
c4c18179 | 14 | * CFLAGS="-m32 -g -O2" ./configure |
9ca52251 MD |
15 | |
16 | Forcing 64-bit build: | |
c4c18179 | 17 | * CFLAGS="-m64 -g -O2" ./configure |
aa8c36e0 | 18 | |
f39cd442 | 19 | Forcing a 32-bit build with 386 backward compatibility: |
58e9f838 | 20 | * CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu |
7d413817 | 21 | |
795d506a MD |
22 | Forcing a 32-bit build for Sparcv9 (typical for Sparc v9) |
23 | * CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure | |
24 | ||
207e8061 | 25 | |
c51e75e6 MD |
26 | ARCHITECTURES SUPPORTED |
27 | ----------------------- | |
28 | ||
aac98a55 MD |
29 | Currently, Linux x86 (i386, i486, i586, i686), x86 64-bit, PowerPC 32/64, |
30 | S390, S390x, ARM, Alpha, ia64 and Sparcv9 32/64 are supported. Tested on | |
31 | Linux, FreeBSD 8.2/9.0, and Cygwin. Should also work on: Android, NetBSD 5, | |
32 | OpenBSD, Darwin (more testing needed before claiming support for these OS). | |
c51e75e6 | 33 | |
aac98a55 MD |
34 | Linux ARM depends on running a Linux kernel 2.6.15 or better, GCC 4.4 or |
35 | better. | |
3b36a2e9 | 36 | |
3b38cfe1 MD |
37 | The gcc compiler versions 3.3, 3.4, 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5 are |
38 | supported, with the following exceptions: | |
39 | ||
40 | - gcc 3.3 and 3.4 have a bug that prevents them from generating volatile | |
41 | accesses to offsets in a TLS structure on 32-bit x86. These versions are | |
42 | therefore not compatible with liburcu on x86 32-bit (i386, i486, i586, i686). | |
43 | The problem has been reported to the gcc community: | |
44 | http://www.mail-archive.com/gcc-bugs@gcc.gnu.org/msg281255.html | |
83e8be52 MD |
45 | - gcc 3.3 cannot match the "xchg" instruction on 32-bit x86 build. |
46 | See: http://kerneltrap.org/node/7507 | |
d68d00f8 GF |
47 | - Alpha, ia64 and ARM architectures depend on gcc 4.x with atomic builtins |
48 | support. For ARM this was introduced with gcc 4.4: | |
49 | http://gcc.gnu.org/gcc-4.4/changes.html | |
7bcbcfb2 | 50 | |
957be43f MD |
51 | Clang version 3.0 (based on LLVM 3.0) is supported. |
52 | ||
0b0d27d0 MD |
53 | For developers using the git tree: |
54 | ||
55 | This source tree is based on the autotools suite from GNU to simplify | |
56 | portability. Here are some things you should have on your system in order to | |
57 | compile the git repository tree : | |
58 | ||
59 | - GNU autotools (automake >=1.10, autoconf >=2.50, autoheader >=2.50) | |
60 | (make sure your system wide "automake" points to a recent version!) | |
61 | - GNU Libtool >=2.2 | |
62 | (for more information, go to http://www.gnu.org/software/autoconf/) | |
63 | ||
64 | If you get the tree from the repository, you will need to use the "bootstrap" | |
65 | script in the root of the tree. It calls all the GNU tools needed to prepare the | |
66 | tree configuration. | |
67 | ||
aac98a55 MD |
68 | Test scripts provided in the tests/ directory of the source tree depend |
69 | on "bash" and the "seq" program. | |
70 | ||
3b38cfe1 | 71 | |
207e8061 MD |
72 | API |
73 | --- | |
74 | ||
75 | See the relevant API documentation files in doc/. The APIs provided by | |
76 | Userspace RCU are, by prefix: | |
77 | ||
9b4ec2a5 | 78 | - rcu_ : Read-Copy Update (see doc/rcu-api.txt) |
207e8061 MD |
79 | - cmm_ : Concurrent Memory Model |
80 | - caa_ : Concurrent Architecture Abstraction | |
9b4ec2a5 MD |
81 | - cds_ : Concurrent Data Structures (see doc/cds-api.txt) |
82 | - uatomic_: Userspace Atomic (see doc/uatomic-api.txt) | |
207e8061 MD |
83 | |
84 | ||
c97ae6eb PMF |
85 | QUICK START GUIDE |
86 | ----------------- | |
aa8c36e0 | 87 | |
0a1d290b MD |
88 | Usage of all urcu libraries |
89 | ||
90 | * Define _LGPL_SOURCE (only) if your code is LGPL or GPL compatible | |
91 | before including the urcu.h or urcu-qsbr.h header. If your application | |
92 | is distributed under another license, function calls will be generated | |
93 | instead of inlines, so your application can link with the library. | |
94 | * Linking with one of the libraries below is always necessary even for | |
95 | LGPL and GPL applications. | |
96 | ||
97 | Usage of liburcu | |
98 | ||
99 | * #include <urcu.h> | |
100 | * Link the application with "-lurcu". | |
fdf01eed MD |
101 | * This is the preferred version of the library, in terms of |
102 | grace-period detection speed, read-side speed and flexibility. | |
103 | Dynamically detects kernel support for sys_membarrier(). Falls back | |
104 | on urcu-mb scheme if support is not present, which has slower | |
105 | read-side. | |
0a1d290b MD |
106 | |
107 | Usage of liburcu-qsbr | |
108 | ||
109 | * #include <urcu-qsbr.h> | |
110 | * Link with "-lurcu-qsbr". | |
111 | * The QSBR flavor of RCU needs to have each reader thread executing | |
112 | rcu_quiescent_state() periodically to progress. rcu_thread_online() | |
113 | and rcu_thread_offline() can be used to mark long periods for which | |
114 | the threads are not active. It provides the fastest read-side at the | |
115 | expense of more intrusiveness in the application code. | |
116 | ||
fdf01eed MD |
117 | Usage of liburcu-mb |
118 | ||
119 | * #include <urcu.h> | |
120 | * Compile any _LGPL_SOURCE code using this library with "-DRCU_MB". | |
121 | * Link with "-lurcu-mb". | |
122 | * This version of the urcu library uses memory barriers on the writer | |
123 | and reader sides. This results in faster grace-period detection, but | |
124 | results in slower reads. | |
125 | ||
126 | Usage of liburcu-signal | |
127 | ||
ee39cfb6 MD |
128 | * #include <urcu.h> |
129 | * Compile any _LGPL_SOURCE code using this library with "-DRCU_SIGNAL". | |
fdf01eed MD |
130 | * Link the application with "-lurcu-signal". |
131 | * Version of the library that requires a signal, typically SIGUSR1. Can | |
132 | be overridden with -DSIGRCU by modifying Makefile.build.inc. | |
133 | ||
fdee2e6d MD |
134 | Usage of liburcu-bp |
135 | ||
136 | * #include <urcu-bp.h> | |
137 | * Link with "-lurcu-bp". | |
138 | * The BP library flavor stands for "bulletproof". It is specifically | |
139 | designed to help tracing library to hook on applications without | |
02be5561 | 140 | requiring to modify these applications. rcu_init(), |
fdee2e6d MD |
141 | rcu_register_thread() and rcu_unregister_thread() all become nops. |
142 | The state is dealt with by the library internally at the expense of | |
143 | read-side and write-side performance. | |
144 | ||
c97ae6eb PMF |
145 | Initialization |
146 | ||
147 | Each thread that has reader critical sections (that uses | |
148 | rcu_read_lock()/rcu_read_unlock() must first register to the URCU | |
4c1471de MD |
149 | library. This is done by calling rcu_register_thread(). Unregistration |
150 | must be performed before exiting the thread by using | |
151 | rcu_unregister_thread(). | |
c97ae6eb PMF |
152 | |
153 | Reading | |
154 | ||
155 | Reader critical sections must be protected by locating them between | |
156 | calls to rcu_read_lock() and rcu_read_unlock(). Inside that lock, | |
157 | rcu_dereference() may be called to read an RCU protected pointer. | |
158 | ||
159 | Writing | |
160 | ||
161 | rcu_assign_pointer() and rcu_xchg_pointer() may be called anywhere. | |
9fb223da MD |
162 | After, synchronize_rcu() must be called. When it returns, the old |
163 | values are not in usage anymore. | |
c97ae6eb | 164 | |
ec4e58a3 MD |
165 | Usage of liburcu-defer |
166 | ||
0376e7b2 PM |
167 | * Follow instructions for either liburcu, liburcu-qsbr, |
168 | liburcu-mb, liburcu-signal, or liburcu-bp above. | |
169 | The liburcu-defer functionality is pulled into each of | |
170 | those library modules. | |
632dd6ba | 171 | * Provides defer_rcu() primitive to enqueue delayed callbacks. Queued |
ec4e58a3 | 172 | callbacks are executed in batch periodically after a grace period. |
632dd6ba | 173 | Do _not_ use defer_rcu() within a read-side critical section, because |
ec4e58a3 | 174 | it may call synchronize_rcu() if the thread queue is full. |
0376e7b2 | 175 | This can lead to deadlock or worse. |
83dd659a MD |
176 | * Requires that rcu_defer_barrier() must be called in library destructor |
177 | if a library queues callbacks and is expected to be unloaded with | |
178 | dlclose(). | |
9c55af9f MD |
179 | * Its API is currently experimental. It may change in future library |
180 | releases. | |
ec4e58a3 | 181 | |
26ba798a PM |
182 | Usage of urcu-call-rcu |
183 | ||
184 | * Follow instructions for either liburcu, liburcu-qsbr, | |
185 | liburcu-mb, liburcu-signal, or liburcu-bp above. | |
186 | The urcu-call-rcu functionality is provided for each of | |
187 | these library modules. | |
188 | * Provides the call_rcu() primitive to enqueue delayed callbacks | |
189 | in a manner similar to defer_rcu(), but without ever delaying | |
190 | for a grace period. On the other hand, call_rcu()'s best-case | |
191 | overhead is not quite as good as that of defer_rcu(). | |
192 | * Provides call_rcu() to allow asynchronous handling of RCU | |
193 | grace periods. A number of additional functions are provided | |
194 | to manage the helper threads used by call_rcu(), but reasonable | |
195 | defaults are used if these additional functions are not invoked. | |
37a9ce52 | 196 | See rcu-api.txt in userspace-rcu documentation for more details. |
26ba798a | 197 | |
dd052bd3 PMF |
198 | Being careful with signals |
199 | ||
0a1d290b | 200 | The liburcu library uses signals internally. The signal handler is |
dd052bd3 PMF |
201 | registered with the SA_RESTART flag. However, these signals may cause |
202 | some non-restartable system calls to fail with errno = EINTR. Care | |
203 | should be taken to restart system calls manually if they fail with this | |
204 | error. A list of non-restartable system calls may be found in | |
0a1d290b MD |
205 | signal(7). The liburcu-mb and liburcu-qsbr versions of the Userspace RCU |
206 | library do not require any signal. | |
c97ae6eb | 207 | |
88ecbe6d MD |
208 | Read-side critical sections are allowed in a signal handler, |
209 | except those setup with sigaltstack(2), with liburcu and | |
210 | liburcu-mb. Be careful, however, to disable these signals | |
7ac06cef | 211 | between thread creation and calls to rcu_register_thread(), because a |
88ecbe6d MD |
212 | signal handler nesting on an unregistered thread would not be |
213 | allowed to call rcu_read_lock(). | |
cee02f0a | 214 | |
0a1d290b MD |
215 | Read-side critical sections are _not_ allowed in a signal handler with |
216 | liburcu-qsbr, unless signals are disabled explicitly around each | |
217 | rcu_quiescent_state() calls, when threads are put offline and around | |
218 | calls to synchronize_rcu(). Even then, we do not recommend it. | |
c97ae6eb | 219 | |
955f5e52 MD |
220 | Interaction with mutexes |
221 | ||
222 | One must be careful to do not cause deadlocks due to interaction of | |
223 | synchronize_rcu() and RCU read-side with mutexes. If synchronize_rcu() | |
224 | is called with a mutex held, this mutex (or any mutex which has this | |
225 | mutex in its dependency chain) should not be acquired from within a RCU | |
226 | read-side critical section. | |
227 | ||
cc558521 MD |
228 | This is especially important to understand in the context of the |
229 | QSBR flavor: a registered reader thread being "online" by | |
230 | default should be considered as within a RCU read-side critical | |
231 | section unless explicitly put "offline". Therefore, if | |
232 | synchronize_rcu() is called with a mutex held, this mutex, as | |
233 | well as any mutex which has this mutex in its dependency chain | |
234 | should only be taken when the RCU reader thread is "offline" | |
235 | (this can be performed by calling rcu_thread_offline()). | |
236 | ||
47c5a84f MD |
237 | Interaction with fork() |
238 | ||
239 | Special care must be taken for applications performing fork() without | |
240 | any following exec(). This is caused by the fact that Linux only clones | |
241 | the thread calling fork(), and thus never replicates any of the other | |
242 | parent thread into the child process. Most liburcu implementations | |
243 | require that all registrations (as reader, defer_rcu and call_rcu | |
244 | threads) should be released before a fork() is performed, except for the | |
245 | rather common scenario where fork() is immediately followed by exec() in | |
246 | the child process. The only implementation not subject to that rule is | |
4cf1675f MD |
247 | liburcu-bp, which is designed to handle fork() by calling |
248 | rcu_bp_before_fork, rcu_bp_after_fork_parent and | |
249 | rcu_bp_after_fork_child. | |
81ad2e19 | 250 | |
ef84facf PM |
251 | Applications that use call_rcu() and that fork() without |
252 | doing an immediate exec() must take special action. The parent | |
253 | must invoke call_rcu_before_fork() before the fork() and | |
254 | call_rcu_after_fork_parent() after the fork(). The child | |
255 | process must invoke call_rcu_after_fork_child(). | |
07bd1a22 MD |
256 | Even though these three APIs are suitable for passing to |
257 | pthread_atfork(), use of pthread_atfork() is *STRONGLY | |
258 | DISCOURAGED* for programs calling the glibc memory allocator | |
259 | (malloc(), calloc(), free(), ...) within call_rcu callbacks. | |
260 | This is due to limitations in the way glibc memory allocator | |
261 | handles calls to the memory allocator from concurrent threads | |
262 | while the pthread_atfork() handlers are executing. | |
263 | Combining e.g.: | |
264 | * call to free() from callbacks executed within call_rcu worker | |
265 | threads, | |
266 | * executing call_rcu atfork handlers within the glibc pthread | |
267 | atfork mechanism, | |
268 | will sometimes trigger interesting process hangs. This usually | |
269 | hangs on a memory allocator lock within glibc. | |
75478b32 MD |
270 | |
271 | Thread Local Storage (TLS) | |
272 | ||
273 | Userspace RCU can fall back on pthread_getspecific() to emulate | |
274 | TLS variables on systems where it is not available. This behavior | |
275 | can be forced by specifying --disable-compiler-tls as configure | |
276 | argument. | |
74093738 MD |
277 | |
278 | Usage of DEBUG_RCU | |
279 | ||
280 | DEBUG_RCU is used to add internal debugging self-checks to the | |
281 | RCU library. This define adds a performance penalty when enabled. | |
282 | Can be enabled by uncommenting the corresponding line in | |
283 | Makefile.build.inc. | |
284 | ||
285 | Usage of DEBUG_YIELD | |
286 | ||
287 | DEBUG_YIELD is used to add random delays in the code for testing | |
288 | purposes. | |
289 | ||
290 | SMP support | |
291 | ||
292 | By default the library is configured to use synchronization primitives | |
293 | adequate for SMP systems. On uniprocessor systems, support for SMP | |
294 | systems can be disabled with: | |
295 | ||
296 | ./configure --disable-smp-support | |
297 | ||
298 | theoretically yielding slightly better performance. |