From 10c54debe0ea6bdf025c6bb587dae8635140ea3e Mon Sep 17 00:00:00 2001 From: Francis Deslauriers Date: Mon, 27 Sep 2021 09:42:54 -0400 Subject: [PATCH] Fix: notification-thread: handling event from a removed tracer event src MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit Issue ===== The issue is caused by a race condition where the `lttng_poll_wait()` returns a _REMOVE_TRACER_EVENT_SOURCE event followed by an actual notification event on the removed event source fd. This causes the notification thread to remove the fd from the potential notification sources list and later fail to find that same fd in the next iteration. This race condition can lead to the notification thread to hang indefinitely or to failed assertions within the `fini_thread_state()` function. Fix === When removing an tracer event source, force the notification thread `lttng_poll_wait()` loop to restart to ignore events from the removed fd. Use the `restart_poll` for that purpose (see note below). Reproducer ========== It's easy to reproduce this issue by adding a `usleep(5000)` just before the `lttng_poll_wait()` call in the notification thread. Note ==== It's the second time that I fix this issue. It was first fixed by this commit by adding the `restart_poll` flag: commit 8b5240601e4ddf6127e4291b7194dd5179cb35b5 Author: Francis Deslauriers Date: Thu Dec 10 15:41:29 2020 -0500 notification-thread: drain all tracer notification on removal and later, that other commit refactored that code but accidently removed the use of the `restart_poll`: commit 34bf4f69e49d8a69331a6aa6826ef1f155e20ede Author: Francis Deslauriers Date: Wed May 26 16:05:16 2021 -0400 notification-thread: remove fd from pollset on LPOLLHUP and friends Signed-off-by: Francis Deslauriers Signed-off-by: Jérémie Galarneau Change-Id: I6da0ed4374b612934adc72fb88d5c142505c5d53 --- src/bin/lttng-sessiond/notification-thread-events.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/bin/lttng-sessiond/notification-thread-events.c b/src/bin/lttng-sessiond/notification-thread-events.c index 747740233..1388b3ce3 100644 --- a/src/bin/lttng-sessiond/notification-thread-events.c +++ b/src/bin/lttng-sessiond/notification-thread-events.c @@ -2153,6 +2153,12 @@ int remove_tracer_event_source_from_pollset( source_element->is_fd_in_poll_set = false; + /* + * Force the notification thread to restart the poll() loop to ensure + * that any events from the removed fd are removed. + */ + state->restart_poll = true; + ret = drain_event_notifier_notification_pipe(state, source_element->fd, source_element->domain); if (ret) { -- 2.34.1