Commit | Line | Data |
---|---|---|
7591bab1 MD |
1 | LTTng Relay Daemon Architecture |
2 | Mathieu Desnoyers, August 2015 | |
3 | ||
4 | This document describes the object model and architecture of the relay | |
5 | daemon, after the refactoring done within the commit "Fix: Relay daemon | |
6 | ownership and reference counting". | |
7 | ||
8 | We have the following object composition hierarchy: | |
9 | ||
10 | relay connection (main.c, for sessiond/consumer) | |
11 | | | |
12 | \-> 0 or 1 session | |
13 | | | |
14 | \-> 0 or many ctf-trace | |
15 | | | |
16 | \-> 0 or many stream | |
17 | | | | |
18 | | \-> 0 or many index | |
19 | | | |
20 | \-------> 0 or 1 viewer stream | |
21 | ||
22 | live connection (live.c, for client) | |
23 | | | |
24 | \-> 1 viewer session | |
25 | | | |
26 | \-> 0 or many session (actually a reference to session as created | |
27 | | by the relay connection) | |
28 | | | |
29 | \-> ..... (ctf-trace, stream, index, viewer stream) | |
30 | ||
31 | There are global tables declared in lttng-relayd.h for sessions | |
32 | (sessions_ht, indexed by session id), streams (relay_streams_ht, indexed | |
33 | by stream handle), and viewer streams (viewer_streams_ht, indexed by | |
34 | stream handle). The purpose of those tables is to allow fast lookup of | |
35 | those objects using the IDs received in the communication protocols. | |
36 | ||
37 | There is also one connection hash table per worker thread. There is one | |
38 | worker thread to receive data (main.c), and one worker thread to | |
39 | interact with viewer clients (live.c). Those tables are indexed by | |
40 | socket file descriptor. | |
41 | ||
42 | A RCU lookup+refcounting scheme has been introduced for all objects | |
43 | (except viewer session which is still an exception at the moment). This | |
44 | scheme allows looking up the objects or doing a traversal on the RCU | |
45 | linked list or hash table in combination with a getter on the object. | |
46 | This getter validates that there is still at least one reference to the | |
47 | object, else the lookup acts just as if the object does not exist. This | |
48 | scheme is protected by a "reflock" mutex in each object. "reflock" | |
49 | mutexes can be nested from the innermost object to the outermost object. | |
50 | IOW, the session reflock can nest within the ctf-trace reflock. | |
51 | ||
52 | The relay_connection (connection between the sessiond/consumer and the | |
53 | relayd) is the outermost object of its hierarchy. | |
54 | ||
55 | The live connection (connection between a live client and the relayd) | |
56 | is the outermost object of its hierarchy. | |
57 | ||
58 | There is also a "lock" mutex in each object. Those are used to | |
59 | synchronize between threads (currently the main.c relay thread and | |
60 | live.c client thread) when objects are shared. Locks can be nested from | |
61 | the outermost object to the innermost object. IOW, the ctf-trace lock can | |
62 | nest within the session lock. | |
63 | ||
64 | A "lock" should never nest within a "reflock". | |
65 | ||
66 | RCU linked lists are used to iterate using RCU, and are protected by | |
67 | their own mutex for modifications. Iterations should be confirmed using | |
68 | the object "getter" to ensure its refcount is not 0 (except in cases | |
69 | where the caller actually owns the objects and therefore can assume its | |
70 | refcount is not 0). | |
71 | ||
72 | RCU hash tables are used to iterate using RCU. Iteration should be | |
73 | confirmed using the object "getter" to ensure its refcount is not 0 | |
74 | (except again if we have ownership and can assume the object refcount is | |
75 | not 0). | |
76 | ||
77 | Object creation has a refcount of 1. Each getter increments the | |
78 | refcount, and needs to be paired with a "put" to decrement it. A final | |
79 | put on "self" (ownership) will allow refcount to reach 0, therefore | |
80 | triggering release, and thus free through call_rcu. | |
81 | ||
82 | In the composition scheme, we find back references from each composite | |
83 | to its container. Therefore, each composite holds a reference (refcount) | |
84 | on its container. This allows following pointers from e.g. viewer stream | |
85 | to stream to ctf-trace to session without performing any validation, | |
86 | due to transitive refcounting of those back-references. | |
87 | ||
88 | In addition to those back references, there are a few key ownership | |
89 | references held. The connection in the relay worker thread (main.c) | |
90 | holds ownership on the session, and on each stream it contains. The | |
91 | connection in the live worker thread (live.c) holds ownership on each | |
92 | viewer stream it creates. The rest is ensured by back references from | |
93 | composite to container objects. When a connection is closed, it puts all | |
94 | the ownership references it is holding. This will then eventually | |
95 | trigger destruction of the session, streams, and viewer streams | |
96 | associated with the connection when all the back references reach 0. | |
97 | ||
98 | RCU read-side locks are now only held during iteration on RCU lists and | |
99 | hash tables, and within the internals of the get (lookup) and put | |
100 | functions. Those functions then use refcounting to ensure existence of | |
101 | the object when returned to their caller. |