Commit | Line | Data |
---|---|---|
5683264c PM |
1 | This file documents how to use memory mapped I/O with netlink. |
2 | ||
3 | Author: Patrick McHardy <kaber@trash.net> | |
4 | ||
5 | Overview | |
6 | -------- | |
7 | ||
8 | Memory mapped netlink I/O can be used to increase throughput and decrease | |
9 | overhead of unicast receive and transmit operations. Some netlink subsystems | |
10 | require high throughput, these are mainly the netfilter subsystems | |
11 | nfnetlink_queue and nfnetlink_log, but it can also help speed up large | |
12 | dump operations of f.i. the routing database. | |
13 | ||
14 | Memory mapped netlink I/O used two circular ring buffers for RX and TX which | |
15 | are mapped into the processes address space. | |
16 | ||
17 | The RX ring is used by the kernel to directly construct netlink messages into | |
18 | user-space memory without copying them as done with regular socket I/O, | |
19 | additionally as long as the ring contains messages no recvmsg() or poll() | |
20 | syscalls have to be issued by user-space to get more message. | |
21 | ||
22 | The TX ring is used to process messages directly from user-space memory, the | |
23 | kernel processes all messages contained in the ring using a single sendmsg() | |
24 | call. | |
25 | ||
26 | Usage overview | |
27 | -------------- | |
28 | ||
29 | In order to use memory mapped netlink I/O, user-space needs three main changes: | |
30 | ||
31 | - ring setup | |
32 | - conversion of the RX path to get messages from the ring instead of recvmsg() | |
33 | - conversion of the TX path to construct messages into the ring | |
34 | ||
35 | Ring setup is done using setsockopt() to provide the ring parameters to the | |
36 | kernel, then a call to mmap() to map the ring into the processes address space: | |
37 | ||
38 | - setsockopt(fd, SOL_NETLINK, NETLINK_RX_RING, ¶ms, sizeof(params)); | |
39 | - setsockopt(fd, SOL_NETLINK, NETLINK_TX_RING, ¶ms, sizeof(params)); | |
40 | - ring = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0) | |
41 | ||
42 | Usage of either ring is optional, but even if only the RX ring is used the | |
43 | mapping still needs to be writable in order to update the frame status after | |
44 | processing. | |
45 | ||
46 | Conversion of the reception path involves calling poll() on the file | |
47 | descriptor, once the socket is readable the frames from the ring are | |
c17cb8b5 | 48 | processed in order until no more messages are available, as indicated by |
5683264c PM |
49 | a status word in the frame header. |
50 | ||
51 | On kernel side, in order to make use of memory mapped I/O on receive, the | |
52 | originating netlink subsystem needs to support memory mapped I/O, otherwise | |
53 | it will use an allocated socket buffer as usual and the contents will be | |
54 | copied to the ring on transmission, nullifying most of the performance gains. | |
55 | Dumps of kernel databases automatically support memory mapped I/O. | |
56 | ||
f884ab15 | 57 | Conversion of the transmit path involves changing message construction to |
5683264c | 58 | use memory from the TX ring instead of (usually) a buffer declared on the |
c17cb8b5 | 59 | stack and setting up the frame header appropriately. Optionally poll() can |
5683264c PM |
60 | be used to wait for free frames in the TX ring. |
61 | ||
62 | Structured and definitions for using memory mapped I/O are contained in | |
63 | <linux/netlink.h>. | |
64 | ||
65 | RX and TX rings | |
66 | ---------------- | |
67 | ||
f884ab15 AP |
68 | Each ring contains a number of continuous memory blocks, containing frames of |
69 | fixed size dependent on the parameters used for ring setup. | |
5683264c PM |
70 | |
71 | Ring: [ block 0 ] | |
72 | [ frame 0 ] | |
73 | [ frame 1 ] | |
74 | [ block 1 ] | |
75 | [ frame 2 ] | |
76 | [ frame 3 ] | |
77 | ... | |
78 | [ block n ] | |
79 | [ frame 2 * n ] | |
80 | [ frame 2 * n + 1 ] | |
81 | ||
82 | The blocks are only visible to the kernel, from the point of view of user-space | |
f884ab15 | 83 | the ring just contains the frames in a continuous memory zone. |
5683264c PM |
84 | |
85 | The ring parameters used for setting up the ring are defined as follows: | |
86 | ||
87 | struct nl_mmap_req { | |
88 | unsigned int nm_block_size; | |
89 | unsigned int nm_block_nr; | |
90 | unsigned int nm_frame_size; | |
91 | unsigned int nm_frame_nr; | |
92 | }; | |
93 | ||
f884ab15 | 94 | Frames are grouped into blocks, where each block is a continuous region of memory |
5683264c PM |
95 | and holds nm_block_size / nm_frame_size frames. The total number of frames in |
96 | the ring is nm_frame_nr. The following invariants hold: | |
97 | ||
98 | - frames_per_block = nm_block_size / nm_frame_size | |
99 | ||
100 | - nm_frame_nr = frames_per_block * nm_block_nr | |
101 | ||
102 | Some parameters are constrained, specifically: | |
103 | ||
104 | - nm_block_size must be a multiple of the architectures memory page size. | |
105 | The getpagesize() function can be used to get the page size. | |
106 | ||
107 | - nm_frame_size must be equal or larger to NL_MMAP_HDRLEN, IOW a frame must be | |
108 | able to hold at least the frame header | |
109 | ||
110 | - nm_frame_size must be smaller or equal to nm_block_size | |
111 | ||
112 | - nm_frame_size must be a multiple of NL_MMAP_MSG_ALIGNMENT | |
113 | ||
114 | - nm_frame_nr must equal the actual number of frames as specified above. | |
115 | ||
f884ab15 | 116 | When the kernel can't allocate physically continuous memory for a ring block, |
3dd17ede | 117 | it will fall back to use physically discontinuous memory. This might affect |
5683264c PM |
118 | performance negatively, in order to avoid this the nm_frame_size parameter |
119 | should be chosen to be as small as possible for the required frame size and | |
120 | the number of blocks should be increased instead. | |
121 | ||
122 | Ring frames | |
123 | ------------ | |
124 | ||
125 | Each frames contain a frame header, consisting of a synchronization word and some | |
126 | meta-data, and the message itself. | |
127 | ||
128 | Frame: [ header message ] | |
129 | ||
130 | The frame header is defined as follows: | |
131 | ||
132 | struct nl_mmap_hdr { | |
133 | unsigned int nm_status; | |
134 | unsigned int nm_len; | |
135 | __u32 nm_group; | |
136 | /* credentials */ | |
137 | __u32 nm_pid; | |
138 | __u32 nm_uid; | |
139 | __u32 nm_gid; | |
140 | }; | |
141 | ||
142 | - nm_status is used for synchronizing processing between the kernel and user- | |
143 | space and specifies ownership of the frame as well as the operation to perform | |
144 | ||
145 | - nm_len contains the length of the message contained in the data area | |
146 | ||
147 | - nm_group specified the destination multicast group of message | |
148 | ||
149 | - nm_pid, nm_uid and nm_gid contain the netlink pid, UID and GID of the sending | |
150 | process. These values correspond to the data available using SOCK_PASSCRED in | |
151 | the SCM_CREDENTIALS cmsg. | |
152 | ||
153 | The possible values in the status word are: | |
154 | ||
155 | - NL_MMAP_STATUS_UNUSED: | |
156 | RX ring: frame belongs to the kernel and contains no message | |
157 | for user-space. Approriate action is to invoke poll() | |
158 | to wait for new messages. | |
159 | ||
160 | TX ring: frame belongs to user-space and can be used for | |
161 | message construction. | |
162 | ||
163 | - NL_MMAP_STATUS_RESERVED: | |
164 | RX ring only: frame is currently used by the kernel for message | |
165 | construction and contains no valid message yet. | |
166 | Appropriate action is to invoke poll() to wait for | |
167 | new messages. | |
168 | ||
169 | - NL_MMAP_STATUS_VALID: | |
170 | RX ring: frame contains a valid message. Approriate action is | |
171 | to process the message and release the frame back to | |
172 | the kernel by setting the status to | |
173 | NL_MMAP_STATUS_UNUSED or queue the frame by setting the | |
174 | status to NL_MMAP_STATUS_SKIP. | |
175 | ||
176 | TX ring: the frame contains a valid message from user-space to | |
177 | be processed by the kernel. After completing processing | |
178 | the kernel will release the frame back to user-space by | |
179 | setting the status to NL_MMAP_STATUS_UNUSED. | |
180 | ||
181 | - NL_MMAP_STATUS_COPY: | |
182 | RX ring only: a message is ready to be processed but could not be | |
183 | stored in the ring, either because it exceeded the | |
184 | frame size or because the originating subsystem does | |
185 | not support memory mapped I/O. Appropriate action is | |
186 | to invoke recvmsg() to receive the message and release | |
187 | the frame back to the kernel by setting the status to | |
188 | NL_MMAP_STATUS_UNUSED. | |
189 | ||
190 | - NL_MMAP_STATUS_SKIP: | |
191 | RX ring only: user-space queued the message for later processing, but | |
192 | processed some messages following it in the ring. The | |
193 | kernel should skip this frame when looking for unused | |
194 | frames. | |
195 | ||
196 | The data area of a frame begins at a offset of NL_MMAP_HDRLEN relative to the | |
197 | frame header. | |
198 | ||
199 | TX limitations | |
200 | -------------- | |
201 | ||
202 | Kernel processing usually involves validation of the message received by | |
203 | user-space, then processing its contents. The kernel must assure that | |
204 | userspace is not able to modify the message contents after they have been | |
205 | validated. In order to do so, the message is copied from the ring frame | |
206 | to an allocated buffer if either of these conditions is false: | |
207 | ||
208 | - only a single mapping of the ring exists | |
209 | - the file descriptor is not shared between processes | |
210 | ||
211 | This means that for threaded programs, the kernel will fall back to copying. | |
212 | ||
213 | Example | |
214 | ------- | |
215 | ||
216 | Ring setup: | |
217 | ||
218 | unsigned int block_size = 16 * getpagesize(); | |
219 | struct nl_mmap_req req = { | |
220 | .nm_block_size = block_size, | |
221 | .nm_block_nr = 64, | |
222 | .nm_frame_size = 16384, | |
223 | .nm_frame_nr = 64 * block_size / 16384, | |
224 | }; | |
225 | unsigned int ring_size; | |
226 | void *rx_ring, *tx_ring; | |
227 | ||
228 | /* Configure ring parameters */ | |
88050049 | 229 | if (setsockopt(fd, SOL_NETLINK, NETLINK_RX_RING, &req, sizeof(req)) < 0) |
5683264c | 230 | exit(1); |
88050049 | 231 | if (setsockopt(fd, SOL_NETLINK, NETLINK_TX_RING, &req, sizeof(req)) < 0) |
5683264c PM |
232 | exit(1) |
233 | ||
c17cb8b5 | 234 | /* Calculate size of each individual ring */ |
5683264c PM |
235 | ring_size = req.nm_block_nr * req.nm_block_size; |
236 | ||
237 | /* Map RX/TX rings. The TX ring is located after the RX ring */ | |
238 | rx_ring = mmap(NULL, 2 * ring_size, PROT_READ | PROT_WRITE, | |
239 | MAP_SHARED, fd, 0); | |
240 | if ((long)rx_ring == -1L) | |
241 | exit(1); | |
242 | tx_ring = rx_ring + ring_size: | |
243 | ||
244 | Message reception: | |
245 | ||
246 | This example assumes some ring parameters of the ring setup are available. | |
247 | ||
248 | unsigned int frame_offset = 0; | |
249 | struct nl_mmap_hdr *hdr; | |
250 | struct nlmsghdr *nlh; | |
251 | unsigned char buf[16384]; | |
252 | ssize_t len; | |
253 | ||
254 | while (1) { | |
255 | struct pollfd pfds[1]; | |
256 | ||
257 | pfds[0].fd = fd; | |
258 | pfds[0].events = POLLIN | POLLERR; | |
259 | pfds[0].revents = 0; | |
260 | ||
261 | if (poll(pfds, 1, -1) < 0 && errno != -EINTR) | |
262 | exit(1); | |
263 | ||
264 | /* Check for errors. Error handling omitted */ | |
265 | if (pfds[0].revents & POLLERR) | |
266 | <handle error> | |
267 | ||
268 | /* If no new messages, poll again */ | |
269 | if (!(pfds[0].revents & POLLIN)) | |
270 | continue; | |
271 | ||
272 | /* Process all frames */ | |
273 | while (1) { | |
274 | /* Get next frame header */ | |
275 | hdr = rx_ring + frame_offset; | |
276 | ||
76237576 | 277 | if (hdr->nm_status == NL_MMAP_STATUS_VALID) { |
5683264c | 278 | /* Regular memory mapped frame */ |
76237576 | 279 | nlh = (void *)hdr + NL_MMAP_HDRLEN; |
5683264c PM |
280 | len = hdr->nm_len; |
281 | ||
282 | /* Release empty message immediately. May happen | |
283 | * on error during message construction. | |
284 | */ | |
285 | if (len == 0) | |
286 | goto release; | |
287 | } else if (hdr->nm_status == NL_MMAP_STATUS_COPY) { | |
288 | /* Frame queued to socket receive queue */ | |
289 | len = recv(fd, buf, sizeof(buf), MSG_DONTWAIT); | |
290 | if (len <= 0) | |
291 | break; | |
292 | nlh = buf; | |
293 | } else | |
294 | /* No more messages to process, continue polling */ | |
295 | break; | |
296 | ||
297 | process_msg(nlh); | |
298 | release: | |
299 | /* Release frame back to the kernel */ | |
300 | hdr->nm_status = NL_MMAP_STATUS_UNUSED; | |
301 | ||
302 | /* Advance frame offset to next frame */ | |
303 | frame_offset = (frame_offset + frame_size) % ring_size; | |
304 | } | |
305 | } | |
306 | ||
307 | Message transmission: | |
308 | ||
309 | This example assumes some ring parameters of the ring setup are available. | |
310 | A single message is constructed and transmitted, to send multiple messages | |
311 | at once they would be constructed in consecutive frames before a final call | |
312 | to sendto(). | |
313 | ||
314 | unsigned int frame_offset = 0; | |
315 | struct nl_mmap_hdr *hdr; | |
316 | struct nlmsghdr *nlh; | |
317 | struct sockaddr_nl addr = { | |
318 | .nl_family = AF_NETLINK, | |
319 | }; | |
320 | ||
321 | hdr = tx_ring + frame_offset; | |
322 | if (hdr->nm_status != NL_MMAP_STATUS_UNUSED) | |
323 | /* No frame available. Use poll() to avoid. */ | |
324 | exit(1); | |
325 | ||
326 | nlh = (void *)hdr + NL_MMAP_HDRLEN; | |
327 | ||
328 | /* Build message */ | |
329 | build_message(nlh); | |
330 | ||
331 | /* Fill frame header: length and status need to be set */ | |
332 | hdr->nm_len = nlh->nlmsg_len; | |
333 | hdr->nm_status = NL_MMAP_STATUS_VALID; | |
334 | ||
335 | if (sendto(fd, NULL, 0, 0, &addr, sizeof(addr)) < 0) | |
336 | exit(1); | |
337 | ||
338 | /* Advance frame offset to next frame */ | |
339 | frame_offset = (frame_offset + frame_size) % ring_size; |