Commit | Line | Data |
---|---|---|
617b586b HR |
1 | INFINIBAND MIDLAYER LOCKING |
2 | ||
3 | This guide is an attempt to make explicit the locking assumptions | |
4 | made by the InfiniBand midlayer. It describes the requirements on | |
5 | both low-level drivers that sit below the midlayer and upper level | |
6 | protocols that use the midlayer. | |
7 | ||
8 | Sleeping and interrupt context | |
9 | ||
10 | With the following exceptions, a low-level driver implementation of | |
11 | all of the methods in struct ib_device may sleep. The exceptions | |
12 | are any methods from the list: | |
13 | ||
14 | create_ah | |
15 | modify_ah | |
16 | query_ah | |
17 | destroy_ah | |
18 | bind_mw | |
19 | post_send | |
20 | post_recv | |
21 | poll_cq | |
22 | req_notify_cq | |
23 | map_phys_fmr | |
24 | ||
25 | which may not sleep and must be callable from any context. | |
26 | ||
27 | The corresponding functions exported to upper level protocol | |
28 | consumers: | |
29 | ||
30 | ib_create_ah | |
31 | ib_modify_ah | |
32 | ib_query_ah | |
33 | ib_destroy_ah | |
34 | ib_bind_mw | |
35 | ib_post_send | |
36 | ib_post_recv | |
37 | ib_req_notify_cq | |
38 | ib_map_phys_fmr | |
39 | ||
40 | are therefore safe to call from any context. | |
41 | ||
42 | In addition, the function | |
43 | ||
44 | ib_dispatch_event | |
45 | ||
46 | used by low-level drivers to dispatch asynchronous events through | |
47 | the midlayer is also safe to call from any context. | |
48 | ||
49 | Reentrancy | |
50 | ||
51 | All of the methods in struct ib_device exported by a low-level | |
52 | driver must be fully reentrant. The low-level driver is required to | |
53 | perform all synchronization necessary to maintain consistency, even | |
54 | if multiple function calls using the same object are run | |
55 | simultaneously. | |
56 | ||
57 | The IB midlayer does not perform any serialization of function calls. | |
58 | ||
59 | Because low-level drivers are reentrant, upper level protocol | |
60 | consumers are not required to perform any serialization. However, | |
61 | some serialization may be required to get sensible results. For | |
62 | example, a consumer may safely call ib_poll_cq() on multiple CPUs | |
63 | simultaneously. However, the ordering of the work completion | |
64 | information between different calls of ib_poll_cq() is not defined. | |
65 | ||
66 | Callbacks | |
67 | ||
68 | A low-level driver must not perform a callback directly from the | |
69 | same callchain as an ib_device method call. For example, it is not | |
70 | allowed for a low-level driver to call a consumer's completion event | |
71 | handler directly from its post_send method. Instead, the low-level | |
72 | driver should defer this callback by, for example, scheduling a | |
73 | tasklet to perform the callback. | |
74 | ||
75 | The low-level driver is responsible for ensuring that multiple | |
76 | completion event handlers for the same CQ are not called | |
77 | simultaneously. The driver must guarantee that only one CQ event | |
78 | handler for a given CQ is running at a time. In other words, the | |
79 | following situation is not allowed: | |
80 | ||
81 | CPU1 CPU2 | |
82 | ||
83 | low-level driver -> | |
84 | consumer CQ event callback: | |
85 | /* ... */ | |
86 | ib_req_notify_cq(cq, ...); | |
87 | low-level driver -> | |
88 | /* ... */ consumer CQ event callback: | |
89 | /* ... */ | |
90 | return from CQ event handler | |
91 | ||
92 | The context in which completion event and asynchronous event | |
93 | callbacks run is not defined. Depending on the low-level driver, it | |
94 | may be process context, softirq context, or interrupt context. | |
95 | Upper level protocol consumers may not sleep in a callback. | |
96 | ||
97 | Hot-plug | |
98 | ||
99 | A low-level driver announces that a device is ready for use by | |
100 | consumers when it calls ib_register_device(), all initialization | |
101 | must be complete before this call. The device must remain usable | |
102 | until the driver's call to ib_unregister_device() has returned. | |
103 | ||
104 | A low-level driver must call ib_register_device() and | |
105 | ib_unregister_device() from process context. It must not hold any | |
106 | semaphores that could cause deadlock if a consumer calls back into | |
107 | the driver across these calls. | |
108 | ||
109 | An upper level protocol consumer may begin using an IB device as | |
110 | soon as the add method of its struct ib_client is called for that | |
111 | device. A consumer must finish all cleanup and free all resources | |
112 | relating to a device before returning from the remove method. | |
113 | ||
114 | A consumer is permitted to sleep in its add and remove methods. |