Commit | Line | Data |
---|---|---|
64db4cff PM |
1 | CONFIG_RCU_TRACE debugfs Files and Formats |
2 | ||
3 | ||
8e79e1f9 PM |
4 | The rcutree and rcutiny implementations of RCU provide debugfs trace |
5 | output that summarizes counters and state. This information is useful for | |
6 | debugging RCU itself, and can sometimes also help to debug abuses of RCU. | |
7 | The following sections describe the debugfs files and formats, first | |
8 | for rcutree and next for rcutiny. | |
64db4cff PM |
9 | |
10 | ||
8e79e1f9 | 11 | CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU debugfs Files and Formats |
64db4cff | 12 | |
90e6ac36 PM |
13 | These implementations of RCU provides several debugfs files under the |
14 | top-level directory "rcu": | |
15 | ||
16 | rcu/rcudata: | |
17 | Displays fields in struct rcu_data. | |
18 | rcu/rcudata.csv: | |
19 | Comma-separated values spreadsheet version of rcudata. | |
20 | rcu/rcugp: | |
21 | Displays grace-period counters. | |
22 | rcu/rcuhier: | |
23 | Displays the struct rcu_node hierarchy. | |
24 | rcu/rcu_pending: | |
25 | Displays counts of the reasons rcu_pending() decided that RCU had | |
26 | work to do. | |
27 | rcu/rcutorture: | |
28 | Displays rcutorture test progress. | |
29 | rcu/rcuboost: | |
30 | Displays RCU boosting statistics. Only present if | |
31 | CONFIG_RCU_BOOST=y. | |
64db4cff PM |
32 | |
33 | The output of "cat rcu/rcudata" looks as follows: | |
34 | ||
d6714c22 | 35 | rcu_sched: |
2036d94a PM |
36 | 0 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=545/1/0 df=50 of=0 ql=163 qs=NRW. kt=0/W/0 ktl=ebc3 b=10 ci=153737 co=0 ca=0 |
37 | 1 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=967/1/0 df=58 of=0 ql=634 qs=NRW. kt=0/W/1 ktl=58c b=10 ci=191037 co=0 ca=0 | |
38 | 2 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1081/1/0 df=175 of=0 ql=74 qs=N.W. kt=0/W/2 ktl=da94 b=10 ci=75991 co=0 ca=0 | |
39 | 3 c=20942 g=20943 pq=1 pgp=20942 qp=1 dt=1846/0/0 df=404 of=0 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=72261 co=0 ca=0 | |
40 | 4 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=369/1/0 df=83 of=0 ql=48 qs=N.W. kt=0/W/4 ktl=e0e7 b=10 ci=128365 co=0 ca=0 | |
41 | 5 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=381/1/0 df=64 of=0 ql=169 qs=NRW. kt=0/W/5 ktl=fb2f b=10 ci=164360 co=0 ca=0 | |
42 | 6 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1037/1/0 df=183 of=0 ql=62 qs=N.W. kt=0/W/6 ktl=d2ad b=10 ci=65663 co=0 ca=0 | |
43 | 7 c=20897 g=20897 pq=1 pgp=20896 qp=0 dt=1572/0/0 df=382 of=0 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=75006 co=0 ca=0 | |
64db4cff | 44 | rcu_bh: |
2036d94a PM |
45 | 0 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=545/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/0 ktl=ebc3 b=10 ci=0 co=0 ca=0 |
46 | 1 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=967/1/0 df=3 of=0 ql=0 qs=.... kt=0/W/1 ktl=58c b=10 ci=151 co=0 ca=0 | |
47 | 2 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1081/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/2 ktl=da94 b=10 ci=0 co=0 ca=0 | |
48 | 3 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1846/0/0 df=8 of=0 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=0 co=0 ca=0 | |
49 | 4 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=369/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/4 ktl=e0e7 b=10 ci=0 co=0 ca=0 | |
50 | 5 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=381/1/0 df=4 of=0 ql=0 qs=.... kt=0/W/5 ktl=fb2f b=10 ci=0 co=0 ca=0 | |
51 | 6 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1037/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/6 ktl=d2ad b=10 ci=0 co=0 ca=0 | |
52 | 7 c=1474 g=1474 pq=1 pgp=1473 qp=0 dt=1572/0/0 df=8 of=0 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=0 co=0 ca=0 | |
64db4cff | 53 | |
bd58b430 PM |
54 | The first section lists the rcu_data structures for rcu_sched, the second |
55 | for rcu_bh. Note that CONFIG_TREE_PREEMPT_RCU kernels will have an | |
56 | additional section for rcu_preempt. Each section has one line per CPU, | |
57 | or eight for this 8-CPU system. The fields are as follows: | |
64db4cff PM |
58 | |
59 | o The number at the beginning of each line is the CPU number. | |
60 | CPUs numbers followed by an exclamation mark are offline, | |
61 | but have been online at least once since boot. There will be | |
62 | no output for CPUs that have never been online, which can be | |
63 | a good thing in the surprisingly common case where NR_CPUS is | |
64 | substantially larger than the number of actual CPUs. | |
65 | ||
66 | o "c" is the count of grace periods that this CPU believes have | |
2fa218d8 PM |
67 | completed. Offlined CPUs and CPUs in dynticks idle mode may |
68 | lag quite a ways behind, for example, CPU 6 under "rcu_sched" | |
69 | above, which has been offline through not quite 40,000 RCU grace | |
70 | periods. It is not unusual to see CPUs lagging by thousands of | |
71 | grace periods. | |
64db4cff PM |
72 | |
73 | o "g" is the count of grace periods that this CPU believes have | |
2fa218d8 PM |
74 | started. Again, offlined CPUs and CPUs in dynticks idle mode |
75 | may lag behind. If the "c" and "g" values are equal, this CPU | |
76 | has already reported a quiescent state for the last RCU grace | |
77 | period that it is aware of, otherwise, the CPU believes that it | |
78 | owes RCU a quiescent state. | |
64db4cff PM |
79 | |
80 | o "pq" indicates that this CPU has passed through a quiescent state | |
81 | for the current grace period. It is possible for "pq" to be | |
82 | "1" and "c" different than "g", which indicates that although | |
83 | the CPU has passed through a quiescent state, either (1) this | |
84 | CPU has not yet reported that fact, (2) some other CPU has not | |
85 | yet reported for this grace period, or (3) both. | |
86 | ||
e4cc1f22 | 87 | o "pgp" indicates which grace period the last-observed quiescent |
64db4cff PM |
88 | state for this CPU corresponds to. This is important for handling |
89 | the race between CPU 0 reporting an extended dynticks-idle | |
90 | quiescent state for CPU 1 and CPU 1 suddenly waking up and | |
91 | reporting its own quiescent state. If CPU 1 was the last CPU | |
92 | for the current grace period, then the CPU that loses this race | |
93 | will attempt to incorrectly mark CPU 1 as having checked in for | |
94 | the next grace period! | |
95 | ||
96 | o "qp" indicates that RCU still expects a quiescent state from | |
2fa218d8 PM |
97 | this CPU. Offlined CPUs and CPUs in dyntick idle mode might |
98 | well have qp=1, which is OK: RCU is still ignoring them. | |
64db4cff | 99 | |
64db4cff PM |
100 | o "dt" is the current value of the dyntick counter that is incremented |
101 | when entering or leaving dynticks idle state, either by the | |
23b5c8fa PM |
102 | scheduler or by irq. This number is even if the CPU is in |
103 | dyntick idle mode and odd otherwise. The number after the first | |
104 | "/" is the interrupt nesting depth when in dyntick-idle state, | |
105 | or one greater than the interrupt-nesting depth otherwise. | |
106 | The number after the second "/" is the NMI nesting depth. | |
64db4cff | 107 | |
64db4cff PM |
108 | o "df" is the number of times that some other CPU has forced a |
109 | quiescent state on behalf of this CPU due to this CPU being in | |
110 | dynticks-idle state. | |
111 | ||
64db4cff PM |
112 | o "of" is the number of times that some other CPU has forced a |
113 | quiescent state on behalf of this CPU due to this CPU being | |
2fa218d8 | 114 | offline. In a perfect world, this might never happen, but it |
64db4cff PM |
115 | turns out that offlining and onlining a CPU can take several grace |
116 | periods, and so there is likely to be an extended period of time | |
117 | when RCU believes that the CPU is online when it really is not. | |
118 | Please note that erring in the other direction (RCU believing a | |
119 | CPU is offline when it is really alive and kicking) is a fatal | |
120 | error, so it makes sense to err conservatively. | |
121 | ||
64db4cff PM |
122 | o "ql" is the number of RCU callbacks currently residing on |
123 | this CPU. This is the total number of callbacks, regardless | |
124 | of what state they are in (new, waiting for grace period to | |
125 | start, waiting for grace period to end, ready to invoke). | |
126 | ||
0ac3d136 PM |
127 | o "qs" gives an indication of the state of the callback queue |
128 | with four characters: | |
129 | ||
130 | "N" Indicates that there are callbacks queued that are not | |
131 | ready to be handled by the next grace period, and thus | |
132 | will be handled by the grace period following the next | |
133 | one. | |
134 | ||
135 | "R" Indicates that there are callbacks queued that are | |
136 | ready to be handled by the next grace period. | |
137 | ||
138 | "W" Indicates that there are callbacks queued that are | |
139 | waiting on the current grace period. | |
140 | ||
141 | "D" Indicates that there are callbacks queued that have | |
142 | already been handled by a prior grace period, and are | |
143 | thus waiting to be invoked. Note that callbacks in | |
144 | the process of being invoked are not counted here. | |
145 | Callbacks in the process of being invoked are those | |
146 | that have been removed from the rcu_data structures | |
147 | queues by rcu_do_batch(), but which have not yet been | |
148 | invoked. | |
149 | ||
150 | If there are no callbacks in a given one of the above states, | |
151 | the corresponding character is replaced by ".". | |
152 | ||
90e6ac36 | 153 | o "kt" is the per-CPU kernel-thread state. The digit preceding |
15ba0ba8 | 154 | the first slash is zero if there is no work pending and 1 |
5ece5bab PM |
155 | otherwise. The character between the first pair of slashes is |
156 | as follows: | |
90e6ac36 PM |
157 | |
158 | "S" The kernel thread is stopped, in other words, all | |
159 | CPUs corresponding to this rcu_node structure are | |
160 | offline. | |
161 | ||
162 | "R" The kernel thread is running. | |
163 | ||
164 | "W" The kernel thread is waiting because there is no work | |
165 | for it to do. | |
166 | ||
15ba0ba8 PM |
167 | "O" The kernel thread is waiting because it has been |
168 | forced off of its designated CPU or because its | |
169 | ->cpus_allowed mask permits it to run on other than | |
170 | its designated CPU. | |
171 | ||
90e6ac36 PM |
172 | "Y" The kernel thread is yielding to avoid hogging CPU. |
173 | ||
174 | "?" Unknown value, indicates a bug. | |
175 | ||
15ba0ba8 PM |
176 | The number after the final slash is the CPU that the kthread |
177 | is actually running on. | |
178 | ||
d5988af5 PM |
179 | This field is displayed only for CONFIG_RCU_BOOST kernels. |
180 | ||
5ece5bab PM |
181 | o "ktl" is the low-order 16 bits (in hexadecimal) of the count of |
182 | the number of times that this CPU's per-CPU kthread has gone | |
183 | through its loop servicing invoke_rcu_cpu_kthread() requests. | |
184 | ||
d5988af5 PM |
185 | This field is displayed only for CONFIG_RCU_BOOST kernels. |
186 | ||
64db4cff PM |
187 | o "b" is the batch limit for this CPU. If more than this number |
188 | of RCU callbacks is ready to invoke, then the remainder will | |
189 | be deferred. | |
190 | ||
269dcc1c PM |
191 | o "ci" is the number of RCU callbacks that have been invoked for |
192 | this CPU. Note that ci+ql is the number of callbacks that have | |
193 | been registered in absence of CPU-hotplug activity. | |
194 | ||
195 | o "co" is the number of RCU callbacks that have been orphaned due to | |
2d999e03 PM |
196 | this CPU going offline. These orphaned callbacks have been moved |
197 | to an arbitrarily chosen online CPU. | |
269dcc1c PM |
198 | |
199 | o "ca" is the number of RCU callbacks that have been adopted due to | |
200 | other CPUs going offline. Note that ci+co-ca+ql is the number of | |
201 | RCU callbacks registered on this CPU. | |
202 | ||
6fd9b3a4 PM |
203 | There is also an rcu/rcudata.csv file with the same information in |
204 | comma-separated-variable spreadsheet format. | |
205 | ||
64db4cff PM |
206 | |
207 | The output of "cat rcu/rcugp" looks as follows: | |
208 | ||
d6714c22 | 209 | rcu_sched: completed=33062 gpnum=33063 |
64db4cff PM |
210 | rcu_bh: completed=464 gpnum=464 |
211 | ||
bd58b430 PM |
212 | Again, this output is for both "rcu_sched" and "rcu_bh". Note that |
213 | kernels built with CONFIG_TREE_PREEMPT_RCU will have an additional | |
214 | "rcu_preempt" line. The fields are taken from the rcu_state structure, | |
215 | and are as follows: | |
64db4cff PM |
216 | |
217 | o "completed" is the number of grace periods that have completed. | |
218 | It is comparable to the "c" field from rcu/rcudata in that a | |
219 | CPU whose "c" field matches the value of "completed" is aware | |
220 | that the corresponding RCU grace period has completed. | |
221 | ||
222 | o "gpnum" is the number of grace periods that have started. It is | |
223 | comparable to the "g" field from rcu/rcudata in that a CPU | |
224 | whose "g" field matches the value of "gpnum" is aware that the | |
225 | corresponding RCU grace period has started. | |
226 | ||
227 | If these two fields are equal (as they are for "rcu_bh" above), | |
228 | then there is no grace period in progress, in other words, RCU | |
229 | is idle. On the other hand, if the two fields differ (as they | |
bd58b430 | 230 | do for "rcu_sched" above), then an RCU grace period is in progress. |
64db4cff PM |
231 | |
232 | ||
233 | The output of "cat rcu/rcuhier" looks as follows, with very long lines: | |
234 | ||
2d999e03 | 235 | c=6902 g=6903 s=2 jfq=3 j=72c7 nfqs=13142/nfqsng=0(13142) fqlh=6 |
12f5f524 PM |
236 | 1/1 ..>. 0:127 ^0 |
237 | 3/3 ..>. 0:35 ^0 0/0 ..>. 36:71 ^1 0/0 ..>. 72:107 ^2 0/0 ..>. 108:127 ^3 | |
238 | 3/3f ..>. 0:5 ^0 2/3 ..>. 6:11 ^1 0/0 ..>. 12:17 ^2 0/0 ..>. 18:23 ^3 0/0 ..>. 24:29 ^4 0/0 ..>. 30:35 ^5 0/0 ..>. 36:41 ^0 0/0 ..>. 42:47 ^1 0/0 ..>. 48:53 ^2 0/0 ..>. 54:59 ^3 0/0 ..>. 60:65 ^4 0/0 ..>. 66:71 ^5 0/0 ..>. 72:77 ^0 0/0 ..>. 78:83 ^1 0/0 ..>. 84:89 ^2 0/0 ..>. 90:95 ^3 0/0 ..>. 96:101 ^4 0/0 ..>. 102:107 ^5 0/0 ..>. 108:113 ^0 0/0 ..>. 114:119 ^1 0/0 ..>. 120:125 ^2 0/0 ..>. 126:127 ^3 | |
64db4cff | 239 | rcu_bh: |
2d999e03 | 240 | c=-226 g=-226 s=1 jfq=-5701 j=72c7 nfqs=88/nfqsng=0(88) fqlh=0 |
12f5f524 PM |
241 | 0/1 ..>. 0:127 ^0 |
242 | 0/3 ..>. 0:35 ^0 0/0 ..>. 36:71 ^1 0/0 ..>. 72:107 ^2 0/0 ..>. 108:127 ^3 | |
243 | 0/3f ..>. 0:5 ^0 0/3 ..>. 6:11 ^1 0/0 ..>. 12:17 ^2 0/0 ..>. 18:23 ^3 0/0 ..>. 24:29 ^4 0/0 ..>. 30:35 ^5 0/0 ..>. 36:41 ^0 0/0 ..>. 42:47 ^1 0/0 ..>. 48:53 ^2 0/0 ..>. 54:59 ^3 0/0 ..>. 60:65 ^4 0/0 ..>. 66:71 ^5 0/0 ..>. 72:77 ^0 0/0 ..>. 78:83 ^1 0/0 ..>. 84:89 ^2 0/0 ..>. 90:95 ^3 0/0 ..>. 96:101 ^4 0/0 ..>. 102:107 ^5 0/0 ..>. 108:113 ^0 0/0 ..>. 114:119 ^1 0/0 ..>. 120:125 ^2 0/0 ..>. 126:127 ^3 | |
64db4cff | 244 | |
bd58b430 PM |
245 | This is once again split into "rcu_sched" and "rcu_bh" portions, |
246 | and CONFIG_TREE_PREEMPT_RCU kernels will again have an additional | |
247 | "rcu_preempt" section. The fields are as follows: | |
64db4cff PM |
248 | |
249 | o "c" is exactly the same as "completed" under rcu/rcugp. | |
250 | ||
251 | o "g" is exactly the same as "gpnum" under rcu/rcugp. | |
252 | ||
253 | o "s" is the "signaled" state that drives force_quiescent_state()'s | |
254 | state machine. | |
255 | ||
256 | o "jfq" is the number of jiffies remaining for this grace period | |
257 | before force_quiescent_state() is invoked to help push things | |
269dcc1c | 258 | along. Note that CPUs in dyntick-idle mode throughout the grace |
64db4cff PM |
259 | period will not report on their own, but rather must be check by |
260 | some other CPU via force_quiescent_state(). | |
261 | ||
262 | o "j" is the low-order four hex digits of the jiffies counter. | |
263 | Yes, Paul did run into a number of problems that turned out to | |
264 | be due to the jiffies counter no longer counting. Why do you ask? | |
265 | ||
266 | o "nfqs" is the number of calls to force_quiescent_state() since | |
267 | boot. | |
268 | ||
269 | o "nfqsng" is the number of useless calls to force_quiescent_state(), | |
270 | where there wasn't actually a grace period active. This can | |
271 | happen due to races. The number in parentheses is the difference | |
272 | between "nfqs" and "nfqsng", or the number of times that | |
273 | force_quiescent_state() actually did some real work. | |
274 | ||
275 | o "fqlh" is the number of calls to force_quiescent_state() that | |
276 | exited immediately (without even being counted in nfqs above) | |
277 | due to contention on ->fqslock. | |
278 | ||
279 | o Each element of the form "1/1 0:127 ^0" represents one struct | |
280 | rcu_node. Each line represents one level of the hierarchy, from | |
281 | root to leaves. It is best to think of the rcu_data structures | |
282 | as forming yet another level after the leaves. Note that there | |
283 | might be either one, two, or three levels of rcu_node structures, | |
284 | depending on the relationship between CONFIG_RCU_FANOUT and | |
285 | CONFIG_NR_CPUS. | |
0edf1a68 | 286 | |
64db4cff PM |
287 | o The numbers separated by the "/" are the qsmask followed |
288 | by the qsmaskinit. The qsmask will have one bit | |
289 | set for each entity in the next lower level that | |
290 | has not yet checked in for the current grace period. | |
291 | The qsmaskinit will have one bit for each entity that is | |
292 | currently expected to check in during each grace period. | |
293 | The value of qsmaskinit is assigned to that of qsmask | |
294 | at the beginning of each grace period. | |
295 | ||
bd58b430 PM |
296 | For example, for "rcu_sched", the qsmask of the first |
297 | entry of the lowest level is 0x14, meaning that we | |
298 | are still waiting for CPUs 2 and 4 to check in for the | |
299 | current grace period. | |
64db4cff | 300 | |
0edf1a68 | 301 | o The characters separated by the ">" indicate the state |
12f5f524 | 302 | of the blocked-tasks lists. A "G" preceding the ">" |
0edf1a68 PM |
303 | indicates that at least one task blocked in an RCU |
304 | read-side critical section blocks the current grace | |
12f5f524 PM |
305 | period, while a "E" preceding the ">" indicates that |
306 | at least one task blocked in an RCU read-side critical | |
307 | section blocks the current expedited grace period. | |
308 | A "T" character following the ">" indicates that at | |
309 | least one task is blocked within an RCU read-side | |
310 | critical section, regardless of whether any current | |
311 | grace period (expedited or normal) is inconvenienced. | |
312 | A "." character appears if the corresponding condition | |
313 | does not hold, so that "..>." indicates that no tasks | |
314 | are blocked. In contrast, "GE>T" indicates maximal | |
315 | inconvenience from blocked tasks. | |
0edf1a68 | 316 | |
64db4cff PM |
317 | o The numbers separated by the ":" are the range of CPUs |
318 | served by this struct rcu_node. This can be helpful | |
319 | in working out how the hierarchy is wired together. | |
320 | ||
321 | For example, the first entry at the lowest level shows | |
322 | "0:5", indicating that it covers CPUs 0 through 5. | |
323 | ||
324 | o The number after the "^" indicates the bit in the | |
325 | next higher level rcu_node structure that this | |
326 | rcu_node structure corresponds to. | |
327 | ||
328 | For example, the first entry at the lowest level shows | |
329 | "^0", indicating that it corresponds to bit zero in | |
330 | the first entry at the middle level. | |
6fd9b3a4 PM |
331 | |
332 | ||
333 | The output of "cat rcu/rcu_pending" looks as follows: | |
334 | ||
d6714c22 | 335 | rcu_sched: |
d21670ac PM |
336 | 0 np=255892 qsp=53936 rpq=85 cbr=0 cng=14417 gpc=10033 gps=24320 nf=6445 nn=146741 |
337 | 1 np=261224 qsp=54638 rpq=33 cbr=0 cng=25723 gpc=16310 gps=2849 nf=5912 nn=155792 | |
338 | 2 np=237496 qsp=49664 rpq=23 cbr=0 cng=2762 gpc=45478 gps=1762 nf=1201 nn=136629 | |
339 | 3 np=236249 qsp=48766 rpq=98 cbr=0 cng=286 gpc=48049 gps=1218 nf=207 nn=137723 | |
340 | 4 np=221310 qsp=46850 rpq=7 cbr=0 cng=26 gpc=43161 gps=4634 nf=3529 nn=123110 | |
341 | 5 np=237332 qsp=48449 rpq=9 cbr=0 cng=54 gpc=47920 gps=3252 nf=201 nn=137456 | |
342 | 6 np=219995 qsp=46718 rpq=12 cbr=0 cng=50 gpc=42098 gps=6093 nf=4202 nn=120834 | |
343 | 7 np=249893 qsp=49390 rpq=42 cbr=0 cng=72 gpc=38400 gps=17102 nf=41 nn=144888 | |
6fd9b3a4 | 344 | rcu_bh: |
d21670ac PM |
345 | 0 np=146741 qsp=1419 rpq=6 cbr=0 cng=6 gpc=0 gps=0 nf=2 nn=145314 |
346 | 1 np=155792 qsp=12597 rpq=3 cbr=0 cng=0 gpc=4 gps=8 nf=3 nn=143180 | |
347 | 2 np=136629 qsp=18680 rpq=1 cbr=0 cng=0 gpc=7 gps=6 nf=0 nn=117936 | |
348 | 3 np=137723 qsp=2843 rpq=0 cbr=0 cng=0 gpc=10 gps=7 nf=0 nn=134863 | |
349 | 4 np=123110 qsp=12433 rpq=0 cbr=0 cng=0 gpc=4 gps=2 nf=0 nn=110671 | |
350 | 5 np=137456 qsp=4210 rpq=1 cbr=0 cng=0 gpc=6 gps=5 nf=0 nn=133235 | |
351 | 6 np=120834 qsp=9902 rpq=2 cbr=0 cng=0 gpc=6 gps=3 nf=2 nn=110921 | |
352 | 7 np=144888 qsp=26336 rpq=0 cbr=0 cng=0 gpc=8 gps=2 nf=0 nn=118542 | |
6fd9b3a4 | 353 | |
bd58b430 PM |
354 | As always, this is once again split into "rcu_sched" and "rcu_bh" |
355 | portions, with CONFIG_TREE_PREEMPT_RCU kernels having an additional | |
356 | "rcu_preempt" section. The fields are as follows: | |
6fd9b3a4 PM |
357 | |
358 | o "np" is the number of times that __rcu_pending() has been invoked | |
359 | for the corresponding flavor of RCU. | |
360 | ||
361 | o "qsp" is the number of times that the RCU was waiting for a | |
362 | quiescent state from this CPU. | |
363 | ||
d21670ac PM |
364 | o "rpq" is the number of times that the CPU had passed through |
365 | a quiescent state, but not yet reported it to RCU. | |
366 | ||
6fd9b3a4 PM |
367 | o "cbr" is the number of times that this CPU had RCU callbacks |
368 | that had passed through a grace period, and were thus ready | |
369 | to be invoked. | |
370 | ||
371 | o "cng" is the number of times that this CPU needed another | |
372 | grace period while RCU was idle. | |
373 | ||
374 | o "gpc" is the number of times that an old grace period had | |
375 | completed, but this CPU was not yet aware of it. | |
376 | ||
377 | o "gps" is the number of times that a new grace period had started, | |
378 | but this CPU was not yet aware of it. | |
379 | ||
380 | o "nf" is the number of times that this CPU suspected that the | |
381 | current grace period had run for too long, and thus needed to | |
382 | be forced. | |
383 | ||
384 | Please note that "forcing" consists of sending resched IPIs | |
385 | to holdout CPUs. If that CPU really still is in an old RCU | |
386 | read-side critical section, then we really do have to wait for it. | |
387 | The assumption behing "forcing" is that the CPU is not still in | |
388 | an old RCU read-side critical section, but has not yet responded | |
389 | for some other reason. | |
390 | ||
391 | o "nn" is the number of times that this CPU needed nothing. Alert | |
392 | readers will note that the rcu "nn" number for a given CPU very | |
393 | closely matches the rcu_bh "np" number for that same CPU. This | |
394 | is due to short-circuit evaluation in rcu_pending(). | |
8e79e1f9 PM |
395 | |
396 | ||
90e6ac36 PM |
397 | The output of "cat rcu/rcutorture" looks as follows: |
398 | ||
399 | rcutorture test sequence: 0 (test in progress) | |
400 | rcutorture update version number: 615 | |
401 | ||
402 | The first line shows the number of rcutorture tests that have completed | |
403 | since boot. If a test is currently running, the "(test in progress)" | |
404 | string will appear as shown above. The second line shows the number of | |
405 | update cycles that the current test has started, or zero if there is | |
406 | no test in progress. | |
407 | ||
408 | ||
409 | The output of "cat rcu/rcuboost" looks as follows: | |
410 | ||
411 | 0:5 tasks=.... kt=W ntb=0 neb=0 nnb=0 j=2f95 bt=300f | |
412 | balk: nt=0 egt=989 bt=0 nb=0 ny=0 nos=16 | |
413 | 6:7 tasks=.... kt=W ntb=0 neb=0 nnb=0 j=2f95 bt=300f | |
414 | balk: nt=0 egt=225 bt=0 nb=0 ny=0 nos=6 | |
415 | ||
416 | This information is output only for rcu_preempt. Each two-line entry | |
417 | corresponds to a leaf rcu_node strcuture. The fields are as follows: | |
418 | ||
419 | o "n:m" is the CPU-number range for the corresponding two-line | |
420 | entry. In the sample output above, the first entry covers | |
421 | CPUs zero through five and the second entry covers CPUs 6 | |
422 | and 7. | |
423 | ||
424 | o "tasks=TNEB" gives the state of the various segments of the | |
425 | rnp->blocked_tasks list: | |
426 | ||
427 | "T" This indicates that there are some tasks that blocked | |
428 | while running on one of the corresponding CPUs while | |
429 | in an RCU read-side critical section. | |
430 | ||
431 | "N" This indicates that some of the blocked tasks are preventing | |
432 | the current normal (non-expedited) grace period from | |
433 | completing. | |
434 | ||
435 | "E" This indicates that some of the blocked tasks are preventing | |
436 | the current expedited grace period from completing. | |
437 | ||
438 | "B" This indicates that some of the blocked tasks are in | |
439 | need of RCU priority boosting. | |
440 | ||
441 | Each character is replaced with "." if the corresponding | |
442 | condition does not hold. | |
443 | ||
444 | o "kt" is the state of the RCU priority-boosting kernel | |
445 | thread associated with the corresponding rcu_node structure. | |
446 | The state can be one of the following: | |
447 | ||
448 | "S" The kernel thread is stopped, in other words, all | |
449 | CPUs corresponding to this rcu_node structure are | |
450 | offline. | |
451 | ||
452 | "R" The kernel thread is running. | |
453 | ||
454 | "W" The kernel thread is waiting because there is no work | |
455 | for it to do. | |
456 | ||
457 | "Y" The kernel thread is yielding to avoid hogging CPU. | |
458 | ||
459 | "?" Unknown value, indicates a bug. | |
460 | ||
461 | o "ntb" is the number of tasks boosted. | |
462 | ||
463 | o "neb" is the number of tasks boosted in order to complete an | |
464 | expedited grace period. | |
465 | ||
466 | o "nnb" is the number of tasks boosted in order to complete a | |
467 | normal (non-expedited) grace period. When boosting a task | |
468 | that was blocking both an expedited and a normal grace period, | |
469 | it is counted against the expedited total above. | |
470 | ||
471 | o "j" is the low-order 16 bits of the jiffies counter in | |
472 | hexadecimal. | |
473 | ||
474 | o "bt" is the low-order 16 bits of the value that the jiffies | |
475 | counter will have when we next start boosting, assuming that | |
476 | the current grace period does not end beforehand. This is | |
477 | also in hexadecimal. | |
478 | ||
479 | o "balk: nt" counts the number of times we didn't boost (in | |
480 | other words, we balked) even though it was time to boost because | |
481 | there were no blocked tasks to boost. This situation occurs | |
482 | when there is one blocked task on one rcu_node structure and | |
483 | none on some other rcu_node structure. | |
484 | ||
485 | o "egt" counts the number of times we balked because although | |
486 | there were blocked tasks, none of them were blocking the | |
487 | current grace period, whether expedited or otherwise. | |
488 | ||
489 | o "bt" counts the number of times we balked because boosting | |
490 | had already been initiated for the current grace period. | |
491 | ||
492 | o "nb" counts the number of times we balked because there | |
493 | was at least one task blocking the current non-expedited grace | |
494 | period that never had blocked. If it is already running, it | |
495 | just won't help to boost its priority! | |
496 | ||
497 | o "ny" counts the number of times we balked because it was | |
498 | not yet time to start boosting. | |
499 | ||
500 | o "nos" counts the number of times we balked for other | |
501 | reasons, e.g., the grace period ended first. | |
502 | ||
503 | ||
8e79e1f9 PM |
504 | CONFIG_TINY_RCU and CONFIG_TINY_PREEMPT_RCU debugfs Files and Formats |
505 | ||
506 | These implementations of RCU provides a single debugfs file under the | |
507 | top-level directory RCU, namely rcu/rcudata, which displays fields in | |
508 | rcu_bh_ctrlblk, rcu_sched_ctrlblk and, for CONFIG_TINY_PREEMPT_RCU, | |
509 | rcu_preempt_ctrlblk. | |
510 | ||
511 | The output of "cat rcu/rcudata" is as follows: | |
512 | ||
513 | rcu_preempt: qlen=24 gp=1097669 g197/p197/c197 tasks=... | |
514 | ttb=. btg=no ntb=184 neb=0 nnb=183 j=01f7 bt=0274 | |
515 | normal balk: nt=1097669 gt=0 bt=371 b=0 ny=25073378 nos=0 | |
516 | exp balk: bt=0 nos=0 | |
517 | rcu_sched: qlen: 0 | |
518 | rcu_bh: qlen: 0 | |
519 | ||
520 | This is split into rcu_preempt, rcu_sched, and rcu_bh sections, with the | |
521 | rcu_preempt section appearing only in CONFIG_TINY_PREEMPT_RCU builds. | |
522 | The last three lines of the rcu_preempt section appear only in | |
523 | CONFIG_RCU_BOOST kernel builds. The fields are as follows: | |
524 | ||
525 | o "qlen" is the number of RCU callbacks currently waiting either | |
526 | for an RCU grace period or waiting to be invoked. This is the | |
527 | only field present for rcu_sched and rcu_bh, due to the | |
528 | short-circuiting of grace period in those two cases. | |
529 | ||
530 | o "gp" is the number of grace periods that have completed. | |
531 | ||
532 | o "g197/p197/c197" displays the grace-period state, with the | |
533 | "g" number being the number of grace periods that have started | |
534 | (mod 256), the "p" number being the number of grace periods | |
535 | that the CPU has responded to (also mod 256), and the "c" | |
536 | number being the number of grace periods that have completed | |
537 | (once again mode 256). | |
538 | ||
539 | Why have both "gp" and "g"? Because the data flowing into | |
540 | "gp" is only present in a CONFIG_RCU_TRACE kernel. | |
541 | ||
542 | o "tasks" is a set of bits. The first bit is "T" if there are | |
543 | currently tasks that have recently blocked within an RCU | |
544 | read-side critical section, the second bit is "N" if any of the | |
545 | aforementioned tasks are blocking the current RCU grace period, | |
546 | and the third bit is "E" if any of the aforementioned tasks are | |
547 | blocking the current expedited grace period. Each bit is "." | |
548 | if the corresponding condition does not hold. | |
549 | ||
550 | o "ttb" is a single bit. It is "B" if any of the blocked tasks | |
551 | need to be priority boosted and "." otherwise. | |
552 | ||
553 | o "btg" indicates whether boosting has been carried out during | |
554 | the current grace period, with "exp" indicating that boosting | |
555 | is in progress for an expedited grace period, "no" indicating | |
556 | that boosting has not yet started for a normal grace period, | |
557 | "begun" indicating that boosting has bebug for a normal grace | |
558 | period, and "done" indicating that boosting has completed for | |
559 | a normal grace period. | |
560 | ||
561 | o "ntb" is the total number of tasks subjected to RCU priority boosting | |
562 | periods since boot. | |
563 | ||
564 | o "neb" is the number of expedited grace periods that have had | |
565 | to resort to RCU priority boosting since boot. | |
566 | ||
567 | o "nnb" is the number of normal grace periods that have had | |
568 | to resort to RCU priority boosting since boot. | |
569 | ||
90e6ac36 | 570 | o "j" is the low-order 16 bits of the jiffies counter in hexadecimal. |
8e79e1f9 | 571 | |
90e6ac36 | 572 | o "bt" is the low-order 16 bits of the value that the jiffies counter |
8e79e1f9 PM |
573 | will have at the next time that boosting is scheduled to begin. |
574 | ||
575 | o In the line beginning with "normal balk", the fields are as follows: | |
576 | ||
577 | o "nt" is the number of times that the system balked from | |
578 | boosting because there were no blocked tasks to boost. | |
579 | Note that the system will balk from boosting even if the | |
580 | grace period is overdue when the currently running task | |
581 | is looping within an RCU read-side critical section. | |
582 | There is no point in boosting in this case, because | |
583 | boosting a running task won't make it run any faster. | |
584 | ||
585 | o "gt" is the number of times that the system balked | |
586 | from boosting because, although there were blocked tasks, | |
587 | none of them were preventing the current grace period | |
588 | from completing. | |
589 | ||
590 | o "bt" is the number of times that the system balked | |
591 | from boosting because boosting was already in progress. | |
592 | ||
593 | o "b" is the number of times that the system balked from | |
594 | boosting because boosting had already completed for | |
595 | the grace period in question. | |
596 | ||
597 | o "ny" is the number of times that the system balked from | |
598 | boosting because it was not yet time to start boosting | |
599 | the grace period in question. | |
600 | ||
601 | o "nos" is the number of times that the system balked from | |
602 | boosting for inexplicable ("not otherwise specified") | |
603 | reasons. This can actually happen due to races involving | |
604 | increments of the jiffies counter. | |
605 | ||
606 | o In the line beginning with "exp balk", the fields are as follows: | |
607 | ||
608 | o "bt" is the number of times that the system balked from | |
609 | boosting because there were no blocked tasks to boost. | |
610 | ||
611 | o "nos" is the number of times that the system balked from | |
612 | boosting for inexplicable ("not otherwise specified") | |
613 | reasons. |