Commit | Line | Data |
---|---|---|
de1ba09b AM |
1 | Fault injection capabilities infrastructure |
2 | =========================================== | |
3 | ||
4 | See also drivers/md/faulty.c and "every_nth" module option for scsi_debug. | |
5 | ||
6 | ||
7 | Available fault injection capabilities | |
8 | -------------------------------------- | |
9 | ||
10 | o failslab | |
11 | ||
12 | injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) | |
13 | ||
14 | o fail_page_alloc | |
15 | ||
16 | injects page allocation failures. (alloc_pages(), get_free_pages(), ...) | |
17 | ||
18 | o fail_make_request | |
19 | ||
5d0ffa2b | 20 | injects disk IO errors on devices permitted by setting |
de1ba09b AM |
21 | /sys/block/<device>/make-it-fail or |
22 | /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) | |
23 | ||
24 | Configure fault-injection capabilities behavior | |
25 | ----------------------------------------------- | |
26 | ||
27 | o debugfs entries | |
28 | ||
29 | fault-inject-debugfs kernel module provides some debugfs entries for runtime | |
30 | configuration of fault-injection capabilities. | |
31 | ||
156f5a78 | 32 | - /sys/kernel/debug/fail*/probability: |
de1ba09b AM |
33 | |
34 | likelihood of failure injection, in percent. | |
35 | Format: <percent> | |
36 | ||
5d0ffa2b DM |
37 | Note that one-failure-per-hundred is a very high error rate |
38 | for some testcases. Consider setting probability=100 and configure | |
156f5a78 | 39 | /sys/kernel/debug/fail*/interval for such testcases. |
de1ba09b | 40 | |
156f5a78 | 41 | - /sys/kernel/debug/fail*/interval: |
de1ba09b AM |
42 | |
43 | specifies the interval between failures, for calls to | |
44 | should_fail() that pass all the other tests. | |
45 | ||
46 | Note that if you enable this, by setting interval>1, you will | |
47 | probably want to set probability=100. | |
48 | ||
156f5a78 | 49 | - /sys/kernel/debug/fail*/times: |
de1ba09b AM |
50 | |
51 | specifies how many times failures may happen at most. | |
52 | A value of -1 means "no limit". | |
53 | ||
156f5a78 | 54 | - /sys/kernel/debug/fail*/space: |
de1ba09b AM |
55 | |
56 | specifies an initial resource "budget", decremented by "size" | |
57 | on each call to should_fail(,size). Failure injection is | |
58 | suppressed until "space" reaches zero. | |
59 | ||
156f5a78 | 60 | - /sys/kernel/debug/fail*/verbose |
de1ba09b AM |
61 | |
62 | Format: { 0 | 1 | 2 } | |
5d0ffa2b DM |
63 | specifies the verbosity of the messages when failure is |
64 | injected. '0' means no messages; '1' will print only a single | |
65 | log line per failure; '2' will print a call trace too -- useful | |
66 | to debug the problems revealed by fault injection. | |
de1ba09b | 67 | |
156f5a78 | 68 | - /sys/kernel/debug/fail*/task-filter: |
de1ba09b | 69 | |
5d0ffa2b DM |
70 | Format: { 'Y' | 'N' } |
71 | A value of 'N' disables filtering by process (default). | |
de1ba09b AM |
72 | Any positive value limits failures to only processes indicated by |
73 | /proc/<pid>/make-it-fail==1. | |
74 | ||
156f5a78 GL |
75 | - /sys/kernel/debug/fail*/require-start: |
76 | - /sys/kernel/debug/fail*/require-end: | |
77 | - /sys/kernel/debug/fail*/reject-start: | |
78 | - /sys/kernel/debug/fail*/reject-end: | |
de1ba09b AM |
79 | |
80 | specifies the range of virtual addresses tested during | |
81 | stacktrace walking. Failure is injected only if some caller | |
329409ae AM |
82 | in the walked stacktrace lies within the required range, and |
83 | none lies within the rejected range. | |
84 | Default required range is [0,ULONG_MAX) (whole of virtual address space). | |
85 | Default rejected range is [0,0). | |
de1ba09b | 86 | |
156f5a78 | 87 | - /sys/kernel/debug/fail*/stacktrace-depth: |
de1ba09b AM |
88 | |
89 | specifies the maximum stacktrace depth walked during search | |
5d0ffa2b DM |
90 | for a caller within [require-start,require-end) OR |
91 | [reject-start,reject-end). | |
de1ba09b | 92 | |
156f5a78 | 93 | - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: |
de1ba09b | 94 | |
5d0ffa2b DM |
95 | Format: { 'Y' | 'N' } |
96 | default is 'N', setting it to 'Y' won't inject failures into | |
de1ba09b AM |
97 | highmem/user allocations. |
98 | ||
156f5a78 GL |
99 | - /sys/kernel/debug/failslab/ignore-gfp-wait: |
100 | - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: | |
de1ba09b | 101 | |
5d0ffa2b DM |
102 | Format: { 'Y' | 'N' } |
103 | default is 'N', setting it to 'Y' will inject failures | |
de1ba09b AM |
104 | only into non-sleep allocations (GFP_ATOMIC allocations). |
105 | ||
156f5a78 | 106 | - /sys/kernel/debug/fail_page_alloc/min-order: |
54114994 AM |
107 | |
108 | specifies the minimum page allocation order to be injected | |
109 | failures. | |
110 | ||
de1ba09b AM |
111 | o Boot option |
112 | ||
113 | In order to inject faults while debugfs is not available (early boot time), | |
114 | use the boot option: | |
115 | ||
116 | failslab= | |
117 | fail_page_alloc= | |
118 | fail_make_request=<interval>,<probability>,<space>,<times> | |
119 | ||
120 | How to add new fault injection capability | |
121 | ----------------------------------------- | |
122 | ||
123 | o #include <linux/fault-inject.h> | |
124 | ||
125 | o define the fault attributes | |
126 | ||
127 | DECLARE_FAULT_INJECTION(name); | |
128 | ||
129 | Please see the definition of struct fault_attr in fault-inject.h | |
130 | for details. | |
131 | ||
5d0ffa2b | 132 | o provide a way to configure fault attributes |
de1ba09b AM |
133 | |
134 | - boot option | |
135 | ||
136 | If you need to enable the fault injection capability from boot time, you can | |
5d0ffa2b | 137 | provide boot option to configure it. There is a helper function for it: |
de1ba09b | 138 | |
5d0ffa2b | 139 | setup_fault_attr(attr, str); |
de1ba09b AM |
140 | |
141 | - debugfs entries | |
142 | ||
143 | failslab, fail_page_alloc, and fail_make_request use this way. | |
5d0ffa2b | 144 | Helper functions: |
de1ba09b | 145 | |
dd48c085 | 146 | fault_create_debugfs_attr(name, parent, attr); |
de1ba09b AM |
147 | |
148 | - module parameters | |
149 | ||
150 | If the scope of the fault injection capability is limited to a | |
151 | single kernel module, it is better to provide module parameters to | |
152 | configure the fault attributes. | |
153 | ||
154 | o add a hook to insert failures | |
155 | ||
5d0ffa2b | 156 | Upon should_fail() returning true, client code should inject a failure. |
de1ba09b | 157 | |
5d0ffa2b | 158 | should_fail(attr, size); |
de1ba09b AM |
159 | |
160 | Application Examples | |
161 | -------------------- | |
162 | ||
18584870 | 163 | o Inject slab allocation failures into module init/exit code |
de1ba09b | 164 | |
de1ba09b AM |
165 | #!/bin/bash |
166 | ||
18584870 | 167 | FAILTYPE=failslab |
156f5a78 GL |
168 | echo Y > /sys/kernel/debug/$FAILTYPE/task-filter |
169 | echo 10 > /sys/kernel/debug/$FAILTYPE/probability | |
170 | echo 100 > /sys/kernel/debug/$FAILTYPE/interval | |
171 | echo -1 > /sys/kernel/debug/$FAILTYPE/times | |
172 | echo 0 > /sys/kernel/debug/$FAILTYPE/space | |
173 | echo 2 > /sys/kernel/debug/$FAILTYPE/verbose | |
174 | echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait | |
de1ba09b | 175 | |
18584870 | 176 | faulty_system() |
de1ba09b | 177 | { |
18584870 | 178 | bash -c "echo 1 > /proc/self/make-it-fail && exec $*" |
de1ba09b AM |
179 | } |
180 | ||
18584870 AM |
181 | if [ $# -eq 0 ] |
182 | then | |
183 | echo "Usage: $0 modulename [ modulename ... ]" | |
184 | exit 1 | |
185 | fi | |
186 | ||
187 | for m in $* | |
188 | do | |
189 | echo inserting $m... | |
190 | faulty_system modprobe $m | |
de1ba09b | 191 | |
18584870 AM |
192 | echo removing $m... |
193 | faulty_system modprobe -r $m | |
194 | done | |
de1ba09b AM |
195 | |
196 | ------------------------------------------------------------------------------ | |
197 | ||
18584870 | 198 | o Inject page allocation failures only for a specific module |
de1ba09b | 199 | |
de1ba09b AM |
200 | #!/bin/bash |
201 | ||
18584870 AM |
202 | FAILTYPE=fail_page_alloc |
203 | module=$1 | |
de1ba09b | 204 | |
18584870 AM |
205 | if [ -z $module ] |
206 | then | |
207 | echo "Usage: $0 <modulename>" | |
208 | exit 1 | |
209 | fi | |
de1ba09b | 210 | |
18584870 | 211 | modprobe $module |
de1ba09b | 212 | |
18584870 AM |
213 | if [ ! -d /sys/module/$module/sections ] |
214 | then | |
215 | echo Module $module is not loaded | |
216 | exit 1 | |
217 | fi | |
218 | ||
156f5a78 GL |
219 | cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start |
220 | cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end | |
18584870 | 221 | |
156f5a78 GL |
222 | echo N > /sys/kernel/debug/$FAILTYPE/task-filter |
223 | echo 10 > /sys/kernel/debug/$FAILTYPE/probability | |
224 | echo 100 > /sys/kernel/debug/$FAILTYPE/interval | |
225 | echo -1 > /sys/kernel/debug/$FAILTYPE/times | |
226 | echo 0 > /sys/kernel/debug/$FAILTYPE/space | |
227 | echo 2 > /sys/kernel/debug/$FAILTYPE/verbose | |
228 | echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait | |
229 | echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem | |
230 | echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth | |
18584870 | 231 | |
156f5a78 | 232 | trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT |
18584870 AM |
233 | |
234 | echo "Injecting errors into the module $module... (interrupt to stop)" | |
235 | sleep 1000000 | |
de1ba09b | 236 |