Commit | Line | Data |
---|---|---|
dc851a0f DW |
1 | ================================================================ |
2 | Documentation for Kdump - The kexec-based Crash Dumping Solution | |
b089f4a6 VG |
3 | ================================================================ |
4 | ||
dc851a0f DW |
5 | This document includes overview, setup and installation, and analysis |
6 | information. | |
b089f4a6 | 7 | |
dc851a0f DW |
8 | Overview |
9 | ======== | |
b089f4a6 | 10 | |
dc851a0f DW |
11 | Kdump uses kexec to quickly boot to a dump-capture kernel whenever a |
12 | dump of the system kernel's memory needs to be taken (for example, when | |
13 | the system panics). The system kernel's memory image is preserved across | |
14 | the reboot and is accessible to the dump-capture kernel. | |
b089f4a6 | 15 | |
f4e87570 | 16 | You can use common commands, such as cp and scp, to copy the |
dc851a0f DW |
17 | memory image to a dump file on the local disk, or across the network to |
18 | a remote system. | |
b089f4a6 | 19 | |
91302143 | 20 | Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, |
16b0371a | 21 | s390x and arm architectures. |
b089f4a6 | 22 | |
dc851a0f DW |
23 | When the system kernel boots, it reserves a small section of memory for |
24 | the dump-capture kernel. This ensures that ongoing Direct Memory Access | |
25 | (DMA) from the system kernel does not corrupt the dump-capture kernel. | |
26 | The kexec -p command loads the dump-capture kernel into this reserved | |
27 | memory. | |
b089f4a6 | 28 | |
dc851a0f DW |
29 | On x86 machines, the first 640 KB of physical memory is needed to boot, |
30 | regardless of where the kernel loads. Therefore, kexec backs up this | |
31 | region just before rebooting into the dump-capture kernel. | |
b089f4a6 | 32 | |
30430134 SH |
33 | Similarly on PPC64 machines first 32KB of physical memory is needed for |
34 | booting regardless of where the kernel is loaded and to support 64K page | |
35 | size kexec backs up the first 64KB memory. | |
36 | ||
91302143 MH |
37 | For s390x, when kdump is triggered, the crashkernel region is exchanged |
38 | with the region [0, crashkernel region size] and then the kdump kernel | |
39 | runs in [0, crashkernel region size]. Therefore no relocatable kernel is | |
40 | needed for s390x. | |
41 | ||
dc851a0f DW |
42 | All of the necessary information about the system kernel's core image is |
43 | encoded in the ELF format, and stored in a reserved area of memory | |
44 | before a crash. The physical address of the start of the ELF header is | |
45 | passed to the dump-capture kernel through the elfcorehdr= boot | |
91302143 MH |
46 | parameter. Optionally the size of the ELF header can also be passed |
47 | when using the elfcorehdr=[size[KMG]@]offset[KMG] syntax. | |
48 | ||
dc851a0f | 49 | |
987bf6fe ZY |
50 | With the dump-capture kernel, you can access the memory image through |
51 | /proc/vmcore. This exports the dump as an ELF-format file that you can | |
52 | write out using file copy commands such as cp or scp. Further, you can | |
53 | use analysis tools such as the GNU Debugger (GDB) and the Crash tool to | |
54 | debug the dump file. This method ensures that the dump pages are correctly | |
55 | ordered. | |
dc851a0f DW |
56 | |
57 | ||
58 | Setup and Installation | |
59 | ====================== | |
60 | ||
9c61a446 VG |
61 | Install kexec-tools |
62 | ------------------- | |
dc851a0f DW |
63 | |
64 | 1) Login as the root user. | |
65 | ||
66 | 2) Download the kexec-tools user-space package from the following URL: | |
67 | ||
db6857c6 | 68 | http://kernel.org/pub/linux/utils/kernel/kexec/kexec-tools.tar.gz |
ea112bd5 | 69 | |
d84a52f6 | 70 | This is a symlink to the latest version. |
dc851a0f | 71 | |
d84a52f6 | 72 | The latest kexec-tools git tree is available at: |
dc851a0f | 73 | |
db6857c6 SH |
74 | git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git |
75 | and | |
76 | http://www.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git | |
77 | ||
78 | There is also a gitweb interface available at | |
79 | http://www.kernel.org/git/?p=utils/kernel/kexec/kexec-tools.git | |
d84a52f6 SH |
80 | |
81 | More information about kexec-tools can be found at | |
b1bdd2eb | 82 | http://horms.net/projects/kexec/ |
dc851a0f | 83 | |
9c61a446 | 84 | 3) Unpack the tarball with the tar command, as follows: |
dc851a0f | 85 | |
d84a52f6 | 86 | tar xvpzf kexec-tools.tar.gz |
dc851a0f | 87 | |
ea112bd5 | 88 | 4) Change to the kexec-tools directory, as follows: |
dc851a0f | 89 | |
d84a52f6 | 90 | cd kexec-tools-VERSION |
dc851a0f | 91 | |
9c61a446 | 92 | 5) Configure the package, as follows: |
dc851a0f DW |
93 | |
94 | ./configure | |
95 | ||
9c61a446 | 96 | 6) Compile the package, as follows: |
dc851a0f DW |
97 | |
98 | make | |
99 | ||
9c61a446 | 100 | 7) Install the package, as follows: |
dc851a0f DW |
101 | |
102 | make install | |
103 | ||
104 | ||
9c61a446 VG |
105 | Build the system and dump-capture kernels |
106 | ----------------------------------------- | |
107 | There are two possible methods of using Kdump. | |
108 | ||
109 | 1) Build a separate custom dump-capture kernel for capturing the | |
110 | kernel core dump. | |
111 | ||
112 | 2) Or use the system kernel binary itself as dump-capture kernel and there is | |
113 | no need to build a separate dump-capture kernel. This is possible | |
19f59460 | 114 | only with the architectures which support a relocatable kernel. As |
16b0371a | 115 | of today, i386, x86_64, ppc64, ia64 and arm architectures support relocatable |
54622f10 | 116 | kernel. |
9c61a446 VG |
117 | |
118 | Building a relocatable kernel is advantageous from the point of view that | |
119 | one does not have to build a second kernel for capturing the dump. But | |
120 | at the same time one might want to build a custom dump capture kernel | |
121 | suitable to his needs. | |
dc851a0f | 122 | |
9c61a446 VG |
123 | Following are the configuration setting required for system and |
124 | dump-capture kernels for enabling kdump support. | |
dc851a0f | 125 | |
9c61a446 VG |
126 | System kernel config options |
127 | ---------------------------- | |
dc851a0f DW |
128 | |
129 | 1) Enable "kexec system call" in "Processor type and features." | |
130 | ||
131 | CONFIG_KEXEC=y | |
132 | ||
133 | 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo | |
134 | filesystems." This is usually enabled by default. | |
135 | ||
136 | CONFIG_SYSFS=y | |
137 | ||
138 | Note that "sysfs file system support" might not appear in the "Pseudo | |
139 | filesystems" menu if "Configure standard kernel features (for small | |
140 | systems)" is not enabled in "General Setup." In this case, check the | |
141 | .config file itself to ensure that sysfs is turned on, as follows: | |
142 | ||
143 | grep 'CONFIG_SYSFS' .config | |
144 | ||
145 | 3) Enable "Compile the kernel with debug info" in "Kernel hacking." | |
146 | ||
147 | CONFIG_DEBUG_INFO=Y | |
148 | ||
149 | This causes the kernel to be built with debug symbols. The dump | |
150 | analysis tools require a vmlinux with debug symbols in order to read | |
151 | and analyze a dump file. | |
152 | ||
9c61a446 VG |
153 | Dump-capture kernel config options (Arch Independent) |
154 | ----------------------------------------------------- | |
dc851a0f | 155 | |
9c61a446 VG |
156 | 1) Enable "kernel crash dumps" support under "Processor type and |
157 | features": | |
dc851a0f | 158 | |
9c61a446 | 159 | CONFIG_CRASH_DUMP=y |
dc851a0f | 160 | |
9c61a446 | 161 | 2) Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems". |
b089f4a6 | 162 | |
9c61a446 VG |
163 | CONFIG_PROC_VMCORE=y |
164 | (CONFIG_PROC_VMCORE is set by default when CONFIG_CRASH_DUMP is selected.) | |
dc851a0f | 165 | |
8bc9d422 BW |
166 | Dump-capture kernel config options (Arch Dependent, i386 and x86_64) |
167 | -------------------------------------------------------------------- | |
168 | ||
169 | 1) On i386, enable high memory support under "Processor type and | |
dc851a0f DW |
170 | features": |
171 | ||
172 | CONFIG_HIGHMEM64G=y | |
173 | or | |
174 | CONFIG_HIGHMEM4G | |
175 | ||
8bc9d422 | 176 | 2) On i386 and x86_64, disable symmetric multi-processing support |
dc851a0f DW |
177 | under "Processor type and features": |
178 | ||
179 | CONFIG_SMP=n | |
9c61a446 | 180 | |
dc851a0f DW |
181 | (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line |
182 | when loading the dump-capture kernel, see section "Load the Dump-capture | |
183 | Kernel".) | |
184 | ||
9c61a446 VG |
185 | 3) If one wants to build and use a relocatable kernel, |
186 | Enable "Build a relocatable kernel" support under "Processor type and | |
187 | features" | |
dc851a0f | 188 | |
9c61a446 | 189 | CONFIG_RELOCATABLE=y |
dc851a0f | 190 | |
9c61a446 VG |
191 | 4) Use a suitable value for "Physical address where the kernel is |
192 | loaded" (under "Processor type and features"). This only appears when | |
193 | "kernel crash dumps" is enabled. A suitable value depends upon | |
194 | whether kernel is relocatable or not. | |
195 | ||
196 | If you are using a relocatable kernel use CONFIG_PHYSICAL_START=0x100000 | |
197 | This will compile the kernel for physical address 1MB, but given the fact | |
198 | kernel is relocatable, it can be run from any physical address hence | |
199 | kexec boot loader will load it in memory region reserved for dump-capture | |
200 | kernel. | |
201 | ||
202 | Otherwise it should be the start of memory region reserved for | |
203 | second kernel using boot parameter "crashkernel=Y@X". Here X is | |
204 | start of memory region reserved for dump-capture kernel. | |
205 | Generally X is 16MB (0x1000000). So you can set | |
206 | CONFIG_PHYSICAL_START=0x1000000 | |
207 | ||
208 | 5) Make and install the kernel and its modules. DO NOT add this kernel | |
209 | to the boot loader configuration files. | |
dc851a0f | 210 | |
9c61a446 VG |
211 | Dump-capture kernel config options (Arch Dependent, ppc64) |
212 | ---------------------------------------------------------- | |
dc851a0f | 213 | |
54622f10 MK |
214 | 1) Enable "Build a kdump crash kernel" support under "Kernel" options: |
215 | ||
216 | CONFIG_CRASH_DUMP=y | |
217 | ||
218 | 2) Enable "Build a relocatable kernel" support | |
219 | ||
220 | CONFIG_RELOCATABLE=y | |
221 | ||
222 | Make and install the kernel and its modules. | |
dc851a0f | 223 | |
9c61a446 VG |
224 | Dump-capture kernel config options (Arch Dependent, ia64) |
225 | ---------------------------------------------------------- | |
ee8bb9ea H |
226 | |
227 | - No specific options are required to create a dump-capture kernel | |
19f59460 | 228 | for ia64, other than those specified in the arch independent section |
ee8bb9ea H |
229 | above. This means that it is possible to use the system kernel |
230 | as a dump-capture kernel if desired. | |
231 | ||
232 | The crashkernel region can be automatically placed by the system | |
233 | kernel at run time. This is done by specifying the base address as 0, | |
234 | or omitting it all together. | |
235 | ||
236 | crashkernel=256M@0 | |
237 | or | |
238 | crashkernel=256M | |
239 | ||
240 | If the start address is specified, note that the start address of the | |
241 | kernel will be aligned to 64Mb, so if the start address is not then | |
242 | any space below the alignment point will be wasted. | |
9c61a446 | 243 | |
16b0371a H |
244 | Dump-capture kernel config options (Arch Dependent, arm) |
245 | ---------------------------------------------------------- | |
246 | ||
247 | - To use a relocatable kernel, | |
248 | Enable "AUTO_ZRELADDR" support under "Boot" options: | |
249 | ||
250 | AUTO_ZRELADDR=y | |
9c61a446 | 251 | |
fb391599 BW |
252 | Extended crashkernel syntax |
253 | =========================== | |
254 | ||
255 | While the "crashkernel=size[@offset]" syntax is sufficient for most | |
256 | configurations, sometimes it's handy to have the reserved memory dependent | |
257 | on the value of System RAM -- that's mostly for distributors that pre-setup | |
258 | the kernel command line to avoid a unbootable system after some memory has | |
259 | been removed from the machine. | |
260 | ||
261 | The syntax is: | |
262 | ||
263 | crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset] | |
264 | range=start-[end] | |
265 | ||
16b0371a H |
266 | Please note, on arm, the offset is required. |
267 | crashkernel=<range1>:<size1>[,<range2>:<size2>,...]@offset | |
268 | range=start-[end] | |
269 | ||
be089d79 ME |
270 | 'start' is inclusive and 'end' is exclusive. |
271 | ||
fb391599 BW |
272 | For example: |
273 | ||
274 | crashkernel=512M-2G:64M,2G-:128M | |
275 | ||
276 | This would mean: | |
277 | ||
278 | 1) if the RAM is smaller than 512M, then don't reserve anything | |
279 | (this is the "rescue" case) | |
be089d79 | 280 | 2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M |
fb391599 BW |
281 | 3) if the RAM size is larger than 2G, then reserve 128M |
282 | ||
283 | ||
be089d79 | 284 | |
9c61a446 VG |
285 | Boot into System Kernel |
286 | ======================= | |
287 | ||
30430134 SH |
288 | 1) Update the boot loader (such as grub, yaboot, or lilo) configuration |
289 | files as necessary. | |
9c61a446 VG |
290 | |
291 | 2) Boot the system kernel with the boot parameter "crashkernel=Y@X", | |
292 | where Y specifies how much memory to reserve for the dump-capture kernel | |
293 | and X specifies the beginning of this reserved memory. For example, | |
294 | "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory | |
295 | starting at physical address 0x01000000 (16MB) for the dump-capture kernel. | |
296 | ||
297 | On x86 and x86_64, use "crashkernel=64M@16M". | |
298 | ||
299 | On ppc64, use "crashkernel=128M@32M". | |
dc851a0f | 300 | |
ee8bb9ea H |
301 | On ia64, 256M@256M is a generous value that typically works. |
302 | The region may be automatically placed on ia64, see the | |
303 | dump-capture kernel config option notes above. | |
797f6a68 | 304 | If use sparse memory, the size should be rounded to GRANULE boundaries. |
ee8bb9ea | 305 | |
91302143 MH |
306 | On s390x, typically use "crashkernel=xxM". The value of xx is dependent |
307 | on the memory consumption of the kdump system. In general this is not | |
308 | dependent on the memory size of the production system. | |
309 | ||
16b0371a H |
310 | On arm, use "crashkernel=Y@X". Note that the start address of the kernel |
311 | will be aligned to 128MiB (0x08000000), so if the start address is not then | |
312 | any space below the alignment point may be overwritten by the dump-capture kernel, | |
313 | which means it is possible that the vmcore is not that precise as expected. | |
314 | ||
315 | ||
dc851a0f DW |
316 | Load the Dump-capture Kernel |
317 | ============================ | |
318 | ||
9c61a446 VG |
319 | After booting to the system kernel, dump-capture kernel needs to be |
320 | loaded. | |
321 | ||
322 | Based on the architecture and type of image (relocatable or not), one | |
323 | can choose to load the uncompressed vmlinux or compressed bzImage/vmlinuz | |
324 | of dump-capture kernel. Following is the summary. | |
325 | ||
8bc9d422 | 326 | For i386 and x86_64: |
9c61a446 VG |
327 | - Use vmlinux if kernel is not relocatable. |
328 | - Use bzImage/vmlinuz if kernel is relocatable. | |
9c61a446 VG |
329 | For ppc64: |
330 | - Use vmlinux | |
331 | For ia64: | |
ee8bb9ea | 332 | - Use vmlinux or vmlinuz.gz |
91302143 MH |
333 | For s390x: |
334 | - Use image or bzImage | |
16b0371a H |
335 | For arm: |
336 | - Use zImage | |
9c61a446 VG |
337 | |
338 | If you are using a uncompressed vmlinux image then use following command | |
339 | to load dump-capture kernel. | |
dc851a0f | 340 | |
9c61a446 | 341 | kexec -p <dump-capture-kernel-vmlinux-image> \ |
dc851a0f | 342 | --initrd=<initrd-for-dump-capture-kernel> --args-linux \ |
9c61a446 | 343 | --append="root=<root-dev> <arch-specific-options>" |
dc851a0f | 344 | |
9c61a446 VG |
345 | If you are using a compressed bzImage/vmlinuz, then use following command |
346 | to load dump-capture kernel. | |
dc851a0f | 347 | |
9c61a446 VG |
348 | kexec -p <dump-capture-kernel-bzImage> \ |
349 | --initrd=<initrd-for-dump-capture-kernel> \ | |
350 | --append="root=<root-dev> <arch-specific-options>" | |
351 | ||
16b0371a H |
352 | If you are using a compressed zImage, then use following command |
353 | to load dump-capture kernel. | |
354 | ||
355 | kexec --type zImage -p <dump-capture-kernel-bzImage> \ | |
356 | --initrd=<initrd-for-dump-capture-kernel> \ | |
357 | --dtb=<dtb-for-dump-capture-kernel> \ | |
358 | --append="root=<root-dev> <arch-specific-options>" | |
359 | ||
360 | ||
ee8bb9ea H |
361 | Please note, that --args-linux does not need to be specified for ia64. |
362 | It is planned to make this a no-op on that architecture, but for now | |
363 | it should be omitted | |
364 | ||
9c61a446 VG |
365 | Following are the arch specific command line options to be used while |
366 | loading dump-capture kernel. | |
367 | ||
ee8bb9ea | 368 | For i386, x86_64 and ia64: |
ac984abe | 369 | "1 irqpoll maxcpus=1 reset_devices" |
9c61a446 VG |
370 | |
371 | For ppc64: | |
ac984abe | 372 | "1 maxcpus=1 noirqdistrib reset_devices" |
dc851a0f | 373 | |
91302143 MH |
374 | For s390x: |
375 | "1 maxcpus=1 cgroup_disable=memory" | |
9c61a446 | 376 | |
16b0371a H |
377 | For arm: |
378 | "1 maxcpus=1 reset_devices" | |
379 | ||
9c61a446 | 380 | Notes on loading the dump-capture kernel: |
dc851a0f DW |
381 | |
382 | * By default, the ELF headers are stored in ELF64 format to support | |
4fd45090 BW |
383 | systems with more than 4GB memory. On i386, kexec automatically checks if |
384 | the physical RAM size exceeds the 4 GB limit and if not, uses ELF32. | |
385 | So, on non-PAE systems, ELF32 is always used. | |
386 | ||
387 | The --elf32-core-headers option can be used to force the generation of ELF32 | |
388 | headers. This is necessary because GDB currently cannot open vmcore files | |
389 | with ELF64 headers on 32-bit systems. | |
dc851a0f DW |
390 | |
391 | * The "irqpoll" boot parameter reduces driver initialization failures | |
392 | due to shared interrupts in the dump-capture kernel. | |
393 | ||
394 | * You must specify <root-dev> in the format corresponding to the root | |
395 | device name in the output of mount command. | |
396 | ||
473e66fd H |
397 | * Boot parameter "1" boots the dump-capture kernel into single-user |
398 | mode without networking. If you want networking, use "3". | |
dc851a0f | 399 | |
9c61a446 VG |
400 | * We generally don' have to bring up a SMP kernel just to capture the |
401 | dump. Hence generally it is useful either to build a UP dump-capture | |
402 | kernel or specify maxcpus=1 option while loading dump-capture kernel. | |
dc851a0f | 403 | |
91302143 MH |
404 | * For s390x there are two kdump modes: If a ELF header is specified with |
405 | the elfcorehdr= kernel parameter, it is used by the kdump kernel as it | |
406 | is done on all other architectures. If no elfcorehdr= kernel parameter is | |
407 | specified, the s390x kdump kernel dynamically creates the header. The | |
408 | second mode has the advantage that for CPU and memory hotplug, kdump has | |
409 | not to be reloaded with kexec_load(). | |
410 | ||
411 | * For s390x systems with many attached devices the "cio_ignore" kernel | |
412 | parameter should be used for the kdump kernel in order to prevent allocation | |
413 | of kernel memory for devices that are not relevant for kdump. The same | |
414 | applies to systems that use SCSI/FCP devices. In that case the | |
415 | "allow_lun_scan" zfcp module parameter should be set to zero before | |
416 | setting FCP devices online. | |
417 | ||
dc851a0f DW |
418 | Kernel Panic |
419 | ============ | |
420 | ||
421 | After successfully loading the dump-capture kernel as previously | |
422 | described, the system will reboot into the dump-capture kernel if a | |
423 | system crash is triggered. Trigger points are located in panic(), | |
424 | die(), die_nmi() and in the sysrq handler (ALT-SysRq-c). | |
425 | ||
426 | The following conditions will execute a crash trigger point: | |
427 | ||
428 | If a hard lockup is detected and "NMI watchdog" is configured, the system | |
429 | will boot into the dump-capture kernel ( die_nmi() ). | |
430 | ||
431 | If die() is called, and it happens to be a thread with pid 0 or 1, or die() | |
432 | is called inside interrupt context or die() is called and panic_on_oops is set, | |
433 | the system will boot into the dump-capture kernel. | |
434 | ||
f4e87570 | 435 | On powerpc systems when a soft-reset is generated, die() is called by all cpus |
30430134 | 436 | and the system will boot into the dump-capture kernel. |
dc851a0f DW |
437 | |
438 | For testing purposes, you can trigger a crash by using "ALT-SysRq-c", | |
30430134 | 439 | "echo c > /proc/sysrq-trigger" or write a module to force the panic. |
dc851a0f DW |
440 | |
441 | Write Out the Dump File | |
442 | ======================= | |
443 | ||
444 | After the dump-capture kernel is booted, write out the dump file with | |
445 | the following command: | |
b089f4a6 VG |
446 | |
447 | cp /proc/vmcore <dump-file> | |
448 | ||
dc851a0f DW |
449 | |
450 | Analysis | |
b089f4a6 | 451 | ======== |
b089f4a6 | 452 | |
dc851a0f DW |
453 | Before analyzing the dump image, you should reboot into a stable kernel. |
454 | ||
455 | You can do limited analysis using GDB on the dump file copied out of | |
456 | /proc/vmcore. Use the debug vmlinux built with -g and run the following | |
457 | command: | |
458 | ||
459 | gdb vmlinux <dump-file> | |
b089f4a6 | 460 | |
dc851a0f DW |
461 | Stack trace for the task on processor 0, register display, and memory |
462 | display work fine. | |
b089f4a6 | 463 | |
dc851a0f DW |
464 | Note: GDB cannot analyze core files generated in ELF64 format for x86. |
465 | On systems with a maximum of 4GB of memory, you can generate | |
466 | ELF32-format headers using the --elf32-core-headers kernel option on the | |
467 | dump kernel. | |
b089f4a6 | 468 | |
dc851a0f DW |
469 | You can also use the Crash utility to analyze dump files in Kdump |
470 | format. Crash is available on Dave Anderson's site at the following URL: | |
a7e670d8 | 471 | |
dc851a0f DW |
472 | http://people.redhat.com/~anderson/ |
473 | ||
9e3961a0 PB |
474 | Trigger Kdump on WARN() |
475 | ======================= | |
476 | ||
477 | The kernel parameter, panic_on_warn, calls panic() in all WARN() paths. This | |
478 | will cause a kdump to occur at the panic() call. In cases where a user wants | |
479 | to specify this during runtime, /proc/sys/kernel/panic_on_warn can be set to 1 | |
480 | to achieve the same behaviour. | |
dc851a0f | 481 | |
dc851a0f | 482 | Contact |
b089f4a6 | 483 | ======= |
dc851a0f | 484 | |
db6857c6 | 485 | Vivek Goyal (vgoyal@redhat.com) |
d58831e4 | 486 | Maneesh Soni (maneesh@in.ibm.com) |
dc851a0f | 487 |