Commit | Line | Data |
---|---|---|
77b44d1b MH |
1 | Kprobe-based Event Tracing |
2 | ========================== | |
d8ec9185 MH |
3 | |
4 | Documentation is written by Masami Hiramatsu | |
5 | ||
6 | ||
7 | Overview | |
8 | -------- | |
77b44d1b MH |
9 | These events are similar to tracepoint based events. Instead of Tracepoint, |
10 | this is based on kprobes (kprobe and kretprobe). So it can probe wherever | |
11 | kprobes can probe (this means, all functions body except for __kprobes | |
12 | functions). Unlike the Tracepoint based event, this can be added and removed | |
13 | dynamically, on the fly. | |
d8ec9185 | 14 | |
77b44d1b | 15 | To enable this feature, build your kernel with CONFIG_KPROBE_TRACING=y. |
d8ec9185 | 16 | |
77b44d1b MH |
17 | Similar to the events tracer, this doesn't need to be activated via |
18 | current_tracer. Instead of that, add probe points via | |
19 | /sys/kernel/debug/tracing/kprobe_events, and enable it via | |
20 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/enabled. | |
d8ec9185 MH |
21 | |
22 | ||
23 | Synopsis of kprobe_events | |
24 | ------------------------- | |
f52487e9 MH |
25 | p[:[GRP/]EVENT] SYMBOL[+offs]|MEMADDR [FETCHARGS] : Set a probe |
26 | r[:[GRP/]EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe | |
d8ec9185 | 27 | |
f52487e9 | 28 | GRP : Group name. If omitted, use "kprobes" for it. |
2fba0c88 MH |
29 | EVENT : Event name. If omitted, the event name is generated |
30 | based on SYMBOL+offs or MEMADDR. | |
31 | SYMBOL[+offs] : Symbol+offset where the probe is inserted. | |
32 | MEMADDR : Address where the probe is inserted. | |
d8ec9185 | 33 | |
2fba0c88 | 34 | FETCHARGS : Arguments. Each probe can have up to 128 args. |
2e06ff63 MH |
35 | %REG : Fetch register REG |
36 | @ADDR : Fetch memory at ADDR (ADDR should be in kernel) | |
d8ec9185 | 37 | @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol) |
2e06ff63 MH |
38 | $stackN : Fetch Nth entry of stack (N >= 0) |
39 | $stack : Fetch stack address. | |
40 | $argN : Fetch function argument. (N >= 0)(*) | |
41 | $retval : Fetch return value.(**) | |
eca0d916 MH |
42 | +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(***) |
43 | NAME=FETCHARG: Set NAME as the argument name of FETCHARG. | |
d8ec9185 MH |
44 | |
45 | (*) aN may not correct on asmlinkaged functions and at the middle of | |
46 | function body. | |
47 | (**) only for return probe. | |
48 | (***) this is useful for fetching a field of data structures. | |
49 | ||
50 | ||
51 | Per-Probe Event Filtering | |
52 | ------------------------- | |
53 | Per-probe event filtering feature allows you to set different filter on each | |
54 | probe and gives you what arguments will be shown in trace buffer. If an event | |
77b44d1b MH |
55 | name is specified right after 'p:' or 'r:' in kprobe_events, it adds an event |
56 | under tracing/events/kprobes/<EVENT>, at the directory you can see 'id', | |
57 | 'enabled', 'format' and 'filter'. | |
d8ec9185 MH |
58 | |
59 | enabled: | |
60 | You can enable/disable the probe by writing 1 or 0 on it. | |
61 | ||
62 | format: | |
eca0d916 | 63 | This shows the format of this probe event. |
d8ec9185 MH |
64 | |
65 | filter: | |
eca0d916 | 66 | You can write filtering rules of this event. |
d8ec9185 | 67 | |
e08d1c65 MH |
68 | id: |
69 | This shows the id of this probe event. | |
d8ec9185 | 70 | |
77b44d1b | 71 | |
cd7e7bd5 MH |
72 | Event Profiling |
73 | --------------- | |
74 | You can check the total number of probe hits and probe miss-hits via | |
75 | /sys/kernel/debug/tracing/kprobe_profile. | |
76 | The first column is event name, the second is the number of probe hits, | |
77 | the third is the number of probe miss-hits. | |
78 | ||
79 | ||
d8ec9185 MH |
80 | Usage examples |
81 | -------------- | |
82 | To add a probe as a new event, write a new definition to kprobe_events | |
83 | as below. | |
84 | ||
2e06ff63 | 85 | echo p:myprobe do_sys_open dfd=$arg0 filename=$arg1 flags=$arg2 mode=$arg3 > /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 MH |
86 | |
87 | This sets a kprobe on the top of do_sys_open() function with recording | |
eca0d916 MH |
88 | 1st to 4th arguments as "myprobe" event. As this example shows, users can |
89 | choose more familiar names for each arguments. | |
d8ec9185 | 90 | |
2e06ff63 | 91 | echo r:myretprobe do_sys_open $retval >> /sys/kernel/debug/tracing/kprobe_events |
d8ec9185 MH |
92 | |
93 | This sets a kretprobe on the return point of do_sys_open() function with | |
99329c44 | 94 | recording return value as "myretprobe" event. |
d8ec9185 MH |
95 | You can see the format of these events via |
96 | /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format. | |
97 | ||
98 | cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format | |
99 | name: myprobe | |
eca0d916 | 100 | ID: 75 |
d8ec9185 MH |
101 | format: |
102 | field:unsigned short common_type; offset:0; size:2; | |
103 | field:unsigned char common_flags; offset:2; size:1; | |
104 | field:unsigned char common_preempt_count; offset:3; size:1; | |
105 | field:int common_pid; offset:4; size:4; | |
106 | field:int common_tgid; offset:8; size:4; | |
107 | ||
108 | field: unsigned long ip; offset:16;tsize:8; | |
109 | field: int nargs; offset:24;tsize:4; | |
eca0d916 MH |
110 | field: unsigned long dfd; offset:32;tsize:8; |
111 | field: unsigned long filename; offset:40;tsize:8; | |
112 | field: unsigned long flags; offset:48;tsize:8; | |
113 | field: unsigned long mode; offset:56;tsize:8; | |
d8ec9185 | 114 | |
6e9f23d1 | 115 | print fmt: "(%lx) dfd=%lx filename=%lx flags=%lx mode=%lx", REC->ip, REC->dfd, REC->filename, REC->flags, REC->mode |
d8ec9185 | 116 | |
d8ec9185 | 117 | |
eca0d916 | 118 | You can see that the event has 4 arguments as in the expressions you specified. |
d8ec9185 MH |
119 | |
120 | echo > /sys/kernel/debug/tracing/kprobe_events | |
121 | ||
5a0d9050 MH |
122 | This clears all probe points. |
123 | ||
124 | Right after definition, each event is disabled by default. For tracing these | |
125 | events, you need to enable it. | |
126 | ||
127 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable | |
128 | echo 1 > /sys/kernel/debug/tracing/events/kprobes/myretprobe/enable | |
129 | ||
130 | And you can see the traced information via /sys/kernel/debug/tracing/trace. | |
d8ec9185 MH |
131 | |
132 | cat /sys/kernel/debug/tracing/trace | |
133 | # tracer: nop | |
134 | # | |
135 | # TASK-PID CPU# TIMESTAMP FUNCTION | |
136 | # | | | | | | |
6e9f23d1 | 137 | <...>-1447 [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0 |
2e06ff63 | 138 | <...>-1447 [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe |
6e9f23d1 | 139 | <...>-1447 [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6 |
2e06ff63 | 140 | <...>-1447 [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
6e9f23d1 | 141 | <...>-1447 [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10 |
2e06ff63 | 142 | <...>-1447 [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3 |
d8ec9185 MH |
143 | |
144 | ||
6e9f23d1 | 145 | Each line shows when the kernel hits an event, and <- SYMBOL means kernel |
d8ec9185 MH |
146 | returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel |
147 | returns from do_sys_open to sys_open+0x1b). | |
148 | ||
149 |