4 **barectf** is a command-line utility which generates pure C99
5 code that is able to write native
6 [CTF](http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.txt;hb=master)
7 (the Common Trace Format) out of a pre-written CTF metadata file.
9 You will find barectf interesting if:
11 1. You need to trace a program.
12 2. You need tracing to be as fast as possible, but also very flexible:
13 record integers of custom sizes, custom floating point numbers,
14 enumerations mapped to a specific integer type, structure fields,
15 NULL-terminated strings, static and dynamic arrays, etc.
16 3. You need to be able to convert the recorded binary events to
17 human-readable text, as well as analyze them with Python scripts
18 ([Babeltrace](http://www.efficios.com/babeltrace) does all that,
20 4. You _cannot_ use [LTTng](http://lttng.org/), an efficient tracing
21 framework for the Linux kernel and Linux/BSD user applications, which
24 The target audience of barectf is developers who need to trace bare metal
25 systems (without an operating system). The code produced by barectf
26 is pure C99 and is lightweight enough to fit on a tiny microcontroller.
27 Each event described in the CTF metadata input becomes one C function with
28 one parameter mapped to one event field. CTF data is recorded in a buffer of
29 any size provided by the user. This buffer corresponds to one CTF packet.
30 The generated tracing functions report when the buffer is full. The user
31 is entirely responsible for the buffering scheme: leave the buffer in memory,
32 save it to some permanent storage, swap it with another empty buffer and
33 concatenate recorded packets, etc.
35 barectf is written in Python 3 and currently uses
36 [pytsdl](https://github.com/efficios/pytsdl) to parse the CTF metadata file
43 Make sure you have `pip` for Python 3. On the latest Ubuntu releases,
46 sudo apt-get install python3-pip
48 On Ubuntu 12.04, you need to install `setuptools` first, then use
49 `easy_install3` to install `pip3`:
51 sudo apt-get install python3-setuptools
52 sudo easy_install3 pip
56 sudo pip3 install barectf
62 Using barectf involves:
64 1. Writing the CTF metadata file describing the various headers,
65 contexts and event fields.
66 2. Running the `barectf` command to generate C99 files out of
67 the CTF metadata file.
68 3. Using the generated C code in your specific application.
70 The following subsections explain the three steps above.
73 ### writing the CTF metadata
75 The **Common Trace Format** is a specialized file format for recording
76 trace data. CTF is designed to be very fast to write and very flexible.
77 All headers, contexts and event fields written in binary files are
78 described using a custom C-like, declarative language, TSDL (Trace
79 Stream Description Language). The file containing this description is
80 called the **CTF metadata**. The latter may be automatically generated
81 by a tracer, like it is the case of LTTng, or written by hand. This
82 metadata file is then used by CTF trace readers to know the layout of
83 CTF binary files containing actual event contexts and fields.
85 The CTF metadata file contains several blocks describing various CTF
86 binary layouts. A CTF trace file is a concatenation of several CTF
87 packets. Here's the anatomy of a CTF packet:
89 ![CTF packet anatomy](doc/ctf-packet.png)
91 A CTF packet belongs to a specific CTF stream. While the packet header
92 is the same for all streams of a given CTF trace, everything else is
93 specified per stream. Following this packet header is a packet context,
94 and then actual recorded events. Each event starts with a mandatory
95 header (same event header for all events of a given stream). The event
96 header is followed by an optional event context with a layout shared
97 by all events of a given stream. Then follows another optional event
98 context, although this one has a layout specific to the event type.
99 Finally, event fields are written.
101 barectf asks you to write the CTF metadata by hand. Although its official
102 [specification](http://git.efficios.com/?p=ctf.git;a=blob_plain;f=common-trace-format-specification.txt;hb=master)
103 is thorough, you will almost always start from this template:
108 /* a few useful standard integer aliases */
109 typealias integer {size = 8; align = 8;} := uint8_t;
110 typealias integer {size = 16; align = 16;} := uint16_t;
111 typealias integer {size = 32; align = 32;} := uint32_t;
112 typealias integer {size = 64; align = 64;} := uint64_t;
113 typealias integer {size = 8; align = 8; signed = true;} := int8_t;
114 typealias integer {size = 16; align = 16; signed = true;} := int16_t;
115 typealias integer {size = 32; align = 32; signed = true;} := int32_t;
116 typealias integer {size = 64; align = 64; signed = true;} := int64_t;
118 /* IEEE 754 standard-precision floating point alias */
119 typealias floating_point {
125 /* IEEE 754 double-precision floating point alias */
126 typealias floating_point {
134 /* CTF version 1.8; leave this as is */
139 * Native byte order (`le` or `be`). This is used by barectf to generate
140 * the appropriate code when writing data to the packet.
145 * Packet header. All packets (buffers) will have the same header.
147 * Special fields recognized by barectf (must appear in this order):
149 * magic: will be set to CTF's magic number (must be the first field)
150 * (32-bit unsigned integer) (mandatory)
151 * stream_id: will be set to the ID of the stream associated with
152 * this packet (unsigned integer of your choice) (mandatory)
154 packet.header := struct {
160 /* environment variables; you may add custom entries */
163 tracer_name = "barectf";
166 tracer_patchlevel = 0;
169 /* clock descriptor */
174 /* clock frequency (Hz) */
177 /* optional clock value offset; offset from Epoch is: offset * (1 / freq) */
181 /* alias for integer used to hold clock cycles */
185 /* map to the appropriate clock using its name */
186 map = clock.my_clock.value;
190 * A stream. You may have as many streams as you want. Events are unique
191 * within their own stream. The main advantage of having multiple streams
192 * is having different event headers, stream event contexts and stream
193 * packet contexts for each one.
197 * Mandatory stream ID (must fit the integer type of
198 * `trace.packet.header.stream_id`).
203 * Mandatory packet context. This structure follows the packet header
204 * (see `trace.packet.header`) immediately in CTF binary streams.
206 * Special fields recognized by barectf:
208 * timestamp_begin: will be set to the current clock value when opening
209 * the packet (same integer type as the clock's value)
210 * timestamp_end: will be set to the current clock value when closing
211 * the packet (same integer type as the clock's value)
212 * content_size: will be set to the content size, in bits, of this
213 * stream (unsigned 32-bit or 64-bit integer) (mandatory)
214 * packet_size: will be set to the packet size, in bits, of this
215 * stream (unsigned 32-bit or 64-bit integer) (mandatory)
216 * cpu_id: if present, the barectf_open_packet() function of this
217 * stream will accept an additional parameter to specify the
218 * ID of the CPU associated with this stream (a given
219 * stream should only be written to by a specific CPU)
220 * (unsigned integer of your choice)
222 * `timestamp_end` must be present if `timestamp_begin` exists.
224 packet.context := struct {
225 my_clock_int_t timestamp_begin;
226 my_clock_int_t timestamp_end;
227 uint64_t content_size;
228 uint64_t packet_size;
233 * Mandatory event header. All events recorded in this stream will start
234 * with this structure.
236 * Special fields recognized by barectf:
238 * id: will be filled by the event ID corresponding to a tracing
239 * function (unsigned integer of your choice)
240 * timestamp: will be filled by the current clock's value (same integer
241 * type as the clock's value)
243 event.header := struct {
245 my_clock_int_t timestamp;
249 * Optional stream event context (you may remove the whole block or leave
250 * the structure empty if you don't want any). This structure follows the
251 * event header (see `stream.event.header`) immediately in CTF binary
254 event.context := struct {
255 int32_t _some_stream_event_context_field;
260 * An event. Events have an ID, a name, an optional context and fields. An
261 * event is associated to a specific stream using its stream ID.
265 * Mandatory event name. This is used by barectf to generate the suffix
266 * of this event's corresponding tracing function, so make sure it follows
267 * the C identifier syntax even though it's a quoted string here.
272 * Mandatory event ID (must fit the integer type of in
273 * `stream.event.header.id` of the associated stream).
277 /* ID of the stream in which this event will be recorded */
281 * Optional event context (you may remove the whole block or leave the
282 * structure empty if you don't want one). This structure follows the
283 * stream event context (if it exists) immediately in CTF binary streams.
286 int32_t _some_event_context_field;
290 * Mandatory event fields (although the structure may be left empty if this
291 * event has no fields). This structure follows the event context (if it
292 * exists) immediately in CTF binary streams.
303 The top `/* CTF 1.8 */` is actually needed right there, and as is, since it
304 acts as a CTF metadata magic number for CTF readers.
306 Only one stream and one event (belonging to this single stream) are described
307 in this template, but you may add as many as you need.
309 The following subsections describe the features of CTF metadata supported
315 The supported structure field types are:
317 * **integers** of any size (64-bit and less), any alignment (power of two)
318 * **floating point numbers** of any total size (64-bit and less), any
319 alignment (power of two)
320 * NULL-terminated **strings** of bytes
321 * **enumerations** associated with a specific integer type
322 * **static** and **dynamic arrays** of any type
323 * **structures** containing only integers, floating point numbers,
324 enumerations and _static_ arrays
326 CTF also supports _variants_ (dynamic selection between different types),
327 but barectf **does not**. Any detected variant will throw an error when
333 CTF integers are defined like this:
337 /* mandatory size in bits (64-bit and less) */
341 * Optional alignment in bits (power of two). Default is 8 when the
342 * size is a multiple of 8, and 1 otherwise.
346 /* optional signedness (`true` or `false`); default is unsigned */
350 * Optional byte order (`le`, `be`, `native` or `network`). `native`
351 * will use the byte order specified by the `trace.byte_order`.
352 * Default is `native`.
357 * Optional display base, used to display the integer value when
358 * reading the trace. Valid values are 2 (or `binary`, `bin` and `b`),
359 * 8 (or `o`, `oct` or `octal`), 10 (or `u`, `i`, `d`, `dec` or
360 * `decimal`), and 16 (or `x`, `X`, `p`, `hex` or `hexadecimal`).
366 * Encoding (if this integer represents a character). Valid values
367 * are `none`, `UTF8` and `ASCII`. Default is `none`.
373 The size (the only mandatory property) does _not_ have to be a power of two:
381 A CTF integer field will make barectf produce a corresponding C integer
382 function parameter with an appropriate size. For example, the 23-bit integer
383 above would produce an `uint32_t` parameter (of which only the first 23
384 least significant bits will be written to the trace), while the first
385 16-bit one will produce an `int16_t` parameter.
387 The `integer` block also accepts a `map` property which is only used
388 when defining the integer used to carry the value of a specified
389 clock. You may always follow the example above.
392 ##### floating point numbers
394 CTF floating point numbers are defined like this:
398 /* exponent size in bits */
401 /* mantissa size in bits */
405 * Optional alignment (power of two). Default is 8 when the total
406 * size (exponent + mantissa) is a multiple of 8, and 1 otherwise.
411 * Optional byte order (`le`, `be`, `native` or `network`). `native`
412 * will use the byte order specified by the `trace.byte_order`.
413 * Default is `native`.
419 If a CTF floating point number is defined with an 8-bit exponent, a 24-bit
420 mantissa and a 32-bit alignment, its barectf C function parameter type will
421 be `float`. It will be `double` for an 11-bit exponent, 53-bit mantissa
422 and 64-bit aligned CTF floating point number. Any other configuration
423 will produce a `uint64_t` function parameter (you will need to cast your
424 custom floating point number to this when calling the tracing function).
429 CTF strings are pretty simple to define:
435 They may also have an encoding property:
439 /* encoding: `none`, `UTF8` or `ASCII`; default is `none` */
444 CTF strings are always byte-aligned.
446 A CTF string field will make barectf produce a corresponding C function
447 parameter of type `const char*`. Bytes will be copied from this pointer
448 until a byte of value 0 is found (which will also be written to the
449 buffer to mark the end of the recorded string).
454 CTF enumerations associate labels to ranges of integer values. They
455 are a great way to trace named states using an integer. Here's an
470 Unless the first entry specifies a value, CTF enumerations are
471 always started at 0. They work pretty much like their C counterpart,
472 although they support ranges and literal strings as labels.
474 CTF enumerations are associated with a CTF integer type (`uint32_t`
475 above). This identifier must be an existing integer type alias.
477 A CTF enumeration field will make barectf produce a corresponding C
478 integer function parameter compatible with the associated CTF integer type.
483 Structure field names may be followed by a subscripted constant to
484 define a static array of the field type:
488 integer {size = 16;} _field[10];
492 In the above structure, `_field` is a static array of ten 16-bit integers.
494 A CTF static array field will make barectf produce a `const void*` C function
495 parameter. Bytes will be copied from this pointer to match the total static
496 array size. In the example above, the integer size is 16-bit, thus its
497 default alignment is 8-bit, so 20 bytes would be copied.
499 The inner element of a CTF static array _must be at least byte-aligned_
500 (8-bit), either by forcing its alignment, or by ensuring it manually
501 when placing fields one after the other. This means the following static
502 array is valid for barectf:
507 integer {size = 5;} _field[10];
511 as long as the very first 5-bit, 1-bit aligned integer element starts
512 on an 8-bit boundary.
517 Just like static arrays, dynamic arrays are defined using a subscripted
518 length, albeit in this case, this length refers to another field using
519 the dot notation. Dynamic arrays are called _sequences_ in the CTF
527 integer {size = 16;} _field[_length];
531 In the above structure, `_field` is a dynamic array of `_length`
534 There are various scopes to which a dynamic array may refer:
536 * no prefix: previous field in the same structure, or in parent
537 structures until found
538 * `event.fields.` prefix: field of the event fields
539 * `event.context.` prefix: field of the event context if it exists
540 * `stream.event.context.` prefix: field of the stream event context
542 * `stream.event.header.` prefix: field of the event header
543 * `stream.packet.context.` prefix: field of the packet context
544 * `trace.packet.header.` prefix: field of the packet header
545 * `env.` prefix: static property of the environment block
547 Here's another, more complex example:
552 string _other_field[stream.event.context.length];
553 float _static_array_of_dynamic_arrays[10][_length];
557 The above examples also demonstrates that dynamic arrays and static
558 arrays may contain eachother. `_other_field` is a dynamic array of
559 `stream.event.context.length` strings. `_static_array_of_dynamic_arrays`
560 is a static array of 10 dynamic arrays of `_length` floating point
561 numbers. This syntax follows the C language.
563 A CTF dynamic array field will make barectf produce a `const void*` C function
564 parameter. Bytes will be copied from this pointer to match the
565 total dynamic array size. The previously recorded length will be
566 found automatically (always an offset from the beginning of the
567 stream packet, or from the beginning of the current event).
569 barectf has a few limitations concerning dynamic arrays:
571 * The inner element of a CTF dynamic array _must be at least byte-aligned_
572 (8-bit), either by forcing its alignment, or by ensuring it manually
573 when placing fields one after the other.
574 * The length type must be a 32-bit, byte-aligned unsigned integer
575 with a native byte order.
580 Structures contain fields associating a name to a type. The fields
581 are recorded in the specified order within the CTF binary stream.
589 string {encoding = ASCII;} _c;
593 The default alignment of a structure is the largest alignment amongst
594 its fields. For example, the following structure has a 32-bit alignment:
598 uint16_t _a; /* alignment: 16 */
599 struct { /* alignment: 32 */
600 uint32_t _a; /* alignment: 32 */
601 string; _b; /* alignment: 8 */
603 integer {size = 64;} _c; /* alignment: 8 */
607 This default alignment may be overridden using a special `align()`
608 option after the structure is closed:
617 integer {size = 64;} _c;
621 You may use structures as field types, although they must have a
622 _known size_ when running barectf. This means they cannot contain
623 sequences or strings.
625 A CTF structure field will make barectf produce a `const void*` C function
626 parameter. The structure (of known size) will be copied as is to the
627 current buffer, respecting its alignment.
629 Note that barectf requires inner structures to be at least byte-aligned.
631 Be careful when using CTF structures for recording binary structures
632 declared in C. You need to make sure your C compiler aligns structure
633 fields and adds padding exactly in the way you define your equivalent
634 CTF structure. For example, using GCC on the x86 architecture, 3 bytes
635 are added after field `a` in the following C structure since `b` is
645 It would be wrong to use the following CTF structure:
649 integer {size = 8; signed = true;} a;
650 integer {size = 32;} b;
654 since field `b` is byte-aligned by default. This one would work fine:
658 integer {size = 8; signed = true;} a;
659 integer {size = 32; align = 32;} b;
663 CTF structures can prove very useful for recording protocols with named
664 fields when reading the trace. For example, here's the CTF structure
665 describing the IPv4 header (excluding options):
669 integer {size = 4;} version;
670 integer {size = 4;} ihl;
671 integer {size = 6;} dscp;
672 integer {size = 2;} ecn;
673 integer {size = 16; byte_order = network;} total_length;
674 integer {size = 16; byte_order = network;} identification;
675 integer {size = 1;} flag_more_fragment;
676 integer {size = 1;} flag_dont_fragment;
677 integer {size = 1;} flag_reserved;
678 integer {size = 13; byte_order = network;} fragment_offset;
679 integer {size = 8;} ttl;
680 integer {size = 8;} protocol;
681 integer {size = 16; byte_order = network;} header_checksum;
682 integer {size = 8;} src_ip_addr[4];
683 integer {size = 8;} dst_ip_addr[4];
687 Although this complex structure has more than ten independent fields,
688 the generated C function would only call a 20-byte `memcpy()`, making
689 it fast to record. Bits will be unpacked properly and values displayed
690 in a human-readable form by the CTF reader thanks to the CTF metadata.
695 Type aliases associate a name with a type definition. Any type may have
696 any name. They are similar to C `typedef`s.
705 byte_order = network;
712 typealias floating_point {
727 typealias enum : uint32_t {
742 float _hello[10][_length];
743 } align(8) := my_struct;
747 ### running the `barectf` command
749 Using the `barectf` command-line utility is easy. In its simplest form,
750 it outputs a few C99 files out of a CTF metadata file:
754 will output in the current working directory:
756 * `barectf_bitfield.h`: macros used by tracing functions to pack bits
757 * `barectf.h`: other macros and prototypes of context/tracing functions
758 * `barectf.c`: context/tracing functions
760 You may also want to produce `static inline` functions if your target
761 system has enough memory to hold the extra code:
763 barectf --static-inline metadata
765 `barectf` is the default name of the files and the default prefix of
766 barectf C functions and structures. You may use a custom prefix:
768 barectf --prefix trace metadata
770 You may also output the files elsewhere:
772 barectf --output /custom/path metadata
774 ### using the generated C99 code
776 This section assumes you ran `barectf` with no options:
780 The command generates C99 structures and functions to initialize
781 and finalize bare CTF contexts. It also generates as many tracing functions
782 as there are events described in the CTF metadata file.
784 Before starting the record events, you must initialize a barectf
785 context. This is done using `barectf_init()`.
787 The clock callback parameter (`clock_cb`) is used to get the clock whenever
788 a tracing function is called. Each platform has its own way of obtaining
789 the a clock value, so this is left to user implementation. The actual
790 return type of the clock callback depends on the clock value CTF integer
791 type defined in the CTF metadata.
793 The `barectf_init()` function name will contain the decimal stream
794 ID if you have more than one stream. You must allocate the context
800 struct barectf_ctx* barectf_ctx = platform_alloc(sizeof(*barectf_ctx));
802 barectf_init(barectf_ctx, buf, 8192, platform_get_clock, NULL);
805 This initializes a barectf context with a buffer of 8192 bytes.
807 After the barectf context is initialized, open a packet using
808 `barectf_open_packet()`. If you have any non-special fields in
809 your stream packet context, `barectf_open_packet()` accepts a
810 parameter for each of them since the packet context is written
814 barectf_open_packet(barectf_ctx);
817 Once the packet is opened, you may call any of the tracing functions to record
818 CTF events into the context's buffer.
820 As an example, let's take the following CTF event definition:
828 integer {size = 32;} _a;
829 integer {size = 14; signed = true;} _b;
830 floating_point {exp_dig = 8; mant_dig = 24; align = 32;} _c;
840 In this example, we assume the stream event context and the event context
841 are not defined for this event. `barectf` generates the following tracing
845 int barectf_trace_my_event(
846 struct barectf_ctx* ctx,
847 uint32_t param_ef__a,
850 const void* param_ef__d,
851 const char* param_ef__e
855 When called, this function first calls the clock callback to get a clock
856 value as soon as possible. It then proceeds to record each field with
857 proper alignment and updates the barectf context. On success, 0 is returned.
858 Otherwise, one of the following negative errors is returned:
860 * `-EBARECTF_NOSPC`: no space left in the context's buffer; the event
861 was **not** recorded. You should call `barectf_close_packet()` to finalize the
864 `barectf_close_packet()` may be called at any time.
865 When `barectf_close_packet()` returns, the packet is complete and ready
866 to be read by a CTF reader. CTF packets may be concatenated in a single
867 CTF stream file. You may reuse the same context and buffer to record another
868 CTF packet, as long as you call `barectf_open_packet()` before calling any
872 ### reading CTF traces
874 To form a complete CTF trace, put your CTF metadata file (it should be
875 named `metadata`) and your binary stream files (concatenations of CTF
876 packets written by C code generated by barectf) in the same directory.
878 To read a CTF trace, use [Babeltrace](http://www.efficios.com/babeltrace).
879 Babeltrace is packaged by most major distributions (`babeltrace`).
880 Babeltrace ships with a command-line utility that can convert a CTF trace
881 to human-readable text output. Also, it includes a Python binding so
882 that you may analyze a CTF trace using a custom script.
884 In its simplest form, the `babeltrace` command-line converter is quite
887 babeltrace /path/to/directory/containing/ctf/files
889 See `babeltrace --help` for more options.
891 You may also use the Python 3 binding of Babeltrace to create custom