Add facilities to test CLI and plugin regressions with expectation files
authorPhilippe Proulx <eeppeliteloop@gmail.com>
Tue, 11 Jun 2019 23:08:46 +0000 (19:08 -0400)
committerPhilippe Proulx <eeppeliteloop@gmail.com>
Mon, 17 Jun 2019 20:56:59 +0000 (16:56 -0400)
commite30aed9ec6bb1906aa2a7972a09adce4269480c0
treeca85358c950fc639690533fd4c59e658fe96b0c8
parent55478183b5a100be028ed476718f0773ec7b4af5
Add facilities to test CLI and plugin regressions with expectation files

This patch adds `tests/utils/diff.sh.in` (which becomes
`tests/utils/diff.sh`) which contains shell functions to use the CLI and
a `sink.text.details` component to easily test the CLI itself and
plugins for regressions with expectation files.

There are three functions in `tests/utils/diff.sh.in`:

bt_diff_cli():
    Compares the output of the CLI with specific arguments to the
    content of a file.

bt_diff_details_ctf_single():
    Compares the output of the CLI using an `src.ctf.fs` component to
    open a specific CTF trace and a `sink.text.details` component to the
    content of a file.

bt_diff_details_ctf_gen_single():
    Like bt_diff_details_ctf_single(), but runs a given CTF trace
    generating program instead of using an existing CTF trace.

When there's any difference, the functions write the full diff to the
standard error. This makes it possible to look at what's wrong in the
CI.

`tests/plugins/ctf` is moved to `tests/plugins/src.ctf.fs` to indicate
that we're testing this specific component class. The specific query
tests are moved to `tests/plugins/src.ctf.fs/query`.

`tests/plugins/src.ctf.fs/diff/test_diff.in` uses `tests/utils/diff.sh`
to show how to use bt_diff_details_ctf_single() and
bt_diff_details_ctf_gen_single().

The functions test_ctf_single() and test_ctf_gen_single() use resp.
bt_diff_details_ctf_single() and bt_diff_details_ctf_gen_single() to
compare the given CTF traces using a `sink.text.details` component to
expectation files ending with `.expect` in the same directory.

In both test_ctf_single() and test_ctf_gen_single(), we make the
`sink.text.details` component hide the trace and stream names because
`src.ctf.fs` puts absolute paths in them. For test_ctf_gen_single(), we
also make the `sink.text.details` component hide UUIDs because CTF
writer creates random ones by default.

More specifically, this line:

    test_ctf_gen_single simple

means:

1. Run the program `gen-trace-simple` in the same directory to generate
   a CTF trace.

2. Get the output of:

       babeltrace2 /path/to/trace -c sink.text.details \
                   -p with-uuid=no,with-trace-name=no,with-stream-name=no

3. Compare the content of 2. to the file `trace-simple.expect` in the
   same directory.

This line:

    test_ctf_single smalltrace

means:

1. Get the output of:

       babeltrace2 "$BTDIR/tests/ctf-traces/succeed/smalltrace" \
                   -c sink.text.details \
                   -p with-trace-name=no,with-stream-name=no

2. Compare the content of 1. to the file `trace-smalltrace.expect` in
   the same directory.

Eventually, we can move the test_ctf_gen_single() and test_ctf_single()
functions outside the `src.ctf.fs` tests to make them available to test
other plugins and parts of the project. They are placed there for the
moment because only the `src.ctf.fs` tests need them.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I53660111c23117926e3ee114b5d0cd060fff87d3
Reviewed-on: https://review.lttng.org/c/babeltrace/+/1414
Tested-by: jenkins
15 files changed:
.gitignore
configure.ac
tests/Makefile.am
tests/plugins/Makefile.am
tests/plugins/ctf/test_ctf_plugin.in [deleted file]
tests/plugins/ctf/test_query_trace_info.py [deleted file]
tests/plugins/src.ctf.fs/Makefile.am [new file with mode: 0644]
tests/plugins/src.ctf.fs/query/test_query.in [new file with mode: 0644]
tests/plugins/src.ctf.fs/query/test_query_trace_info.py [new file with mode: 0644]
tests/plugins/src.ctf.fs/succeed/Makefile.am [new file with mode: 0644]
tests/plugins/src.ctf.fs/succeed/gen-trace-simple.c [new file with mode: 0644]
tests/plugins/src.ctf.fs/succeed/test_succeed.in [new file with mode: 0644]
tests/plugins/src.ctf.fs/succeed/trace-simple.expect [new file with mode: 0644]
tests/plugins/src.ctf.fs/succeed/trace-smalltrace.expect [new file with mode: 0644]
tests/utils/diff.sh.in [new file with mode: 0644]
This page took 0.026335 seconds and 4 git commands to generate.