Fix REUSE licensing/copyright issues in `tests/utils`

This patch makes `reuse lint` happy for the files in `tests/utils`.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I472089852d8840653c0a4a2350a4d4981329f43e
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11385
Tested-by: jenkins <jenkins@lttng.org>
CI-Build: Michael Jeanson <mjeanson@efficios.com>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Fix REUSE licensing/copyright issues in `src`

This patch makes `reuse lint` happy for the files in `src`.

Putting the info into a separate `.license` file for
`src/cpp-common/optional.hpp` to avoid modifying the original file which
comes from an external project.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I073afa1b5de0906f4f9f2a809ef387c05833ca17
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11384
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Add SPDX info for `tests/utils/python/typing/typing.py`

Putting the info in a separate `.license` file to avoid modifying the
original one (which comes from the CPython project).

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: If183d9662f239da27a00b7f8ba8887eaec489270
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11383
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Add SPDX info to Python bindings documentation

The text and images have the `CC-BY-SA-4.0` license.

`LICENSES/CC-BY-SA-4.0.txt` already exists, but wasn't used.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I66428778d69cc4b44816cb6588283984d16f3d78
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11382
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Add missing REUSE licenses

`reuse lint` says:

    # MISSING LICENSES

    'FSFAP' found in:
    * m4/ax_append_compile_flags.m4
    * m4/ax_append_flag.m4
    * m4/ax_check_compile_flag.m4
    * m4/ax_cxx_compile_stdcxx.m4
    * m4/ax_require_defined.m4
    'FSFULLR' found in:
    * m4/ae_python_modules.m4
    'GPL-2.0-or-later' found in:
    * m4/ax_c___attribute__.m4
    * m4/ax_pthread.m4
    * tests/utils/tap-driver.sh
    'LicenseRef-Autoconf-exception-macro' found in:
    * m4/ax_c___attribute__.m4
    * m4/ax_pthread.m4

Download and add them.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I71576e2ed2281a1890a49a5cb9261c11fe30706e
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11381
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

LICENSES: add `.txt` extension to license files

`reuse lint` says:

    # LICENSES WITHOUT FILE EXTENSION

    The following licenses have no file extension:
    * BSD-2-Clause
    * BSD-4-Clause
    * BSL-1.0
    * CC-BY-SA-4.0
    * GPL-2.0-only
    * GPL-3.0-or-later
    * LGPL-2.1-only
    * MIT
    * PSF-2.0

Make it happy.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I6c06d74c37fe4230e5c302783183211eeff2b8a5
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11379
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>
CI-Build: Michael Jeanson <mjeanson@efficios.com>

Use `LICENSES/GPL-3.0-or-later` (`GPL-3.0` is deprecated)

As of this date, the SPDX website says [1] about "GNU General Public
License v3.0 only":

> This license identifier has been deprecated since license list
> version 3.0.

Therefore, use `GPL-3.0-or-later`.

That's already the short ID we use in `tests/utils/tap/tap.sh`, not
`GPL-3.0`.

[1]: https://spdx.org/licenses/GPL-3.0.html

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I72349b1631cabfa16a59e0228871205978d5dd91
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11378
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

Use `LICENSES/GPL-2.0-only` (`GPL-2.0` is deprecated)

As of this date, the SPDX website says [1] about "GNU General Public
License v2.0 only":

> This license identifier has been deprecated since license list
> version 3.0.

Therefore, use `GPL-2.0-only`.

That's already the short ID we use everywhere, not `GPL-2.0`.

[1]: https://spdx.org/licenses/GPL-2.0.html

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ifa073ef2a833f5bc6fb1051d50146ca352965688
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11377
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Use `LICENSES/LGPL-2.1-only` (`LGPL-2.1` is deprecated)

As of this date, the SPDX website says [1] about "GNU Lesser General
Public License v2.1 only":

> This license identifier has been deprecated since license list
> version 3.0.

Therefore, use `LGPL-2.1-only`.

That's already the short ID we use in `src/common/list.h`.

[1]: https://spdx.org/licenses/LGPL-2.1.html

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I2a6193f9b9b03275505424ecebc9896d9faabb0e
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11376
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

Remove `src/plugins/ctf/common/bfcr/btr.gdb`

We don't use nor maintain that. I'm not even sure what it does. We used
it mostly while developing Babeltrace 2.0, but things are stable now.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I3ced5c5e57ee55013820cac77cdb456f3f984b0a
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11375
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

tests/utils/python/normand.py: add SPDX headers

This is still the file as is from the upstream project.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ide3014910bd169f7fdaf05e2bbafd517a6ff6a93
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11374
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

tools: add lint-py.sh

Add lint-py.sh, running the various static analysis tools we use for
Python.

Change-Id: I6ccd5f7fb484506a0f936d7fe276042cccd186a6
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11269
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Re-format C++ files

The last few commits introduced some formatting errors, re-formating
using `tools/format-cpp`.

Change-Id: Ib9e4075d52b95f505bc6cbe2a2d5cd70d6926f84
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11268
CI-Build: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: add and use bt_grep_ok

When a test that consists of matching a pattern in a file using grep
fails (sometimes just in the CI), it would be useful to see the file
contents to understand what happened.  Add the `bt_grep_ok` utility
function.  It tries to match a pattern in a file using `grep`, then
issues a test result.  If the test failed, it prints the content of the
file on stderr.

Change-Id: I464e414b334052677799ce201483095fa9bff4c3
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11169
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: add and use bt_grep

Add the `bt_grep` function, which passes all its arguments to the grep
binary specified by BT_TESTS_GREP_BIN.  Use everywhere where applicable
in the testsuite.

Change-Id: Ia166106fd85312e681fa2692e1f5409c9556f897
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11248
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Michael Jeanson <mjeanson@efficios.com>

build: move common headers to common_libcommon_la_SOURCES

Since we have an libcommon convenience library, I think it would make
sense to put all common headers there, instead of in noinst_HEADERS.

Change-Id: I161e5dc234c46599efbaffcea4e759e2df0ffc96
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11291
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

filter.utils.trimmer: remove unused parameter from set_bound_from_param

Change-Id: I17e3a3cb7542c292f7eaa5e349725e2dca664abc
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11217
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

sink.text.details: remove unused parameter from configure_bool_opt

Change-Id: I71601439e92bdd678b45731f7f08a7c4d59f66eb
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11216
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

flt.lttng-utils.debug-info: remove unused parameter from handle_msg_iterator_inactivity

Change-Id: I007b1d2a121ce1cf15abffda7485aaa396c3c7b2
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11215
CI-Build: Simon Marchi <simon.marchi@efficios.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

src.ctf.lttng-live: remove unused parameter from lttng_live_metadata_create_stream

Change-Id: Id70429d549c25aca07e6b6819e0f6f0a5fe01e4d
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11214
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

src.ctf.lttng-live: remove unused parameter from live_get_msg_ts_ns

Change-Id: I5cd8569817f83b080af0ca0ff9fdb9218d3f5b13
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11213
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

src.ctf.fs: remove parameter from create_one_port_for_trace

Change-Id: Ia58cc038ba597dff3c5e2fc6d9d29e18f71d5c89
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11212
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

src.ctf.fs: remove unused parameter from ctf_fs_component_create

Change-Id: I32ed8d398a1292a48ebccb27578dee9e20d3b4f3
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11211
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

sink.ctf.fs: remove unused parameter from append_string_field_class

Change-Id: I262bcd035bdbdc81b80c80dc74f8ca9aebf5af34
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11210
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

sink.ctf.fs: remove unused parameter from write_string_field

Change-Id: I5098245393516b4a37daae1d0c0fe37fcf9bb2f5
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11209
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

lib: remove unused parameter from set_integer_value

Change-Id: I6334e430afce5170908f28a2d579806f87eff860
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11208
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

lib: remove unused parameters bt_util_ns_from_origin_inline

Change-Id: I54bb5b1677f9e5761cb9e394d25706086f1a38ec
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11207
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

lib: remove `extended` parameters from format_* functions

Change-Id: Iad81577eebfc0306afbaf6b15b8f7ad398a84959
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11206
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

lib: remove unused internal component destroy functions

These destroy functions are empty, remove them.  They can always be
added back later if needed, but I don't see the value in keeping then
"just in case".

Change-Id: I5845d2834294583367c85a840b016a536bf0b120
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11205
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

compat: remove unused `addr` parameter from bt_mmap

Change-Id: Iecf7764f5fd669369064aa3b3fafe2575d933f28
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11204
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/' by one more

The libcommon Makefile was the only one left of the convenience library
Makefiles to ensure that the version header was generated before the
other libraries were built.

By using the 'BUILT_SOURCES' variable we can ensure that 'version.i' is
generated first and merge this in 'src/Makefile'.

Change-Id: I0a6510af65674d1c8bb6a3a08d9a10b07e00fdd7
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11203
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

lib: remove unused parameters in internal component creation functions

I noticed that these 3 functions didn't use the `class` parameter,
remove it.

Change-Id: I34b798dfbd8b1e54e7ddf633b61d6d205c0132cb
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10408
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

Reduce the number of Makefiles in 'src/' even MOAR!

Change-Id: I3dbb16250f62513ba131592c596fbaaf1ba97ffe
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11192
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

doc: fix passing non-event messages in distill examples

The distill filter example means to let non-event messages go through,
but it actually drops them.  Fix it so it actually lets them through.

Verified by sending the output of the filter to `sink.text.details`.
Before the patch, there is no stream beginning message.  After the
patch, there is.

Change-Id: I2414a37ca7fc5fa6e5ed0fe179ee39cadf261f31
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10444
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

configure: re-enable '-Wunused-parameter'

This warning is part of '-Wextra' in GCC.

Change-Id: Ifb514a41cb9c62cf703d415eeb2ccefc331dee77
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/7559
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

configure: re-enable '-Wmissing-field-initializers'

This warning is part of '-Wextra' in GCC.

Change-Id: Ie6170cb06900e93011ddedc102a9964c63edad55
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/7558
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>

Fix: tests: conditionally skip Python bindings test that requires Python plugins support

As exposed by bug 1272 [1], there are some tests in
test_trace_collection_message_iterator.py that depend on Python plugins
being enabled.  Running the tests with `--enable-python-bindings` and
`--disable-python-plugins` therefore leads to these tests failing.

Fix it by skipping them if the support for Python plugins is not
enabled.

[1] https://bugs.lttng.org/issues/1272

Change-Id: I4881a3b0a336b9078715837cb7f8f423bbc2b982
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/3621
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Michael Jeanson <mjeanson@efficios.com>

tests: remove test dir in test-query-metadata-info.sh

Change-Id: Ib3415ae8370bec53d0eb1ffa807f011b6b0c933c
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11182
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Fix: tests: use local typing.py for Python 3.5

`typing.NoReturn` was introduced in Python 3.5.4.  Therefore, there are
releases of Python 3.5 with which we get errors like:

    # Running: python3 /home/smarchi/src/babeltrace/tests/utils/python/mctf.py --base-dir /tmp/test-stored-value.jJOHoog/stored-values /home/smarchi/src/babeltrace/tests/data/ctf-traces/live/stored-values.mctf
    Traceback (most recent call last):
      File "/home/smarchi/src/babeltrace/tests/utils/python/mctf.py", line 13, in <module>
        import normand
      File "/home/smarchi/src/babeltrace/tests/utils/python/normand.py", line 61, in <module>
        from typing import Any, Set, Dict, List, Union, Pattern, Callable, NoReturn, Optional
    ImportError: cannot import name 'NoReturn'

Fix that by using our local typing module for Python 3.5.

Change-Id: I5660a5e0e2014a71c5e6b98b6ce1970b5bb1226e
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11172
CI-Build: Michael Jeanson <mjeanson@efficios.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

Fix: tests: use run_python in gen_mctf_trace

gen_mctf_trace calls the python3 interpreter, without using the
`run_python` wrapper.  PYTHONPATH is therefore not set appropriately
for Python to find our local `typing.py` module, if needed.

Add a call to `run_python` to fix that.

Change-Id: Id813169905d511e5600a70cebad681b00ebb49d8
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11171
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: retry os.rename on PermissionError failure in lttng_live_server.py

On the Windows CI jobs, we get random failures like:

    # plugins/src.ctf.lttng-live/test-live.sh: python3 /c/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/src/babeltrace/tests/data/plugins/src.ctf.lttng-live/lttng_live_server.py /c/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/src/babeltrace/tests/data/plugins/src.ctf.lttng-live/inactivity-discarded-packet.json --port-file /c/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/tmp/test-live-server-port.Rn2dyS --trace-path-prefix C:\Users\jenkins\workspace\dev_review_babeltrace_master_winbuild\build\std\conf\std\platform\msys2-mingw64\src\babeltrace\tests\data\ctf-traces
    Traceback (most recent call last):
      File "C:/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/src/babeltrace/tests/data/plugins/src.ctf.lttng-live/lttng_live_server.py", line 1951, in <module>
        LttngLiveServer(port, port_filename, sessions, max_query_data_response_size)
      File "C:/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/src/babeltrace/tests/data/plugins/src.ctf.lttng-live/lttng_live_server.py", line 1667, in __init__
        self._write_port_to_file(port_filename)
      File "C:/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/src/babeltrace/tests/data/plugins/src.ctf.lttng-live/lttng_live_server.py", line 1792, in _write_port_to_file
        os.replace(tmp_port_file.name, port_filename)
    PermissionError: [WinError 5] Access is denied: 'C:/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/tmp/tmpt13jh6sp' -> 'C:/Users/jenkins/workspace/dev_review_babeltrace_master_winbuild/build/std/conf/std/platform/msys2-mingw64/tmp/test-live-server-port.Rn2dyS'

The PermissionError exception is raised when trying to move the port
file from its temporary location to its final location, where the bash
script expects it to appear.

I don't understand the root cause of the issue.  When exiting the `with`
scope, the temporary file is supposed to be closed, and it should be
fine to move it.  I suppose it's possible that something in the Windows
kernel hasn't completely finished using the file when we try to move it.

Implement a wait-and-retry scheme as a (bad) workaround.

Change-Id: Ia8dcefca9538aa5e58438bf84a3fa67e5e05a49a
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11174
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

cleanup: tests: remove unused files from lf / crlf traces

Change-Id: I259bd6a88f48d00471bd6a779d0e15150d4c3d4e
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11175
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tools/format-cpp: add `.sh` extension to identify it

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ifb57e0a26a4d7a2da99bd74fdf24d8aadcec4cf6
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11173

tools/format-cpp: exclude generated `parser.*` and `lexer.*` files

We don't need to format them because they're not tracked.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I0df8ffe9cef64b62b9327ee05acc2f4c5d48401b
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11150
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>

tools/format-cpp: accept optional starting directory

This patch makes `tools/format-cpp` accept an optional starting
directory (instead of the root directory of the project) as its first
argument:

    $ pwd
    /home/dboucher/Mes Logiciels/babeltrace/src/plugins/ctf

    $ ../../../tools/format-cpp .
    clang-format -i /home/eepp/dev/babeltrace/src/plugins/ctf/fs-src/fs.cpp
    clang-format -i /home/eepp/dev/babeltrace/src/plugins/ctf/fs-src/file.cpp
    clang-format -i /home/eepp/dev/babeltrace/src/plugins/ctf/fs-src/query.cpp
    ...
    clang-format -i /home/eepp/dev/babeltrace/src/plugins/ctf/lttng-live/metadata.hpp
    clang-format -i /home/eepp/dev/babeltrace/src/plugins/ctf/lttng-live/data-stream.hpp

Using realpath(1) to transform the starting directory because find(1)
will only consider file names relative to its starting point, therefore
if the starting point is `.` (and the CWD is `src/cpp-common`),
`*/src/cpp-common/optional.hpp` and so on don't match anything and the
script will format files to be excluded.

Using `-path` instead of `-wholename` which is more precise and also
seems more portable.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I46219035e9b815dffe9200bb924478e4a7b02a04
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11145

tools/format-cpp: use a group of `echo` lines with a single `>& 2`

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I4202fa48c0fe6909aeb3b1ea34870e920908917f
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11144
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>

tools/format-cpp: add `-t` to `xargs`

`tools/format-cpp` can be a bit long, so show the progress by making
xargs print the executed command.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ifb5e40017a2bb429ce7e900a2076532bc038a792
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11143
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

tools/format-cpp: make shellcheck happy again

This patch:

* Checks the `version` variable assignment directly to avoid relying on
  `$?`, therefore removing the `shellcheck disable` comment.

* Makes `formatter` an array to satisfy SC2086, therefore removing the
  `shellcheck disable` comment.

* Quotes `$(nproc)` to satisfy SC2068.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ib7c5fe530faad12704ee8b91ae55a6bfcb1aee5c
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11142
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>

tests: normalize names of files and more

The current naming situation is somewhat of a mess in the `tests`
directory, making it difficult to deduce (or write) a convention.

Use hyphens to separate words as much as possible, following what we
usually like for source file names (see `tree src`).

Pretty much the only things remaining with underscores are Python file
names because of the `import` constraints and because it's Python's own
convention.

Also make all shell scripts have the `.sh` extension. This makes any
worthy editor highlight the shell syntax correctly and tells you
immediately it's not a compiled ("binary") executable. I'd prefer
`.bash` for Bash code, but I don't think we need to go as far.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I43cd279d78c240a7e3686ad8afd6759bad9f575f
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11170
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>
CI-Build: Simon Marchi <simon.marchi@efficios.com>

Reduce the number of Makefiles in 'doc/'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds.

Change-Id: I7a4a8543a2f1e2781f09fbe0bdea2cc0395d239d
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11156
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds.

Change-Id: I3d87dea9979c09d1a6416c7b9950264a8c4d81dc
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11155
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/plugins/text'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds. Also reduce the number of
convenience libraries for faster linking.

Change-Id: I1e706eb0bb5d52d0172ab6d0773f861865562158
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11154
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/plugins/lttng-utils'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds.

Change-Id: I339d190fea78fcefd6b3927cccb89db1b4fec6f4
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11153
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/plugins/common'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds.

Change-Id: Ic90c2c4e60f0af76bb0fde784d3cb64030aa7f44
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11152
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Reduce the number of Makefiles in 'src/plugins/utils'

Continue the effort started by Simon to reduce the overall number of
Makefiles to allow for faster parallel builds. Also reduce the number of
convenience libraries for faster linking.

Change-Id: I772572c3dc1d2481ec94526a4301d4bb79f9d855
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11151
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Remove `src/cpp-tester`

This was never meant to be added.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I4ba129b99145cceef6f785d07f7c84341754e827
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11157

Update autoconf archive macros

Change-Id: I899d955dd1ff53a7bde9729a13396da7514f35cc
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10342
CI-Build: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

Add test to parse lf/crlf metadata

Unix-like platforms represent line endings in text file with a '\n' or
LF while Windows uses '\r\n' or CRLF. The CTF standard doesn't specify a
canonical representation for metadata files and only requires whitespace.

Test both styles of line endings on all platforms to ensure
interoperability of the produced metadata files.

Change-Id: Ib4080b1d95be2fb0d5844e7772bd6c2832de378a
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11078
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

fix: tests: use a portable invocation of 'sed -i'

The GNU and BSD sed implementations slightly differ in the handling of
an argument to the '-i' switch to use as a suffix to a backup file. BSD
sed requires an argument while it's optional for GNU sed, if present it
must be placed directly after the switch with no space.

To make both implementations work properly and not create a backup file,
a zero length argument must be placed directly after the switch with no
space.

Change-Id: Ifd2cd5b27df05e3301e527ef1d8e718b434d2236
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11132
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

ctf: reduce the number of `Makefile.am`s

This should help speeding parallel builds a bit, by reducing the time
spent in directories with few source files.  The time necessary to build
the intermediary .la libraries is also non-negligible (at least on
Windows), so having less of those should help.

I kept `common/metadata/Makefile.am`, because it has a lot of complexity
specific to the use of bison, I think it's good to keep it in its own
file.

Change-Id: I6b7a5ca370d6b4481bda3649ab51009e70a0e393
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11137
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

ctf: grow stored_values array when necessary

The CTF message iterator accesses the `stored_values` array out of
bounds in the following situation:

 - In the context of a src.ctf.lttng-live source, a ctf_trace_class gets
   created from some metadata.
 - At this point, ctf_trace_class->stored_value_count indicates that a
   certain number of stored values are necessary given the metadata
   parsed up to now.
 - The message iterators are created with `stored_values` arrays of that
   size.
 - The source receives more metadata, which requires more stored
   values.
 - The message iterator reads some event described by the new metadata,
   that requires the use of a stored value.
 - Since the stored value arrays have not been resized to reflect the
   necessary number of stored value considering the new metadata, the
   message iterator tries to store a value past the end of the array.

Fix this by ensuring the `stored_values` array is large enough before
storing a value in it.

Add a test with a hand-crafted trace that replicates the scenario
described in the simplest manner possible:

 - send a bit of metadata
 - send a bit of data that uses that metadata
 - send a bit more metadata (that requires a new stored value)
 - send a bit of data that uses that new metadata

Without the fix, we get (when babeltrace is built in debug mode):

    (╯°□°)╯︵ ┻━┻  /home/smarchi/src/babeltrace/src/plugins/ctf/common/msg-iter/msg-iter.cpp:1865: bfcr_unsigned_int_cb(): Assertion `(uint64_t) int_fc->storing_index < msg_it->stored_values->len` failed.

... showing the out of bounds array access.

Change-Id: I78e3ca57ac6cae1959425df3c8ffdbfeb534f348
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10866
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: add `tests/utils/python/mctf.py`, a text to CTF trace generator

Add `mctf.py`, a utility to generate a test trace from a moultipart file
(see `moultipart.py`).

Any part having the header info `metadata` is written as-is in a file
named `metadata`.  Other parts are considered Normand inputs, and the
output is written to a file of which the name is the exact header info
of the part.  For example:

    --- metadata
    /* CTF 1.8 */
    [...]

    --- stream0
    "Normand input here"

    --- .index/stream0
    "Normand input here too"

Variables and labels are carried from one normand.parse()
call to the other, allowing parts to refer to variables/labels from
previous parts.

All files are written relative to the value of the `--base-dir` option,
or relative to the current working directory if omitted.  Any
nonexistent parent directory is created if needed (like `make -p`).

Add a function in `utils.sh  that runs `mctf.py`.

Change-Id: I345b77ac382f268e8becd553e2e301982d80ced3
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11056
CI-Build: Simon Marchi <simon.marchi@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: add moultipart.py

Add moultipart.py, a Python module to help read text files meant to be
split in different parts.  A moultipart-compliant file is a succession
of parts, where each part is made of a header line followed with zero or
more lines of content.

A header line consists of `---` optionally followed by a space and some
custom information.

The moultipart module exposes the `parse` function, which takes a TextIO
object and returns a list of Part objects.

For instance, the following input:

    --- Victoria
    Parenteau
    ---
    Taillon
    --- This part is empty
    --- Josianne
    Gervais

results in the following parts:

    [Part('Victoria', 'Parenteau\n', 2),
     Part('', 'Taillon\n', 4),
     Part('This part is empty', '', 6),
     Part('Josianne', 'Gervais\n', 7)]

Change-Id: I50fb88ed6e064c09deaf1d56187415bc26002a14
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10912
Tested-by: jenkins <jenkins@lttng.org>

tests: use /dev/null instead of empty files in src.ctf.lttng-live test

Instead of creating empty files, use /dev/null.

Change-Id: I4193da1fb60e9bd25df61afde09a7684b6cdf8f7
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10906
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: pass trace path prefix down in test_live

A future test case will need to use a different trace path prefix than
the existing tests.  Add a parameter for that and pass the value down.

Change-Id: I2b95ce098461f32503cafabfc640d99fe120e582
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10919
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: pass live server args down using arrays

Pass arguments to lttng_live_server.py all the way down using arrays.

Change-Id: I9efd02539fd7c790bb941069bceadb7b7f5315a2
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10918
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: move --port-file and --trace-path-prefix arguments up one level

Specify the --port-file and --trace-path-prefix arguments in function
get_cli_output_with_lttng_live_server rather than function
lttng_live_server.

Change-Id: Iddb781b8a085b85468f4148708409c774c79893f
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10917
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

common: introduce bt_g_array_index

Introduce bt_g_array_index, a wrapper to g_array_index, that adds a
check that the specified index does not go past the end of the array.

Use BT_ASSERT_DBG, so that it's enabled only in builds with debug mode
enabled.

The difficulty is that g_array_index is expected to yield an lvalue,
since some call sites take the address of what it yields, or assign to
it.  The way I found to make it work is to add a statement expression
that has no effect, and put the BT_ASSERT_DBG in it.

Change-Id: I280c0657ec6f1cadd01328eec79da5cefada5c31
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10905
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

Fix: set proper size when creating new strings

After adding a wrapper to g_array_index that validates that accessed
indices don't go past the end of the arrays, I got this:

    (╯°□°)╯︵ ┻━┻  /home/smarchi/src/babeltrace/src/lib/trace-ir/field.c:392: create_string_field(): Assertion `0 < string_field->buf->len` failed.

The problem is that arrays backing string fields are created with:

    string->buf = g_array_sized_new(FALSE, FALSE, sizeof(char), 1);

This g_array_sized_new call reserves space for one element, but doesn't
actually make the length of the array one.  Add a call to
g_array_set_size to fix that.  Fix another occurence of the same bug in
ctf-writer.

Change-Id: I2b90b26c37d70d64bcd825b5b07b204bcbb5baf5
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10904
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: make lttng_live_server.py write temp port filename in same directory

When running lttng_live_server.py with `--port-filename yoyoyo`, I get:

    Traceback (most recent call last):
      File "lttng_live_server.py", line 1982, in <module>
        LttngLiveServer(port, port_filename, sessions, max_query_data_response_size)
      File "lttng_live_server.py", line 1716, in __init__
        self._write_port_to_file(port_filename)
      File "lttng_live_server.py", line 1839, in _write_port_to_file
        os.replace(tmp_port_file.name, port_filename)
    OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpe70085jz' -> 'yoyoyo'

Fix this by placing the temporary file in the same directory as its
final location.

Change-Id: Ibe6fee5dc99860ab3cd4464a2f490e05c65df1eb
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10901
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: make lttng_live_server.py print session URLs

To help with interactive use, make lttng_live_server.py print session
URLs that you can pass directly to babeltrace.

For instance:

    $ PYTHONPATH=/home/smarchi/src/babeltrace/tests/utils/python /opt/python/3.4.10/bin/python3 lttng_live_server.py --trace-path-prefix /home/smarchi/src/babeltrace/tests/data/ctf-traces  split_metadata.json --port 1245
    Listening on port 1245
    net://localhost:1245/host/hostname/split_metadata

Change-Id: Ib4d806804bfdc00132ec7987c62016ad976b50d2
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10900
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: remove isdir/isfile asserts from lttng_live_server.py

When lttng_live_server.py receives a trace path that doesn't exist
(perhaps because the user forgot to pass a --trace-path-prefix), it
gives:

    Traceback (most recent call last):
      File "lttng_live_server.py", line 1969, in <module>
        trace_path_prefix,
      File "lttng_live_server.py", line 1892, in _session_descriptors_from_path
        traces.append(LttngTrace(path, metadata_sections, beacons))
      File "lttng_live_server.py", line 1044, in __init__
        assert os.path.isdir(trace_dir)
    AssertionError

This is not very helpful, it hides details about the nature of the
problem.  Remove the assert, and let the subsequent open fail, saying
why it failed.  After the patch, it becomes:

    Traceback (most recent call last):
      File "lttng_live_server.py", line 1968, in <module>
trace_path_prefix,
      File "lttng_live_server.py", line 1891, in _session_descriptors_from_path
traces.append(LttngTrace(path, metadata_sections, beacons))
      File "lttng_live_server.py", line 1045, in __init__
self._create_metadata_stream(trace_dir, metadata_sections)
      File "lttng_live_server.py", line 1099, in _create_metadata_stream
metadata_path, config_metadata_sections
      File "lttng_live_server.py", line 974, in _split_metadata_sections
with open(metadata_file_path, "r") as metadata_file:
    FileNotFoundError: [Errno 2] No such file or directory: 'live/split_metadata/metadata'

Remove an isfile assert for the same reason.

Change-Id: If4289515b2c27748dd90209049c623e02c940a03
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10899
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: change lttng_live_server.py's --sessions-filename to be a positional argument

When running lttng_live_server.py with no arguments, we get:

    Traceback (most recent call last):
      File "lttng_live_server.py", line 1968, in <module>
        trace_path_prefix,
      File "lttng_live_server.py", line 1871, in _session_descriptors_from_path
        with open(sessions_filename, "r") as sessions_file:
    TypeError: invalid file: None

The --sessions-filename argument is effectively mandatory.  Change it to
be a positional argument.

Change-Id: I48e67cbf1da8adfaa87d5024d91a922f9b67a7fd
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10898
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: make --port-filename optional in lttng_live_server.py

While port_filename is useful for automation, it isn't essential when
using lttng_live_server.py manually.  Make that argument non-required.

Also, print a message indicating the port used to listen, which is handy
for interactive use.

Change how lttng_live_server.py is invoked to redirect stdout to stderr,
hopefully avoiding messing up the tap output format.

Change-Id: Ie185ced440c51fb1bb8355553f7b4a7725a13542
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10896
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: remove UnexpectedInput from lttng_live_server.py

I don't think this exception is useful, versus just raising a
RuntimeError.

The current code catches exceptions at the top-level, logs it (using
logging.error) and prints it to stderr.  I don't think it's useful to
log using logging.error, since it also goes to stderr.  And I don't
think it's useful to manually print the exception.  Instead, let them
escape and let Python print them using its regular uncaught exception
mechanism, which ends up printing them on stderr too.

Change-Id: Ice797d4b672dc5eba7312c4cabb2740806c37a2a
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10894
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: add typing annotations to lttng_live_server.py

Add typing annotations throughout lttng_live_server.py, such that
pyright passes cleanly in strict mode.

Specify in a comment in lttng_live_server.py to run in strict mode, but
disable these two warnings:

 - reportTypeCommentUsage: we need to use the comment form to support
   Python 3.4
 - reportMissingTypeStubs: it would complain about not having a stub
   file for the utils module (although it can inspect utils.py just
   fine)

Note that the sessions_filename variable near the bottom is marked as a
string, even though it's currently an optional argument (so it should be
`str | None`).  But the rest of the code treats it as non-optional.
This is corrected in a subsequent patch that makes the sessions filename
mandatory.

Change the way lttng_live_server.py is launched to use run_python, to
have access to Python modules in tests/utils/python, to have access to
jsonw.py and possible the typing module shim for Python 3.4.

Change-Id: I842f4c696eea7c99932876af26c195bf99d5cfff
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10869
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: add strongly-typed JSON wrappers

When using the result of `json.load` as-is, the returned values are not
strongly-typed, so it is up to the code to check that values are of the
expected type.  To facilitate this introduce small wrappers around that,
for objects and arrays, to add automatic type-checking when accessing
children.

    # Obtain the child of an object with type check
    obj: tjson.ObjVal = ...
    child = obj.at("child-key", tjson.BoolVal)

    # Iterate on children of an array with type check
    arr: tjson.ArrayVal = ...
    for integers in arr.iter(tjson.IntVal):
      ...

These expressions throw TypeError if the children are not of the
expected type. `tjson.ArrayVal` and `tjson.ObjVal` are resp. sequence
and mapping types so that you may still use the typical operators to
disable type checking:

    obj: tjson.ObjVal = ...
    child = obj["child-key"]

    arr: tjson.ArrayVal = ...
    for integers in arr:
      ...

The wrappers also help generate relatively precise error messages, by
recording the path to each object, and outputting it in jq filter
format.  For instance:

    TypeError: `.[1].traces[0].path`: expecting a string value

    KeyError: `.[1].traces[0]`: no value has the key `path`

Change-Id: Ic6fbb2de5731851af3b90a476af009315f829665
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10909
Tested-by: jenkins <jenkins@lttng.org>

src.ctf.fs: improve `metadata-info` query error message with non-existent metadata file

A colleague pointed out that the message babeltrace2 outputs when
pointing the `metadata-info` query to a directory without a metadata
file (or a non-existent directory) is not clear as it could be.

    ERROR:    [Babeltrace CLI] (/home/smarchi/src/babeltrace/src/cli/babeltrace2.c:651)
      Failed to query component class: unknown error: plugin-name="ctf", comp-cls-name="fs", comp-cls-type=SOURCE object="metadata-info"
    CAUSED BY [libbabeltrace2] (/home/smarchi/src/babeltrace/src/lib/graph/query-executor.c:234)
      Component class's "query" method failed: query-exec-addr=0x60b000000880, cc-addr=0x60f000000130, cc-type=SOURCE, cc-name="fs", cc-partial-descr="Read CTF traces from the file sy",
      cc-is-frozen=0, cc-so-handle-addr=0x607000000020, cc-so-handle-path="/home/smarchi/build/babeltrace/src/plugins/ctf/babeltrace-plugin-ctf.la", object="metadata-info",
      params-addr=0x606000000500, params-type=MAP, params-element-count=1, log-level=WARNING
    CAUSED BY ['source.ctf.fs'] (/home/smarchi/src/babeltrace/src/plugins/ctf/fs-src/query.cpp:92)
      Cannot open trace metadata: path="yoyoyo".

In the ctf_fs_metadata_open_file function, we could have access to the
errno that explains the failure, but currently don't use it.  Also, it
would be good to tell which file we tried to open exactly.  Make
ctf_fs_metadata_open_file log an error and append an error cause.

The new error stack is the same as the above, with this new cause at the
end:

    CAUSED BY ['source.ctf.fs'] (/home/smarchi/src/babeltrace/src/plugins/ctf/fs-src/metadata.cpp:41)
      Failed to open metadata file: No such file or directory: path="yoyoyo/metadata"

Add a test that validates that this error cause is present in the error
message.

Change-Id: I111848cc72e6f3b2cdfb251914eb337ea02a19f4
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11079
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Simon Marchi <simon.marchi@efficios.com>

Fix: make gen-version-i.sh work with shallow clones

With a shallow git clone (a clone done with `--depth=1`),
gen-version-i.sh fails with:

    $ TOP_SRCDIR=$PWD ./src/common/gen-version-i.sh
    fatal: No names found, cannot describe anything.

The underlying git command that fails is:

    $ git describe --tags --dirty --abbrev=12
    fatal: No names found, cannot describe anything.

git-describe fails because no tags are available in the shallow clone.
Work around that by using `--always`, which makes git-describe
generate string with only the commit hash as a fallback.   In my
non-shallow clone:

    $ git describe --tags --dirty --abbrev=12 --always
    v1.2.0-3854-g0b538efadd88

And in my shallow clone:

    $ git describe --tags --dirty --abbrev=12 --always
    0b538efadd88

Change-Id: Ie1c15239497bda248c9742e74a12a14af6e32928
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Signed-off-by: Thomas Applencourt <tapplencourt@anl.gov>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11073
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

fix: test_message_iterator.py hangs on Python 3.12

Starting with Python 3.12, `None` is immortal: its refcount operations
are NO-OP and sys.getrefcount() will return a static value of UINT_MAX
on 64-bit and UINT_MAX >> 2 on 32-bit.

This basically transform `test_try_again_many_times` in an almost
infinite loop and hangs the testsuite.

Detect this by checking if the refcount on `None` is incremented after
assigning to a variable and skip the test if it's not the case.

See PEP-0683[1] for the gory details.

[1] https://peps.python.org/pep-0683/

Change-Id: Id07658245d524288ce7606cb0a011ad97068dad1
Signed-off-by: Michael Jeanson <mjeanson@debian.org>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10379
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Fix: src.ctf.fs: omitting to add ctf signature

When depacketizing a metadata stream we may need to add the CTF 1.8
signature at the beginning of the file if it's absent.

This fixes a regression introduces by commit:
  commit 06be99464988953e390114337ccdd20622fb50d1
  Author: Philippe Proulx <eeppeliteloop@gmail.com>
  Date:   Tue Jul 9 19:40:45 2019 -0400

      ctf: refactor metadata decoder to always have an instance

This regression was not caught earlier because the test case for that
feature was disabled.

This commit reintroduces that behaviour and enables the test case.

Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Change-Id: I3621a7f8a21ffe51f29dfa344f30039fdf5d16c4
Reviewed-on: https://review.lttng.org/c/babeltrace/+/8204
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
CI-Build: Simon Marchi <simon.marchi@efficios.com>

tests: fix file comment of plugins/src.ctf.lttng-live/test_live

That comment appears to be a copy-paste leftover from a `src.ctf.fs`
test.  Adjust it to make it relevant to that file.

Change-Id: I943cef771d01589ec432a75128860c7c55aed314
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/3599
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

fix: python bindings: use stdlib distutils when setuptools is installed

When the setuptools package is installed, it monkey patches the standard
library distutils even if the user code doesn't import setuptools.

This results in a failure to install the Python agent in a directory
which isn't in the current PYTHONPATH. To allow this, setuptools requires
the '--single-version-externally-managed' options which is not
implemented in distutils.

To resolve this, force the use of distutils for Python < 3.12 even when
setuptools is installed with the 'SETUPTOOLS_USE_DISTUTILS' environment
variable and use the previously mentionned setuptools option with Python
>= 3.12 which doesn't include distutils anymore.

Change-Id: I9e8412021c6ec79b0a9ea38759c475113e4ea018
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10371
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

fix: python bindings: install on Debian python >= 3.10

Starting with Debian's Python 3.10, the default install scheme is
'posix_local' which is a Debian specific scheme based on 'posix_prefix'
but with an added 'local' prefix. This is the default so users doing
system wide manual installations of python modules end up in
'/usr/local'. This interferes with our autotools based install which
already defaults to '/usr/local' and expects a provided prefix to be
used verbatim.

Monkeypatch sysconfig to override this scheme and use 'posix_prefix'
instead.

Change-Id: I41973f6db4966519aa1c7e74a9d1e9b5d90bf7ea
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10343
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

ctf: remove some unused `ctx` parameters in visitor-generate-ir.cpp

My editor pointed these out.

Change-Id: Ia0c3b244c0037a9a2111f53cfd7fbf2ede7bb2c7
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10453
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: remove unused MyFilter in UserMessageIteratorTestCase.test_try_again_many_times

This MyFilter class (and associated MyFilterIter class) is not used,
remove it.

Change-Id: Ib148361606ac8814145b2924a940e262eb7d4bd7
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10380
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
CI-Build: Michael Jeanson <mjeanson@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

Remove dev-requirements.txt entry for click

Since we don't use black 22 anymore, this entry for click is not
required anymore.

Change-Id: Icf89fca7c7af6d71bb00c0fc4f6ca1d18e17b653
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11057
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
CI-Build: Michael Jeanson <mjeanson@efficios.com>

Add `tests/utils/python/normand.py` (Normand 0.23)

This is an unmodified copy of the `normand.py` module [1], version 0.23.

Normand, an EfficiOS project, is both a template language to generate
binary data and a processor of said language.

`normand.py` is part of the `normand` package, but it's completely
independent, without external dependencies, therefore we can copy it
right here. `normand.py` works with Python 3.4. The module itself offers
a Python 3 API as well as a command-line tool.

We plan to use Normand in Babeltrace 2 to write human-readable and
Git-trackable plain text CTF data streams, and eventually other types of
binary data, for testing purposes.

Although Normand is an EfficiOS project, it's considered an external
project from the point of view of the Babeltrace project. Therefore, we
make Black, isort, and Flake8 ignore it, even though Normand uses the
same settings in its own repository.

[1]: https://github.com/efficios/normand

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I6e8937cf40320c3cf1226c3447f71640ed9f91bf
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10910
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>

tests: use a local `typing` module with Python 3.4 only

This patch changes the strategy to support the `typing` module in tests.

To make it possible to include compatible third-party/portable Python
modules as is, Python needs to find the `typing` module. Python 3.5+
will find it, but Python 3.4 won't.

To fix this:

1. `tests/utils/utils.sh` calls the Python interpreter specified by
   BT_TESTS_PYTHON_BIN to get the Python version.

2. The run_python() function in `tests/utils/utils.sh` adds the
   `tests/utils/python/typing` directory to `PYTHONPATH` if the version
   of Python is 3.4.

3. Move `tests/utils/python/local_typing.py` to
   `tests/utils/python/typing/typing.py`.

4. Remove `tests/utils/python/utils.py` (not needed anymore).

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Ifc939f617627ff11cd6a7cbf8642c7b33006d303
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10915
Tested-by: jenkins <jenkins@lttng.org>

Fix: Windows DLL path lookup with Python >= 3.8

Starting with Python 3.8 on Windows [1] the DLLs lookup when loading
native modules was changed to not search PATH like everything else on
this platform. Restore this behavior by doing it manually.

Simon adds: it only started to matter now because MSYS2 dropped a local
patch [2] that retained the old behaviour.

[1]: https://docs.python.org/3/library/os.html#os.add_dll_directory
[2]: https://www.msys2.org/news/#2023-08-06-python-changed-behavior-when-loading-dll-dependencies-of-extension-modules

Change-Id: Id436244a717f440db11c33b6d418f31c7838591d
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/11043

tests/utils/utils.sh: add run_python helper function

Add a run_python function, which runs the passed command with
`tests/utils/python` in PYTHONPATH.  Re-implement run_python_bt2 on top
of run_python.

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: I30dcee2e85d581340881bc108e4cc8bbc65ba798
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10914
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
CI-Build: Simon Marchi <simon.marchi@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

configure.ac: require Python 3.4+

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Change-Id: Iec0fc9ec23016069a30b08bb63c808ed6d3143cb
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10913
Reviewed-by: Michael Jeanson <mjeanson@efficios.com>
Reviewed-by: Simon Marchi <simon.marchi@efficios.com>
Tested-by: jenkins <jenkins@lttng.org>

tests: fix all basic type check issues of lttng_live_server.py

Do the necessary changes such that this comment passes cleanly:

    $ PYTHONPATH=/home/smarchi/src/babeltrace/tests/utils/python pyright tests/data/plugins/src.ctf.lttng-live/lttng_live_server.py

Pyright pointed out that field _trace_session_state didn't exist, in:

    @property
    def trace_session_state(self):
        return self._trace_session_state

The actual field name is _ts_state.  However, it shows that this
property is never used, so remove it.

Change-Id: I115fe473310ecdb1e2fd2e5d369aae33afe963f1
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10893
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests/python: add local copy of typing module

Add a local copy of the typing module [1], in order to be able to use
the typing module features with Python 3.4.

Add notes about the license in the top-level LICENSE file.

Add exclusions to avoid this file being reformatted by black, analyzed
by flake8 and its imports sorted by isort.

Add the tests/utils/python/utils.py with an import
er files will be able to use the
typing module using:

    typing = utils._typing_mod

... and not care where the typing module comes from.

[1] https://raw.githubusercontent.com/python/typing/3.10.0.2/src/typing.py

Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Change-Id: I7648e700f7e350acc595004c697a89e1fc97de4a
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10892
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

tests: move classes around in lttng_live_server.py

In preparation for an upcoming patch, which fixes typing issues in
lttng_live_server.py, move some types around to avoid forward-references
when possible.

Change-Id: If7c14db977fa8b5c5d050badf2d8f1254fb1279a
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10870
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>
Tested-by: jenkins <jenkins@lttng.org>

python: run isort on everything

Run isort on the whole repo, to nicely sort Python imports.

Add a mention in CONTRIBUTING.adoc, that all Python imports must be
sorted using the version of isort specified in dev-requirements.txt.

Add `isort ~= 5.12` to dev-requirements.txt.  5.12 is the latest stable
version available on PyPi as of writing.

Tell isort to use its `black` profile, in pyproject.toml, so that
black remains happy.  The details of that profile are here:

  https://pycqa.github.io/isort/docs/configuration/profiles.html

Tell isort to ignore `tests/utils/python/tap`, since it's code copied
from elsewhere.

Use the `length_sort` option, because Philippe prefers it like that.
See:

  https://pycqa.github.io/isort/docs/configuration/custom_sections_and_ordering.html

Change-Id: I8778f908342d0feeb97d4442506113d9252559e7
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10871
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

Document usage of Black in CONTRIBUTING.adoc

Mention that all Python code must be formatted using Black.

Change-Id: I420c4e5a75b34eae40da83ae09d5fd76d8eddb0f
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10888
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

python: remove internal `import bt2` imports

Standardize the internal imports to import specific modules like this:

    from bt2 import foo as bt2_foo

... instead of importing the complete bt2 package.

Change-Id: I32eb223a8bcfe841020a2aafb976f121dadd8680
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10388
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

python: move exception types out of __init__.py

Move exception types defined in __init__.py.

 - Move _MemoryError to error.py
 - Move _IncompleteUserClass to component.py
 - Move the others to utils.py

I initially moved them to a new exceptions.py file, but I think that
it's just fine to move them in the existing files.  But it's a
possibility too, if y'all prefer that.

The goal of this commit is to make it easier to use these types
internally, using the form:

    from bt2 import utils as bt2_utils

    ... use bt2_utils.TryAgain ...

... in order to get rid of the internal `import bt2` imports.

Note that it would be possible to keep the exception types in
__init__.py and do:

    from bt2 import TryAgain

... and then use `TryAgain` in the code, but I think we prefer using a
module qualifier for whatever is imported.

Change-Id: I7d8dbdc28a3a2e3e38093903d622487a45fe7637
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10387
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>

python: standardize intra-bt2 imports

A subsequent patch wants to refer to `object` (the Python type), but it
is shadowed by our `from bt2 import object` import.  It seems like a
good time to standardize how we import intra-bt2 modules, to use the
pattern:

    from bt2 import potato as bt2_potato

Note that I didn't add the bt2_ prefix to the native_bt import, since it
would be a bit redundant.

Change-Id: Icdb4339075a1888463c4f84b292deb272ad49943
Signed-off-by: Simon Marchi <simon.marchi@efficios.com>
Reviewed-on: https://review.lttng.org/c/babeltrace/+/10386
Tested-by: jenkins <jenkins@lttng.org>
Reviewed-by: Philippe Proulx <eeppeliteloop@gmail.com>