deliverable/lttng-tools.git
5 years agoFix: remove inode from inode registry ht fd-tracking
Jérémie Galarneau [Fri, 6 Jul 2018 23:14:43 +0000 (19:14 -0400)] 
Fix: remove inode from inode registry ht

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: unbalanced fd references
Jérémie Galarneau [Fri, 6 Jul 2018 22:44:41 +0000 (18:44 -0400)] 
Fix: unbalanced fd references

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: unlink stream files through the fd-tracker
Jérémie Galarneau [Fri, 6 Jul 2018 22:43:37 +0000 (18:43 -0400)] 
relayd: unlink stream files through the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: crash on close of partially initialized handle
Jérémie Galarneau [Fri, 6 Jul 2018 22:11:27 +0000 (18:11 -0400)] 
Fix: crash on close of partially initialized handle

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: unlink index files through the fd-tracker
Jérémie Galarneau [Fri, 6 Jul 2018 21:40:30 +0000 (17:40 -0400)] 
relayd: unlink index files through the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: rlim_cur and rlim_max printing causes a warning on some archs
Jérémie Galarneau [Fri, 6 Jul 2018 20:07:12 +0000 (16:07 -0400)] 
Fix: rlim_cur and rlim_max printing causes a warning on some archs

rlim_cur and rlim_max were assumed to be unsigned longs, but they
are explicitly 64-bits long on 32 bits archs (warning seen for
powerpc 32 and arm 32 builds).

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoTests: add fd-tracker tests for the unlink operation
Jérémie Galarneau [Fri, 6 Jul 2018 19:50:25 +0000 (15:50 -0400)] 
Tests: add fd-tracker tests for the unlink operation

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker test: register rcu thread of test application
Jérémie Galarneau [Fri, 6 Jul 2018 16:04:13 +0000 (12:04 -0400)] 
fd-tracker test: register rcu thread of test application

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: use lttng_inode to store fs_handle's path
Jérémie Galarneau [Fri, 6 Jul 2018 16:03:13 +0000 (12:03 -0400)] 
fd-tracker: use lttng_inode to store fs_handle's path

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: remove duplicate clear of O_CREAT flag
Jérémie Galarneau [Fri, 6 Jul 2018 15:40:20 +0000 (11:40 -0400)] 
fd-tracker: remove duplicate clear of O_CREAT flag

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker build fix: missing parameter in poll compat function signature
Jérémie Galarneau [Fri, 6 Jul 2018 15:35:29 +0000 (11:35 -0400)] 
fd-tracker build fix: missing parameter in poll compat function signature

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: perform unsuspendable_fd release through call_rcu
Jérémie Galarneau [Fri, 6 Jul 2018 05:01:48 +0000 (01:01 -0400)] 
fd-tracker: perform unsuspendable_fd release through call_rcu

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add the lttng-inode interface
Jérémie Galarneau [Fri, 6 Jul 2018 02:29:12 +0000 (22:29 -0400)] 
fd-tracker: add the lttng-inode interface

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add the unlink operation to fs handles
Jérémie Galarneau [Fri, 6 Jul 2018 02:29:59 +0000 (22:29 -0400)] 
fd-tracker: add the unlink operation to fs handles

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: remove unneeded header inclusion
Jérémie Galarneau [Fri, 6 Jul 2018 02:45:11 +0000 (22:45 -0400)] 
fd-tracker: remove unneeded header inclusion

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add an optimization note to the fd-tracker
Jérémie Galarneau [Fri, 6 Jul 2018 02:43:33 +0000 (22:43 -0400)] 
fd-tracker: add an optimization note to the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoLTTNG-RELAYD(8): document the --fd-pool-size option
Jérémie Galarneau [Thu, 5 Jul 2018 01:28:18 +0000 (21:28 -0400)] 
LTTNG-RELAYD(8): document the --fd-pool-size option

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: rename fd-cap parameter to fd-pool-size
Jérémie Galarneau [Tue, 3 Jul 2018 17:49:43 +0000 (13:49 -0400)] 
relayd: rename fd-cap parameter to fd-pool-size

Rename the fd-cap parameter and change its default behaviour.
The minimum number of file descriptor is raised to 100 and a
"reserve" amount of 10 fds is allowed to accomodate transient
fd uses that can't be tracked by the relay daemon.

The --fd-pool-size will accept parameters in the
[100, fileno soft limit] interval.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: log tracker capacity on creation
Jérémie Galarneau [Tue, 3 Jul 2018 17:48:29 +0000 (13:48 -0400)] 
fd-tracker: log tracker capacity on creation

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: replace lttng_index_file with relay_index_file
Jérémie Galarneau [Sun, 1 Jul 2018 03:20:43 +0000 (23:20 -0400)] 
relayd: replace lttng_index_file with relay_index_file

lttng_index_file is shared between the consumer and relay daemon.
However, the introduction of the fd-tracker in the relay daemon
makes it hard to cleanly share this piece of code between both
daemons.

The ctf-index.h header is still shared by both daemons which
is the most important part. The lttng/relay_index_file class
is a fairly thin wrapper around file system operations (unlink,
read, and write an index) so there is little value gained in
sharing the code vs heavily modifying it to handle the presence
of an fd-tracker in the process.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoMove index initialization to ctf-index.h
Jérémie Galarneau [Sat, 30 Jun 2018 18:51:55 +0000 (14:51 -0400)] 
Move index initialization to ctf-index.h

This initialization code is moved to a common header to re-use
it in a follow-up patch.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: fully initialize viewer stream before publishing it
Jérémie Galarneau [Fri, 29 Jun 2018 22:05:47 +0000 (18:05 -0400)] 
Fix: fully initialize viewer stream before publishing it

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: use the fd-tracker to track stream_fd fds
Jérémie Galarneau [Fri, 29 Jun 2018 21:48:58 +0000 (17:48 -0400)] 
relayd: use the fd-tracker to track stream_fd fds

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the live client connections socket
Jérémie Galarneau [Thu, 28 Jun 2018 05:22:06 +0000 (01:22 -0400)] 
relayd: track the live client connections socket

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track relayd control connection sockets
Jérémie Galarneau [Thu, 28 Jun 2018 05:16:56 +0000 (01:16 -0400)] 
relayd: track relayd control connection sockets

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track relayd data connection sockets
Jérémie Galarneau [Thu, 28 Jun 2018 05:16:43 +0000 (01:16 -0400)] 
relayd: track relayd data connection sockets

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the data listener socket
Jérémie Galarneau [Thu, 28 Jun 2018 04:15:54 +0000 (00:15 -0400)] 
relayd: track the data listener socket

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the control listener socket
Jérémie Galarneau [Thu, 28 Jun 2018 04:15:40 +0000 (00:15 -0400)] 
relayd: track the control listener socket

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the live listener socket
Jérémie Galarneau [Thu, 28 Jun 2018 04:14:52 +0000 (00:14 -0400)] 
relayd: track the live listener socket

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track stdio output fds
Jérémie Galarneau [Thu, 28 Jun 2018 03:16:44 +0000 (23:16 -0400)] 
relayd: track stdio output fds

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the live viewer worker thread's epoll fd
Jérémie Galarneau [Thu, 28 Jun 2018 02:38:52 +0000 (22:38 -0400)] 
relayd: track the live viewer worker thread's epoll fd

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the live listener thread's epoll fd
Jérémie Galarneau [Thu, 28 Jun 2018 02:36:05 +0000 (22:36 -0400)] 
relayd: track the live listener thread's epoll fd

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the live_conn_pipe with the fd-tracker
Jérémie Galarneau [Thu, 28 Jun 2018 02:30:37 +0000 (22:30 -0400)] 
relayd: track the live_conn_pipe with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track listener's epoll fd using the fd-tracker
Jérémie Galarneau [Wed, 27 Jun 2018 19:48:53 +0000 (15:48 -0400)] 
relayd: track listener's epoll fd using the fd-tracker

This addresses the bogus fd report mentionned in a previous
patch of this series as the clean-up of the listener thread's
epoll fd now occurs through the fd-tracker.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track worker thread's epoll fd using the fd-tracker
Jérémie Galarneau [Wed, 27 Jun 2018 19:42:38 +0000 (15:42 -0400)] 
relayd: track worker thread's epoll fd using the fd-tracker

This commit introduces an fd leak report (bogus) which is caused
by another thread using the same poll initialization functions as
the worker thread.

The fd is cleaned-up by that other thread, but the fd-tracker
is not aware of this, thus causing the report.

This is adressed in a follow-up patch.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the health thread's poll fd with fd-tracker
Jérémie Galarneau [Wed, 27 Jun 2018 18:58:48 +0000 (14:58 -0400)] 
relayd: track the health thread's poll fd with fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track clients of the health unix socket with the fd-tracker
Jérémie Galarneau [Tue, 26 Jun 2018 21:38:22 +0000 (17:38 -0400)] 
relayd: track clients of the health unix socket with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the health unix socket with the fd-tracker
Jérémie Galarneau [Tue, 26 Jun 2018 21:20:47 +0000 (17:20 -0400)] 
relayd: track the health unix socket with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the health quit pipe with the fd-tracker
Jérémie Galarneau [Tue, 26 Jun 2018 19:17:16 +0000 (15:17 -0400)] 
relayd: track the health quit pipe with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the relay_conn_pipe with the fd-tracker
Jérémie Galarneau [Tue, 26 Jun 2018 19:07:08 +0000 (15:07 -0400)] 
relayd: track the relay_conn_pipe with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: track the quit pipe with the fd-tracker
Jérémie Galarneau [Tue, 26 Jun 2018 18:51:22 +0000 (14:51 -0400)] 
relayd: track the quit pipe with the fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: Don't bypass the fd tracker when closing file descriptors
Jérémie Galarneau [Tue, 19 Jun 2018 16:22:31 +0000 (12:22 -0400)] 
relayd: Don't bypass the fd tracker when closing file descriptors

There is no reason to close all file descriptors at this point in
the relay daemon as we know for a fact that the only open fds
are stdin, stdout, and stderr. If the relayd was to depend on a
library that opens other file descriptors, it would be unadvisable
to perform this kind of bulk closing of all possible file descriptors.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: close stdin
Jérémie Galarneau [Tue, 19 Jun 2018 02:45:27 +0000 (22:45 -0400)] 
relayd: close stdin

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agorelayd: initialize the global fd tracker from fd_cap parameter
Jérémie Galarneau [Wed, 27 Jun 2018 19:04:49 +0000 (15:04 -0400)] 
relayd: initialize the global fd tracker from fd_cap parameter

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoAdd fd-cap option to the relay daemon
Jérémie Galarneau [Mon, 18 Jun 2018 22:39:07 +0000 (18:39 -0400)] 
Add fd-cap option to the relay daemon

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: remove extra space in function call
Jérémie Galarneau [Mon, 18 Jun 2018 22:37:04 +0000 (18:37 -0400)] 
Cleanup: remove extra space in function call

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoTest: add a unit test for the fd tracker
Jérémie Galarneau [Tue, 19 Jun 2018 18:27:01 +0000 (14:27 -0400)] 
Test: add a unit test for the fd tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add pipe management wrappers to fd-tracker
Jérémie Galarneau [Wed, 27 Jun 2018 20:38:17 +0000 (16:38 -0400)] 
fd-tracker: add pipe management wrappers to fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add epoll/poll management wrappers to fd-tracker
Jérémie Galarneau [Wed, 27 Jun 2018 19:05:06 +0000 (15:05 -0400)] 
fd-tracker: add epoll/poll management wrappers to fd-tracker

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agofd-tracker: add an fd-tracker util to common
Jérémie Galarneau [Thu, 14 Jun 2018 02:55:29 +0000 (22:55 -0400)] 
fd-tracker: add an fd-tracker util to common

This commit adds an fd-tracker utility to the common libs.
This interface allows a process to keep track of its open
file descriptors and enforce a limit to the number of file
descriptors that may be simultaneously opened.

The intent is to use this interface as part of the relay daemon
to mitigate file descriptors exhaustion problems that are
encountered when the relay has to handle a large number of streams.

The fd-tracker defines two classes of file descriptors: suspendable
and unsuspendable file descriptors.

Suspendable file descriptors are handles to filesystem objects
(e.g. regular files) that may be closed and re-opened later without
affecting the application.

A suspendable file descriptor can be opened by creating a filesystem
handle (fs_handle) using the fd-tracker. The raw file descritptor
must then be obtained and released using that handle. Closing the
handle will effectively ensure that the file descritptor is closed.

Unsuspendable file descriptors are file descriptors that cannot
be closed without affecting the application's state. For instance,
it is not possible to close and re-open a pipe, a TCP socket, or
an epoll fd without involving some app-specific logic. Thus, the
fd-tracker considers those file descriptors as unsuspendable.

Opening an unsuspendable file descritptor will return a raw file
decriptor to the application. It is its responsability to notify the
fd-tracker of the file descriptor's closing to ensure the number
of active file descriptors can be tracked accurately.

If a request to open a new file descriptors is made to the fd-tracker
and the process has already reached its maximal count of
simultaneously opened file descriptors, an attempt will be made to
suspend a suspendable file descriptor to release a slot.

Suspending a file descriptor involves:
  - verifying that the file is still available on the FS (restorable),
  - sampling its current position,
  - closing the file descriptor.

Note that suspending a file descriptor eliminates the POSIX guarantee
that a file may be unlinked at any time without affecting the
application (provided that it holds an open FD to that
file). Applications using the fd-tracker that need to maintain this
guarantee should open those files as unsuspendable file descriptors.

To protect against unlinking and file replacement scenarios, the
fd-tracker samples the files' inode number when a fs_handle is
created. This inode number will then be checked anytime the handle
is suspended or restored to ensure that the application is made
aware of the file's unavailability. This is preferable to
inadvertently opening another file of the same name if the original
file was unlinked and/or replaced between a fs_handle's suspension
and restoration.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoClean-up relayd: session_release can be marked as static
Jérémie Galarneau [Thu, 14 Jun 2018 02:56:09 +0000 (22:56 -0400)] 
Clean-up relayd: session_release can be marked as static

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: check for removal of session's shm_path in destroy()
Jérémie Galarneau [Wed, 13 Jun 2018 00:13:48 +0000 (20:13 -0400)] 
Fix: check for removal of session's shm_path in destroy()

When a session is created with an explicit shm_path,
the consumer daemon will create its shared memory files
at that location and will *not* unlink them. This is normal
as the intention of that feature is to make it possible
to retrieve the content of those files should a crash occur.

To ensure the content of those files can be used, the
sessiond daemon will replicate the content of the metadata
cache in a metadata file.

On clean-up, it is expected that the consumer daemon will
unlink the shared memory files and that the session daemon
will unlink the metadata file. Then, the session's directory
in the shm path can be removed.

Unfortunately, a flaw in the design of the sessiond's and
consumerd's tear down of channels makes it impossible to
determine when the sessiond _and_ the consumerd have both
destroyed their representation of a channel. For one, the
unlinking, close, and rmdir happen in deferred 'call_rcu'
callbacks in both daemons.

However, it is also impossible for the sessiond to know when
the consumer daemon is done destroying its channel(s) since
it occurs as a reaction to the closing of the channel's file
descriptor. There is no resulting communication initiated
from the consumerd to the sessiond to confirm that the
operation is completed (and was successful).

Until this is all fixed, the session daemon checks for the
removal of the session's shm path which makes it possible
to safely advertise a session as having been destroyed.

Prior to this fix, it was not possible to reliably save
a session making use of the --shm-path option, destroy it,
and load it again. This is because the creation of the
session would fail upon seeing the session's shm path
already in existence.

Note that none of the error paths in the check for the
directory's existence return an error. This is normal
as there isn't much that can be done. The session will
be destroyed properly, except that we can't offer the
guarantee that the same session can be re-created.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoDocument the contents of the shm_path fields of the ust session registry
Jérémie Galarneau [Tue, 12 Jun 2018 21:16:32 +0000 (17:16 -0400)] 
Document the contents of the shm_path fields of the ust session registry

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoImplement MI and save/load support for callstack contexts
Francis Deslauriers [Tue, 5 Jun 2018 16:06:53 +0000 (12:06 -0400)] 
Implement MI and save/load support for callstack contexts

Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoTests: Add callstack contexts tests
Francis Deslauriers [Thu, 8 Jun 2017 21:15:06 +0000 (17:15 -0400)] 
Tests: Add callstack contexts tests

Tests callstack-user and callstack-kernel contexts by tracing with those
contexts an application that calls multiple functions in chain before
executing a system call.

callstack-user addresses are tested against the output of coreutils'
addr2line.
callstack-kernel addresses are tested against the addresses of kernel
symbols available in the /proc/kallsyms procfile.

Both these tests need to be run by root because those contexts are made
available by the kernel tracer. The callstack-kernel test also need to
access /proc/kallsyms.

Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoAdd kernel and userspace callstack event context
Francis Giraldeau [Wed, 17 Jul 2013 21:06:23 +0000 (17:06 -0400)] 
Add kernel and userspace callstack event context

LTTng callstack context. The callstack context can be added to any kernel
channel. It records either the kernel or the userspace callstack, up to a
max depth. The context is a CTF sequence, such that it uses only the space
required for the number of callstack entries.

The symbol name resolution is left to the trace reader.

Signed-off-by: Francis Giraldeau <francis.giraldeau@gmail.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: missing context enum values in session xml schema
Francis Deslauriers [Tue, 5 Jun 2018 14:57:48 +0000 (10:57 -0400)] 
Fix: missing context enum values in session xml schema

    Handling of the following enum are added:
        LTTNG_EVENT_CONTEXT_INTERRUPTIBLE
        LTTNG_EVENT_CONTEXT_PREEMPTIBLE
        LTTNG_EVENT_CONTEXT_NEED_RESCHEDULE
        LTTNG_EVENT_CONTEXT_MIGRATABLE

Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agolttng-enable-event(1): update the Filter expression section
Philippe Proulx [Mon, 25 Sep 2017 19:18:41 +0000 (15:18 -0400)] 
lttng-enable-event(1): update the Filter expression section

This patch documents new features of filter expressions:

* Bracket and dot notations to get nested fields.
* New bitwise operators.
* Custom operator precedence table.
* Formal rules for how integers are represented for the sake of
  the evaluation (casted to signed/unsigned 64-bit integers depending
  on the operator).

Signed-off-by: Philippe Proulx <eeppeliteloop@gmail.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: document ust app ctx limitation
Mathieu Desnoyers [Mon, 25 Sep 2017 20:22:57 +0000 (16:22 -0400)] 
Filter: document ust app ctx limitation

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: add FILTER_OP_RETURN_S64 instruction
Mathieu Desnoyers [Mon, 25 Sep 2017 13:50:49 +0000 (09:50 -0400)] 
Filter: add FILTER_OP_RETURN_S64 instruction

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: make bitwise and, or, xor higher prio than relational expressions
Mathieu Desnoyers [Fri, 22 Sep 2017 21:38:13 +0000 (17:38 -0400)] 
Filter: make bitwise and, or, xor higher prio than relational expressions

It seems that the usual use-case for bitwise ops are to apply them on
values rather than on the result of equality/inequality. Therefore, move
away from the C operator precedence order, and ensure bitwise and, or,
xor are higher priority than relational expressions (equality and
inequality).

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: Update shifting tests
Mathieu Desnoyers [Fri, 22 Sep 2017 21:14:16 +0000 (17:14 -0400)] 
Filter: Update shifting tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoAdd () for bitwise and comparator tests
Mathieu Desnoyers [Fri, 22 Sep 2017 00:53:01 +0000 (20:53 -0400)] 
Add () for bitwise and comparator tests

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: Implement rshift, lshift, bit not operators
Mathieu Desnoyers [Fri, 22 Sep 2017 00:13:17 +0000 (20:13 -0400)] 
Filter: Implement rshift, lshift, bit not operators

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilters: generate backward compatible "get field" and "get context" instructions
Mathieu Desnoyers [Thu, 21 Sep 2017 20:29:10 +0000 (16:29 -0400)] 
Filters: generate backward compatible "get field" and "get context" instructions

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFilter: index array, sequences, implement bitwise binary operators
Mathieu Desnoyers [Tue, 4 Jul 2017 20:28:54 +0000 (16:28 -0400)] 
Filter: index array, sequences, implement bitwise binary operators

Add load expressions, and produce bytecode allowing indexing of array
and sequence of integers, as well as bitwise binary operators &, |, ^.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoImplement support for brackets in filter expressions
Mathieu Desnoyers [Thu, 22 Jun 2017 20:17:54 +0000 (16:17 -0400)] 
Implement support for brackets in filter expressions

Extends the bytecode with new instructions.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoTests: add session auto-loading test cases
Anders Wallin [Thu, 17 May 2018 20:50:41 +0000 (22:50 +0200)] 
Tests: add session auto-loading test cases

lttng-sessiond can auto load sessions at startup;
- with "--load" option to lttng-sessiond, load one file
  or all sessions files in that directory
- from session files in $LTTNG_HOME/.lttng/sessions/auto/
- from session files in $sysconfdir/lttng/sessions/auto

This test case validates the two first scenarios.

Signed-off-by: Anders Wallin <wallinux@gmail.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoReplace deprecated readdir_r() with readdir()
Michael Jeanson [Tue, 5 Jun 2018 16:11:20 +0000 (12:11 -0400)] 
Replace deprecated readdir_r() with readdir()

readdir_r() has been deprecated since glibc 2.24 [1].

We used readdir_r() in load_session_from_path() to be thread-safe
since this function is part of liblttng-ust-ctl. However, according
to readdir()'s man page, it's thread-safe as long as the directory
stream it operates on is not shared across threads :

  In the current POSIX.1 specification (POSIX.1-2008), readdir(3) is
  not required to be thread-safe. However, in modern
  implementations (including the glibc implementation), concurrent
  calls to readdir(3) that specify different directory streams are
  thread-safe. Therefore, the use of readdir_r() is generally
  unnecessary in multithreaded programs. In cases where multiple
  threads must read from the same directory stream, using readdir(3)
  with external synchronization is still preferable to the use of
  readdir_r(), for the reasons given in the points above.

In our use-case where we open the directory stream in the same function,
we know it won't be shared across threads and thus it's safe to use
readdir(). Here is the relevevant bit from the POSIX.1 [2] spec :

  The returned pointer, and pointers within the structure, might be
  invalidated or the structure or the storage areas might be overwritten
  by a subsequent call to readdir() on the same directory stream. They shall
  not be affected by a call to readdir() on a different directory stream.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=19056
[2] http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html

Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoBash completion: ignore namespace for xmllint parsing
Jonathan Rajotte [Mon, 28 May 2018 21:31:48 +0000 (17:31 -0400)] 
Bash completion: ignore namespace for xmllint parsing

xmllint cli does not "easily" support namespace.

One can use the local_name() xpath function and other "trick".
The simplest trick for bash completion is to ignore the namespace
altogether.

Replacing "xmlns" by "ignore" does the job.

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoUse https in links to the lttng.org website
Jérémie Galarneau [Tue, 5 Jun 2018 15:38:08 +0000 (11:38 -0400)] 
Use https in links to the lttng.org website

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoLog the session to which a ROTATE_PENDING command applies
Jérémie Galarneau [Thu, 10 May 2018 21:31:10 +0000 (17:31 -0400)] 
Log the session to which a ROTATE_PENDING command applies

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoInitialize relay_stream chunk_id to its session's current trace archive id
Jérémie Galarneau [Thu, 10 May 2018 21:14:39 +0000 (17:14 -0400)] 
Initialize relay_stream chunk_id to its session's current trace archive id

Initializing the relayd's streams with a stream_chunk_id allows the
relayd to differentiate between a stream created before the first
rotation (at chunk id == 0) vs. a stream that has been created
after the last (or pending) rotation.

Before this fix, the relayd can fail to identify that a rotation
has been completed.

This is caused by the fact that a stream's chunk id is initialized
to 0 and updated by the RELAYD_ROTATE_STREAM command to the
id of the chunk that is currently being rotated.

The 'stream->current_chunk_id.value < chunk_id' check performed
by the RELAYD_ROTATE_PENDING will cause rotations to never
complete for streams that are created between the launch of a
rotation and the check for its completion.

For example, when the relayd is checking whether the rotation id
'3' is completed, it may see streams with the default value of
their chunk id set to '0' and determine that a rotation is still
pending.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoPass the consumerd stream's trace archive id to the relayd
Jérémie Galarneau [Thu, 10 May 2018 19:00:34 +0000 (15:00 -0400)] 
Pass the consumerd stream's trace archive id to the relayd

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: propagate archive id to the consumer daemon on stream creation
Jérémie Galarneau [Wed, 9 May 2018 01:41:08 +0000 (21:41 -0400)] 
Fix: propagate archive id to the consumer daemon on stream creation

This is the first of a series of fixes addressing a number of problems
with the way session rotation completions handled.

Those issues can result in:
  - A stop never completing,
  - A rotation never completing,
  - A rotation being marked as completed while the consumerd/relayd
      are still writing to the completed chunk's trace archive,
      resulting in a temporarily corrupted trace.

This first commit performs a relatively simple modification
to ensure that the session's current archive id is propagated to the
consumer daemon.

Detailed description of the problems
---

At the core of the problem is the fact that in per-pid buffering, we
are not guaranteed that the sessiond will be able to see an
application's channel(s) if it was torn down before (or even during)
the rotation.

When an application is torn down, it is removed from the ust_app_ht.
That doesn't mean its buffers were received by the relayd or even
consumed by the consumerd. The session daemon issues a "flush channel"
command, but there is no guarantee/synchronization to ensure the
buffers have been consumed.

The current design assumes that the sessiond knows all the channels to
rotate and that we can monitor those channels for the completion of
a rotation. Given that an application can disappear or appear while
we iterate on the ust_app_ht, this assumption does not hold. We also
don't want to prevent/delay applications from registering or exiting
just because a rotation is ongoing.

* Problem 1 *

A rename can happen before the relay has received all data for a given
chunk, leading to the data pending issue explained previously.

Rename should be performed as the last action after the rotation
has been completed since data can still be in-flight,
causing the creation of indexes upon its arrival on the relayd's end.

See: https://github.com/lttng/lttng-tools/blob/cea6c68/src/bin/lttng-sessiond/rotation-thread.c#L392

Currently, the rotation thread waits for all channels (known to the
sessiond at the start of the rotation) to have reached their rotation
point. More specifically, the consumer will write to the
channel_rotation pipe everytime a channel's subbuffers have been read
up to the point of the rotation position. This does not guarantee that
the data has been commited to disk on the relay's end.

At that point, the command to rename the destination folder is sent to
the relayd and the sessiond checks for the pending rotation
periodically (every 200ms) if the output was to a relayd.

That check is assumed not to be needed when tracing locally since
reaching the rotation point implies the contents being written to
disk.

This scheme is not safe. If the sessiond sees no channel to iterate
on, it will issue the rename command immediately. If an application's
buffers were being flushed by the consumerd, the relayd will receive
the data, attempt to create index files, and fail since the folder has
been moved.

From an architectural standpoint, the rename command also leaves the
'path' of streams that were unknown to the sessiond pointing to a path
that does not exist anymore.

* Problem 2 *

In per-pid tracing mode, an application can appear after the rotation
was initiated and cause the rotate pending check to never complete.

A RELAYD_ROTATE_PENDING command is applied to a unique session id and
a chunk id.

When handling a RELAYD_ROTATE_PENDING commands, the relayd will perform
the following check:
- Iterate on every stream known at that point:
- Check if the stream is rotating (stream->rotate_at_seq_num != -1ULL)
- If the stream is not rotating, "stream->chunk_id < chunk_id" is checked.
- If true, the rotation is considered incomplete.

See: https://github.com/lttng/lttng-tools/blob/cea6c68/src/bin/lttng-relayd/main.c#L2850

Given that streams, at their creation, are initialized with their
current "chunk_id" set to 0, the rotation will never be considered
complete if a stream is created between a ROTATE_STREAM and
ROTATE_PENDING command.

This can happen whenever an application is registered during a rotation.

* Problem 3 *

Since the sessiond can't accurately monitor the channels that have to
be rotated, the "rotation completed" notification (and state, if
queried with the lttng_rotation_handle_get_state() interface) is not
reliable.

A client could see that the rotation is marked as completed and
attempt to read a trace archive that has not been completely written.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoTypo in ust consumer log message (channek -> channel)
Jérémie Galarneau [Mon, 4 Jun 2018 21:28:56 +0000 (17:28 -0400)] 
Typo in ust consumer log message (channek -> channel)

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoUse dynamic payload for the add stream realyd command
Jonathan Rajotte [Thu, 10 May 2018 15:02:25 +0000 (11:02 -0400)] 
Use dynamic payload for the add stream realyd command

Move away from static constant defined char array.

Keep backward compatibility.

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoDynamic payload for relayd create session command
Jonathan Rajotte [Thu, 10 May 2018 14:13:22 +0000 (10:13 -0400)] 
Dynamic payload for relayd create session command

Move away from static constant defined char array.

Perform the length check based on constant defined value on reception.

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: backward relayd communication compatibility.
Jonathan Rajotte [Tue, 8 May 2018 21:23:48 +0000 (17:23 -0400)] 
Fix: backward relayd communication compatibility.

The size of LTTNG_HOST_NAME_MAX changed from 64 to 256 in commit id
b867041c62b48e89c9f00430cde4c33f13a2da09.

This change result in breaking compatibility with older relayd.

Freeze size of struct used for relayd communication.

Confirmed by lttng-ivc project.

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoAdd unused attribute to lttng_to_index_major param
Jérémie Galarneau [Mon, 4 Jun 2018 16:50:51 +0000 (12:50 -0400)] 
Add unused attribute to lttng_to_index_major param

Suppresses an unused variable warning. The parameter is kept
since this does depend on the connection's full version. the
'minor' parameter is unused for now since there is only one
major version to support and only one major index version.

However, this could change in the future and we don't want to
have to modify all the version conversion sites.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoReplace strncpy by lttng_strncpy in lttngctl session configuration API
Jérémie Galarneau [Mon, 4 Jun 2018 15:27:37 +0000 (11:27 -0400)] 
Replace strncpy by lttng_strncpy in lttngctl session configuration API

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoReplace strncpy by lttng_strncpy in utils_stream_file_name()
Jérémie Galarneau [Mon, 4 Jun 2018 15:27:08 +0000 (11:27 -0400)] 
Replace strncpy by lttng_strncpy in utils_stream_file_name()

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoUse dynamic buffer to build session configuration path
Jérémie Galarneau [Mon, 4 Jun 2018 15:24:26 +0000 (11:24 -0400)] 
Use dynamic buffer to build session configuration path

Re-use the dynamic buffer interface to build the session
configuration path. The main benefit here is silencing a source
string truncation warning emitted by GCC 8. However, this
interface is also simpler to use than manually building the
path.

The LTTNG_PATH_MAX is still enforced, but there is no real
need to restrict paths to that size. This could be removed if
it ever poses a problem.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoReplace strncpy by lttng_strncpy in session config
Jérémie Galarneau [Mon, 4 Jun 2018 15:23:18 +0000 (11:23 -0400)] 
Replace strncpy by lttng_strncpy in session config

This eliminates a warning produced by GCC 8 in that repeated
code pattern (potential truncation of the source string) and
using the lttng_strncpy macro reduces code duplication.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoSilence strncpy warning emitted by GCC 8 in XSD path construction
Jérémie Galarneau [Fri, 1 Jun 2018 17:01:42 +0000 (13:01 -0400)] 
Silence strncpy warning emitted by GCC 8 in XSD path construction

The size of the XSD's path is fully determined in this function
which makes strcpy() safe to use.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoSilence strncpy warning emitted by GCC 8 in lttng_strncpy()
Jérémie Galarneau [Fri, 1 Jun 2018 16:54:00 +0000 (12:54 -0400)] 
Silence strncpy warning emitted by GCC 8 in lttng_strncpy()

GCC 8 warns if the destination's length is passed to strncpy,
since that could cause the destination to not be NULL-terminated.

This is not a concern for the lttng_strncpy since it checks
that the source must not be truncated. Therefore, it is safe
to simply use strcpy().

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoSilence strncpy warning emitted by GCC 8 in ini parser
Jérémie Galarneau [Fri, 1 Jun 2018 16:50:40 +0000 (12:50 -0400)] 
Silence strncpy warning emitted by GCC 8 in ini parser

While copying 'dst len' bytes in strncpy is normally risky
as the dst may not be NULL-terminated, this function ensures
that the last byte of 'dst' is NULL.

Therefore, this change is mostly made to silence GCC.

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoFix: use signed variable for refcounting of consumer_relayd_sock_pair
Jonathan Rajotte [Wed, 14 Mar 2018 18:54:21 +0000 (14:54 -0400)] 
Fix: use signed variable for refcounting of consumer_relayd_sock_pair

Otherwise refcount check after decreasing have no meaning as in
consumer_stream_relayd_close function.

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: sobjd is never used by reply_ust_register_channel()
Jonathan Rajotte [Wed, 14 Mar 2018 17:46:09 +0000 (13:46 -0400)] 
Cleanup: sobjd is never used by reply_ust_register_channel()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: chan is never used by save_agent_events()
Jonathan Rajotte [Wed, 14 Mar 2018 17:29:56 +0000 (13:29 -0400)] 
Cleanup: chan is never used by save_agent_events()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: open_memstream and close_memstream compat is never used
Jonathan Rajotte [Wed, 14 Mar 2018 17:26:46 +0000 (13:26 -0400)] 
Cleanup: open_memstream and close_memstream compat is never used

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
5 years agoRemove unnecessary inclusions of version.h
Jérémie Galarneau [Fri, 1 Jun 2018 09:31:12 +0000 (05:31 -0400)] 
Remove unnecessary inclusions of version.h

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoAdd multilib test files to .gitignore
Jérémie Galarneau [Fri, 1 Jun 2018 09:28:45 +0000 (05:28 -0400)] 
Add multilib test files to .gitignore

Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: ua_sess is never used create_ust_app_channel_context()
Jonathan Rajotte [Tue, 13 Mar 2018 22:23:26 +0000 (18:23 -0400)] 
Cleanup: ua_sess is never used create_ust_app_channel_context()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: consumer_data is never used by update_kernel_stream()
Jonathan Rajotte [Tue, 13 Mar 2018 22:21:14 +0000 (18:21 -0400)] 
Cleanup: consumer_data is never used by update_kernel_stream()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: app is never used by alloc_ust_app_session()
Jonathan Rajotte [Tue, 13 Mar 2018 22:19:48 +0000 (18:19 -0400)] 
Cleanup: app is never used by alloc_ust_app_session()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: ust_session_id unused by buffer_reg_uid_consumer_channel_key
Jonathan Rajotte [Tue, 13 Mar 2018 22:19:32 +0000 (18:19 -0400)] 
Cleanup: ust_session_id unused by buffer_reg_uid_consumer_channel_key

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: wpipe already contain kernel_tracer_fd
Jonathan Rajotte [Tue, 13 Mar 2018 22:13:39 +0000 (18:13 -0400)] 
Cleanup: wpipe already contain kernel_tracer_fd

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: domain type is never used by send_consumer_relayd_socket()
Jonathan Rajotte [Tue, 13 Mar 2018 22:06:57 +0000 (18:06 -0400)] 
Cleanup: domain type is never used by send_consumer_relayd_socket()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
5 years agoCleanup: uid and gid are never used by run_as_noworker()
Jonathan Rajotte [Tue, 13 Mar 2018 21:41:34 +0000 (17:41 -0400)] 
Cleanup: uid and gid are never used by run_as_noworker()

Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
This page took 0.044883 seconds and 5 git commands to generate.