Move to kernel style SPDX license identifiers The SPDX identifier is a legally binding shorthand, which can be used instead of the full boiler plate text. See https://spdx.org/ids-how for details. Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Change-Id: I62e7038e191a061286abcef5550b58f5ee67149d Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
relayd: replace uses of block FDs by the fs_handle interface Replace all usage of "raw" block-device file descriptors for relay_streams, viewer_streams, and index files by the fs_handle. Wrappers are introduced for read, write, seek and truncate operations in order to reduce code duplication as all uses of fs_handles implies getting an fd, using it, and putting it back. Those operations allow the fd-tracker to suspend and restore fs_handles as needed. The stream_fd util is eliminated as it is completely replaced by the fs_handle interface. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com> Change-Id: Iedff88d27aeba3891d4e8818b9e08e4b16a927cc
fd-tracker: refactor: extract fs_handle interface from fd_tracker Make the fs_handle interface a proper abstract interface containing overridable callbacks. The objective of this refactor is to make it possible for lttng_trace_chunk to return fs_handles which are tracked by an fd_tracker (or not) depending on the execution context (which daemon). In effect, the relay daemon will provide a trace chunk with an fd_tracker to use and then rely on the fs_handle interface to track the use of file descriptors. The other daemons using the lttng_trace_chunk interface will use a dummy implementation of fs_handle which basically directly returns the underlying file descriptor and performs the unlink/close operations directly. This makes is possible to share code interacting with files between the various daemons without carrying a plethora of optional parameters in every util. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com> Change-Id: Iaafa0f4442442bdfdaf220ce33a966978877df23
Refactoring: use an opaque lttng_tracker_id type Move the tracker and tracker id related API to tracker.h and tracker-internal.h. The use of an opaque object mimics the new API for rotation and trigger etc. Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Change-Id: I00b876c618d7dcb0dd940189e5250c3f3d64c7e0 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix build: fd-tracker is not part of librelayd The fd-tracker library is not part of the internal librelayd; it is its own stand-alone library. Add an autoconf conditional to build libfd-tracker by default and require it when building the relay daemon. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com> Change-Id: I40934eeaa62e61f5da4508c1f2c91daaa9e57829
Format lists in src/common/Makefile.am I'm currently dealing with src/common/Makefile.am, and would find it more pleasant to modify if the lists were formatted consistently and sorted (except DIST_SUBDIRS, for which the order is important). I have taken the liberty to put the corresponding .c and .h on the same line, since they will always be next to each other. It reduces a bit the number of lines. If you don't like that, I would also be fine with a completely flat list. Change-Id: I8a65fe5e0fea37230f1376c51c7d4a547f651c7e Signed-off-by: Simon Marchi <simon.marchi@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Replace libuuid with internal implementation We use a very small subset of libuuid features to transform UUIDs between their string and binary representation. Plus we have a lot of compat code for different platforms with some unspecified default behavior regarding the use of upper/lower case. Drop the dependency on libuuid and replace it with a minimal internal implementation that respects RFC4122. Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Change-Id: I744e3cf65d6a22d0acf7a9943c10943ba64e8468 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
fd-tracker: add an fd-tracker util to common This commit adds an fd-tracker utility to the common libs. This interface allows a process to keep track of its open file descriptors and enforce a limit to the number of file descriptors that may be simultaneously opened. The intent is to use this interface as part of the relay daemon to mitigate file descriptors exhaustion problems that are encountered when the relay has to handle a large number of streams. The fd-tracker defines two classes of file descriptors: suspendable and unsuspendable file descriptors. Suspendable file descriptors are handles to filesystem objects (e.g. regular files) that may be closed and re-opened later without affecting the application. A suspendable file descriptor can be opened by creating a filesystem handle (fs_handle) using the fd-tracker. The raw file descritptor must then be obtained and released using that handle. Closing the handle will effectively ensure that the file descritptor is closed. Unsuspendable file descriptors are file descriptors that cannot be closed without affecting the application's state. For instance, it is not possible to close and re-open a pipe, a TCP socket, or an epoll fd without involving some app-specific logic. Thus, the fd-tracker considers those file descriptors as unsuspendable. Opening an unsuspendable file descritptor will return a raw file decriptor to the application. It is its responsability to notify the fd-tracker of the file descriptor's closing to ensure the number of active file descriptors can be tracked accurately. If a request to open a new file descriptors is made to the fd-tracker and the process has already reached its maximal count of simultaneously opened file descriptors, an attempt will be made to suspend a suspendable file descriptor to release a slot. Suspending a file descriptor involves: - verifying that the file is still available on the FS (restorable), - sampling its current position, - closing the file descriptor. Note that suspending a file descriptor eliminates the POSIX guarantee that a file may be unlinked at any time without affecting the application (provided that it holds an open FD to that file). Applications using the fd-tracker that need to maintain this guarantee should open those files as unsuspendable file descriptors. To protect against unlinking and file replacement scenarios, the fd-tracker samples the files' inode number when a fs_handle is created. This inode number will then be checked anytime the handle is suspended or restored to ensure that the application is made aware of the file's unavailability. This is preferable to inadvertently opening another file of the same name if the original file was unlinked and/or replaced between a fs_handle's suspension and restoration. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: bindings import segfaults on missing hash_key_u64 A mishandled error in SWIG-generated code causes the Python3 interpreter to segfault when a missing symbol is reported during the linking (at runtime) against liblttng-ctl. libcommon makes use of the internal libhashtable.la since the addition of the lttng_trace_chunk interface. This introduces a transitive dependency to libhashtable.la in liblttng-ctl. Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Add the trace chunk and trace chunk registry interfaces A trace chunk is a set of stream files. It maps to the user-visible concept of "trace archive chunks" produced following a tracing session rotation. The concept of a "chunk" is introduced to make it possible to associate a group of stream files together, store common properties (e.g. the epoch, base path, list of files, credentials, etc.), and perform an action once all files have been closed/released. The "trace chunk" interface is to be used by the session, consumer, and relay daemons in different ways, through the OWNER or USER roles. The lttng_dynamic_pointer_array, lttng_dynamic_array, and optional utils are added since they are used by the trace chunk implementation. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Add mkdirat utils and runas wrappers The lttng_directory_handle allows its user to keep a handle to a directory and to create subdirectories relative to it. On platforms implementing POSIX.2008, a directory file descriptor is used to maintain a handle to an existing directory and used in conjunction with mkdirat() to create subdirectories. Derelict platforms (such as Solaris 10) use an alternative implementation which carries the location of the root directory and builds subdirectory paths on creation. The existing mkdir utils are re-implemented using this new interface (using the special AT_FDCWD file descriptor value, when applicable) to limit code duplication. The implementation of the directory handle and its users is automatically selected based on the presence of the dirfd() function, but can also be explicitly chosen using the --enable/disable-dirfd configuration option. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Generate session name and default output on sessiond's end The lttng client currently generates the default session name and output parameters. This has, over time, resulted in a number of problems. Notably, it is possible for scripts to create session too quickly using automatically-generated session names that would clash since the session's creation timestamp is the only variable part of a session "automatic" name. Hence, sessions created in the same second would clash and result in spurious session creation failures. More importantly, generating session names and outputs on the client end makes it impossible to reliably differentiate output locations that were automatically generated vs. those that were explicitly provided. This causes destinations to be "opaque" to the LTTng daemons as the subdir, session name, and session's creation timestamp are all "cooked" as part of the output destination path/subdir. Keeping these path components separate will make it easier to implement output path configurations that allow the grouping of session outputs by name, by host, etc. Since a session's creation time is used as part of its shm-path, an accessor to the session's creation time is added to the public API: lttng_session_get_creation_time(). This creation time attribute can be accessed when an lttng_session structure is created using the session listing API. Note that existing session creation functions are preserved to maintain the binary compatibility with existing liblttng-ctl users. The session creation functions are reimplemented on top of this newly-introduced API. The only function for which compatibility is dropped is the hidden _lttng_create_session_ext(). Overhaul of path separation --- Not generating paths on the client-end has uncovered a number of problems in the path handling of the session daemon, especially when a network output was used. A lot of code presumed that a network session would be created with a URL containing a sub-directory of the form "session_name-timestamp". While this is true for remote sessions created by the lttng client, a sub-directory is not required when liblttng-ctl is used directly. Hence, this commit ensures that session directories are split as base path, chunk directory, domain directory, application directory. A number of changes in this fix ensure that a session's base path contains everything up to the "session" path element _or_ up to the user-specified output directory. For example, creating a local session using default output settings, the session base output is: /home/user/lttng-traces/session-timestamp Creating a remote session using default output settings, the session base output path is: /hostname/session-timestamp/ Using custom output directories, whether locally or remotely, causes the session base path to be set to that custom output directory. For example, using a local output path of /tmp/my_path will result in a session base path of the form: /tmp/my_path Whereas creating a session with a network output of net://localhost/my_path will result in a session base path of the form: /hostname/my_path Another problematic element is the subdir of the kernel_session and ust_session consumer output which in different scenarios contained chunk names and arbitrary parts of the path hierarchy. The consumer output subdir has been renamed to 'domain_subdir' and now only ever contains: "kernel/", "ust/", or "". Finally, the chunk_path session attribute only contains the name of the current chunk directory being produced. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Revert stubbing of runas functions All the runas functions were stubbed on builds where the sessiond isn't built which is the case for all platforms except Linux. This was done because of 2 new commands that require elf.h which is not present on MacOSX. However the other commands can be used by the relayd. Revert this and and only stub the relevant commands when "elf.h" is not present on the system. Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Revert stubbing of unix socket functions Instead of stubbing useful UNIX socket functions to work around Linux-only credential passing, ifdef the relevant parts like it was already done for other functions. Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
run_as: add extract ELF symbol offset command We use a run_as command for this work because the input is controlled by the user and it should be parsed using the permission of that user. Also, parsing ELF files is tricky and doing it in a run_as process has the benefit of isolating potential crashes. Signed-off-by: Francis Deslauriers <francis.deslauriers@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>