Move to kernel style SPDX license identifiers The SPDX identifier is a legally binding shorthand, which can be used instead of the full boiler plate text. See https://spdx.org/ids-how for details. Signed-off-by: Michael Jeanson <mjeanson@efficios.com> Change-Id: I62e7038e191a061286abcef5550b58f5ee67149d Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Refactoring: introduce lttng_tracker_ids data structure This data structure is opaque to allow for back-end implementation change in the future. For now, only the following functions concerning lttng_tracker_ids are public: lttng_list_tracker_ids lttng_tracker_ids_get_count lttng_tracker_ids_get_at_index lttng_tracker_ids_destroy Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Change-Id: Iae1c10d0b578b402ab91378dd49f69f605b316b2 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Refactoring: use an opaque lttng_tracker_id type Move the tracker and tracker id related API to tracker.h and tracker-internal.h. The use of an opaque object mimics the new API for rotation and trigger etc. Signed-off-by: Jonathan Rajotte <jonathan.rajotte-julien@efficios.com> Change-Id: I00b876c618d7dcb0dd940189e5250c3f3d64c7e0 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: sessiond: check for lttng-modules ABI 2.1 rather than 2.8 The clear patchset introduces a regression that breaks session rotations when the kernel domain is used. The 2.8 lttng-modules ABI does not exist yet. The packet sequence number functionality was introduced in LTTng 2.8, which introduced the 2.1 kernel tracer ABI. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com> Change-Id: Ief97e7e25f6f0fcb5b5b97b39abb417c1eb327ec
Send ust and kernel domain directory handle to consumer Send a directory handle to ust/ and kernel/ subdirectories of the chunk to the consumer daemon, and alter the paths sent to consumer so they do not include the domain subdir. This will facilitate implementation of the clear command when no rotation has been performed. The clear command will need to move the ust/ and kernel/ directories to a "deleted" temporary directory while the consumer is still actively using those directories. Making sure that the consumer uses a handle on those subdirectories only ensures that the consumer will not observe any transient state when those directories are moved. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I2357ad702f810561c4d11405edc3ef204fc6c455 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: consumerd: use packet sequence number for rotation position Refer to "Fix: relayd: use packet sequence number for rotation position" for context of this change. This commit introduces the changes required in the consumerd. Some notable points related to this commit: - Internally, the rotate_position (per-stream) is now a 64-bit value rather than an unsigned long. - The scheme to rotate a stream is changed to allow using the backward-compatible lttng_consumer_take_snapshot() rather than the newer lttng_consumer_get_produced_snapshot(), thus allowing backward compatibility of the implicit rotation on destroy with pre-2.10 lttng-modules. - The rotate position used as pivot point for the rotation is based on the packet_seq_num of the last packet that has been send over the network by consumerd, incremented by the number of packets between the sampled produced_pos and the consumed_pos. In the worse case scenario where an overwrite mode ring buffer overwrites its contents enough to trigger a 4GB overflow on a 32-bit producer since the last packet was sent (e.g. due to a slow network), the difference between produced_pos and consumed_pos will be lower that what would have been expected. However, because this pivot position is used as a lower bound, being smaller than the real value is fine: the data that would have been misplaced in the wrong trace chunk were actually overwritten, and will therefore never be consumed. - When interacting with pre-2.8 lttng-modules, the packet sequence number is not available. The current approach is to disallow rotations performed on sessions which have kernel tracing active with a pre-2.8 lttng-modules. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I8600cb5e2e9c05f3dfba0499a5fc4a3bb85dec24 Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: reintroduce lazy kernel modules load, fix empty syscall list Up and including to lttng 2.10, if the lttng-modules are not available when the session daemon starts, an attempt is made to load them each time a command attempts to interact with the kernel domain. 2.11-rc introduces a change in behavior which removes this lazy initialization. This is an issue for distribution packaging (e.g. Ubuntu) where there is no strong dependency between lttng-tools and lttng-modules (because either are optional). So we can be unlucky and find ourselves in a situation where the modules are not available when the session daemon is launched at installation, but only afterwards. Re-introduce the lazy kernel module load behavior, since this is expected by users. Also, fix an issue with empty syscall list in those lazy initialization scenario by moving invocation of syscall_init_table() from main() to init_kernel_tracer(). While we are there, cleanup the following in session daemon: - move kernel_tracer_fd from globals.c to kernel.c. It becomes a static variable, - move module_proc_lttng from main.c to kernel.c, - move init_kernel_tracer() from main.c to kernel.c, - introduce kernel.c:cleanup_kernel_tracer(), invoke it from program cleanup, - introduce kernel_tracer_is_initialized() to check the state of kernel tracer initialization, - adapt kernel.c functions to use the static kernel_tracer_fd rather than expect it as parameter, - update syscall_init_table, invoked from kernel.c, to pass the kernel_tracer_fd as parameter. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: kernel/ust snapshot backward compat for 2.10 relay Ensure the snapshots are saved in the right subdirectory on older relay daemons (2.10 and before). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: keep ust/kernel session items around for destroy notifier Split the destruction and release operations for ust and kernel inner-sessions as they may be accessed by session destruction notifiers registered against their parent ltt_session. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
relayd: create stream files relative to a session's trace chunk Like to the consumer daemon, the relay daemon now creates its stream files through the lttng_trace_chunk interface. This requires changes across the session daemon and relay daemon. The changes to the relay daemon mainly target the file creation. In the session daemon, the snapshot command has been heavily modified to fully initialize the ltt_session's ust and kernel consumer_outputs _before_ the snapshot trace chunk is created. This way, the session is setup in the same way a regular network streaming session would be, a temporary trace chunk is created to contain the snapshot's content, the actual snapshot is taken, and the session's original consumer_outputs are restored. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Create stream files relative to a stream's current trace chunk Create stream (and metadata) files relative to a session's current trace chunk using the lttng_trace_chunk_open/unlink[...] functions in the consumer daemons. Four new commands are added to the sessiond <-> consumerd protocol: - CREATE_TRACE_CHUNK Command parameters: - relayd_id: uint64_t Unique ìd of the session's associated relay daemon connection - override_name: optional char[] Overriden chunk name. This field is not used by the consumer daemon; it is forwarded to the relay daemon in order to set the name of a trace chunk's directory when it should not follow the `<ts begin>-<ts end>-<index>` form used by trace archives (i.e. as produced by session rotations). This is used to preserve the existing format of snapshot output directory names. - sessiond_id: uint64_t Unique id of the session (from the sessiond's perspective) to which this trace chunk belongs. - chunk_id: uint64_t Unique id the the session's trace chunk. - credentials: pair of uint32_t uid/gid Credentials the consumer daemon should use in order to create files within the trace chunk. The session daemon maintains the current lttng_trace_chunk of an ltt_session. When a session that has an output (`output_traces` == 1), an lttng_trace_chunk is created. In local tracing modes, the current trace chunk of a session, on the session daemon's end, holds the ownership of the chunk's output directory. The CREATE_TRACE_CHUNK command is used to replicate the session daemon's current trace chunk in the consumer daemon. This representation of the current trace chunk has a different role. It is created in "user" mode. Essentialy, the trace chunk's location is opaque to the consumer daemon; it receives a directory file descriptor from which a number of stream files will be created. The trace chunk registry, as used by the consumer daemon, implicitly owns the trace chunks on behalf of the session daemon. This is only needed in the consumer since the consumer has no notion of a session beyond session IDs being used to identify other objects. When a channel is created, its session_id and initial chunk_id are provided. This allows the consumer daemon to retrieve the session's current trace chunk and associate it with the newly-created channel. The channel holds a reference to its current trace chunk. Streams created from a channel also hold a reference to their current trace chunk, as retrived from their "parent" channel. The life time of trace chunks in the consumer daemon is cooperatively managed with the session daemon. This means session daemon through the LTTNG_CONSUMER_CREATE_TRACE_CHUNK and LTTNG_CONSUMER_CLOSE_TRACE_CHUNK commands. - CLOSE_TRACE_CHUNK [... TODO ...] This command is used to release the global reference to a given trace chunk in the consumer daemon. Relasing the consumer daemon's global reference to the trace chunk leaves only the streams to hold references until the moment when they are either closed or they switch-over to another chunk in the event of a session rotation. - TRACE_CHUNK_EXISTS [... TODO ...] - ADD_TRACE_CHUNK_CLOSE_COMMAND [... TODO ...] This commit changes a lot of code since it essentialy changes how files and directories are created. A number of commands no longer need to specify a `trace_archive_id` since the CREATE_TRACE_CHUNK and CLOSE_TRACE_CHUNK allow the consumer daemon to keep track of the current trace chunk of a channel at any given time. Creation and ownership of channel sub-directories --- The path expressed in consumer channel objects is now relative to the current trace chunk rather than being absolute. For example, the `pathname` of a consumer channel is now of the form `ust/1000/64-bit` rather than containing the full output path `/home/me/lttng-traces/session-[...]/ust/1000/64-bit/`. The subdirectory of a channel (relative to a trace chunk, e.g. `ust/1000/64-bit`) is lazily created when a stream's output files are created. To do so, the `lttng_consumer_channel` now has a `most_recent_chunk_id` attribute. When a stream creates its output files (i.e. at the beginning of a session, or during a session rotation), the stream's current trace chunk `id` is compared to the channel's `most_recent_chunk_id`. If it is determined that the channel is entering a new trace chunk, its channel subdirectory is created relative to the stream's chunk. Since this new state is within the `lttng_consumer_channel`, the channel lock must be held on code paths that may result in the creation of a new set of output files for a given stream. Note that as of this commit, there is now a clear ownership boundary between directories, owned by the session daemon through its trace chunk, and files, owned by the consumer daemon. Down-scoping of channel credentials --- Since files are now created relative to their stream's current trace chunk (which has credentials set), the fewer sites need access to the channel's credentials. The only reason credentials are kept as part of the consumer channel structure is the need to open and unlink UST shared memory mappings. Since the credentials must only be used for this purpose, they are now stored as an `LTTNG_OPTIONAL` field, buffer_credentials, that is only set for UST channels. Stream files should never need those credentials to be created. The following commands sessiond <-> consumerd commands have been removed: - LTTNG_CONSUMER_ROTATE_RENAME - LTTNG_CONSUMER_CHECK_ROTATION_PENDING_LOCAL - LTTNG_CONSUMER_CHECK_ROTATION_PENDING_RELAY - LTTNG_CONSUMER_MKDIR Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Cleanup: remove duplicated code in snapshot record command The session daemon's snapshot record command contains duplicated code depending on the configuration of the snapshot output. This makes it harder to modify the code in a follow-up commit. Those code paths are made const-correct by the same occasion. No behaviour change is intended. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Create an initial trace chunk on first start of an ltt_session Add a current trace chunk to an ltt_session on session start. The session base output directory is created at that moment and an handle to it is provided to the session daemon's trace chunk when it is set in 'owner' mode. Setting the trace chunk in "owner" mode results in the creation of the chunk's output directory. In the case, of the first chunk, the chunk output directory matches the session output directory. Hence, this has no visible effect. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Generate session name and default output on sessiond's end The lttng client currently generates the default session name and output parameters. This has, over time, resulted in a number of problems. Notably, it is possible for scripts to create session too quickly using automatically-generated session names that would clash since the session's creation timestamp is the only variable part of a session "automatic" name. Hence, sessions created in the same second would clash and result in spurious session creation failures. More importantly, generating session names and outputs on the client end makes it impossible to reliably differentiate output locations that were automatically generated vs. those that were explicitly provided. This causes destinations to be "opaque" to the LTTng daemons as the subdir, session name, and session's creation timestamp are all "cooked" as part of the output destination path/subdir. Keeping these path components separate will make it easier to implement output path configurations that allow the grouping of session outputs by name, by host, etc. Since a session's creation time is used as part of its shm-path, an accessor to the session's creation time is added to the public API: lttng_session_get_creation_time(). This creation time attribute can be accessed when an lttng_session structure is created using the session listing API. Note that existing session creation functions are preserved to maintain the binary compatibility with existing liblttng-ctl users. The session creation functions are reimplemented on top of this newly-introduced API. The only function for which compatibility is dropped is the hidden _lttng_create_session_ext(). Overhaul of path separation --- Not generating paths on the client-end has uncovered a number of problems in the path handling of the session daemon, especially when a network output was used. A lot of code presumed that a network session would be created with a URL containing a sub-directory of the form "session_name-timestamp". While this is true for remote sessions created by the lttng client, a sub-directory is not required when liblttng-ctl is used directly. Hence, this commit ensures that session directories are split as base path, chunk directory, domain directory, application directory. A number of changes in this fix ensure that a session's base path contains everything up to the "session" path element _or_ up to the user-specified output directory. For example, creating a local session using default output settings, the session base output is: /home/user/lttng-traces/session-timestamp Creating a remote session using default output settings, the session base output path is: /hostname/session-timestamp/ Using custom output directories, whether locally or remotely, causes the session base path to be set to that custom output directory. For example, using a local output path of /tmp/my_path will result in a session base path of the form: /tmp/my_path Whereas creating a session with a network output of net://localhost/my_path will result in a session base path of the form: /hostname/my_path Another problematic element is the subdir of the kernel_session and ust_session consumer output which in different scenarios contained chunk names and arbitrary parts of the path hierarchy. The consumer output subdir has been renamed to 'domain_subdir' and now only ever contains: "kernel/", "ust/", or "". Finally, the chunk_path session attribute only contains the name of the current chunk directory being produced. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Fix: wrong error code returned by kernel_snapshot_record() On snapshot error, kernel_snapshot_record() can return LTTNG_ERR_KERN_CONSUMER_FAIL which means that the kernel consumer daemon failed to launch. In this path, the appropriate error to return is LTTNG_ERR_KERN_META_FAIL. Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>