From: Mathieu Desnoyers Date: Tue, 5 Nov 2019 15:40:58 +0000 (-0500) Subject: Fix: relayd: use packet sequence number for rotation position X-Git-Url: http://git.efficios.com/?p=lttng-tools.git;a=commitdiff_plain;h=0f83d1cc4a38b1c18db30ff6cd1d543401c0fc4c;hp=0f83d1cc4a38b1c18db30ff6cd1d543401c0fc4c Fix: relayd: use packet sequence number for rotation position The "network" sequence number (net_seq_num) is a 64-bit sequence number tagging each packet sent over the network. The net_seq_num increments monotonically (+1) for each packet sent from consumer daemon to relay daemon, on a per-stream basis. It is tagged by the consumer daemon when sending a trace packet to the relay daemon. The LTTng kernel and user-space ring buffer "consumed position" (consumed_pos) and "produced position" (produced_pos) are free-running counters counting the number of bytes consumed and produced so far by each stream. Because those counters are updated atomically, they are limited to a size of 32-bit on 32-bit architectures. The "packet" sequence number (packet_seq_num) is a sequence number found in the packet header starting from LTTng 2.8. It is a 64-bit sequence number assigned by the lttng-modules and lttng-ust ring buffers. It increments monotonically (+1) for each packet produced within a given ring buffer (stream). Using produced_pos as rotation position and comparing it to the net_seq_num has a few issues: 1) It breaks on 32-bit producers after generating more than 4GB of data per stream, due to overflow. The net_seq_num is a 64-bit counter, which does not overflow, but the produced_pos overflows after 4GB on 32-bit architectures. This can lead to never-completing rotations. 2) It breaks scenarios where ring buffers are configured in overwrite mode, and streaming to a relay daemon. Indeed, when the ring buffer moves the consumed_pos ahead, actually overwriting data within the ring buffer, it introduces an offset between the produced_pos and the net_seq_num. Therefore, if producers are generating a low- (or no-) throughput in some streams, the rotation may never complete, even on 64-bit architectures. The solution proposed for this issue is to use the packet_seq_num as rotation position rather than the net_seq_num. It takes care of the two problematic scenarios, since the counter is always 64-bit (even on 32-bit architectures), and because the counter is managed by the producer, which therefore tracks progress of the ring buffer overwrites. This commit introduces changes required at the relayd side. A separate commit introduces the changes required in the consumerd. In relayd, one major restriction is the fact that the packet_seq_num is not sent over the data socket, only through the control socket receiving the indexes. Therefore, in order to figure out the pivot position for the data socket for a given stream, the associated index first needs to be received. At that point, the corresponding net_seq_num is known, which provides the pivot position for the data stream. Given that the data and index sockets provide no ordering guarantees with respect to their arrival, we handle the fact that data might have been saved to disk in the wrong (previous) trace chunk by moving it to the next trace chunk when the pivot position is known. In order to allow "jumps" in the sequence numbers produced by overwrite mode buffers, try_rotate_stream_index(), which previously asserted that each sequence number was received in sequence, now uses the packet_seq_num pivot position as a lower (inclusive) bound. Signed-off-by: Mathieu Desnoyers Change-Id: I755329e313f0980655a164b7bdb57e4f3d8e944a Signed-off-by: Jérémie Galarneau ---