=== Event Source Lookup ===
-For CTF traces using specification v1.8.2 or above, information can optionally be embedded in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
+Some trace types can optionally embed information in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
==== Source Code ====
[[Image:images/kernelMemoryUsage/KernelMemoryUsageChart.png]]
+== Process Wait Analysis ==
+
+TraceCompass can recover wait causes of local and distributed processes using operating system events. The analysis highlights the tasks and devices causing wait. Wait cause recovery is recursive, comprise all tasks running on the system and works across computers using packet trace synchronization.
+
+The analysis details are available in the paper [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7294678&isnumber=4359390 Wait analysis of distributed systems using kernel tracing].
+
+=== Prerequisites ===
+
+The analysis requires a Linux kernel trace. Additional instrumentation may be required for specific kernel version and for distributed tracing. This instrumentation is available in [https://github.com/giraldeau/lttng-modules/tree/addons LTTng modules addons] on GitHub.
+
+The required events are:
+* '''sched_switch, sched_wakeup''': Scheduling events indicate when a process is blocked and the wake-up event indicates the task or resource that unblocked the task. For kernel versions comprised between 3.8 and 4.1, the event '''sched_ttwu''' (which stands for Try To Wake-Up) is provided for backward compatibility in LTTng modules addons.
+* '''IRQ, SoftIRQ and IPI''': Interrupt events are required to distinguish the context of the wake-up. When a wake-up occurs inside an interrupt handler, it must be associated with the device causing the interrupt and not the interrupted task. For that reason, interrupt entry and exit events are required.
+* '''inet_sock_local_in, inet_sock_local_out''': The network events record a subset of TCP/IP packet header using a netfilter hook in the kernel. The send and receive events are matched to show the communication between distributed processes. Network events are mandatory for analyzing wait in TCP/IP programs, whether they are executing locally or on different computers. They also used to synchronize traces recorded on multiple computers. For further details, refer to the [[#Trace synchronization]] section.
+
+To analyze a distributed program, all computers involved in the processing must be traced simultaneously. The LTTng Tracer Control of TraceCompass can trace a remote computer, but controlling simultaneous tracing is not supported at the moment, meaning that all sessions must be started separately and interactively. TraceCompass will support this feature in the future. For now, it is suggested to use [https://github.com/giraldeau/lttng-cluster lttng-cluster] command line tool to control simultaneous tracing sessions on multiple computers. This tool is based on [http://www.fabfile.org/ Fabric] and uses SSH to start the tracing sessions, execute a workload, stop the sessions and gather traces on the local computer. For more information, refer to the lttng-cluster documentation.
+
+We use the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] as an example to demonstrate the wait analysis. [https://www.djangoproject.com/ Django] is a popular Web framework. The application is the [https://docs.djangoproject.com/en/1.9/intro/tutorial01/ Django Poll app tutorial]. The traces were recorded on three computers, namely the client (implemented with Python Mechanize), the Web server (Apache with WSGI) and the database server (PostgreSQL). The client simulates a vote in the poll.
+
+=== Running the analysis ===
+
+To open all three traces simultaneously, we first create an experiment containing these traces and then synchronize the traces, such that they have a common time base. Then, the analysis is done by selecting a task in the '''Control Flow View'''. The result is displayed in the '''Critical Flow View''', which works like the '''Control Flow View'''. The steps to load the Django example follows.
+
+# Download and extract the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] archive.
+# In TraceCompass, open the [[#LTTng Kernel Perspective]].
+# Create a new tracing project. Select '''File -> New -> Tracing -> Tracing Project''', choose a name and click '''Finish'''.
+# Under the created tracing project, right-click on '''Traces''' and select '''Import...'''. In the import dialog, select the root directory containing the extracted trace by clicking on '''Browse'''. Three traces should be listed. Select the traces and click '''Finish'''. After the import is completed, the traces should be listed below '''Traces'''.
+# Right-click on '''Experiments''', select '''New...''' and enter a name for the experiment, such as '''django'''.
+# Right-click on the '''django''' experiment and click on '''Select Traces...'''. In the dialog, check the three traces '''django-client''', '''django-httpd''' and '''django-db'''. These traces will appear below the experiment. If the experiment is opened at this stage, the traces are not synchronized and there will be a large time gap between events from different traces.
+# To synchronize the traces, right-click on the '''django''' experiment and select '''Synchronize Traces'''. In the '''Select reference trace''' dialog, select any available trace and click '''Finish'''. Once the synchronization is completed, a new entry with an underline suffix will appear for each modified trace. The created trace entries have a function which is applied to the timestamps of events in order to shift the time according to the reference trace. The '''Project Explorer''' after the import is shown below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisProjectExplorer.png]]
+# Open the experiment '''django'''. The '''Control Flow''' and the '''Resources''' views should display the three traces simultaneously.
+# In the main menu, select '''Window -> Show View -> Other...''' and under '''LTTng''' select '''Critical Flow View'''. The view is empty for the moment.
+# In the '''Critical Flow View''', right-click on the '''Process''' entry to analyze and select '''Follow''', as shown in the figure below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisFollow.png]]
+#:The analysis will execute and the result will appear in the '''Critical Flow View'''. For the Django example, use the '''View Filters''' to search for the python process with TID 2327. When zooming on the execution, the view displays the work done by the Web server and the database to process the request of the python client. Vertical arrows represent synchronization and communication between processes. The legend [[Image:images/show_legend.gif]] displays the colors associated with the processes states.
+
+[[Image:images/waitAnalysis/KernelWaitAnalysisDjango.png]]
+
+== Input/Output Analysis ==
+
+TraceCompass can analyse disk input/output through the read/write system calls to get the read/write per processes, but also with the disk request events, to get the actual reads and writes to disk.
+
+=== Get the trace ===
+
+The following tracepoints should be enabled to get the disk read/write data. Also, enabling syscalls will allow to match the reads and writes per processes.
+
+ # sudo lttng list -k
+ Kernel events:
+ -------------
+ ...
+ block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ ...
+
+For full disk request tracking, some extra tracepoints are necessary. They are not required for the I/O analysis, but make the analysis more complete. Here is the procedure to get those tracepoints that are not yet part of the mainline kernel.
+
+ # git clone https://github.com/giraldeau/lttng-modules.git
+ # cd lttng-modules
+
+Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.
+
+ # git checkout addons
+ # make
+ # sudo make modules_install
+ # sudo depmod -a
+
+The lttng addons modules must be inserted manually for the extra tracepoints to be available:
+
+ # sudo modprobe lttng-addons
+ # sudo modprobe lttng-elv
+
+And enable the following tracepoint
+
+ addons_elv_merge_requests
+
+=== Input/Output Views ===
+
+The following views are available for input/output analyses:
+
+* Disk I/O Activity
+A time aligned XY chart of the read and write speed for the different disks on the system. This view is useful to see where there was more activity on the disks and whether it was mostly reads or writes.
+
+ [[Image:images/io/diskIoActivity.png| Disk I/O Activity Example]]
+
== LTTng Kernel Events Editor ==
The LTTng Kernel Events editor '''is''' the plain TMF [[#Events_Editor | Events Editor]], except that it provides its own specialized viewer to replace the standard one. In short, it has exactly the same behaviour but the layout is slightly different:
Please note this view will not show shared memory or stack memory usage.
+== Source Lookup (for LTTng-UST 2.8+) ==
+
+Starting with LTTng 2.8, the tracer can now provide enough information to
+associate trace events with their location in the original source code.
+
+To make use of this feature, first make sure your binaries are compiled with
+debug information (-g), so that the instruction pointers can be mapped to source
+code locations. This lookup is made using the ''addr2line'' command-line utility,
+which needs to be installed and on the '''$PATH''' of the system running Trace
+Compass. ''addr2line'' is available in most Linux distributions, Mac OS X, Windows using Cygwin and others.
+
+The following trace events need to be present in the trace:
+
+* lttng_ust_statedump:start
+* lttng_ust_statedump:end
+* lttng_ust_statedump:bin_info
+* lttng_ust_statedump:build_id
+
+as well as the following contexts:
+
+* vpid
+* ip
+
+For ease of use, you can simply enable all the UST events when setting up your
+session:
+
+ lttng enable-event -u -a
+ lttng add-context -u -t vpid -t ip
+
+Note that you can also create and configure your session using the [[#Control View | Control View]].
+
+If you want to track source locations in shared libraries loaded by the
+application, you also need to enable the "lttng_ust_dl:*" events, as well
+as preload the UST library providing them when running your program:
+
+ LD_PRELOAD=/path/to/liblttng-ust-dl.so ./myprogram
+
+If all the required information is present, then the ''Source Location'' column
+of the Event Table should be populated accordingly, and the ''Open Source Code''
+action should be available. Refer to the section [[#Event Source Lookup]] for
+more details.
+
+The ''Binary Location'' information should be present even if the original
+binaries are not available, since it only makes use of information found in the
+trace. A '''+''' denotes a relative address (i.e. an offset within the object
+itself), whereas a '''@''' denotes an absolute address, for
+non-position-independent objects.
+
+[[Image:images/sourceLookup/trace-with-debug-info.png]]
+
+''Example of a trace with debug info and corresponding Source Lookup information, showing a tracepoint originating from a shared library''
+
+=== Binary file location configuration ===
+
+To resolve addresses to function names and source code locations, the analysis
+makes use of the binary files (executables or shared libraries) present on the
+system. By default, it will look for the file paths as they are found in the
+trace, which means that it should work out-of-the-box if the trace was taken on
+the same machine that Trace Compass is running.
+
+It is possible to configure a ''root directory'' that will be used as a prefix
+for all file path resolutions. The button to open the configuration dialog is
+called '''Configure how addresses are mapped to function names''' and is
+currently located in the [[#Call Stack View]]. Note that the Call Stack View
+will also make use of this configuration to resolve its function names.
+
+[[Image:images/sourceLookup/symbol-mapping-config-ust28.png]]
+
+''The symbol configuration dialog for LTTng-UST 2.8+ traces''
+
+This can be useful if a trace was taken on a remote target, and an image of that
+target is available locally.
+
+If a binary file is being traced on a target, the paths in the trace will refer
+to the paths on the target. For example, if they are:
+
+* /usr/bin/program
+* /usr/lib/libsomething.so
+* /usr/local/lib/libcustom.so
+
+and an image of that target is copied locally on the system at
+''/home/user/project/image'', which means the binaries above end up at:
+
+* /home/user/project/image/usr/bin/program
+* /home/user/project/image/usr/lib/libsomething.so
+* /home/user/project/image/usr/local/lib/libcustom.so
+
+Then selecting the ''/home/user/project/image'' directory in the configuration
+dialog above will allow Trace Compass to read the debug symbols correctly.
+
+Note that this path prefix will apply to both binary file and source file
+locations, which may or may not be desirable.
+
= Trace synchronization =
It is possible to synchronize traces from different machines so that they have the same time reference. Events from the reference trace will have the same timestamps as usual, but the events from traces synchronized with the first one will have their timestamps transformed according to the formula obtained after synchronization.
<pre>
<head>
- <analysis id="org.eclipse.tracecompass.lttng2.kernel.analysis" />
+ <analysis id="org.eclipse.tracecompass.analysis.os.linux.kernel" />
<label value="CPU status XY view" />
</head>
</pre>
projects / deliverable / tracecompass.git / blobdiff