timing.core/ui: make flamegraphs aggregate threads by default

[deliverable/tracecompass.git] / doc / org.eclipse.tracecompass.doc.user / doc / User-Guide.mediawiki

= Table of Contents =

__TOC__

= Overview =

Trace Compass is a Java tool for viewing and analyzing any type of logs or traces. Its goal is to provide views, graphs, metrics, etc. to help extract useful information from traces, in a way that is more user-friendly and informative than huge text dumps.

== About Tracing ==

Tracing is a troubleshooting technique used to understand the behavior of an instrumented application by collecting information on its execution path. A tracer is the software used for tracing. Tracing can be used to troubleshoot a wide range of bugs that are otherwise extremely challenging. These include, for example, performance problems in complex parallel systems or real-time systems.

Tracing is similar to logging: it consists in recording events that happen in a system at selected execution locations. However, compared to logging, it is generally aimed at developers and it usually records low-level events at a high rate. Tracers can typically generate thousands of events per second. The generated traces can easily contain millions of events and have sizes from many megabytes to tens of gigabytes. Tracers must therefore be optimized to handle a lot of data while having a small impact on the system.

Traces may include events from the operating system kernel (IRQ handler entry/exit, system call entry/exit, scheduling activity, network activity, etc). They can also consists of application events (a.k.a UST - User Space Tracing) or a mix of the two.

For the maximum level of detail, tracing events may be viewed like a log file. However, trace analyzers and viewers are available to derive useful information from the raw data coupled with knowledge of the traced program. These programs must be specially designed to handle quickly the enormous amount of data a trace may contain.

== Features ==

Trace Compass has a number of features to allow efficient handling of very large traces (and sets of large traces):

* Support for arbitrarily large traces (larger than available memory)
* Support for correlating multiple time-ordered traces
* Support for zooming down to the nanosecond on any part of a trace or set of traces
* Views synchronization of currently selected time or time range, and window time range
* Efficient searching and filtering of events
* Support for trace bookmarks
* Support for importing and exporting trace packages

There is also support for the integration of non-LTTng trace types:

* Built-in CTF parser
* Dynamic creation of customized parsers (for XML and text traces)
* Dynamic creation of customized state systems (from XML files)
* Dynamic creation of customized views (from XML files)

Trace Compass provides the following main views:

* ''Project Explorer'' - an extension to the standard Eclipse Project view tailored for tracing projects
* ''Events'' - a versatile view that presents the raw events in tabular format with support for searching, filtering and bookmarking
* ''Statistics'' - a view that that provides simple statistics on event occurrences by type
* ''Histogram'' - a view that displays the event density with respect to time in traces

These views can be extended or tailored for specific trace types (e.g. kernel, HW, user app).

== LTTng integration ==

One of the main features of Trace Compass is the LTTng integration. LTTng (Linux Trace Toolkit, next generation) is a highly efficient tracing tool for Linux that can be used to track down kernel and application performance issues as well as troubleshoot problems involving multiple concurrent processes and threads. It consists of a set of kernel modules, daemons - to collect the raw tracing data - and a set of tools to control, visualize and analyze the generated data. It also provides support for user space application instrumentation.
For more information about LTTng, refer to the project [http://lttng.org site]

'''Note''': This User Guide covers the integration of the latest LTTng (up to v2.4) in Eclipse.

The LTTng plug-ins provide an integration for the control of the LTTng tracer as well as fetching and visualization of the traces produced. It also provides the foundation for user-defined analysis tools.

At present, the LTTng plug-ins support the following kernel-oriented views:

* ''Control Flow'' - to visualize processes state transitions
* ''Resources'' - to visualize system resources state transitions
* ''CPU Usage'' - to visualize the usage of the processor with respect to the time in traces
* ''Kernel Memory Usage'' - to visualize the relative usage of system memory
* ''IO Usage'' - to visualize the usage of input/output devices
* ''System Calls'' - presents all the system calls in a table view
* ''System Call Statistics'' - present all the system calls statistics
* ''System Call Density'' - to visualize the system calls displayed by duration
* ''System Call vs Time'' - to visualize when system calls occur

Also, the LTTng plug-ins supports the following User Space traces views:

* ''Memory Usage'' - to visualize the memory usage per thread with respect to time in the traces
* ''Call Stack'' - to visualize the call stack's evolution over time
* ''Function Duration Density'' - to visualize function calls displayed by duration
* ''Flame Graph'' - to visualize why the CPU is busy

Finally, the LTTng plug-ins supports the following Control views:
* ''Control'' - to control the tracer and configure the tracepoints

Although the control and fetching parts are targeted at the LTTng tracer, the underlying framework can also be used to process any trace that complies with the ''Common Trace Format'' ([http://www.efficios.com/ctf CTF]). CTF specifies a very efficient and compact binary trace format that is meant to be application-, architecture-, and language-agnostic.

= Installation =

This section describes the installation of the LTTng tracer and the Trace Compass plug-ins as well as their dependencies.

== LTTng Tracer ==

While the Eclipse plug-ins can run on the standard Eclipse platforms (Linux, Mac, Windows), the LTTng tracer and its accompanying tools run on Linux.

The tracer and tools have been available for download in Ubuntu since 12.04. They can easily be installed with the following command:

<pre>
  > sudo apt-get install lttng-tools
</pre>

For other distributions, older Ubuntu distributions, or the latest, bleeding edge LTTng tracer, please refer to the [http://lttng.org/download LTTng website] for installation information.

'''Note''': The LTTng tracer (and accompanying tools) is required only if you want to create your own traces (the usual case). If you intend to simply analyze existing traces then it is not necessary to install the tracer.

== Trace Compass Plug-ins ==

The easiest way to install the Trace Compass plug-ins for Eclipse is through the Install New Software menu. For information on how to use this menu, refer to this [http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.platform.doc.user%2Ftasks%2Ftasks-124.htm link].

The Trace Compass main plug-ins are structured as a stack of features/plug-ins as following:

* '''CTF''' - A CTF parser that can also be used as a standalone component
** ''Feature'': org.eclipse.tracecompass.ctf
** ''Plug-ins'': org.eclipse.tracecompass.ctf.core, org.eclipse.tracecompass.ctf.parser

* '''State System Core''' - State system for TMF
** ''Plug-ins'': org.eclipse.tracecompass.statesystem.core

* '''TMF''' - ''Tracing and Monitoring Framework'' a framework for generic trace processing
** ''Feature'': org.eclipse.tracecompass.tmf
** ''Plug-ins'': org.eclipse.tracecompass.tmf.core, org.eclipse.tracecompass.tmf.ui. org.eclipse.tracecompass.tmf.analysis.xml.core, org.eclipse.tracecompass.tmf.analysis.xml.ui

* '''CTF support for TMF''' - CTF support for the TMF Feature
** ''Feature'': org.eclipse.tracecompass.tmf.ctf
** ''Plug-ins'': org.eclipse.tracecompass.tmf.ctf.core

* '''LTTng Control''' - The wrapper for the LTTng tracer control. Can be used for kernel or application tracing.
** ''Feature'': org.eclipse.tracecompass.lttng2.control
** ''Plug-ins'': org.eclipse.tracecompass.lttng2.control.core, org.eclipse.tracecompass.lttng2.control.ui

* '''LTTng Kernel''' - Analysis components specific to Linux kernel traces
** ''Feature'': org.eclipse.tracecompass.lttng2.kernel
** ''Plug-ins'': org.eclipse.tracecompass.analysis.os.linux.core, org.eclipse.tracecompass.analysis.os.linux.ui, org.eclipse.tracecompass.lttng2.kernel.core, org.eclipse.tracecompass.lttng2.kernel.ui

* '''LTTng UST''' - Analysis components specific to Linux userspace traces
** ''Feature'': org.eclipse.tracecompass.lttng2.ust
** ''Plug-ins'': org.eclipse.tracecompass.lttng2.ust.core, org.eclipse.tracecompass.lttng2.ust.ui

== LTTng Control Dependencies ==

The Eclipse LTTng Control feature controls the LTTng tracer through an ''ssh'' connection, if the tracer is running locally it can use or bypass the ''ssh'' connection.

When using ''ssh'', the target system (where the tracer runs) needs to run an ''ssh'' server as well as ''sftp'' server (for file transfer) to which you have permission to connect.

On the host side (where Eclipse is running), you also need to have Eclipse Remote Services installed to handle the SSH connection and transport. The Remote Services are installed for you as a dependency of the LTTng Control feature. If necessary, it can be installed manually with the standard way (''Help'' > ''Install New Software...'' > ''General Purpose Tools'' > ''Remote Services'').

== Installation Verification ==

If you do not have any traces, sample LTTng traces can be found here [http://lttng.org/files/samples]. This page contains links to some sample LTTng 2.0 kernel traces. The trace needs to be uncompressed to be opened. The traces can also be imported directly as archive, see the [[#Importing|Importing]] section for more detail.

Here are the quick steps to verify that your installation is functional using a LTTng trace:

* Start Eclipse
* Open the LTTng Kernel perspective
* Create a Tracing project
** Right-click in the Project Explorer view and select New, Tracing Project
** Enter the name of your project (e.g. "MyLTTngProject")
** The project will be created. It will contain 2 empty folders: "Traces" and "Experiments"
* Open and visualize a sample trace
** Right-click on the newly created project "Traces" folder and select "Open Trace..."
** Navigate to the sample LTTng trace that you want to visualize and select any file in the trace folder
** The newly imported trace should appear under the Traces folder
** The trace should load and the views be populated

If an error message is displayed, you might want to double-check that the trace type is correctly set (right-click on the trace and "Select Trace Type...").

Refer to [[#Tracing Perspective]] for detailed description of the views and their usage.

= Trace Compass Main Features =

== Tracing Perspective ==

The '''Tracing''' perspective is part of the  '''Tracing and Monitoring Framework (TMF)''' and groups the following views:

* [[#Project Explorer_View | Project Explorer View]]
* [[#Events_Editor | Events Editor]]
* [[#Histogram_View | Histogram View]]
* [[#Statistics_View   | Statistics View]]

The views are synchronized i.e. selecting an event, a timestamp, a time range, etc will update the other views accordingly.

[[Image:images/TracingPerspective.png]]

The perspective can be opened from the Eclipse Open Perspective dialog ('''Window > Open Perspective... > Other''').

[[Image:images/ShowTracingPerspective.png]]

In addition to these views, the '''Tracing and Monitoring Framework (TMF)''' feature provides a set of generic tracing specific views, such as:

* [[#Colors_View | Colors View]]
* [[#Filters_View | Filters View]]
* [[#Time_Chart_View  | Time Chart View]]
* [[#State_System_Explorer_View | State System Explorer View]]
* [[#Call_Stack_View | Call Stack View]]

The framework also supports user creation of [[#Custom_Parsers  | Custom Parsers]].

To open one of the above '''Tracing''' views, use the Eclipse Show View dialog ('''Window > Show View > Other...'''). Then select the relevant view from the '''Tracing''' category.

[[Image:images/ShowTracingViews.png]]

Additionally, the '''LTTng Control''' feature provides an '''LTTng Tracer Control''' functionality. It comes with a dedicated '''Control View'''.

* [[#LTTng_Tracer_Control | LTTng Tracer Control]]

== Project Explorer View ==

The Project Explorer view is the standard Eclipse Project Explorer. '''Tracing''' projects are well integrated in the Eclipse's Common Navigator Framework. The Project Explorer shows '''Tracing''' project with a small "T" decorator in the upper right of the project folder icon.

=== Creating a Tracing Project ===

A new '''Tracing''' project can be created using the New Tracing Project wizard. To create a new '''Tracing'''  select '''File > New > Project...''' from the main menu bar or alternatively form the context-sensitive menu (click with right mouse button in the '''Project Explorer'''.

The first page of project wizard will open.

[[Image:images/NewTracingProjectPage1.png]]

In the list of project categories, expand category '''Tracing''' and select '''Tracing Project''' and the click on '''Next >'''. A second page of the wizard will show. Now enter the a name in the field '''Project Name''', select a location if required and the press on '''Finish'''.

[[Image:images/NewTracingProjectPage2.png]]

A new project will appear in the '''Project Explorer''' view.

[[Image:images/NewProjectExplorer.png]]

Tracing projects have two sub-folders: '''Traces''' which holds the individual traces, and '''Experiments''' which holds sets of traces that we want to correlate.

=== Importing Traces to the Project ===

The '''Traces''' folder holds the set of traces available for a tracing project. It can optionally contain a tree of trace folders to organize traces into sub-folders. The following chapters will explain different ways to import traces to the '''Traces''' folder of a tracing project.

* [[#Opening a Trace | Opening a Trace]]
* [[#Importing | Importing]]
* [[#Drag and Drop | Drag and Drop]]

==== Opening a Trace ====

To open a trace, right-click on a target trace folder and select '''Open Trace...'''.

[[Image:images/OpenTraceFile.png]]

A new dialog will show for selecting a trace to open. Select a trace file and then click on '''OK'''. Note that for traces that are directories (such as Common Trace Format (CTF) traces) any file in the trace directory can be selected to open the trace. Now, the trace viewer will attempt to detect the trace types of the selected trace. The auto detection algorithm will validate the trace against all known trace types. If multiple trace types are valid, a trace type is chosen based on a confidence criteria. The validation process and the computation of the confidence level are trace type specific. After successful validation the trace will be linked into the selected target trace folder and then opened with the detected trace type.

Depending of the trace types enabled in the [[#Trace_Types_Preference_Page | Trace Types preference page]], the list of available trace types can vary.

==== Importing ====

To import a set of traces to a trace folder, right-click on the target folder and select '''Import...''' from the context-sensitive menu.

[[Image:images/ProjectImportTraceAction.png]]

At this point, the '''Import Trace Wizard''' will show for selecting traces to import. By default, it shows the correct destination directory where the traces will be imported to. Now, specify the location of the traces in the '''Root directory'''. For that click on the button '''Browse''', browse the media to the location of the traces and click on '''OK'''. Then select the traces to import in the list of files and folders. If the selected files include archive files (tar, zip), they will be extracted automatically and imported as well.

Traces can also be imported directly from an archive file such as a zip or a tar file by selecting the '''Select archive file''' option then by clicking '''Browse'''. Then select the traces to import in the list of files and folders as usual.

Optionally, select the '''Trace Type''' from the drop-down menu. If '''Trace Type''' is set to '''<Automatic Detection>''', the wizard will attempt to detect the trace types of the selected files. The automatic detection algorithm validates a trace against all the trace types enabled in the [[#Trace_Types_Preference_Page | Trace Types preference page]]. If multiple trace types are valid, a trace type is chosen based on a confidence criteria. The validation process and the computation of the confidence level are trace type specific. Optionally, '''Import unrecognized traces''' can be selected to import trace files that could not be automatically detected by '''<Automatic Detection>'''.

Select or deselect the checkboxes for '''Overwrite existing trace without warning''', '''Create links in workspace''' and '''Preserve folder structure'''. When all options are configured, click on '''Finish'''.

Note that traces of certain types (e.g. LTTng Kernel) are actually a composite of multiple channel traces grouped under a folder. Either the folder or its files can be selected to import the trace.

The option '''Preserve folder structure''' will create, if necessary, the structure of folders relative to (and excluding) the selected '''Root directory''' (or '''Archive file''') into the target trace folder.

The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the '''Root directory''' name, when importing from directory, or the ''' Archive file''' name, when importing from archive. One can change the experiment name by typing a new name in the text box beside the option.

[[Image:images/ProjectImportTraceDialog.png]]

If a trace already exists with the same name in the target trace folder, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example smalltrace becomes smalltrace(2).

[[Image:images/ProjectImportTraceDialogRename.png]]

If one selects '''Rename All''', '''Overwrite All''' or '''Skip All''' the choice will be applied for all traces with a name conflict.

Upon successful importing, the traces will be stored in the target trace folder. If a trace type was associated to a trace, then the corresponding icon will be displayed. If no trace type is detected the default editor icon associated with this file type will be displayed. Linked traces will have a little arrow as decorator on the right bottom corner.

Depending of the trace types enabled in the [[#Trace_Types_Preference_Page | Trace Types preference page]], the list of available trace types can vary.

Alternatively, one can open the '''Import...''' menu from the '''File''' main menu, then select '''Tracing''' > '''Trace Import''' and click on '''Next >'''.

[[Image:images/ProjectImportWizardSelect.png]]

At this point, the '''Import Trace Wizard''' will show. To import traces to the tracing project, follow the instructions that were described above.

==== Drag and Drop ====

Traces can be also be imported to a project by dragging from another tracing project and dropping to the project's target trace folder. The trace will be copied and the trace type will be set.

Any resource can be dragged and dropped from a non-tracing project, and any file or folder can be dragged from an external tool, into a tracing project's trace folder. The resource will be copied or imported as a new trace and it will be attempted to detect the trace types of the imported resource. The automatic detection algorithm validates a trace against all known trace types. If multiple trace types are valid, a trace type is chosen based on a confidence criteria. The validation process and the computation of the confidence level are trace type specific. If no trace type is detected the user needs to set the trace type manually.

To import the trace as a link, use the platform-specific key modifier while dragging the source trace. A link will be created in the target project to the trace's location on the file system.

If a folder containing traces is dropped on a trace folder, the full directory structure will be copied or linked to the target trace folder. The trace type of the contained traces will not be auto-detected.

It is also possible to drop a trace, resource, file or folder into an existing experiment. If the item does not already exist as a trace in the project's trace folder, it will first be copied or imported, then the trace will be added to the experiment.

=== Trace Package Exporting and Importing ===

A trace package is an archive file that contains the trace itself and can also contain its bookmarks and its supplementary files. Including supplementary files in the package can improve performance of opening an imported trace but at the expense of package size.

==== Exporting ====

The '''Export Trace Package Wizard''' allows users to select a trace and export its files and bookmarks to an archive on a media.

The '''Traces''' folder holds the set of traces available for a tracing project. To export traces contained in the '''Traces''' folder, one can open the '''Export...''' menu from the '''File''' main menu. Then select '''Trace Package Export'''

[[Image:images/tracePackageImages/fileExport.png]]

At this point, the '''Trace Package Export''' is opened. The project containing the traces has to be selected first then the traces to be exported.

[[Image:images/tracePackageImages/chooseTrace.png]]

One can also open the wizard and skip the first page by expanding the project, selecting traces or trace folders under the '''Traces''' folder, then right-clicking and selecting the '''Export Trace Package...''' menu item in the context-sensitive menu.

[[Image:images/tracePackageImages/exportSelectedTrace.png]]

Next, the user can choose the content to export and various format options for the resulting file.

[[Image:images/tracePackageImages/exportPackage.png]]

The '''Trace''' item is always selected and represents the files that constitute the trace. The '''Supplementary files''' items represent files that are typically generated when a trace is opened by the viewer. Sharing these files can speed up opening a trace dramatically but also increases the size of the exported archive file. The ''Size'' column can help to decide whether or not to include these files. Lastly, by selecting '''Bookmarks''', the user can export all the bookmarks so that they can be shared along with the trace.

The '''To archive file''' field is used to specify the location where to save the resulting archive.

The '''Options''' section allows the user to choose between a tar archive or a zip archive. Compression can also be toggled on or off.

When Finish button is clicked, the package is generated and saved to the media. The folder structure of the selected traces relative to the '''Traces''' folder is preserved in the trace package.

==== Importing ====

The '''Import Trace Package Wizard''' allows users to select a previously exported trace package from their media and import the content of the package in the workspace.

The '''Traces''' folder holds the set of traces for a tracing project. To import a trace package to the '''Traces''' folder, one can open the '''Import...''' menu from the '''File''' main menu. Then select '''Trace Package Import'''.

[[Image:images/tracePackageImages/fileImport.png]]

One can also open the wizard by expanding the project name, right-clicking on a target folder under the '''Traces''' folder then selecting '''Import Trace Package...''' menu item in the context-sensitive menu.

[[Image:images/tracePackageImages/importTraceFolder.png]]

At this point, the '''Trace Package Import Wizard''' is opened.

[[Image:images/tracePackageImages/importPackage.png]]

The '''From archive file''' field is used to specify the location of the trace package to export. The user can choose the content to import in the tree.

If the wizard was opened using the File menu, the destination project has to be selected in the '''Into project''' field.

When Finish is clicked, the trace is imported in the target folder. The folder structure from the trace package is restored in the target folder.

=== Refreshing of Trace and Trace Folder ===
Traces and trace folders in the workspace might be updated on the media. To refresh the content, right-click the trace or trace folder and select menu item '''Refresh'''. Alternatively, select the trace or trace folder and press key '''F5'''.

=== Remote Fetching ===

It is possible to import traces automatically from one or more remote hosts according to a predefined remote profile by using the '''Fetch Remote Traces''' wizard.

To start the wizard, right-click on a target trace folder and select '''Fetch Remote Traces...'''.

[[Image:images/FetchRemoteTracesMenu.png]]

The wizard opens on the '''Remote Profile''' page.

[[Image:images/RemoteProfileWizardPageBlank.png]]

If the remote profile already exists, it can be selected in the '''Profile name''' combo box. Otherwise, click '''Manage Profiles''' to open the '''Remote Profiles''' preferences page.

==== Remote Profile elements ====

[[Image:images/RemoteProfilesPreferencesPage.png]]

Click '''Add''' to create a new remote profile. A default remote profile template appears.

[[Image:images/RemoteProfilesPreferencesPageDefault.png]]

===== Profile =====

Edit the '''Profile name''' field to give a unique name to the new profile.

Under the Profile element, at least one Connection Node element must be defined.

===== Connection Node =====

'''Node name''': Unique name for the connection within the scope of the Remote Services provider.

'''URI''': URI for the connection. Its scheme maps to a particular Remote Services provider. If the connection name already exists for that provider, the URI must match its connection information. The scheme '''ssh''' can be used for the Built-In SSH provider. The scheme '''file''' can be used for the local file system.

To view or edit existing connections, see the '''Remote Development''' > '''Remote Connections''' preferences page. On this page the user can enter a password for the connection.

Under the Connection Node element, at least one Trace Group element must be defined.

===== Trace Group =====

'''Root path''': The absolute root path from where traces will be fetched. For example, ''/home/user'' or ''/C/Users/user''.

'''Recursive''': Check this box to search for traces recursively in the root path.

Under the Trace Group element, at least one Trace element must be defined.

===== Trace =====

'''File pattern''': A regular expression pattern to match against the file name of traces found under the root path. If the '''Recursive''' option is used, the pattern must match against the relative path of the trace, using forward-slash as a path separator. Files that do not match this pattern are ignored. If multiple Trace elements have a matching pattern, the first matching element will be used, and therefore the most specific patterns should be listed first. Following are some pattern examples:

* <pre><nowiki>.*</nowiki></pre> matches any trace in any folder
* <pre><nowiki>[^/]*\.log</nowiki></pre> matches traces with .log extension in the root path folder
* <pre><nowiki>.*\.log</nowiki></pre> matches traces with .log extension in any folder
* <pre><nowiki>folder-[^/]*/[^/]*\.log</nowiki></pre> matches traces with .log extension in folders matching a pattern
* <pre><nowiki>(.*/)?filename</nowiki></pre> matches traces with a specific name in any folder

'''Trace Type''': The trace type to assign to the traces after fetching, or '''<Automatic Detection>''' to determine the trace type automatically. Note that traces whose trace type can not be assigned according to this setting are not deleted after fetching.

==== Profile editing and management ====

Right-click a profile element to bring up its context menu. A '''New''' child element of the appropriate type can be created. Select '''Delete''' to delete a node, or '''Cut''', '''Copy''' and '''Paste''' to move or copy elements from one profile element to another. The keyboard shortcuts can also be used.

Press the '''Add''' button to create a new element of the same type and following the selected element, or a new profile if the selection is empty.

Press the '''Remove''' button to delete the selected profile elements.

Press the '''Import''' button to import profiles from a previously exported XML file.

Press the '''Export''' button to export the selected profiles to an XML file.

Press the '''Move Up''' or '''Move Down''' buttons to reorder the selected profile element.

The filter text box can be used to filter profiles based on the profile name or connection node.

When the remote profile information is valid and complete, press the '''OK''' button to save the remote profiles preferences.

[[Image:images/RemoteProfilesPreferencesPageFull.png]]

==== Selecting remote traces ====

Back in the '''Remote Profiles''' wizard page, select the desired profile and click '''Next >'''. Clicking '''Finish''' at this point will automatically select and download all matching traces.

[[Image:images/RemoteProfileWizardPageNext.png]]

If required, the selected remote connections are created and connection is established. The user may be prompted for a password. This can be avoided by storing the password for the connection in the '''Remote Connections''' preference page.

[[Image:images/FetchRemoteTracesPassword.png]]

The root path of every Trace Group is scanned for matching files. The result is shown in the '''Remote Traces''' wizard page.

[[Image:images/RemoteTracesWizardPage.png]]

Select the traces to fetch by checking or unchecking the desired connection node, trace group, folder or individual trace. Click '''Finish''' to complete the operation.

If any name conflict occurs, the user will be prompted to rename, overwrite or skip the trace, unless the '''Overwrite existing trace without warning''' option was checked in the '''Remote Profiles''' wizard page.

The downloaded traces will be imported to the initially selected project folder. They will be stored under a folder structure with the pattern ''<connection name>/<path>/<trace name>'' where the path is the trace's remote path relative to its trace group's root path.

[[Image:images/FetchRemoteTracesProject.png]]

=== Selecting a Trace Type ===

If no trace type was selected a trace type has to be associated to a trace before it can be opened. To select a trace type select the relevant trace and click the right mouse button. In the context-sensitive menu, select '''Select Trace Type...''' menu item. A sub-menu will show will all available trace type categories. From the relevant category select the required trace type. The examples, below show how to select the '''Common Trace Format''' types '''Linux Kernel Trace''' and '''Generic CTF trace'''.

[[Image:images/SelectLTTngKernelTraceType.png]]

[[Image:images/SelectGenericCTFTraceType.png]]

After selecting the trace type, the trace icon will be updated with the corresponding trace type icon.

[[Image:images/ExplorerWithAssociatedTraceType.png]]

The user can edit the [[#Trace_Types_Preference_Page | Trace Types preference page]] to choose which trace types will be available under the '''Select Trace Type...''' menu.

=== Trace Types Preference Page ===
The '''Trace Types''' preference page lists all the available trace types and allows the user to enable/disable the trace types.

Trace type is an extension point of the '''Tracing and Monitoring Framework (TMF)'''. Depending on which features are loaded and which custom parsers have been created, the list of trace types can vary.

The user needs to press the '''Apply''' button or '''OK'''' to save the changes.

Only the enabled trace types will be used for trace importing, auto-detection, as well as shown under the [[#Selecting_a_Trace_Type | Select Trace Type...]] menu.

To access the '''Trace Types''' preference page, select the '''Window''' menu. Then Select '''Preferences''' and search for '''Trace Type''' under '''Tracing''' group. The example below shows the '''Trace Types''' preference page.

[[Image:images/TraceTypePreferencePage.png]]

=== Opening a Trace or Experiment ===

A trace or experiment can be opened by double-clicking the left mouse button on the trace or experiment in the '''Project Explorer''' view. Alternatively, select the trace or experiment in the in the '''Project Explorer''' view and click the right mouse button. Then select '''Open''' menu item of the context-sensitive menu. If there is no trace type set for a file resource then the file will be opened in the default editor associated with this file type.

[[Image:images/OpenTraceAction.png]]

If there is a default perspective associated with the opened trace or experiment and it is not the active perspective, the user will be prompted to switch to this perspective. The user can choose to remember this decision. The user preference can be later updated in the '''Perspectives''' preference page. Select '''Window > Preferences''' in the main menu, then select '''Tracing > Perspectives''' in the tree, and choose one of the options under '''Open the associated perspective when a trace is opened'''..

When opening a trace or experiment, all currently opened views which are relevant for the corresponding trace type will be updated.

If a trace resource is a file (and not a directory), then the '''Open With''' menu item is available in the context-sensitive menu and can be used to open the trace source file with any applicable internal or external editor. In that case the trace will not be processed by the tracing application.

=== Creating a Experiment ===

An experiment consists in an arbitrary number of aggregated traces for purpose of correlation. In the degenerate case, an experiment can consist of a single trace. The experiment provides a unified, time-ordered stream of the individual trace events.

To create an experiment, select the folder '''Experiments''' and click the right mouse button. Then select '''New...'''.

[[Image:images/NewExperimentAction.png]]

A new display will open for entering the experiment name. Type the name of the experiment in the text field '''Experiment Name''' and the click on '''OK'''.

[[Image:images/NewExperimentDialog.png]]

=== Selecting Traces for an Experiment ===

After creating an experiment, traces need to be added to the experiment. To select traces for an experiment select the newly create experiment and click the right mouse button. Select '''Select Traces...''' from the context sensitive menu.

[[Image:images/SelectTracesAction.png]]

A new dialog box will open with a list of available traces. The filter text box can be used to quickly find traces. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Select the traces to add from the list and then click on '''Finish'''.

[[Image:images/SelectTracesDialog.png]]

Now the selected traces will be linked to the experiment and will be shown under the '''Experiments''' folder.

[[Image:images/ExplorerWithExperiment.png]]

Alternatively, traces can be added to an experiment using [[#Drag_and_Drop | Drag and Drop]].

=== Removing Traces from an Experiment ===

To remove one or more traces for an experiment select the trace(s) to remove under the Experiment folder and click the right mouse button. Select '''Remove''' from the context sensitive menu.

[[Image:images/RemoveTracesAction.png]]

After that the selected trace(s) are removed from the experiment. Note that the traces are still in the '''Traces''' folder.

=== Renaming a Trace or Experiment ===

Traces and Experiment can be renamed from the '''Project Explorer''' view. To rename a trace or experiment select the relevant trace and click the right mouse button. Then select '''Rename...''' from the context sensitive menu. The trace or experiment needs to be closed in order to do this operation.

[[Image:images/RenameTraceAction.png]]

A new dialog box will show for entering a new name. Enter a new trace or experiment name respectively in the relevant text field and click on '''OK'''. If the new name already exists the dialog box will show an error and a different name has to be entered.

[[Image:images/RenameTraceDialog.png]]

[[Image:images/RenameExperimentDialog.png]]

After successful renaming the new name will show in the '''Project Explorer'''. In case of a trace all reference links to that trace will be updated too. Note that linked traces only changes the display name, the underlying trace resource will stay the original name.

Note that all supplementary files will be also handled accordingly (see also [[#Deleting Supplementary Files | Deleting Supplementary Files]]).

=== Copying a Trace or Experiment ===

To copy a trace or experiment select the relevant trace or experiment in the '''Project Explorer''' view and click the right mouse button. Then select '''Copy...''' from the context sensitive menu.

[[Image:images/CopyTraceAction.png]]

A new dialog box will show for entering a new name. Enter a new trace or experiment name respectively in the relevant text field and click on '''OK'''. If the new name already exists the dialog box will show an error and a different name has to be entered.

[[Image:images/CopyTraceDialog.png]]

[[Image:images/CopyExperimentDialog.png]]

After successful copy operation the new trace or experiment respectively will show in the '''Project Explorer'''. In case of a linked trace, the copied trace will be a link to the original trace too.

Note that the directory for all supplementary files will be copied, too. (see also [[#Deleting Supplementary Files | Deleting Supplementary Files]]).

=== Deleting a Trace or Experiment ===

To delete a trace or experiment select the relevant trace or experiment in the '''Project Explorer''' view and click the right mouse button. Then select '''Delete...''' from the context sensitive menu. The trace or experiment needs to be closed in order to do this operation.

[[Image:images/DeleteExperimentAction.png]]

A confirmation dialog box will open. To perform the deletion press '''OK''' otherwise select '''Cancel'''.

[[Image:images/DeleteExperimentConfirmationDialog.png]]

After successful operation the selected trace or experiment will be removed from the project. In case of a linked trace only the link will be removed. The actual trace resource remain on the disk.

Note that the directory for all supplementary files will be deleted, too. (see also [[#Deleting Supplementary Files | Deleting Supplementary Files]]).

=== Deleting Supplementary Files ===

Supplementary files are by definition trace specific files that accompany a trace. These file could be temporary files, persistent indexes or any other persistent data files created by the LTTng integration in Eclipse during parsing a trace. For the LTTng 2.0 trace viewer a persistent state history of the Linux Kernel is created and is stored under the name '''stateHistory.ht'''. The statistics for all traces are stored under '''statistics.ht'''. Other state systems may appear in the same folder as more custom views are added.

All supplementary file are hidden from the user and are handled internally by the TMF. However, there is a possibility to delete the supplementary files so that there are recreated when opening a trace.

To delete all supplementary files from one or many traces and experiments, select the relevant traces and experiments in the '''Project Explorer''' view and click the right mouse button. Then select the '''Delete Supplementary Files...''' menu item from the context-sensitive menu.

[[Image:images/DeleteSupplementaryFilesAction.png]]

A new dialog box will open with a list of supplementary files, grouped under the trace or experiment they belong to. Select the file(s) to delete from the list and press '''OK'''. The traces and experiments that need to be closed in order to do this operation will automatically be closed.

[[Image:images/DeleteSupplementaryFilesDialog.png]]

=== Displaying the trace's time range ===

The trace's time range can be displayed alongside the trace's name. The time is displayed in the Time Format from Tracing preferences.

To enable this feature, head to '''Preferences > Tracing''' and check the box '''Show trace time range in Project Explorer'''.

[[Image:images/ProjectExplorerDisplayRangePreferences.png]]

If a trace is empty or its type unknown, nothing will be shown. It the range has not been fully read from the trace or the supplementary files, '''[...]''' will be shown. If the trace is being read and only its start time '''start''' is known, '''[start - ...]''' will be shown. Finally, when the end time '''end''' is also known, '''[start - end]''' will be shown.

[[Image:images/ProjectExplorerDisplayRange.png]]

=== Link with Editor ===

The tracing projects support the feature '''Link With Editor''' of the Project Explorer view. With this feature it is now possible to<br/>
* select a trace element in the Project Explorer view and the corresponding [[#Events Editor | Events Editor]] will get focus if the relevant trace is open.
* select an [[#Events Editor | Events Editor]] and the corresponding trace element will be highlighted in the Project Explorer view.

To enable or disable this feature toggle the '''Link With Editor''' button of the Project Explorer view as shown below.

[[Image:images/TMF_LinkWithEditor.png]]

== Events Editor ==

The Events editor shows the basic trace data elements (events) in a tabular format. The editors can be dragged in the editor area so that several traces may be shown side by side. These traces are synchronized by timestamp.

[[Image:images/LTTng2EventsEditor.png]]

The header displays the current trace (or experiment) name.

The columns of the table are defined by the fields (aspects) of the specific trace type. These are the defaults:

* '''Timestamp''': the event timestamp
* '''Event Type''': the event type
* '''Contents''': the fields (or payload) of this event

The first row of the table is the header row a.k.a. the Search and Filter row.

The highlighted event is the ''current event'' and is synchronized with the other views. If you select another event, the other views will be updated accordingly. The properties view will display a more detailed view of the selected event.

An event range can be selected by holding the '''Shift''' key while clicking another event or using any of the cursor keys ('''Up'''', '''Down''', '''PageUp''', '''PageDown''', '''Home''', '''End'''). The first and last events in the selection will be used to determine the current selected time range for synchronization with the other views.

[[Image:images/LTTng2EventProperties.png]]

The Events editor can be closed, disposing a trace. When this is done, all the views displaying the information will be updated with the trace data of the next event editor tab. If all the editor tabs are closed, then the views will display their empty states.

Column order and size is preserved when changed. If a column is lost due to it being resized to 0 pixels, right click on the context menu and select '''Show All''', it will be restored to a visible size.

=== Searching and Filtering ===

Searching and filtering of events in the table can be performed by entering matching conditions in one or multiple columns in the header row (the first row below the column header).

To apply a matching condition to a specific column, click on the column's header row cell, type in a [http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html regular expression]. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.

Press the '''Enter''' key to apply the condition as a search condition. It will be added to any existing search conditions.

Press the '''Ctrl+Enter''' key to immediately add the condition (and any other existing search conditions) as a filter instead.

When matching conditions are applied to two or more columns, all conditions must be met for the event to match (i.e. 'and' behavior).

A preset filter created in the [[#Filters_View | Filters]] view can also be applied by right-clicking on the table and selecting '''Apply preset filter...''' > ''filter name''

==== Searching ====

When a searching condition is applied to the header row, the table will select the next matching event starting from the top currently displayed event. Wrapping will occur if there is no match until the end of the trace.

All matching events will have a 'search match' icon in their left margin. Non-matching events will be dimmed. The characters in each column which match the regular expression will be highlighted.

[[Image:images/TraceEditor-Search.png]]

Pressing the '''Enter''' key will search and select the next matching event. Pressing the '''Shift+Enter''' key will search and select the previous matching event. Wrapping will occur in both directions.

Press '''Esc''' to cancel an ongoing search.

To add the currently applied search condition(s) as filter(s), click the '''Add as Filter''' [[Image:images/filter_add.gif]] button in the header row margin, or press the '''Ctrl+Enter''' key.

Press '''Delete''' to clear the header row and reset all events to normal.

==== Filtering ====

When a new filter is applied, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace. The characters in each column which match the regular expression will be highlighted.

A status row will be displayed before and after the matching events, dynamically showing how many matching events were found and how many events were processed so far. Once the filtering is completed, the status row icon in the left margin will change from a 'stop' to a 'filter' icon.

[[Image:images/TraceEditor-Filter.png]]

Press '''Esc''' to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.

The header bar will be displayed above the table and will show a label for each applied filter. Clicking on a label will highlight the matching strings in the events that correspond to this filter condition. Pressing the '''Delete''' key will clear this highlighting.

To remove a specific filter, click on the [[Image:images/delete_button.gif]] icon on its label in the header bar. The table will be updated with the events matching the remaining filters.

The header bar can be collapsed and expanded by clicking on the [[Image:images/expanded_ovr.gif]][[Image:images/collapsed_ovr.gif]] icons in the top-left corner or on its background. In collapsed mode, a minimized version of the filter labels will be shown that can also be used to highlight or remove the corresponding filter.

Right-click on the table and select '''Clear Filters''' from the context menu to remove all filters. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filters are removed.

You can also search on the subset of filtered events by entering a search condition in the header row while a filter is applied. Searching and filtering conditions are independent of each other.

==== Bookmarking ====

Any event of interest can be tagged with a bookmark.

To add a bookmark, double-click the left margin next to an event, or right-click the margin and select '''Add bookmark...'''. Alternatively use the '''Edit''' > '''Add bookmark...''' menu. Edit the bookmark description as desired and press '''OK'''.

The bookmark will be displayed in the left margin, and hovering the mouse over the bookmark icon will display the description in a tooltip.

The bookmark will be added to the '''Bookmarks''' view. In this view the bookmark description can be edited, and the bookmark can be deleted. Double-clicking the bookmark or selecting '''Go to''' from its context menu will open the trace or experiment and go directly to the event that was bookmarked.

To remove a bookmark, double-click its icon, select '''Remove Bookmark''' from the left margin context menu, or select '''Delete''' from the Bookmarks view.

[[Image:images/Bookmarks.png]]

=== Copy to Clipboard ===

The text of selected events can be copied to the clipboard by right-clicking on the table and selecting '''Copy to Clipboard''' in the context menu. The clipboard contents will be prefixed by the column header names. For every event in the table selection, the column text will be copied to the clipboard. The column text will be tab-separated. Hidden columns will not be included in the clipboard contents.

=== Event Source Lookup ===

Some trace types can optionally embed information in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.

==== Source Code ====

If a source file is available in the trace for the selected event, the item '''Open Source Code''' is shown in the context menu. Selecting this menu item will attempt to find the source file in all opened projects in the workspace. If multiple candidates exist, a selection dialog will be shown to the user. The selected source file will be opened, at the correct line, in its default language editor. If no candidate is found, an error dialog is shown displaying the source code information.

==== EMF Model ====

If an EMF model URI is available in the trace for the selected event, the item '''Open Model Element''' is shown in the context menu. Selecting this menu item will attempt to open the model file in the project specified in the URI. The model file will be opened in its default model editor. If the model file is not found, an error dialog is shown displaying the URI information.

=== Exporting To Text ===

It is possible to export the content of the trace to a text file based on the columns displayed in the events table. If a filter (see '''[[#Filtering| Filtering]]''') was defined prior exporting only events that match the filter will be exported to the file. To export the trace to text, press the right mouse button on the events table. A context-sensitive menu will show. Select the '''Export To Text...''' menu option. A file locater dialog will open. Fill in the file name and location and then press on '''OK'''. A window with a progress bar will open till the export is finished.

''Note'':  The columns in the text file are separated by tabs.

=== Refreshing of Trace ===

It's possible to refresh the content of the trace and resume indexing in case the current open trace was updated on the media. To refresh the trace, right-click into the table and select menu item '''Refresh'''. Alternatively, press key '''F5'''.

=== Collapsing of Repetitive Events ===

The implementation for collapsing of repetitive events is trace type specific and is only available for certain trace types. For example, a trace type could allow collapsing of consecutive events that have the same event content but not the same timestamp. If a trace type supports this feature then it is possible to select the '''Collapse Events''' menu item after pressing the right mouse button in the table.

When the collapsing of events is executing, the table will clear all events and fill itself with all relevant events. If the collapse condition is met, the first column of the table will show the number of times this event was repeated consecutively.

[[Image:images/TablePreCollapse.png]]

A status row will be displayed before and after the events, dynamically showing how many non-collapsed events were found and how many events were processed so far. Once the collapsing is completed, the status row icon in the left margin will change from a 'stop' to a 'filter' icon.

[[Image:images/TablePostCollapse.png]]

To remove the collapse filter, press the ([[Image:images/delete_button.gif]]) icon on the '''Collapse''' label in the header bar, or press the right mouse button in the table and select menu item '''Clear Filters''' in the context sensitive menu (this will also remove any other filters).

=== Customization ===

The table columns can be reordered by the user by dragging the column headers. This column order is saved when the editor is closed. The setting applies to all traces of the same trace type.

The table columns can be hidden or restored by right-clicking on any column header and clicking on an item in the context menu to toggle its state. Clicking '''Show All''' will restore all table columns.

The table font can be customized by the user by changing the preference in '''Window''' > '''Preferences''' > '''General''' > '''Appearance''' > '''Colors and Fonts''' > '''Tracing''' > '''Trace event table font'''.

The search and filter highlight color can be customized by the user by changing the preference in '''Window''' > '''Preferences''' > '''General''' > '''Appearance''' > '''Colors and Fonts''' > '''Tracing''' > '''Trace event table highlight color'''.

== Histogram View ==

The Histogram View displays the trace events distribution with respect to time. When streaming a trace, this view is dynamically updated as the events are received. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).

[[Image:images/HistogramView.png]]

The '''Align Views''' toggle button [[Image:images/link.gif]] in the local toolbar allows to disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the Histogram view will disable this feature across all the views because it's a workspace preference.

The '''Hide Lost Events''' toggle button [[Image:images/hide_lost_events.gif]] in the local toolbar allows to hide the bars of lost events. When the button is selected it can be toggled again to show the lost events.

The '''Activate Trace Coloring''' toggle button [[Image:images/show_hist_traces.gif]] in the local toolbar allows to use separate colors for each trace of an experiment. Note that this feature is not available if your experiment contains more than twenty two traces. When activated, a legend is displayed at the bottom on the histogram view.

On the top left, there are three text controls:

* '''Selection Start''': Displays the start time of the current selection
* '''Selection End''': Displays the end time of the current selection
* '''Window Span''': Displays the current zoom window size in seconds

The controls can be used to modify their respective value. After validation, the other controls and views will be synchronized and updated accordingly. To modify both selection times simultaneously, press the link icon [[Image:images/link.gif]] which disables the '''Selection End''' control input.

The large (full) histogram, at the bottom, shows the event distribution over the whole trace or set of traces. It also has a smaller semi-transparent orange window, with a cross-hair, that shows the current zoom window.

The smaller (zoom) histogram, on top right, corresponds to the current zoom window, a sub-range of the event set. The window size can be adjusted by dragging the sash left beside the zoom window.

The x-axis of each histogram corresponds to the event timestamps. The start time and end time of the histogram range is displayed. The y-axis shows the maximum number of events in the corresponding histogram bars.

The vertical blue line(s) show the current selection time (or range). If applicable, the region in the selection range will be shaded.

The mouse can be used to control the histogram:

* '''Left-click''': Set a selection time
* '''Left-drag''': Set a selection range
* '''Shift-left-click or drag''': Extend or shrink the selection range

* '''Middle-click or Ctrl-left-click''': Center the zoom window on mouse (full histogram only)
* '''Middle-drag or Ctrl-left-drag''': Move the zoom window

* '''Right-drag''': Set the zoom window
* '''Shift-right-click or drag''': Extend or shrink the zoom window (full histogram only)

* '''Mouse wheel up''': Zoom in
* '''Mouse wheel down''': Zoom out

Hovering the mouse over an histogram bar pops up an information window that displays the start/end time of the corresponding bar, as well as the number of events (and lost events) it represents. If the mouse is over the selection range, the selection span in seconds is displayed.

In each histogram, the following keys are handled:

* '''Left Arrow''': Moves the current event to the previous non-empty bar
* '''Right Arrow''': Moves the current event to the next non-empty bar
* '''Home''': Sets the current time to the first non-empty bar
* '''End''': Sets the current time to the last non-empty histogram bar
* '''Plus (+)''': Zoom in
* '''Minus (-)''': Zoom out

== Statistics View ==

The Statistics View displays the various event counters that are collected when analyzing a trace. After opening a trace, the element '''Statistics''' is added under the '''Tmf Statistics Analysis''' tree element in the Project Explorer. To open the view, double-click the '''Statistics''' tree element. Alternatively, select '''Statistics''' under '''Tracing''' within the '''Show View''' window ('''Window''' -> '''Show View''' -> '''Other...'''). The statistics is collected for the whole trace. This view is part of the '''Tracing and Monitoring Framework (TMF)''' and is generic. It will work for any trace type extensions.

The view is separated in two sides. The left side of the view presents the Statistics in a table. The table shows 3 columns: ''Level'' ''Events total'' and ''Events in selected time range''. The data is organized per trace. After parsing a trace the view will display the number of events per event type in the second column and in the third, the currently selected time range's event type distribution is shown. The cells where the number of events are printed also contain a colored bar with a number that indicates the percentage of the event count in relation to the total number of events.

[[Image:images/LTTng2StatisticsTableView.png]]

The right side illustrates the proportion of types of events into two pie charts. The legend of each pie chart gives the representation of each color in the chart.
* The ''Global'' pie chart displays the general proportion of the events in the trace.
* When there is a range selection, the ''Events in selection'' pie chart appears next to the ''Global'' pie chart and displays the proportion the event in the selected range of the trace.

[[Image:images/LTTng2StatisticsPieChartView.png]]

By default, the statistics use a state system, therefore will load very quickly once the state system is written to the disk as a supplementary file.

== Colors View ==

[[Image:images/ColorsView.png]]

The Colors view allows the user to define a prioritized list of color settings.

A color setting associates a foreground and background color (used in any events table), and a tick color (used in the Time Chart view), with an event filter.

In an events table, any event row that matches the event filter of a color setting will be displayed with the specified foreground and background colors. If the event matches multiple filters, the color setting with the highest priority will be used.

The same principle applies to the event tick colors in the Time Chart view. If a tick represents many events, the tick color of the highest priority matching event will be used.

Color settings can be inserted, deleted, reordered, imported and exported using the buttons in the Colors view toolbar. Changes to the color settings are applied immediately, and are persisted to disk.

== Filters View ==

[[Image:images/FiltersView.png]]

The Filters view allows the user to define preset filters that can be applied to any events table.

The filters can be more complex than what can be achieved with the filter header row in the events table. The filter is defined in a tree node structure, where the node types can be any of '''TRACETYPE''', '''AND''', '''OR''', '''CONTAINS''', '''EQUALS''', '''MATCHES''' or '''COMPARE'''. Some nodes types have restrictions on their possible children in the tree.

The '''TRACETYPE''' node filters against the trace type of the trace as defined in a plug-in extension or in a custom parser. When used, any child node will have its ''type'' combo box fixed and its ''aspect'' combo box restricted to the possible aspects of that trace type. Depending of the Trace Types enabled in the [[#Trace_Types_Preference_Page | Trace Types preference page]], the list of available trace types for the filtering can vary.

The '''AND''' node applies the logical ''and'' condition on all of its children. All children conditions must be true for the filter to match. A ''not'' operator can be applied to invert the condition.

The '''OR''' node applies the logical ''or'' condition on all of its children. At least one children condition must be true for the filter to match. A ''not'' operator can be applied to invert the condition.

The '''CONTAINS''' node matches when the specified event ''aspect'' value contains the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive. The ''type'' combo box restricts the possible aspects to those of the specified trace type.

The '''EQUALS''' node matches when the specified event ''aspect'' value equals exactly the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive. The ''type'' combo box restricts the possible aspects to those of the specified trace type.

The '''MATCHES''' node matches when the specified event ''aspect'' value matches against the specified ''regular expression''. A ''not'' operator can be applied to invert the condition. The ''type'' combo box restricts the possible aspects to those of the specified trace type.

The '''COMPARE''' node matches when the specified event ''aspect'' value compared with the specified ''value'' gives the specified ''result''. The result can be set to ''smaller than'', ''equal'' or ''greater than''. The type of comparison can be numerical, alphanumerical or based on time stamp. A ''not'' operator can be applied to invert the condition. The ''type'' combo box restricts the possible aspects to those of the specified trace type.

For numerical comparisons, strings prefixed by "0x", "0X" or "#" are treated as hexadecimal numbers and strings prefixed by "0" are treated as octal numbers.

For time stamp comparisons, strings are treated as seconds with or without fraction of seconds. This corresponds to the '''TTT''' format in the '''Time Format''' preferences. The value for a selected event can be found in the '''Properties''' view under the ''Timestamp'' property. The common 'Timestamp' aspect can always be used for time stamp comparisons regardless of its time format.

Filters can be added, deleted, imported and exported using the buttons in the Filters view toolbar. The nodes in the view can be Cut (Ctrl-X), Copied (Ctrl-C) and Pasted (Ctrl-V) by using the buttons in the toolbar or by using the key bindings. This makes it easier to quickly build new filters from existing ones. Changes to the preset filters are only applied and persisted to disk when the '''Save filters''' button is pressed.

To apply a saved preset filter in an events table, right-click on the table and select '''Apply preset filter...''' > ''filter name''.

== Time Chart View ==

[[Image:images/TimeChartView.png]]

The Time Chart view allows the user to visualize every open trace in a common time chart. Each trace is display in its own row and ticks are display for every punctual event. As the user zooms using the mouse wheel or by right-clicking and dragging in the time scale, more detailed event data is computed from the traces. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).

Time synchronization is enabled between the time chart view and other trace viewers such as the events table.

Color settings defined in the Colors view can be used to change the tick color of events displayed in the Time Chart view.

When a search is applied in the events table, the ticks corresponding to matching events in the Time Chart view are decorated with a marker below the tick.

When a bookmark is applied in the events table, the ticks corresponding to the bookmarked event in the Time Chart view is decorated with a bookmark above the tick.

When a filter is applied in the events table, the non-matching ticks are removed from the Time Chart view.

The Time Chart only supports traces that are opened in an editor. The use of an editor is specified in the plug-in extension for that trace type, or is enabled by default for custom traces.

The '''Align Views''' toggle button [[Image:images/link.gif]] in the local toolbar allows to disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference.

== State System Explorer View ==

The State System Explorer view allows the user to inspect the state interval values of every attribute of a state system at a particular time.

The view shows a tree of currently selected traces and their registered state system IDs. For each state system the tree structure of attributes is displayed. The attribute name, quark, value, start and end time, and full attribute path are shown for each attribute.

To modify the time of attributes shown in the view, select a different current time in other views that support time synchronization (e.g. event table, histogram view). When a time range is selected, this view uses the begin time.

== External Analyses ==

Trace Compass supports the execution of '''external analyses''' conforming to the [https://github.com/lttng/lami-spec/blob/v1.0.1/lami.adoc LAMI 1.0.x specification]. This includes recent versions of the [https://github.com/lttng/lttng-analyses LTTng-Analyses project].

An external analysis is a [[#Running an External Analysis|program executed by Trace Compass]]. When the program is done analyzing, Trace Compass generates a '''[[#Opening a Report|report]]''' containing its results. A report contains one or more tables which can also be viewed as bar and scatter [[#Creating a Chart from a Result Table|charts]].

'''Note''': The program to execute is found by searching the directories listed in the standard <code>$PATH</code> environment variable when no path separator (<code>/</code> on Unix and OS X, <code>\</code> on Windows) is found in its command.

Trace Compass ships with a default list of ''descriptors'' of external analyses (not the analyses themselves), including the descriptors of the [http://github.com/lttng/lttng-analyses LTTng analyses]. If the LTTng analyses project is installed, its analyses are available when opening or importing an LTTng kernel trace.

=== Running an External Analysis ===

To run an external analysis:

# [[#Importing Traces to the Project|Import a trace to the project]].
# Make sure the trace is opened by double-clicking its name in the [[#Project Explorer View]].
# Under the trace in the [[#Project Explorer View]], expand ''External Analyses'' to view the list of available external analyses.<p>The external analyses which are either missing or not compatible with the trace are stroke and cannot be executed.</p><p>[[Image:images/externalAnalyses/external-analyses-list.png]]</p>
# '''Optional''': If you want the external analysis to analyze a specific time range of the current trace, make a time range selection.<p>You can use views like the [[#Histogram View]] and the [[#Control Flow View]] (if it's available for this trace) to make a time range selection.</p><p>External analyses are executed on the current time range selection if there is one, or on the whole trace otherwise.</p>
# Right-click the external analysis to run and click '''Run External Analysis'''.<p>[[Image:images/externalAnalyses/run-external-analysis.png]]</p>
# In the opened ''External Analysis Parameters'' window, optionally enter extra parameters to pass to the program.<p>[[Image:images/externalAnalyses/external-analysis-parameters-dialog.png]]</p>
# Click '''OK''' to start the analysis.

Note that many external analyses can be started concurrently.

When the external analysis is done analyzing, its results are saved as a [[#Opening a Report|report]] in Trace Compass. The tables contained in this report are also automatically opened into a new report view when the analysis is finished.

=== Opening a Report ===

A '''report''' is created after a successful [[#Running an External Analysis|execution of an external analysis]].

To open a report:

* Under ''Reports'' under a trace in the [[#Project Explorer View]], double-click the report to open.<p>Each result table generated by the external analysis is shown in its own tab in the opened report view.</p><p>[[Image:images/externalAnalyses/report-view.png]]</p>

=== Creating a Chart from a Result Table ===

To create a bar or a scatter chart from the data of a given result table:

# [[#Opening a Report|Open the report]] containing the result table to use for creating the chart.
# In the opened report view, click the tab of the result table to use for creating the chart.
# Click the ''View Menu'' button, then click either '''New custom bar chart''' or '''New custom scatter chart'''.<p>[[Image:images/externalAnalyses/new-custom-scatter-chart-menu.png]]</p>
# In the opened ''Bar chart series creation'' or ''Scatter chart series creation'' window, under ''Series creator'', select a column to use for the X axis of the chart, and one or more columns to use for the Y axis of the chart, then click '''Add''' to create a series.<p>[[Image:images/externalAnalyses/chart-configuration-dialog.png]]</p><p>Repeat this step to create more series.</p>
# Click '''OK''' to create the chart.<p>The chart is created and shown at the right of its source result table.</p><p>[[Image:images/externalAnalyses/table-and-chart.png]]</p>

=== Showing or Hiding a Result Table ===

To show or hide a result table once a [[#Creating a Chart from a Result Table|chart]] has been created:

* In the report view, click the ''Toggle the Table view of the results'' button.<p>[[Image:images/externalAnalyses/table-and-chart-toggle-button.png]]</p><p>If the result table was visible, it is now hidden:</p><p>[[Image:images/externalAnalyses/chart-only.png]]</p>

=== Adding and Removing a User-Defined External Analysis ===

You can add a user-defined external analysis to the current list of external analyses. Note that the command to invoke must conform to the machine interface of [http://github.com/lttng/lttng-analyses LTTng analyses] 0.4.

'''Note''': If you want to create your own external analysis, consider following the [http://lttng.org/files/lami/lami-1.0.1.html LAMI 1.0 specification], which is supported by later versions of Trace Compass.

To add a user-defined external analysis:

# Under any trace in the [[#Project Explorer View]], right-click ''External Analyses'' and click '''Add External Analysis'''.<p>[[Image:images/externalAnalyses/add-external-analysis.png]]</p>
# In the opened ''Add External Analysis'' window, enter the name of the new external analysis and the associated command to run.<p>[[Image:images/externalAnalyses/add-external-analysis-dialog.png]]</p><p>The name is the title of the external analysis as shown under ''External Analyses'' in the [[#Project Explorer View]].</p><p>The command is the complete command line to execute. You can put arguments containing spaces or other special characters in double quotes.</p><p>'''Note''': If the command is not a file system path, then it must be found in the directories listed in the <code>$PATH</code> environment variable.</p>
# Click '''OK''' to add the user-defined external analysis.<p>A user-defined external analysis with a green icon is created under ''External Analyses'' in the [[#Project Explorer View]].</p><p>[[Image:images/externalAnalyses/user-defined-external-analysis.png]]</p>

'''Note''': The new external analysis entry is saved in the workspace.

To remove a user-defined external analysis:

* Under ''External Analyses'' in the [[#Project Explorer View]], right-click the external analysis to remove and click '''Remove External Analysis'''.<p>[[Image:images/externalAnalyses/remove-external-analysis.png]]</p><p>'''Note''': Only user-defined (green icon) external analyses can be removed.</p>

== Custom Parsers ==

Custom parser wizards allow the user to define their own parsers for text or XML traces. The user defines how the input should be parsed into internal trace events and identifies the event fields that should be created and displayed. Traces created using a custom parser can be correlated with other built-in traces or traces added by plug-in extension.

=== Creating a custom text parser ===

The '''New Custom Text Parser''' wizard can be used to create a custom parser for text logs. It can be launched several ways:

* Select '''File''' &gt; '''New''' &gt; '''Other...''' &gt; '''Tracing''' &gt; '''Custom Text Parser'''
* Open the '''[[#Managing custom parsers|Manage Custom Parsers]]''' dialog, select the '''Text''' radio button and click the '''New...''' button

[[Image:images/CustomTextParserInput.png]]

Fill out the first wizard page with the following information:

* '''Category:''' Enter a category name for the trace type.
* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser. This will also be the default event type name.
* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp, or leave blank to use the default Time Format preference.<br>
Note: information about date and time patterns can be found here: [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]

Click the '''Add next line''', '''Add child line''' or '''Remove line''' buttons to create a new line of input or delete it. For each line of input, enter the following information:

* '''Regular expression:''' Enter a regular expression that should match the input line in the log, using capturing groups to extract the data.<br>
Note: information about regular expression patterns can be found here: [http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html]

* '''Cardinality:''' Enter the minimum and maximum number of lines matching this line's regular expression that must be found in the log. At least the minimum number of lines must be found before the parser will consider the next line. Child lines will always be considered first.

* '''Event type:''' Optionally enable this text field to enter an event type name that will override the default (trace type) when this line matches.

<u>Important note:</u> The custom parsers identify a log entry when the first line's regular expression matches (Root Line n). Each subsequent text line in the log is attempted to be matched against the regular expression of the parser's input lines in the order that they are defined (Line n.*). Only the first matching input line will be used to process the captured data to be stored in the log entry. When a text line matches a Root Line's regular expression, a new log entry is started.

Click the '''Add group''' or '''Remove group''' buttons to define the data extracted from the capturing groups in the line's regular expression. For each group, enter the following information:

* '''Name combo:''' Select a name for the extracted data:
** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
** '''Event type''': Select this option to identify the event type name. This will override the default or line-specific event type name.
** '''Message''': Select this option to identify the main log entry's message. This is usually a group which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box.

* '''Action combo:''' Select the action to be performed on the extracted data:
** '''Set''': Select this option to overwrite the data for the chosen name when there is a match for this group.
** '''Append''': Select this option to append to the data with the chosen name, if any, when there is a match for this group.
** '''Append with |''' : Select this option to append to the data with the chosen name, if any, when there is a match for this group, using a | separator between matches.

The '''Preview input''' text box can be used to enter any log data that will be processed against the defined custom parser. When the wizard is invoked from a selected log file resource, this input will be automatically filled with the file contents.

The '''Preview:''' text field of each capturing group and of the Time Stamp will be filled from the parsed data of the first matching log entry.

In the '''Preview input''' text box, the matching entries are highlighted with different colors:

* <code><span style="background:#FFFF00">&nbsp;Yellow&nbsp;</span></code> : indicates uncaptured text in a matching line.
* <code><span style="background:#00FF00">&nbsp;Green&nbsp;&nbsp;</span></code> : indicates a captured group in the matching line's regular expression for which a custom parser group is defined. This data will be stored by the custom parser.
* <code><span style="background:#FF00FF">&nbsp;Magenta</span></code> : indicates a captured group in the matching line's regular expression for which there is no custom parser group defined. This data will be lost.
* <code>&nbsp;White&nbsp;&nbsp;</code> : indicates a non-matching line.

The first line of a matching entry is highlighted with darker colors.

By default only the first matching entry will be highlighted. To highlight all matching entries in the preview input data, click the '''Highlight All''' button. This might take a few seconds to process, depending on the input size.

Click the '''Next >''' button to go to the second page of the wizard.

[[Image:images/CustomTextParserOutput.png]]

On this page, the list of default and custom data is shown, along with a preview of the custom parser log table output.

The custom data output can be modified by the following options:

* '''Visibility:''' Select or unselect the checkbox to display the custom data or hide it.

* '''Column order:''' Click '''Move before''' or '''Move after''' to change the display order of custom data.

The table at the bottom of the page shows a preview of the custom parser log table output according to the selected options, using the matching entries of the previous page's '''Preview input''' log data.

Click the '''Finish''' button to close the wizard and save the custom parser.

=== Creating a custom XML parser ===

The '''New Custom XML Parser''' wizard can be used to create a custom parser for XML logs. It can be launched several ways:

* Select '''File''' &gt; '''New''' &gt; '''Other...''' &gt; '''Tracing''' &gt; '''Custom XML Parser'''
* Open the '''[[#Managing custom parsers|Manage Custom Parsers]]''' dialog, select the '''XML''' radio button and click the '''New...''' button

[[Image:images/CustomXMLParserInput.png]]

Fill out the first wizard page with the following information:

* '''Category:''' Enter a category name for the trace type.
* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser. This will also be the default event type name.
* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp, or leave blank to use the default Time Format preference.<br>
Note: information about date and time patterns can be found here: [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]

Click the '''Add document element''' button to create a new document element and enter a name for the root-level document element of the XML file.

Click the '''Add child''' button to create a new element of input to the document element or any other element. For each element, enter the following information:

* '''Element name:''' Enter a name for the element that must match an element of the XML file.
* '''Log entry:''' Select this checkbox to identify an element which represents a log entry. Each element with this name in the XML file will be parsed to a new log entry. At least one log entry element must be identified in the XML document. Log entry elements cannot be nested.
* '''Name combo:''' Select a name for the extracted data:
** '''Ignore''': Select this option to ignore the extracted element's data at this level. It is still possible to extract data from this element's child elements.
** '''Event type''': Select this option to identify the event type name. This will override the default or element-specific event type name.
** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
** '''Message''': Select this option to identify the main log entry's message. This is usually an input which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box. It does not have to match the element name.
* '''Action combo:''' Select the action to be performed on the extracted data:
** '''Set''': Select this option to overwrite the data for the chosen name when there is a match for this element.
** '''Append''': Select this option to append to the data with the chosen name, if any, when there is a match for this element.
** '''Append with |''' : Select this option to append to the data with the chosen name, if any, when there is a match for this element, using a | separator between matches.
* '''Event type:''' Optionally enable this text field to enter an event type name that will override the default (trace type) when this element is present.

Note: An element's extracted data 'value' is a parsed string representation of all its attributes, children elements and their own values. To extract more specific information from an element, ignore its data value and extract the data from one or many of its attributes and children elements.

Click the '''Add attribute''' button to create a new attribute input from the document element or any other element. For each attribute, enter the following information:

* '''Attribute name:''' Enter a name for the attribute that must match an attribute of this element in the XML file.
* '''Name combo:''' Select a name for the extracted data:
** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
** '''Event type''': Select this option to identify the event type name. This will override the default or element-specific event type name.
** '''Message''': Select this option to identify the main log entry's message. This is usually an input which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box. It does not have to match the element name.
* '''Action combo:''' Select the action to be performed on the extracted data:
** '''Set''': Select this option to overwrite the data for the chosen name when there is a match for this element.
** '''Append''': Select this option to append to the data with the chosen name, if any, when there is a match for this element.
** '''Append with |''' : Select this option to append to the data with the chosen name, if any, when there is a match for this element, using a | separator between matches.

Note: A log entry can inherited input data from its parent elements if the data is extracted at a higher level.

Click the '''Feeling lucky''' button to automatically and recursively create child elements and attributes for the current element, according to the XML element data found in the '''Preview input''' text box, if any.

Click the '''Remove element''' or '''Remove attribute''' buttons to remove the extraction of this input data. Take note that all children elements and attributes are also removed.

The '''Preview input''' text box can be used to enter any XML log data that will be processed against the defined custom parser. When the wizard is invoked from a selected log file resource, this input will be automatically filled with the file contents.

The '''Preview:''' text field of each capturing element and attribute and of the Time Stamp will be filled from the parsed data of the first matching log entry. Also, when creating a new child element or attribute, its element or attribute name will be suggested if possible from the preview input data.

Click the '''Next >''' button to go to the second page of the wizard.

[[Image:images/CustomXMLParserOutput.png]]

On this page, the list of default and custom data is shown, along with a preview of the custom parser log table output.

The custom data output can be modified by the following options:

* '''Visibility:''' Select or unselect the checkbox to display the custom data or hide it.
* '''Column order:''' Click '''Move before''' or '''Move before''' to change the display order of custom data.

The table at the bottom of the page shows a preview of the custom parser log table output according to the selected options, using the matching entries of the previous page's '''Preview input''' log data.

Click the '''Finish''' button to close the wizard and save the custom parser.

=== Managing custom parsers ===

The '''Manage Custom Parsers''' dialog is used to manage the list of custom parsers used by the tool. To open the dialog:

* Open the '''Project Explorer''' view.
* Select '''Manage Custom Parsers...''' from the '''Traces''' folder context menu, or from a trace's '''Select Trace Type...''' context sub-menu.

[[Image:images/ManageCustomParsers.png]]

The ordered list of currently defined custom parsers for the selected type is displayed on the left side of the dialog.

To change the type of custom parser to manage, select the '''Text''' or '''XML''' radio button.

The following actions can be performed from this dialog:

* New...

Click the '''New...''' button to launch the '''New Custom Parser''' wizard.

* Edit...

Select a custom parser from the list and click the '''Edit...''' button to launch the '''Edit Custom Parser''' wizard.

* Delete

Select a custom parser from the list and click the '''Delete''' button to remove the custom parser.

* Import...

Click the '''Import...''' button and select a file from the opened file dialog to import all its custom parsers. If any parser conflicts with an existing built-in or custom trace type, the user will be prompted to skip or rename the imported parser.

* Export...

Select a custom parser from the list, click the '''Export...''' button and enter or select a file in the opened file dialog to export the custom parser. Note that if an existing file containing custom parsers is selected, the custom parser will be appended to the file.

=== Opening a trace using a custom parser ===

Once a custom parser has been created, any [[#Importing Traces to the Project|imported trace]] file can be opened and parsed using it.

To do so:

* Select a trace in the '''Project Explorer''' view
* Right-click the trace and select '''Select Trace Type...''' &gt; ''category name'' &gt; ''parser name''
* Double-click the trace or right-click it and select '''Open'''

The trace will be opened in an editor showing the events table, and an entry will be added for it in the Time Chart view.

== Automatic Time Axis Alignment ==

Trace Compass supports automatic alignment of the time axis for time base views. The user now can resize the time window of one view and all other open views will align to the new window size and position. The automatic alignment is optional and can be disabled and enabled using the '''Align Views''' toolbar button. Disabling or enabling it in one view it will disable and enable it for all view since it's a workspace wide setting.

[[Image:images/TimeAlignment_sash.png]]

== Searching in Time Graph Views ==

Search for an entry in a '''Time Graph view''', e.g. [[#Control_Flow_View | Control Flow View]] or [[#Resources_View | Resources View]], using the ''' Find ''' dialog. To use the dialog :

* Select the time graph view you want to search in
* Press ''' Ctrl + F '''. The following screen will be shown :

[[Image:images/FindDialog.png]]

* Enter the string to find in the ''' Find ''' text drop down and select the ''' Options ''' and ''' Direction ''' you need.
* Press the ''' Find ''' button or ''' Enter ''' or ''' Alt + n '''. The next match in the selected time graph view will be selected.

Various options are available in the ''' Options ''' group :
* ''' Case sensitive ''' makes the search case sensitive.
* ''' Wrap search ''' restarts the search from the first index, depending of the direction, when no entry were found.
* ''' Whole word ''' allows to search for whole words, delimited by spaces or special character, that are identical to the search text.
* ''' Regular expression ''' specifies that the search text is a regular expression or not.

The ''' Direction ''' group allows to select the search direction : ''' Forward ''' or ''' Backward '''.

== Configurable Marker Sets ==

Time graph views can allow the user to display periodic markers over time graphs by selecting a marker set. The marker sets are user-configurable by editing the ''markers.xml'' file.

From the view menu, select '''Marker Set''' &gt; '''Edit...'''.  The ''markers.xml'' file will be opened in an editor. After editing the file, save the modifications, and then select '''Marker Set''' &gt; ''marker set name'' to activate the marker set. Select '''Marker set''' &gt; '''None''' to deactivate the marker set.

=== Marker Set Configuration XML Format ===

The format of the ''markers.xml'' file is defined as follows:

 <marker-sets> (marker-set*)
   <market-set> (marker*)
     <marker> ((submarker | segments))*
       <submarker> ((submarker | segments))*
       </submarker>
       <segments> (segment+, ((submarker | segments))*)
         <segment/> ((submarker | segments))*
       </segments>
     </marker>
   </marker-set>
 </marker-sets>

The ''<marker>'' element defines a fixed-period marker at the root of the marker set. Optionally, a ''<marker>'' can have child ''<submarker>'' elements, which split each marker into a number of equal sub-markers, and/or child ''<segments>'' elements, which split each marker into segments of defined weights defined by the list of child ''<segment>'' elements. Each of these elements can recursively have their own ''<submarker>'' and ''<segments>'' child elements.

The element attributes are defined as follows:

 <marker-set name="name" id="id">
;name (required)
:The name of the marker set.
;id (required)
:The unique id of the marker set.

 <marker name="name" label="label" id="id" referenceid="referenceid" color="color" period="period" unit="unit" range="range" offset="offset" index="index">
;name (required)
:The category name for this marker.
;label (optional)
:The Java String format for this marker, where the first and only argument is the marker index. When omitted, the default value is "%d".
;id (optional)
:The unique id of this marker.
;referenceid (optional)
:The reference id that can be used by a trace to provide a marker reference by adapting the IMarkerReferenceProvider interface. When omitted, the marker reference is time zero.
;color (required)
:An RGB value in format #rrggbb, or an X11 color name.
;period (required)
:The marker period in units, as a decimal number.
;unit (required)
:One of ms, us, ns or cycles. If cycles is used, the trace should adapt the ICyclesConverter interface.
;range (optional)
:The marker index range, in format min..max where min and max are optional. If max is not present, the index does not wrap. If omitted, the default range of 0.. is used.
;offset (optional)
:The offset in units, relative to the marker reference. If omitted, the offset is zero.
;index (optional)
:The set of valid index ranges, as a comma-separated list of (index|min..max). Index values not in this range set will not generate any markers or child markers. If omitted, are index values are valid.

 <submarker name="name" label="label" id="id" color="color" range="range" index="index">
;name (required)
:The category name for this sub-marker.
;label (optional)
:The Java String format for this marker, where the first and only argument is the marker index. When omitted, the default value is "%d".
;id (optional)
:The unique id of this sub-marker.
;color (optional)
:An RGB value in format #rrggbb, or an X11 color name. If omitted, the parent marker's color will be used.
;range (required)
:The marker index range, in format min..max. The range determines the number of equal sub-markers in which the parent marker will be split.
;index (optional)
:The set of valid index ranges, as a comma-separated list of (index|min..max). Index values not in this range set will not generate any markers or child markers. If omitted, are index values are valid.

 <segments name="name">
;name (required)
:The category name for the segments.

 <segment label="label" id="id" color="color" length="length">
;label (optional)
:The Java String format for this marker, where the first and only argument is the marker index. When omitted, the default value is "%d". The segment elements index is sequential, starting at zero for the first segment.
;id (optional)
:The unique id of this segment.
;color (optional)
:An RGB value in format #rrggbb, or an X11 color name. If omitted, the segment will not generate any markers or child markers.
;length (required)
:The length of this segment, as an integer number relative to the total of all segments' lengths. The length determines the fraction of the parent marker to be used for this segment.

An example marker set configuration can be found below:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<marker-sets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="markers.xsd">
	<marker-set name="Example" id="example.id">
		<marker name="Frame" color="#ff0000" period="10" unit="ms" range="0..4095">
			<submarker name="Subframe" color="#00ff00" label="#%d Normal" range="0..9" index="0,2..9">
				<submarker name="Slot" color="#008800" range="0..1"/>
			</submarker>
			<submarker name="Subframe" color="#ff8800" label="#%d Special" range="0..9" index="1">
				<segments name="Slot">
					<segment label="A" color="#884400" length="1"/>
					<segment label="B" color="#884400" length="3"/>
					<segment label="C" color="#884400" length="2"/>
				</segments>
			</submarker>
		</marker>
	</marker-set>
</marker-sets>
</pre>

= LTTng Tracer Control =

The LTTng Tracer Control in Eclipse for the LTTng Tracer toolchain version v2.0 (or later) is done using SSH and requires an SSH server to be running on the remote host. For the SSH connection the SSH implementation of Remote Services is used. The functions to control the LTTng tracer (e.g. start and stop), either locally or remotely, are available from a dedicated Control View.

In the following sections the LTTng 2.0 tracer control integration in Eclipse is described. Please refer to the LTTng 2.0 tracer control command line manual for more details and descriptions about all commands and their command line parameters [[#References | References]].

== Control View ==
To open the Control View, select '''Window->Show View->Other...->LTTng->Control View''.

[[Image:images/LTTngControlView.png]]

=== Creating a New Connection to a Remote Host ===

To connect to a remote host, select the '''New Connection''' button in the Control View.

[[Image:images/LTTngControlViewConnect.png]]

A new dialog is opened for selecting a remote connection. You can also edit or define a remote connection from here.

[[Image:images/LTTng2NewConnection.png]]

To define a new remote host using the default SSH service, select '''Buit-in SSH''' and then select '''Create...'''. This will start the standard '''New Connection''' wizard provided by the Remote Services plugin. Similar, to edit the definition of a remote connection, select '''Edit...''' and use the '''Edit Connection''' wizard provided by the SSH service. In case you have installed an additional adapter for the Remote Services, you can choose to define a remote connection based on this adapter.

[[Image:images/LTTng2NewRemoteConnection.png]]

To use an existing connection definition, select the relevant entry in the tree and then select '''Ok'''.

[[Image:images/LTTng2SelectConnection.png]]

A new display will show for providing the user name and password. This display only opens if no password had been saved before. Enter user name and password in the '''Password Required''' dialog box and select '''Ok'''.

[[Image:images/LTTng2EnterPassword.png]]

After pressing '''Ok''' the SSH connection will be established and after successful login the Control View implementation retrieves the LTTng Tracer Control information. This information will be displayed in the Control View in form of a tree structure.

[[Image:images/LTTng2ControlViewFilled.png]]

The top level tree node is the representation of the remote connection (host). The connection name of the connection will be displayed. Depending on the connection state different icons are displayed. If the node is '''CONNECTED''' the icon is shown [[Image:images/Target_connected.gif]], otherwise (states '''CONNECTING''', '''DISCONNNECTING''' or '''DISCONNECTED''' the icon is [[Image:images/Target_disconnected.gif]].

Under the host level two folder groups are located. The first one is the '''Provider''' group. The second one is the '''Sessions''' group.

Under the '''Provider''' group all trace providers are displayed. Trace providers are '''Kernel''' and any user space application that supports UST tracing. Under each provider a corresponding list of events are displayed.

Under the '''Sessions''' group all current sessions will be shown. The level under the sessions show the configured domains. Currently the LTTng Tracer Toolchain supports domain '''Kernel''', '''UST global''', '''JUL''', '''Log4j''' and '''Python'''. Under the domains '''Kernel''' and '''UST Global''' the configured channels will be displayed. The last level is under the channels where the configured events are displayed.

Each session can be '''ACTIVE''' or '''INACTIVE'''. Active means that tracing has been started, inactive means that the tracing has been stopped. Depending on the state of a session a different icon is displayed. The icon for an active session is [[Image:images/Session_active.gif]]. The icon for an inactive session is [[Image:images/Session_inactive.gif]].

Each channel can be '''ENABLED''' or '''DISABLED'''. An enabled channel means that all configured events of that channel will be traced and a disabled channel won't trace any of its configured events. Different icons are displayed depending on the state of the channel. The icon for an enabled channel is  [[Image:images/Channel.gif]] and the icon for a disabled channel is [[Image:images/Channel_disabled.gif]].

Events within a channel can be in state '''ENABLED''' or '''DISABLED'''. Enabled events are stored in the trace when passed during program execution. Disabled events on the other hand won't be traced. Depending on the state of the event the icons for the event is different. An enabled event has the icon [[Image:images/Event_enabled.gif]] and a disabled event the icon [[Image:images/Event_disabled.gif]].

=== Disconnecting from a Remote Host ===

To disconnect from a remote host, select the host in the Control View and press the '''Disconnect''' button. Alternatively, press the right mouse button. A context-sensitive menu will show. Select the '''Disconnect''' button.

[[Image:images/LTTng2ControlViewDisconnect.png]]

=== Connecting to a Remote Host ===

To connect to a remote host, select the host in the Control View and press the '''Connect''' button. Alternatively, press the right mouse button. A context-sensitive menu will show. Select the '''Connect''' button. This will start the connection process as discribed in [[#Creating a New Connection to a Remote Host | Creating a New Connection to a Remote Host]].

[[Image:images/LTTng2ControlViewConnect.png]]

=== Deleting to a Remote Host Connection ===

To delete a remote host connection, select the host in the Control View and press the '''Delete''' button. Alternatively, press the right mouse button. A context-sensitive menu will show. Select the '''Delete''' button. For that command to be active the connection state has to be '''DISCONNECTED''' and the trace has to be closed.

[[Image:images/LTTng2ControlViewDelete.png]]

=== Creating a Tracing Session ===
To create a tracing session, select the tree node '''Sessions''' and press the right mouse button. Then select the '''Create Session...''' button of the context-sensitive menu.

[[Image:images/LTTng2CreateSessionAction.png]]

A dialog box will open for entering information about the session to be created.

[[Image:images/LTTng2CreateSessionDialog.png]]

Fill in the '''Session Name''' and optionally the '''Session Path''' and press '''Ok'''. Upon successful operation a new session will be created and added under the tree node '''Sessions'''.

=== Creating a Tracing Session With Advanced Options ===
LTTng Tools version v2.1.0 introduces the possibility to configure the trace output location at session creation time. The trace can be stored in the (tracer) local file system or can be transferred over the network.

To create a tracing session and configure the trace output, open the trace session dialog as described in chapter [[#Creating a Tracing Session | Creating a Tracing Session]]. A dialog box will open for entering information about the session to be created.

[[Image:images/LTTng2CreateSessionDialog_Advanced.png]]

The button '''Advanced >>>''' will only show if the remote host has LTTng Tools v2.1.0 installed. To configure the trace output select the '''Advanced >>>''' button. The Dialog box will be shown new fields to configure the trace output location.

[[Image:images/LTTng2CreateSessionDialog_TracePath.png]]

By default, the button '''Use same protocol and address for data and control''' is selected which allows to configure the same '''Protocol''' and '''Address''' for both data URL and control URL.

If button '''Use same protocol and address for data and control''' is selected the '''Protocol''' can be '''net''' for the default network protocol which is TCP (IPv4), '''net6''' for the default network protocol which is TCP (IPv6) and '''file''' for the local file system. For '''net''' and '''net6''' the port can be configured. Enter a value in '''Port''' for data and control URL or keep them empty for the default port to be used. Using '''file''' as protocol no port can be configured and the text fields are disabled.

If button '''Use same protocol and address for data and control''' is not selected the '''Protocol''' can be '''net''' for the default network protocol which is TCP (IPv4), '''net6''' for the default network protocol which is TCP (IPv6), '''tcp''' for the network protocol TCP (IPv4) and '''tcp6''' for the network protocol TCP (IPv6). Note that for '''net''' and '''net6''' always the default port is used and hence the port text fields are disabled. To configure non-default ports use '''tcp''' or '''tcp6'''.

The text field '''Trace Path''' allows for specifying the path relative to the location defined by the '''relayd''' or relative to the location specified by the '''Address''' when using protocol '''file'''. For more information about the '''relayd''' see '''LTTng relayd User Manual''' in chapter [[#References | References]].

To create a session with advanced options, fill in the relevant parameters and press '''Ok'''. Upon successful operation a new session will be created and added under the tree node '''Sessions'''.

=== Creating a Snapshot Tracing Session ===
LTTng Tools version v2.3.0 introduces the possibility to create snapshot tracing sessions. After starting tracing the trace events are not stored on disk or over the network. They are only transfered to disk or over the network when the user records a snapshot. To create such a snapshot session, open the trace session dialog as described in chapter [[#Creating a Tracing Session | Creating a Tracing Session]].

[[Image:images/LTTng2CreateSessionDialog_Snapshot.png]]

Fill in all necessary information, select the radio button for '''Snapshot Mode''' and press '''Ok'''. By default, the location for the snapshot output will be on the host where the host is located.

Refer to chapter [[#Recording a Snapshot | Recording a Snapshot]] for how to create a snapshot.

<!--=== Creating a Live Tracing Session ===
LTTng Tools version v2.4.0 introduces the possibility to create live tracing sessions. The live mode allows you to stream the trace and view it while it's being recorded. To create such a live session, open the trace session dialog as described in chapter [[#Creating a Tracing Session | Creating a Tracing Session]].

[[Image:images/LTTng2CreateSessionDialog_Live.png]]

In the advanced options, it is possible to set the '''Live Delay'''. The '''Live Delay''' is the delay in micro seconds before the data is flushed and streamed.

[[Image:images/LTTng2CreateSessionDialog_Live_Advanced.png]]

Fill in all necessary information, select the radio button for '''Live Mode''' and press '''Ok'''.
-->
=== Enabling Channels - General ===

Enabling channels can be done using a session tree node when the domain hasn't be created in the session or, alternatively on a domain tree node of a session in case the domain is already available.

=== Enabling Channels On Session Level ===

To enable a channel, select the tree node of the relevant session and press the right mouse button. Then select the '''Enable Channel...''' button of the context-sensitive menu.

[[Image:images/LTTng2CreateChannelAction.png]]

A dialog box will open for entering information about the channel to be created.

[[Image:images/LTTng2CreateChannelDialog.png]]

By default the domain '''Kernel''' is selected. To create a UST channel, select '''UST''' under the domain section. The label <Default> in any text box indicates that the default value of the tracer will be configured. To initialize the dialog box press button '''Default'''.

'''Note''': You cannot create a channel under the '''JUL''', '''LOG4J''' and '''Python''' domain. Instead those domains uses a default channel under the '''UST global''' domain named '''lttng_jul_channel''', '''lttng_log4j_channel''' or '''lttng_python_channel'''. Those are the channels that LTTng uses to trace Java or Python application and you cannot add '''UST''' events to those channels.

If required update the following channel information and then press '''Ok'''.

* '''Channel Name''': The name of the channel.
* '''Sub Buffer size''': The size of the sub-buffers of the channel (in bytes).
* '''Number of Sub Buffers''': The number of sub-buffers of the channel.
* '''Switch Timer Interval''': The switch timer interval.
* '''Read Timer Interval''': The read timer interval.
* '''Discard Mode''': '''Overwrite''' events in buffer or '''Discard''' new events when buffer is full.

Upon successful operation, the requested domain will be created under the session tree node as well as the requested channel will be added under the domain. The channel will be '''ENABLED'''.

=== Configuring Trace File Rotation ===

Since LTTng Tools v2.2.0 it is possible to set the maximum size of trace files and the maximum number of them. These options are located in the same dialog box that is used for enabling channels.

[[Image:images/LTTng2CreateChannelDialogFileRotation.png]]

* '''Maximum size of trace files''': The maximum size of trace files
* '''Maximum number of trace files''': The maximum number of trace files

=== Configuring per UID and per PID Buffers (UST only) ===

Since LTTng Tools v2.2.0 it is possible to configure the type of buffers for '''UST''' application. It is now possible to choose between per '''UID''' buffers (per user ID) and per '''PID''' buffers (per process ID) using the dialog box for enabling channels.

[[Image:images/LTTng2CreateChannelDialogPerUIDBuffers.png]]

* '''Per PID buffers''': To activate the per PID buffers option for UST channels
* '''Per UID buffers''': To activate the per UID buffers option for UST channels

If no buffer type is selected then the default value of the tracer will be configured.

Note that '''Global shared buffers''' is only for kernel channel and is pre-selected when '''Kernel''' is selected in the dalog box.

=== Configuring Periodical Flush for metadata Channel ===

Since LTTng Tools v2.2.0 it is possible to configure periodical flush for the metadata channel. To set this, use the checkbox '''Configure metadata channel''' then fill the switch timer interval.

[[Image:images/LTTng2CreateChannelDialogMetadataFlush.png]]

=== Enabling Channels On Domain Level ===

Once a domain is available, channels can be enabled directly using the domain. To enable a channel under an existing domain, select the tree node of the relevant domain and press the right mouse button. Then select the '''Enable Channel...''' button of the context-sensitive menu.

[[Image:images/LTTng2CreateChannelOnDomainAction.png]]

The dialog box for enabling channel will open for entering information about the channel to be created. Note that the domain is pre-selected and cannot be changed. Fill the relevant information and press '''Ok'''.

=== Enabling and Disabling Channels ===

To disable one or more enabled channels, select the tree nodes of the relevant channels and press the right mouse button. Then select the '''Disable Channel''' menu item of the context-sensitive menu.

[[Image:images/LTTng2DisableChannelAction.png]]

Upon successful operation, the selected channels will be '''DISABLED''' and the icons for the channels will be updated.

To enable one or more disabled channels, select the tree nodes of the relevant channels and press the right mouse button. Then select the '''Enable Channel''' menu item of the context-sensitive menu.

[[Image:images/LTTng2EnableChannelAction.png]]

Upon successful operation, the selected channels will be '''ENABLED''' and the icons for the channels will be updated.

=== Enabling Events - General ===

Enabling events can be done using different levels in the tree node. It can be done on the session, domain level and channel level. For the case of session or domain, i.e. when no specific channels is assigned then enabling of events is done on the default channel with the name '''channel0''' which created, if not already exists, by the LTTng tracer control on the server side.

=== Enabling Kernel Events On Session Level ===

To enable events, select the tree node of the relevant session and press the right mouse button. Then select the '''Enable Event (default channel)...''' button of the context-sensitive menu.

[[Image:images/LTTng2EventOnSessionAction.png]]

A dialog box will open for entering information about events to be enabled.

[[Image:images/LTTng2EventOnSessionDialog.png]]

By default the domain '''Kernel''' is selected and the kernel specific data sections are created. From this dialog box kernel '''Tracepoint''' events, '''System calls (Syscall)''', a '''Dynamic Probe''' or a '''Dynamic Function entry/return''' probe can be enabled. Note that events of one of these types at a time can be enabled.

To enable all '''Tracepoints''' and all '''System calls (Syscall)''', select the button '''Select''' of section '''All Tracepoint Events and Syscalls''' and press '''Ok'''.

[[Image:images/LTTng2EnableAllEventsDialog.png]]

Upon successful operation, the domain '''Kernel''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all a wildcard event '''*''' of type  '''TRACEPOINT''' under the channel and a wildcard event '''*''' of type '''SYSCALL''' . The channel and events will be '''ENABLED'''.

To enable '''Tracepoint''' events, first select the corresponding '''Select''' button, then select either all tracepoins (select '''All''') or select selectively one or more tracepoints in the displayed tree of tracepoints. You can also enter directly the name of the events you want to enable (comma separated list and wildcards are supported). Finally press '''Ok'''.

[[Image:images/LTTng2TracepointEventsDialog.png]]

Upon successful operation, the domain '''Kernel''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all requested events of type '''TRACEPOINT''' under the channel. The channel and events will be '''ENABLED'''.

[[Image:images/LTTng2EnabledKernelTracepoints.png]]

To enable '''Syscall''' events, first select the corresponding '''Select''' button, then select either all syscalls (select '''All''') or select selectively one or more syscalls in the displayed tree of syscalls. You can also enter directly the name of the events you want to enable (comma separated list and wildcards are supported). Finally press '''Ok'''.

[[Image:images/LTTng2SyscallsDialog.png]]

Upon successful operation, the domain '''Kernel''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all requested events of type '''SYSCALL''' under the channel. The channel and events will be '''ENABLED'''.

[[Image:images/LTTng2EnabledKernelSyscalls.png]]

To enable a '''Dynamic Probe''' event, select the corresponding '''Select''' button, fill the '''Event Name''' and '''Probe''' fields and press '''Ok'''. Note that the probe can be an address, symbol or a symbol+offset where the address and offset can be octal (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...).

[[Image:images/LTTng2ProbeEventDialog.png]]

Upon successful operation, the dynamic probe event with the given name and event type '''PROBE''' will be added under the default channel (channel0). If necessary the domain '''Kernel''' and the channel '''channel0''' will be created.

[[Image:images/LTTng2EnabledKernelProbeEvent.png]]

To enable a '''Dynamic Function entry/return Probe''' event, select the corresponding '''Select''' button, fill the '''Event Name''' and '''Function''' fields and press '''Ok'''. Note that the funtion probe can be an address, symbol or a symbol+offset where the address and offset can be octal (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...).

[[Image:images/LTTng2FunctionEventDialog.png]]

Upon successful operation, the dynamic function probe event with the given name and event type '''PROBE''' will be added under the default channel (channel0). If necessary the domain '''Kernel''' and the channel '''channel0''' will be created.

[[Image:images/LTTng2EnabledFunctionProbeEvent.png]]

=== Enabling UST Events On Session Level ===

For enabling UST events, first open the enable events dialog as described in section [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]] and select domain '''UST'''.

To enable '''Tracepoint''' events, first select the corresponding '''Select''' button, then select either all tracepoins (select '''All''') or select selectively one or more tracepoints in the displayed tree of tracepoints and finally press '''Ok'''.

[[Image:images/LTTng2UstTracepointEventsDialog.png]]

Upon successful operation, the domain '''UST global''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all requested events under the channel. The channel and events will be '''ENABLED'''. Note that for the case that '''All''' tracepoints were selected the wildcard '''*''' is used which will be shown in the Control View as below.

[[Image:images/LTTng2EnabledAllUstTracepoints.png]]

For UST it is possible to enable '''Tracepoint''' events using a wildcard. To enable '''Tracepoint''' events with a wildcard, select first the corresponding '''Select''' button, fill the '''Wildcard''' field and press '''Ok'''.

[[Image:images/LTTng2UstWildcardEventsDialog.png]]

Upon successful operation, the event with the given wildcard and event type '''TRACEPOINT''' will be added under the default channel (channel0). If necessary the domain '''UST global''' and the channel '''channel0''' will be created.

[[Image:images/LTTng2EnabledUstWildcardEvents.png]]

When enabling '''Tracepoint''' with wildcard, it is possible to specify event(s) (comma separated list) that we want to '''exclude''' from that wildcard selection. To '''exclude''' '''Tracepoint''' events, check the corresponding '''Select''' check box, fill the '''Event Names''' field and press '''Ok'''.

[[Image:images/LTTng2UstExcludeEventsDialog.png]]

For UST it is possible to enable '''Tracepoint''' events using log levels. To enable '''Tracepoint''' events using log levels, select first the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.

* '''Event Name''': Name to display
* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
* '''loglevel-only''': To specify that only the specified log level shall be configured

[[Image:images/LTTng2UstLoglevelEventsDialog.png]]

Upon successful operation, the event with the given event name and event type '''TRACEPOINT''' will be added under the default channel (channel0). If necessary the domain '''UST global''' and the channel '''channel0''' will be created.

[[Image:images/LTTng2EnabledUstLoglevelEvents.png]]

=== Enabling JUL Events On Session Level ===

For enabling JUL loggers, first open the enable events dialog as described in section [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]] and select domain '''JUL'''.

To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers and finally press '''Ok'''.

[[Image:images/LTTng2JulLoggerEventsDialog.png]]

Upon successful operation, the domain '''JUL''' will be created in the tree (if neccessary). With JUL loggers there is no channel, you see the enabled loggers directly under the '''JUL''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.

[[Image:images/LTTng2EnabledAllJulLoggers.png]]

For JUL it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.

* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
* '''loglevel-only''': To specify that only the specified log level shall be configured

[[Image:images/LTTng2JulLoglevelEventsDialog.png]]

=== Enabling LOG4J Events On Session Level ===

For enabling LOG4J loggers, first open the enable events dialog as described in section [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]] and select domain '''LOG4J'''.

To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers and finally press '''Ok'''.

[[Image:images/LTTng2Log4jLoggerEventsDialog.png]]

Upon successful operation, the domain '''LOG4J''' will be created in the tree (if neccessary). With LOG4J loggers there is no channel, you see the enabled loggers directly under the '''LOG4J''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.

[[Image:images/LTTng2EnabledAllLog4jLoggers.png]]

For LOG4J it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.

* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
* '''loglevel-only''': To specify that only the specified log level shall be configured

[[Image:images/LTTng2Log4jLoglevelEventsDialog.png]]

=== Enabling Python Events On Session Level ===

For enabling Python loggers, first open the enable events dialog as described in section [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]] and select domain '''Python'''.

To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers. You can also enter the name of your logger in the text field. Finally press '''Ok'''.

[[Image:images/LTTng2PythonLoggerEventsDialog.png]]

Upon successful operation, the domain '''Python''' will be created in the tree (if neccessary). With Python loggers there is no channel, you see the enabled loggers directly under the '''Python''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.

[[Image:images/LTTng2EnabledAllPythonLoggers.png]]

For Python it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.

* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
* '''loglevel-only''': To specify that only the specified log level shall be configured

[[Image:images/LTTng2PythonLoglevelEventsDialog.png]]

=== Enabling Events On Domain Level ===

Kernel events can also be enabled on the domain level. For that select the relevant domain tree node, click the right mouse button and the select '''Enable Event (default channel)...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''', '''UST global''', '''JUL''', '''LOG4J''' or '''Python''', the domain specific fields are shown and the domain selector is preselected and read-only.

[[Image:images/LTTng2EventOnDomainAction.png]]

Instructions for enalbing events for a particular domain can be found here:
* '''Kernel''' domain: [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]]
* '''UST global''' domain: [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]]
* '''JUL''' domain: [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]]
* '''LOG4J''' domain: [[#Enabling LOG4J Events On Session Level | Enabling LOG4J Events On Session Level]]
* '''Python''' domain: [[#Enabling Python Events On Session Level | Enabling Python Events On Session Level]]

The events will be added to the default channel '''channel0'''. This channel will be created by on the server side if necessary.

=== Enabling Events On Channel Level ===

Kernel events can also be enabled on the channel level. If necessary, create a channel as described in sections [[#Enabling Channels On Session Level | Enabling Channels On Session Level]] or [[#Enabling Channels On Domain Level | Enabling Channels On Domain Level]].

Then select the relevant channel tree node, click the right mouse button and the select '''Enable Event...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''' or '''UST global''', the domain specific fields are shown and the domain selector is preselected and read-only. Since there is no channel under the '''JUL''', '''LOG4J''' or '''Python''' domain you cannot enable those loggers directly from a channel. 

[[Image:images/LTTng2EventOnChannelAction.png]]

To enable events for domain '''Kernel''' follow the instructions in section [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]], for domain '''UST global''' [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]].

When enabling events on the channel level, the events will be add to the selected channel.

=== Enabling and Disabling Events ===

To disable one or more enabled events, select the tree nodes of the relevant events and click the right mouse button. Then select '''Disable Event''' menu item in the context-sensitive menu.

[[Image:images/LTTng2DisableEventAction.png]]

Upon successful operation, the selected events will be '''DISABLED''' and the icons for these events will be updated.

To enable one or more disabled events, select the tree nodes of the relevant events and press the right mouse button. Then select the '''Enable Event''' menu item of the context-sensitive menu.

[[Image:images/LTTng2EnableEventAction.png]]

Upon successful operation, the selected events will be '''ENABLED''' and the icons for these events will be updated.

'''Note''': There is currently a limitation for kernel event of type '''SYSCALL'''. This kernel event can not be disabled. An error will appear when trying to disable this type of event. A work-around for that is to have the syscall event in a separate channel and disable the channel instead of the event.

=== Enabling Tracepoint Events From Provider ===

It is possible to enable events of type '''Tracepoint''' directly from the providers and assign the enabled event to a session and channel. Before doing that a session has to be created as described in section [[#Creating a Tracing Session | Creating a Tracing Session]]. Also, if other than default channel '''channel0''' is required, create a channel as described in sections [[#Enabling Channels On Session Level | Enabling Channels On Session Level]] or [[#Enabling Channels On Domain Level | Enabling Channels On Domain Level]].

To assign tracepoint events to a session and channel, select the events to be enabled under the provider (e.g. provider '''Kernel'''), click right mouse button and then select '''Enable Event...''' menu item from the context sensitive menu.

[[Image:images/LTTng2AssignEventAction.png]]

A new display will open for defining the session and channel.

[[Image:images/LTTng2AssignEventDialog.png]]

Select a session from the '''Session List''' drop-down menu, a channel from the '''Channel List''' drop-down menu and the press '''Ok'''. Upon successful operation, the selected events will be added to the selected session and channel of the domain that the selected provider belongs to. In case that there was no channel available, the domain and the default channel '''channel0''' will be created for corresponding session. The newly added events will be '''ENABLED'''.

[[Image:images/LTTng2AssignedEvents.png]]

=== Configuring Filter Expression When Enabling Events ===

It is possible to provide a filter expression when enabling events for UST or Kernel. This feature has been available for UST since LTTng v2.1.0 and for Kernel since v2.7.0. To configure a filter expression, open the enable event dialog as described in previous chapters [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]], [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]], [[#Enabling Events On Domain Level | Enabling Events On Domain Level]] or [[#Enabling Events On Channel Level | Enabling Events On Channel Level]]. Then configure the relevant events and enter the filter expression in the '''Filter Expression''' text field.

[[Image:images/LTTng2EnableEventWithFilter.png]] [[Image:images/LTTng2EnableEventWithKernelFilter.png]]

Alternatively, open the dialog box for assigning events to a session and channel described in [[#Enabling Tracepoint Events From Provider | Enabling Tracepoint Events From Provider]] and enter the filter expression in the '''Filter Expression''' text field.

[[Image:images/LTTng2AssignEventDialogWithFilter.png]]

For the syntax of the filter expression refer to the '''LTTng Tracer Control Command Line Tool User Manual''' of chapter [[#References |References]].

=== Adding Contexts to Channels and Events of a Domain ===

It is possible to add contexts to channels and events. Adding contexts on channels and events from the domain level, will enable the specified contexts to all channels of the domain and all their events. To add contexts on the domain level, select a domain, click right mouse button on a domain tree node (e.g. provider '''Kernel''') and select the menu item '''Add Context...''' from the context-sensitive menu.

[[Image:images/LTTng2AddContextOnDomainAction.png]]

A new display will open for selecting one or more contexts to add.

[[Image:images/LTTng2AddContextDialog.png]]

The tree shows all available context that can be added. Select one or more context and the press '''Ok'''. Upon successful operation, the selected context will be added to all channels and their events of the selected domain.

'''Note''': The LTTng UST tracer only supports  contexts '''procname''', '''pthread_id''', '''vpid''' '''vtid'''. Adding any other contexts in the UST domina will fail.

=== Adding Contexts to All Events of a Channel ===

Adding contexts on channels and events from the channel level, will enable the specified contexts to all events of the selected channel. To add contexts on the channel level, select a channel, click right mouse button on a channel tree node and select the menu item '''Add Context...''' from the context-sensitive menu.

[[Image:images/LTTng2AddContextOnChannelAction.png]]

A new display will open for selecting one or more contexts to add. Select one or more contexts as described in chapter [[#Adding Contexts to Channels and Events of a Domain | Adding Contexts to Channels and Events of a Domain]]. Upon successful operation, the selected context will be added to all channels and their events of the selected domain. '''Note''' that the LTTng 2.0 tracer control on the remote host doesn't provide a way to retrieve added contexts. Hence it's not possible to display the context information in the GUI.

=== Adding Contexts to an Event of a Specific Channel ===

Adding contexts to an event of a channel is only available in LTTng Tools versions v2.0.0-2.1.x. The menu option won't be visible for LTTng Tools version v2.2.0 or later. To add contexts on an event select an event of a channel, click right mouse button on the corresponding event tree node and select the menu item '''Add Context...''' from the context-sensitive menu.

[[Image:images/LTTng2AddContextToEventsAction.png]]

A new display will open for selecting one or more contexts to add. Select one or more contexts as described in chapter [[#Adding Contexts to Channels and Events of a Domain | Adding Contexts to Channels and Events of a Domain]]. Upon successful operation, the selected context will be added to the selected event.

=== Start Tracing ===

To start tracing, select one or more sessions to start in the Control View and press the '''Start''' button. Alternatively, press the right mouse button on the session tree nodes. A context-sensitive menu will show. Then select the '''Start''' menu item.

[[Image:images/LTTng2StartTracingAction.png]]

Upon successful operation, the tracing session will be '''ACTIVE''' and the icon of the session will be updated.

=== Recording a Snapshot ===

LTTng Tools version v2.3.0 introduces the possibility to create snapshot tracing sessions. After creating a snapshot session (see [[#Creating a Snapshot Tracing Session | Creating a Snapshot Tracing Session]]) and starting tracing (see [[#Start Tracing | Start Tracing]]) it possible to record snapshots. To record a snapshot select one or more sessions and press the '''Record Snapshot''' button. Alternatively, press the right mouse button on the session tree nodes. A context-sensitive menu will show. Then select the '''Recored Snapshot''' menu item.

[[Image:images/LTTng2RecordSnapshotAction.png]]

This action can be executed many times. It is possible to import the recorded snpshots to a tracing project. The trace session might be '''ACTIVE''' or '''INACTIVE''' for that. Refer to section [[#Importing Session Traces to a Tracing Project | Importing Session Traces to a Tracing Project]] on how to import a trace to a tracing project.

=== Stop Tracing ===

To stop tracing, select one or more sessions to stop in the Control View and press the '''Stop''' button. Alternatively, click the right mouse button on the session tree node. A context-sensitive menu will show. Then select the '''Stop''' menu item.

[[Image:images/LTTng2StopTracingAction.png]]

Upon successful operation, the tracing session will be '''INACTIVE''' and the icon of the session will be updated.

=== Destroying a Tracing Session ===

To destroy a tracing session, select one or more sessions to destroy in the Control View and press the '''Destroy''' button. Alternatively, click the right mouse button on the session tree node. A context-sensitive menu will show. Then select the '''Destroy...''' menu item. Note that the session has to be '''INACTIVE''' for this operation.

[[Image:images/LTTng2DestroySessionAction.png]]

A confirmation dialog box will open. Click on '''Ok''' to destroy the session otherwise click on '''Cancel'''.

[[Image:images/LTTng2DestroyConfirmationDialog.png]]

Upon successful operation, the tracing session will be destroyed and removed from the tree.

=== Refreshing the Node Information ===

To refresh the remote host information, select any node in the tree of the Control View and press the '''Refresh''' button. Alternatively, click the right mouse button on any tree node. A context-sensitive menu will show. Then select the '''Refresh''' menu item.

[[Image:images/LTTng2RefreshAction.png]]

Upon successful operation, the tree in the Control View will be refreshed with the remote host configuration.

=== Importing Session Traces to a Tracing Project ===

To import traces from a tracing session, select the relevant session and click on the '''Import''' Button. Alternatively, click the right mouse button on the session tree node and select the menu item '''Import...''' from the context-sensitive menu.

[[Image:images/LTTng2ImportAction.png]]

A new display will open for selecting the traces to import.

[[Image:images/LTTng2ImportDialog.png]]

By default all traces are selected. A default project with the name '''Remote''' is selected which will be created if necessary. Update the list of traces to be imported, if necessary, by selecting and deselecting the relevant traces in the tree viewer. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Also if needed, change the tracing project from the '''Available Projects''' combo box. The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the session name. One can change the experiment name by typing a new name in the text box beside the option. 

Then press button '''Finish'''. Upon successful import operation the selected traces will be stored in the '''Traces''' directory of the specified tracing project. A directory with the connection name will be created under the '''Traces'''  directory. Underneath that, the session directory structure as well as the trace names will be preserved in the destination tracing project. For '''Kernel''' traces the trace type '''Linux Kernel Trace''' and for '''UST''' traces the trace type '''LTTng UST Trace''' will be set. From the '''Project Explorer''' view, the trace can be analyzed further.

'''Note''': If a trace already exists with the same name in the destination directory, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example kernel becomes kernel(2).

[[Image:images/LTTng2ImportOverwriteConfirmationDialog.png]]

If one selects '''Rename All''', '''Overwrite All''' or '''Skip All''' the choice will be applied for all traces with a name conflict.

=== Importing Network Traces to a Tracing Project ===

Since LTTng Tools v2.1.0 it is possible to store traces over the network. To import network traces, execute the '''Import''' action as described in chapter [[#Importing Session Traces to a Tracing Project | Importing Session Traces to a Tracing Project]]. For network traces the '''Import Trace Wizard''' will be displayed. Follow the instructions in chapter [[#Importing | Importing]] to import the network traces of the current session.

=== Saving Tracing Sessions ===
Since LTTng Tools v2.5.0 it is possible to save tracing sessions. The LTTng Tools command-line tool will save the sessions to XML files located by default in a subdirectory of the user's home directory. The Trace Compass '''Control''' view integration for this feature will also store this session profile file into the user's Trace Compass workspace. This will allow user's to re-use session profiles across remote nodes. To save one or more sessions, select the tree nodes of the relevant sessions and press the right mouse button. Then select the '''Save...''' entry of the context-sensitive menu.  

[[Image:images/LTTng2SaveAction.png]]

A new display will open for saving the sessions.

[[Image:images/LTTng2SaveDialog.png]]

By default the '''force''' button is selected that will overwrite any conflicting session profile files on the remote node. Click on '''Ok''' to save the session(s) otherwise click on '''Cancel'''. Upon successful operation, the session profile files will be saved on the remote node and then will be downloaded to the user's Trace Compass workspace. In the case that a session XML file already exists in the workspace the user will be prompted to skip or overwrite the existing profile file.

=== Loading Tracing Sessions ===
Since LTTng Tools v2.5.0 it is possible to load tracing sessions. The Trace Compass '''Control''' view integrations for this feature will allow to load session profiles that are located in the user's Trace Compass workspace, or alternatively, that are located on the remote node. In the first case the session profiles will be uploaded to the remote node before the load command is executed.

To load one or more sessions, select the tree node '''Sessions''' and press the right mouse button. Then select the '''Load...''' entry of the context-sensitive menu.

[[Image:images/LTTng2LoadAction.png]]

A new display will open for loading session profiles.

[[Image:images/LTTng2LoadDialog.png]]

By default the '''Local''' button and '''force''' buttons are selected and session profile files of the user's workspace will be listed. Select one or more profiles, update the '''force''' button if needed and then click '''Ok'''. This will upload the session profile files to the remote node. If a session profile file with the same name already exist on the remote node, it will be overwritten. If the '''force''' button is selected any existing session with a conflicting name will be destroyed and a new one will be created.

Alternatively, one can select the '''Remote''' button to list all available session profile files on the remote node. To load one of the remote session profiles, select one or more profiles, update the '''force''' button if needed and then click '''Ok'''.

[[Image:images/LTTng2LoadRemoteDialog.png]]

Upon successful operation, the tracing sessions of the selected session profiles are created and added under the tree node '''Sessions'''  the '''Control''' view.

=== Managing Tracing Session Profiles ===
The '''LTTng Remote Profiles''' preference page is used to manage the list of LTTng session profiles that are stored in the user's Trace Compass workspace. To open the preference page, select the '''Manage...''' button of the '''Load Sessions''' dialog described in chapter [[#Loading Tracing Sessions |Loading Tracing Sessions]]. Alternatively, select '''Window -> Preferences''' from the top level menu and go to '''Tracing -> LTTng Remote Profiles'''. 

[[Image:images/LTTng2ManageSessionConfig.png]]

The following actions can be performed from this dialog:

* Delete

Select one or more LTTng session profiles from the list and click the '''Delete''' button to remove the profile from the Trace Compass workspace. The user will be prompted to confirm the deletion.

* Import...

Click the '''Import...''' button and select a file from the opened file dialog to import a session profile file. If the file name conflicts with an existing profile file, the user will be prompted to skip or overwrite the existing profile file.
* Export...

Select one or more session profile files from the list, click the '''Export...''' button and enter or select a directory in the opened directory dialog to export the profile files. If the file name conflicts with an existing profile file in the destination directory, the user will be prompted to skip or overwrite the existing profile file.

== Properties View ==

The Control View provides property information of selected tree component. Depending on the selected tree component different properties are displayed in the property view. For example, when selecting the node level the property view will be filled as followed:

[[Image:images/LTTng2PropertyView.png]]

'''List of properties''':

* '''Host''' Properties
** '''Connection Name''': The alias name to be displayed in the Control View.
** '''Host Name''': The IP address or DNS name of the remote system.
** '''State''': The state of the connection ('''CONNECTED''', '''CONNECTING''', '''DISCONNNECTING''' or '''DISCONNECTED''').
* '''Kernel Provider''' Properties
** '''Provider Name''': The name of the provider.
* '''UST Provider''' Properties
** '''Provider Name''': The name of the provider.
** '''Process ID''': The process ID of the provider.
* '''Event''' Properties (Provider)
** '''Event Name''': The name of the event.
** '''Event Type''': The event type ('''TRACEPOINT''' only).
** '''Fields''': Shows a list of fields defined for the selected event. (UST only, since support for LTTng Tools v2.1.0)
** '''Log Level''': The log level of the event.
* '''Logger''' Properties (Provider)
** '''Logger Name''': The name of the logger.
** '''Logger Type''': The event type ('''TRACEPOINT''' only).
* '''Session''' Properties
** '''Session Name''': The name of the Session.
** '''Session Path''': The path on the remote host where the traces will be stored. (Not shown for snapshot sessions).
** '''State''': The state of the session ('''ACTIVE''' or '''INACTIVE''')
** '''Snapshot ID''': The snapshot ID. (Only shown for snapshot sessions).
** '''Snapshot Name''': The name of the snapshot output configuration. (Only shown for snapshot sessions).
** '''Snapshot Path''': The path where the snapshot session is located. (Only shown for snapshot sessions).
* '''Domain''' Properties
** '''Domain Name''': The name of the domain.
** '''Buffer Type''': The buffer type of the domain.
* '''Channel''' Properties
** '''Channel Name''': The name of the channel.
** '''Number of Sub Buffers''': The number of sub-buffers of the channel.
** '''Output type''': The output type for the trace (e.g. ''splice()'' or ''mmap()'')
** '''Overwrite Mode''': The channel overwrite mode ('''true''' for overwrite mode, '''false''' for discard)
** '''Read Timer Interval''': The read timer interval.
** '''State''': The channel state ('''ENABLED''' or '''DISABLED''')
** '''Sub Buffer size''': The size of the sub-buffers of the channel (in bytes).
** '''Switch Timer Interval''': The switch timer interval.
** '''Number of Discarded Events''': The number of discarded events of the channel.
** '''Number of Lost Packets''': The number of lost packets of the channel.
* '''Event''' Properties (Channel)
** '''Event Name''': The name of the event.
** '''Event Type''': The event type ('''TRACEPOINT''', '''SYSCALL''' or '''PROBE''').
** '''Log Level''': The log level of the event. (For LTTng Tools v2.4.0 or later, '''<=''' prior the log level name will indicate a range of log levels and '''==''' a single log level.)
** '''State''': The Event state ('''ENABLED''' or '''DISABLED''')
** '''Filter''': Shows '''with filter''' if a filter expression is configured else property '''Filter''' is omitted. (since support for LTTng Tools v2.1.0)
* '''Logger''' Properties (Domain)
** '''Logger Name''': The name of the logger.
** '''Logger Type''': The logger type ('''TRACEPOINT''').
** '''Log Level''': The log level of the logger. (For LTTng Tools v2.4.0 or later, '''<=''' prior the log level name will indicate a range of log levels and '''==''' a single log level.)
** '''State''': The logger state ('''ENABLED''' or '''DISABLED''')

== LTTng Tracer Control Preferences ==

Several LTTng 2.0 tracer control preferences exists which can be configured. To configure these preferences, select '''Window -> Preferences''' from the top level menu. The preference display will open. Then select '''Tracing -> LTTng Tracer Control Preferences'''. This preferences page allows the user to specify the tracing group of the user and to specify the command execution timeout as well as it allows the user to configure the logging of LTTng 2.0 tracer control commands and results to a file.

[[Image:images/LTTng2Preferences.png]]

To change the tracing group of the user which will be specified on each command line, enter the new group name in the '''Tracing Group''' text field and click button '''OK'''. The default tracing group is '''tracing''' and can be restored by pressing the '''Restore Defaults''' button.

[[Image:images/LTTng2PreferencesGroup.png]]

To configure logging of trace control commands and the corresponding command result to a file, selected the button  '''Logging'''. To append to an existing log file, select the '''Append''' button. Deselect the '''Append''' button to overwrite any existing log file. It's possible to specify a verbose level. There are 3 levels with inceasing verbosity from '''Level 1''' to '''Level 3'''. To change the verbosity level, select the relevant level or select '''None'''. If '''None''' is selected only commands and command results are logged. Then press on button '''OK'''. The log file will be stored in the users home directory with the name ''lttng_tracer_control.log''. The name and location cannot be changed. To reset to default preferences, click on the button '''Restore Defaults'''.

[[Image:images/LTTng2PreferencesLogging.png]]

To configure the LTTng command execution timeout, select '''Tracing -> Remote Connection Preferences''' and enter a timeout value into the text field '''Command Timeout (in seconds)'''. Then press on button '''OK'''. To reset to the default value of 15 seconds, click on the button '''Restore Defaults'''.

[[Image:images/LTTng2PreferencesTimeout.png]]


= LTTng Kernel Analysis =

Historically, LTTng was developed to trace the Linux kernel and, over time, a number of kernel-oriented analysis views were developed and organized in a perspective.

This section presents a description of the '''OS Tracing Overview''' perspective and the '''LTTng Kernel''' perspective.

== OS Tracing Overview Perspective ==

The '''OS Tracing Overview''' perspective groups the following views:

* [[#Project Explorer_View | Project Explorer View]]
* [[#Events_Editor | Events Editor]]
* [[#Histogram_View | Histogram View]]
* [[#LTTng CPU Usage View | CPU Usage View]]
* [[#Disk I/O Activity View | Disk I/O Activity View]]
* [[#Kernel Memory Usage View | Kernel Memory Usage View]]

The perspective can be opened from the Eclipse Open Perspective dialog ('''Window > Open Perspective... > Other''').

[[Image:images/osOverview/select_os_overview.png]]

This perspective is intended to be used to locate performance issues by observing resource usage.

The perspective can show times resource usage is anomalous. This can help locating the causes of system slowdowns in throughput or response time.

An example can be program that is doing a lot of processing then slows down due to a database access. The user will see a dip in CPU usage and maybe a slight rise in I/O access. The user should consider both spike and slums to be an indication of an area worth investigating.

[[Image:images/osOverview/os_overview_perspective.png]]

Once a performance issue has been localized, it can be further investigated with the #LTTng kernel Perspective.

== LTTng Kernel Perspective ==

The '''LTTng Kernel''' perspective is built upon the [[#Tracing_Perspective | Tracing Perspective]], re-organizes them slightly and adds the following views:

* [[#Control_Flow_View | Control Flow View]] - to visualize processes state transitions
* [[#Resources_View | Resources View]] - to visualize system resources state transitions
* [[#LTTng_Tracer_Control | LTTng Tracer Control]] - to configure LTTng tracing sessions remotely

[[Image:images/LTTngKernelPerspective.png]]


The perspective can be opened from the Eclipse Open Perspective dialog ('''Window > Open Perspective... > Other''').


[[Image:images/OpenLTTngKernelPerspective.png]]

== Control Flow View ==

The '''''Control Flow''''' view is a LTTng-specific view that shows per-process events graphically. The Linux Kernel Analysis is executed the first time a LTTng Kernel is opened. After opening the trace, the element '''Control Flow''' is added under the '''Linux Kernel Analysis''' tree element in the Project Explorer. To open the view,  double-click the '''Control Flow''' tree element.

[[Image:images/Cfv_show_view.png]]

Alternatively, select ''Control Flow'' under ''LTTng'' within the ''Show View'' window ('''Window''' -> '''Show View''' -> '''Other...'''):

You should get something like this:

[[Image:images/Cfv_global.png]]

The view is divided into the following important sections: '''process tree and information''', '''control flow''' and the '''toolbar'''. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).

The following sections provide detailed information for each part of the Control Flow View.

=== Process tree and information ===

Processes are organized as a tree within this view. This way, child and parent processes are easy to identify.

[[Image:images/Cfv_process_tree.png]]

The layout is based on the states computed from the trace events.

A given process may be shown at different places within the tree since the nodes are '''unique (TID, birth time) couples'''. This means that if process B of parent A dies, you'll still see it in the tree. If process A forks process B again, it will be shown as a different node since it won't have the same birth time (and probably not the same TID). This has the advantage that the tree, once loaded, never changes: horizontal scrolling within the [[#Control flow|control flow]] remains possible.

The TID column shows the process node's '''thread ID''' and the PTID column shows its '''parent thread ID''' (nothing is shown if the process has no parent).

It is possible to sort the columns of the tree by clicking on the column header. Subsequent clicking will change the sort order. The hierarchy, i.e. the parent-child relationship is kept. When opening a trace for the first time, the processes are sorted by '''birth time'''. The sort order and column will be preserved when switching between open traces. Note that when opening an experiment the processes will be sorted within each trace.

=== Control flow ===

This part of the Control Flow View is probably the most interesting one. Using the mouse, you can navigate through the trace (go left, right) and zoom on a specific region to inspect its details.

The colored bars you see represent '''states''' for the associated process node. When a process state changes in time, so does the color. For state '''SYSCALL''' the name of the system call is displayed in the state bar. States colors legend is available through a [[#Toolbar|toolbar button]]:

[[Image:images/Cfv_legend.png]]

This dark yellow is what you'll see most of the time since scheduling puts processes on hold while others run.

The vertical blue line with T1 above it is the '''current selection indicator'''. When a time range is selected, the region between the begin and end time of the selection will be shaded and two lines with T1 and T2 above will be displayed. The time stamps corresponding to T1, T2 and their delta are shown in the status line when the mouse is hovering over the control flow.

Arrows can be displayed that follow the execution of each CPU across processes. The arrows indicate when the scheduler switches from one process to another for a given CPU. The CPU being followed is indicated on the state tooltip. When the scheduler switches to and from the idle process, the arrow skips to the next process which executes on the CPU after the idle process. Note that an appropriate zoom level is required for all arrows to be displayed.

The display of arrows is optional and can be toggled using the '''Hide Arrows''' toolbar button. It is also possible to follow a CPU's execution across state changes and the scheduler's process switching using the '''Follow CPU Forward/Backward''' toolbar buttons.

==== Using the mouse ====

The following mouse actions are available:

* '''left-click''': select a time or time range begin time
* '''Shift-left-click or drag''': Extend or shrink the selection range

* '''left-drag horizontally''': select a time range or change the time range begin or end time
* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''click on a colored bar''': the associated process node is selected and the current time indicator is moved where the click happened
* '''mouse wheel up/down''': scroll up or down
* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically
* '''drag the time ruler horizontally''': zoom in or out with fixed start time
* '''double-click the time ruler''': reset zoom to full range

When the current time indicator is changed (when clicking in the states flow), all the other views are '''synchronized'''. For example, the [[#LTTng Kernel Events Editor|Events Editor]] will show the event matching the current time indicator. The reverse behaviour is also implemented: selecting an event within the Events View will update the Control Flow View current time indicator.

==== Using the keyboard ====

The following keyboard shortcuts are available:

*'''arrow-right key''': selects the next state for the selected process
*'''arrow-left key''': selects the previous state for the selected process
*'''Shift + arrow-right key''': updates the selection end time of the current selection range by selecting the next state of the current process
*'''Shift + arrow-left key''': updates the selection end time of the current selection range by selecting the previous state of the current process
*'''.''': selects the next active marker
*''',''': selects the previous active marker
*'''Shift + .''': updates the selection end time of the current selection range by selecting the next active marker boundary
*'''Shift + ,''': updates the selection end time of the current selection range by selecting the previous active marker boundary
*'''arrow-down''': selects the next process
*'''arrow-up''': selects the previous process
*'''Page Down''': selects the process down one page
*'''Page Up''': selects the process up one page
*'''Home''': selects the first process
*'''End''': selects the last process
*'''Enter''': toggles the expansion state of the current process in the tree
*'''+''': Zoom-in horizontally
*'''-''': Zoom-out horizontally
*'''Ctrl + +''': Zoom-in vertically
*'''Ctrl + -''': Zoom-out vertically
*'''Ctrl + 0''': Reset the vertical zoom
*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
When the mouse cursor is over entries (left pane):
*'''-''': Collapse selected entry
*'''+''': Expand selected entry
*'''*''': Expand selected entry to the level with at least one collapsed entry

'''Please note that the behavior of some shortcuts can slightly differ based on the operating system.'''

When the selection indicators are changed, all the other views are '''synchronized'''. For example, the [[#LTTng Kernel Events Editor|Events Editor]] will show the event matching the current time indicator. The reverse behaviour is also implemented: selecting an event within the Events View will update the Control Flow View current time indicator.

==== Incomplete regions ====

You'll notice '''small dots''' over the colored bars at some places:

[[Image:images/Cfv_small_dots.png]]

Those dots mean the underlying region is '''incomplete''': there's not enough pixels to view all the events. In other words, you have to zoom in.

When zooming in, small dots start to disappear:

[[Image:images/Cfv_zoom.png]]

When no dots are left, you are viewing '''all the events and states''' within that region.

==== Zoom region ====

To zoom in on a specific region, '''right-click and drag''' in order to draw a time range:

[[Image:images/Cfv_zoom_region.png]]

The states flow horizontal space will only show the selected region.

==== Tooltips ====

Hover the cursor over a colored bar and a '''tooltip''' will pop up:

[[Image:images/Cfv_tooltip.png]]

The tooltip indicates:

* the process name
* the pointed state name
* the CPU (if applicable)
* the system call name (if applicable)
* the pointed state date and start/stop times
* the pointed state duration (seconds)

=== Toolbar ===

<!-- Keep in sync with ref:resource-view-toolbar -->

The Control Flow View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:

{|
| [[Image:images/link.gif]]
| Align Views
| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in this view will disable this feature across all the views because it's a workspace preference.
|-
| [[Image:images/filter_items.gif]]
| Show View Filter
| Opens the process filter dialog. Filter settings will be preserved when switching between open traces.
|-
| [[Image:images/show_legend.gif]]
| Show Legend
| Displays the states legend.
|-
| [[Image:images/Optimization.png]]
| Optimize
| "Optimize" the row order of the control flow view. This groups the threads by minimizing the distance in the graph of transitions by CPU for a given time range. The scheduler will often keep tightly coupled threads on the same CPU to preserve data locality. An interesting side effect of this is that threads communicating together in that time range will be moved closer together when pressing the button. The button moves the rows only when pressed. When the time range is moved, the optimization action can be re-applied.
|-
| [[Image:images/home_nav.gif]]
| Reset the Time Scale to Default
| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
| Select Previous State Change
| Selects the previous state for the selected process. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_event.gif]]
| Select Next State Change
| Selects the next state for the selected process. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/add_bookmark.gif]]
| Add Bookmark...
| Adds a bookmark at the current selection range. A bookmark is a user-defined interval marker. The '''Add Bookmark''' dialog is opened where the user can enter a description and choose the highlighting color and alpha (transparency) value. This button is replaced by the '''Remove Bookmark''' button if the current selection range corresponds to an existing bookmark. The bookmarks can also be managed in the '''Bookmark View'''.
|-
| [[Image:images/remove_bookmark.gif]]
| Remove Bookmark
| Removes the bookmark at the current selection range. This button replaces the '''Add Bookmark''' when the current selection range corresponds to an existing bookmark.
|-
| [[Image:images/prev_bookmark.gif]]
| Previous Marker
| Selects the previous active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_bookmark.gif]]
| Next Marker
| Selects the next active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range. Clicking the button drop-down arrow will open a menu where marker categories can be made active or inactive for navigation. To toggle the display of a marker category, use the '''View Menu''' instead.
|-
| [[Image:images/prev_menu.gif]]
| Select Previous Process
| Selects the previous process.
|-
| [[Image:images/next_menu.gif]]
| Select Next Process
| Selects the next process.
|-
| [[Image:images/zoomin_nav.gif]]
| Zoom In
| Zooms in on the selection by 50%.
|-
| [[Image:images/zoomout_nav.gif]]
| Zoom Out
| Zooms out on the selection by 50%.
|-
| [[Image:images/hide_arrows.gif]]
| Hide Arrows
| Toggles the display of arrows on or off.
|-
| [[Image:images/follow_arrow_bwd.gif]]
| Follow CPU Backward
| Selects the previous state following CPU execution across processes. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/follow_arrow_fwd.gif]]
| Follow CPU Forward
| Selects the next state following CPU execution across processes. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/shift_l_edit.gif]]
| Go to previous event of the selected thread
| Move to the closest previous event belonging to the selected thread. This action looks through all trace events, unlike the ''Select Previous State Change'' action which only stops at state changes.
|-
| [[Image:images/shift_r_edit.gif]]
| Go to next event of the selected thread
| Move to the closest following event belonging to the selected thread. This action looks through all trace events, unlike the ''Select Next State Change'' action which only stops at state changes.
|}

View Menu

{|
|
| Show Markers
| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
|-
|
| Marker Set
| The user can select from one of the configured market sets, or choose '''None''' to use no marker set. The setting applies to all views that support marker sets. The marker set configuration can be edited by selecting the '''Edit...''' menu item (see [[#Marker Set Configuration XML Format]]). After saving the changes in the opened editor, the marker set should then be re-selected to update the view.
|-
|
| Thread Presentation
| Select the threads layout. Two layouts are available. '''Flat''' layout lists the threads in a flat list per trace. '''Hierarchical''' layout shows the threads in a parent-child tree per trace.
|}

=== Marker Axis ===

The marker axis is visible only when at least one marker category with markers for the current trace is shown.

The marker axis displays one row per marker category. Each marker's time range and/or label (if applicable) are drawn on the marker axis.

Clicking on any marker's time range or label will set the current time selection to the marker's time or time range.

Clicking on the "X" icon to the left of the marker category name will hide this marker category from the time graph. It can be shown again using the corresponding '''Show Markers''' menu item in the view menu.

The marker axis can be collapsed and expanded by clicking on the icon at the top left of the marker axis. The marker axis can be completely removed by hiding all available marker categories.

== Resources View ==

This view is specific to LTTng kernel traces.  The Linux Kernel Analysis is executed the first time a LTTng Kernel is opened. After opening the trace, the element '''Resources''' is added under the '''Linux Kernel Analysis''' tree element of the Project Explorer. To open the view, double-click the '''Resources''' tree element.

Alternatively, go in '''Window''' -> '''Show View''' -> '''Other...''' and select '''LTTng/Resources''' in the list.

[[Image:images/Rv_example.png|Example of resources view with all trace points and syscalls enabled]]

This view shows the state of system resources i.e. if changes occurred during the trace either on '''CPUs''', '''IRQs''' or '''soft IRQs''', it will appear in this view. The left side of the view present a list of resources that are affected by at least one event of the trace. The right side illustrate the state in which each resource is at some point in time. For state '''USERMODE''' it also prints the process name in the state bar. For state '''SYSCALL''' the name of the system call is
displayed in the state region.

When an '''IRQ''' is handled by a '''CPU''', its states are shown under the corresponding '''CPU''' entry. Similarly, the '''CPU''' that was handling an '''IRQ''' is shown under the handled '''IRQ'''. Therefore, the trace can be visualized from a '''CPU''' point of view or from an '''IRQ''' point of view.

Just like other views, according to which trace points and system calls are activated, the content of this view may change from one trace to another.

The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).

Each state are represented by one color so it is faster to say what is happening.

[[Image:images/Rv_legend.png|Color for each state]]

To go through the state of a resource, you first have to select the resource and the timestamp that interest you. For the latter, you can pick some time before the interesting part of the trace.

[[Image:images/RV_infobox1.png|Shows the state of an IRQ]]

Then, by selecting '''Next Event''', it will show the next state transition and the event that occurred at this time.

[[Image:images/RV_infobox2.png|Shows the next state of the IRQ]]

This view is also synchronized with the others : [[#Histogram_View | Histogram View]], [[#LTTng_Kernel_Events_Editor | Events Editor]], [[#Control_Flow_View | Control Flow View]], etc.

=== Follow CPU ===
It is possible to follow a CPU by right-clicking on its entry in the view, then selecting ''Follow CPU X'' where X is the number of the CPU. Following a CPU will filter the [[#LTTng CPU Usage View | CPU Usage View]] to display only usage for the selected CPU. To unfollow a CPU, one needs to right-click on any CPU entry and select ''Unfollow CPU''.

=== Navigation ===

See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.

=== Incomplete regions ===

See Control Flow View's '''[[#Incomplete_regions | Incomplete regions]]'''.

=== Toolbar ===

<!-- ref:resource-view-toolbar -->

The Resources View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:

{|
| [[Image:images/link.gif]]
| Align Views
| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in this view will disable this feature across all the views because it's a workspace preference.
|-
| [[Image:images/filter_items.gif]]
| Show View Filter
| Opens the resources filter dialog. Filter settings will be preserved when switching between open traces.
|-
| [[Image:images/show_legend.gif]]
| Show Legend
| Displays the states legend.
|-
| [[Image:images/home_nav.gif]]
| Reset the Time Scale to Default
| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
| Select Previous State Change
| Selects the previous state for the selected resource. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_event.gif]]
| Select Next State Change
| Selects the next state for the selected resource. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/add_bookmark.gif]]
| Add Bookmark...
| Adds a bookmark at the current selection range. A bookmark is a user-defined interval marker. The '''Add Bookmark''' dialog is opened where the user can enter a description and choose the highlighting color and alpha (transparency) value. This button is replaced by the '''Remove Bookmark''' button if the current selection range corresponds to an existing bookmark. The bookmarks can also be managed in the '''Bookmark View'''.
|-
| [[Image:images/remove_bookmark.gif]]
| Remove Bookmark
| Removes the bookmark at the current selection range. This button replaces the '''Add Bookmark''' when the current selection range corresponds to an existing bookmark.
|-
| [[Image:images/prev_bookmark.gif]]
| Previous Marker
| Selects the previous active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_bookmark.gif]]
| Next Marker
| Selects the next active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range. Clicking the button drop-down arrow will open a menu where marker categories can be made active or inactive for navigation.
|-
| [[Image:images/prev_menu.gif]]
| Select Previous Resource
| Selects the previous resource
|-
| [[Image:images/next_menu.gif]]
| Select Next Resource
| Selects the next resource
|-
| [[Image:images/zoomin_nav.gif]]
| Zoom In
| Zooms in on the selection by 50%.
|-
| [[Image:images/zoomout_nav.gif]]
| Zoom Out
| Zooms out on the selection by 50%.
|}

View Menu

{|
|
| Show Markers
| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.

| Marker Set
| The user can select from one of the configured market sets, or choose '''None''' to use no marker set. The setting applies to all views that support marker sets. The marker set configuration can be edited by selecting the '''Edit...''' menu item (see [[#Marker Set Configuration XML Format]]). After saving the changes in the opened editor, the marker set should then be re-selected to update the view.
|}

=== Marker Axis ===

See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.

== LTTng CPU Usage View ==

The CPU Usage analysis and view is specific to LTTng Kernel traces. The CPU usage is derived from a kernel trace as long as the '''sched_switch''' event was enabled during the collection of the trace. This analysis is executed the first time that the CPU Usage view is opened after opening the trace. To open the view, double-click on the '''CPU Usage''' tree element under the '''Linux Kernel Analysis''' tree element of the Project Explorer.

[[Image:images/LTTng_OpenCpuUsageView.png]]

Now, the CPU Usage view will show:

[[Image:images/LTTng_CpuUsageView.png]]

The view is divided into the following important sections: '''Process Information''' and the '''CPU Usage Chart'''. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).


=== Process Information ===

The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID), process name (Process), the average (%) execution time and the actual execution time (Time) during the current time range. It shows all threads that were executing on the CPUs in the current time range.


=== CPU Usage Chart ===

The CPU Usage Chart on the right side of the view, plots the total time spent on all CPUs of all processes and the time of the selected process.


==== Using the mouse ====

The CPU Usage chart is usable with the mouse. The following actions are set:

* '''left-click''': select a time or time range begin time
* '''Shift-left-click or drag''': Extend or shrink the selection range

* '''left-drag horizontally''': select a time range or change the time range begin or end time
* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally

==== Tooltips ====

Hover the cursor over a line of the chart and a tooltip will pop up with the following information:
* '''time''': current time of mouse position
* '''Total''': The total CPU usage


==== Toolbar ====

The CPU Usage View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:

{|
| [[Image:images/link.gif]]
| Align Views
| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference
|-
|}

[[Image:images/LTTng_CpuUsageViewToolTip.png]]

==== CPU Filtering ====
[[#Follow CPU | Follow a CPU]] will filter the CPU Usage View and will display only usage for the followed CPU.

== Kernel Memory Usage View ==

The Kernel Memory Usage and view is specific to kernel traces. To open the view, double-click on the '''Kernel Memory Usage Analysis''' tree element under the '''Kernel''' tree element of the Project Explorer.

[[Image:images/kernelMemoryUsage/OpenKernelMemoryUsageView.png]]

Now, the Kernel memory usage view will show:

[[Image:images/kernelMemoryUsage/KernelMemoryUsageView.png]]

Where:

* '''TID''': The ID of the thread this event belongs to
* '''Process''': The process of the TID that belongs to it

The view is divided into the following important sections: '''Process Information''' and the '''Relative Kernel Memory Usage'''. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).


=== Process Information ===

The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID) and the process name (Process).


=== Relative Kernel Memory Chart ===

The Relative Kernel Memory Chart on the right side of the view, plots the relative amount of memory that was allocated and deallocated during that period of time.


==== Using the mouse ====

The Relative Kernel Memory chart is usable with the mouse. The following actions are set:

* '''left-click''': select a time or time range begin time
* '''Shift-left-click or drag''': Extend or shrink the selection range

* '''left-drag horizontally''': select a time range or change the time range begin or end time
* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally


==== Tooltips ====

Hover the cursor over a line of the chart and a tooltip will pop up with the following information:
* '''time''': current time of mouse position
* '''Total''': The total CPU usage

[[Image:images/kernelMemoryUsage/KernelMemoryUsageChart.png]]

== Process Wait Analysis ==

TraceCompass can recover wait causes of local and distributed processes using operating system events. The analysis highlights the tasks and devices causing wait. Wait cause recovery is recursive, comprise all tasks running on the system and works across computers using packet trace synchronization.

The analysis details are available in the paper [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7294678&isnumber=4359390 Wait analysis of distributed systems using kernel tracing].

=== Prerequisites ===

The analysis requires a Linux kernel trace. Additional instrumentation may be required for specific kernel version and for distributed tracing. This instrumentation is available in [https://github.com/giraldeau/lttng-modules/tree/addons LTTng modules addons] on GitHub.

The required events are:
* '''sched_switch, sched_wakeup''': Scheduling events indicate when a process is blocked and the wake-up event indicates the task or resource that unblocked the task. For kernel versions comprised between 3.8 and 4.1, the event '''sched_ttwu''' (which stands for Try To Wake-Up) is provided for backward compatibility in LTTng modules addons.
* '''IRQ, SoftIRQ and IPI''': Interrupt events are required to distinguish the context of the wake-up. When a wake-up occurs inside an interrupt handler, it must be associated with the device causing the interrupt and not the interrupted task. For that reason, interrupt entry and exit events are required.
* '''inet_sock_local_in, inet_sock_local_out''': The network events record a subset of TCP/IP packet header using a netfilter hook in the kernel. The send and receive events are matched to show the communication between distributed processes. Network events are mandatory for analyzing wait in TCP/IP programs, whether they are executing locally or on different computers. They also used to synchronize traces recorded on multiple computers. For further details, refer to the [[#Trace synchronization]] section.

To analyze a distributed program, all computers involved in the processing must be traced simultaneously. The LTTng Tracer Control of TraceCompass can trace a remote computer, but controlling simultaneous tracing is not supported at the moment, meaning that all sessions must be started separately and interactively. TraceCompass will support this feature in the future. For now, it is suggested to use [https://github.com/giraldeau/lttng-cluster lttng-cluster] command line tool to control simultaneous tracing sessions on multiple computers. This tool is based on [http://www.fabfile.org/ Fabric] and uses SSH to start the tracing sessions, execute a workload, stop the sessions and gather traces on the local computer. For more information, refer to the lttng-cluster documentation.

We use the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] as an example to demonstrate the wait analysis. [https://www.djangoproject.com/ Django] is a popular Web framework. The application is the [https://docs.djangoproject.com/en/1.9/intro/tutorial01/ Django Poll app tutorial]. The traces were recorded on three computers, namely the client (implemented with Python Mechanize), the Web server (Apache with WSGI) and the database server (PostgreSQL). The client simulates a vote in the poll.

=== Running the analysis ===

To open all three traces simultaneously, we first create an experiment containing these traces and then synchronize the traces, such that they have a common time base. Then, the analysis is done by selecting a task in the '''Control Flow View'''. The result is displayed in the '''Critical Flow View''', which works like the '''Control Flow View'''. The steps to load the Django example follows.

# Download and extract the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] archive.
# In TraceCompass, open the [[#LTTng Kernel Perspective]].
# Create a new tracing project. Select '''File -> New -> Tracing -> Tracing Project''', choose a name and click '''Finish'''.
# Under the created tracing project, right-click on '''Traces''' and select '''Import...'''. In the import dialog, select the root directory containing the extracted trace by clicking on '''Browse'''. Three traces should be listed. Select the traces and click '''Finish'''. After the import is completed, the traces should be listed below '''Traces'''.
# Right-click on '''Experiments''', select '''New...''' and enter a name for the experiment, such as '''django'''.
# Right-click on the '''django''' experiment and click on '''Select Traces...'''. In the dialog, check the three traces '''django-client''', '''django-httpd''' and '''django-db'''. These traces will appear below the experiment. If the experiment is opened at this stage, the traces are not synchronized and there will be a large time gap between events from different traces.
# To synchronize the traces, right-click on the '''django''' experiment and select '''Synchronize Traces'''. In the '''Select reference trace''' dialog, select any available trace and click '''Finish'''. Once the synchronization is completed, a new entry with an underline suffix will appear for each modified trace. The created trace entries have a function which is applied to the timestamps of events in order to shift the time according to the reference trace. The '''Project Explorer''' after the import is shown below.
#:[[Image:images/waitAnalysis/KernelWaitAnalysisProjectExplorer.png]]
# Open the experiment '''django'''. The '''Control Flow''' and the '''Resources''' views should display the three traces simultaneously.
# In the main menu, select '''Window -> Show View -> Other...''' and under '''LTTng''' select '''Critical Flow View'''. The view is empty for the moment.
# In the '''Critical Flow View''', right-click on the '''Process''' entry to analyze and select '''Follow''', as shown in the figure below.
#:[[Image:images/waitAnalysis/KernelWaitAnalysisFollow.png]]
#:The analysis will execute and the result will appear in the '''Critical Flow View'''. For the Django example, use the '''View Filters''' to search for the python process with TID 2327. When zooming on the execution, the view displays the work done by the Web server and the database to process the request of the python client. Vertical arrows represent synchronization and communication between processes. The legend [[Image:images/show_legend.gif]] displays the colors associated with the processes states.

[[Image:images/waitAnalysis/KernelWaitAnalysisDjango.png]]

== Input/Output Analysis ==

TraceCompass can analyse disk input/output through the read/write system calls to get the read/write per processes, but also with the disk request events, to get the actual reads and writes to disk.

=== Get the trace ===

The following tracepoints should be enabled to get the disk read/write data. Also, enabling syscalls will allow to match the reads and writes per processes.

    # sudo lttng list -k
    Kernel events:
    -------------
      ...
      block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
      block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
      ...

For full disk request tracking, some extra tracepoints are necessary. They are not required for the I/O analysis, but make the analysis more complete. Here is the procedure to get those tracepoints that are not yet part of the mainline kernel.

    # git clone https://github.com/giraldeau/lttng-modules.git
    # cd lttng-modules

Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.

    # git checkout addons
    # make
    # sudo make modules_install
    # sudo depmod -a

The lttng addons modules must be inserted manually for the extra tracepoints to be available:

    # sudo modprobe lttng-addons
    # sudo modprobe lttng-elv

And enable the following tracepoint

    addons_elv_merge_requests

=== Input/Output Views ===

The following views are available for input/output analyses:

==== Disk I/O Activity View ====
A time aligned XY chart of the read and write speed for the different disks on the system. This view is useful to see where there was more activity on the disks and whether it was mostly reads or writes.

 [[Image:images/io/diskIoActivity.png| Disk I/O Activity Example]]

== System Call Latency Analysis ==

The '''System Call Latency Analysis''' measures the system call latency between system call entry and exit per type of system call. The durations are visualized using the '''Latency''' views. For more information about the '''Latency''' views see chapter [[#Latency_Analyses | Latency Analyses]].

== Futex Contention Latency Analysis ==

The '''Futex Contention Latency Analysis''' measures the futexes contention latency between futex entry and exit event for a thread. The durations are visualized using the '''Latency''' views. For more information about the '''Latency''' views see chapter [[#Latency_Analyses | Latency Analyses]].

The following views are also available for the Futex Contention Latency Analysis:

=== Uaddr vs Waiter ===
A timegraph view of the waiters by futex uaddr. This view is useful to see which threads are waiting on a specific futex and understand blocked threads.

[[Image:images/futex/uaddrVsWaiter.png| uaddr vs waiter example]]

=== Scenarios ===
A timegraph view of the futex wait/lock and wake/unlock scenarios (from futex entry to futex exit). This view is useful to suss up the general level of contention in a given trace. It highlights futex lifespans.

[[Image:images/futex/scenario.png| scenario example]]

== LTTng Kernel Events Editor ==

The LTTng Kernel Events editor '''is''' the plain TMF [[#Events_Editor | Events Editor]], except that it provides its own specialized viewer to replace the standard one. In short, it has exactly the same behaviour but the layout is slightly different:

* '''Timestamp''': the event timestamp
* '''Channel''': the event channel (data collector)
* '''CPU''': the CPU on which the event was taken
* '''Event Type''': the event type (or kernel marker)
* '''Contents''': the fields (or payload) of this event
* '''TID''': The ID of the thread this event belongs to
* '''Prio''': The priority of the thread this event belongs to

[[Image:images/LTTng2EventsEditor.png]]

= LTTng-UST Analyses =

The Userspace traces are taken on an application level. With kernel traces, you know what events you will have as the domain is known and cloistered. Userspace traces can contain pretty much anything. Some analyses are offered if certain events are enabled.

== Call Stack View ==

The Call Stack view allows the user to visualize the call stack per thread over time, if the application and trace provide this information.

To open this view go in '''Window''' -> '''Show View''' -> '''Other...''' and select '''Tracing/Call Stack''' in the list. The view shows the call stack information for the currently selected trace. Conversely, you can select a trace and expand it in the '''Project Explorer''' then expand '''LTTng-UST CallStack Analysis''' (the trace must be loaded) and open '''Call Stack'''.

The table on the left-hand side of the view shows the threads and call stack. The function name, depth, entry and exit time and duration are shown for the call stack at the selected time.

Double-clicking on a function entry in the table will zoom the time graph to the selected function's range of execution.

The time graph on the right-hand side of the view shows the call stack state graphically over time. The function name is visible on each call stack event if size permits. The color of each call stack event is randomly assigned based on the function name, allowing for easy identification of repeated calls to the same function.

Clicking on the time graph will set the current time and consequently update the table with the current call stack information.

Shift-clicking on the time graph will select a time range. When the selection is a time range, the begin time is used to update the stack information.

Double-clicking on a call stack event will zoom the time graph to the selected function's range of execution.

Clicking the '''Select Next State Change''' or '''Select Previous State Change''' or using the left and right arrows will navigate to the next or previous call stack event, and select the function currently at the top of the call stack. Note that pressing the '''Shift''' key at the same time will update the selection end time of the current selection.

Clicking the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) icon will open a file selection dialog, allowing you to import a text file containing mappings from function addresses to function names. If the callstack provider for the current trace type only provides function addresses, a mapping file will be required to get the function names in the view. See the following sections for an example with LTTng-UST traces.

=== Using the Callstack View with LTTng-UST traces ===

There is support in the LTTng-UST integration plugin to display the callstack
of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see
the ''liblttng-ust-cyg-profile'' man page for additional information). To do
so, you need to:

* Recompile your application with "''-g -finstrument-functions''".
* Set up a tracing session with the the ''vpid'', ''vtid'' and ''procname'' contexts. See the [[#Enabling UST Events On Session Level]] and [[#Adding Contexts to Channels and Events of a Domain]] sections. Or if using the command-line:
** <pre>lttng enable-event -u -a</pre>
** <pre>lttng add-context -u -t vpid -t vtid -t procname</pre>
* Preload the ''liblttng-ust-cyg-profile'' library when running your program:
** <pre>LD_PRELOAD=/usr/lib/liblttng-ust-cyg-profile.so ./myprogram</pre>

Once you load the resulting trace, the Callstack View should be populated with
the relevant information.

Note that for non-trivial applications, ''liblttng-ust-cyg-profile'' generates a
'''lot''' of events! You may need to increase the channel's subbuffer size to
avoid lost events. Refer to the
[http://lttng.org/docs/#doc-fine-tuning-channels LTTng documentation].

For traces taken with LTTng-UST 2.8 or later, the Callstack View should show the
function names automatically, since it will make use of the debug information
statedump events (which are enabled when using ''enable-event -u -a'').

For traces taken with prior versions of UST, you would need to set the path to
the binary file or mapping manually:

=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===

''For LTTng-UST 2.8+, if it doesn't resolve symbols automatically, see the [[#Binary file location configuration | Source Lookup's Binary file location configuration]].''

If you followed the steps in the previous section, you should have a Callstack
View populated with function entries and exits. However, the view will display
the function addresses instead of names in the intervals, which are not very
useful by themselves. To get the actual function names, you need to:

* Click the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) button in the Callstack View.

Then either:
* Point to the binary that was used for taking the trace
OR
* Generate a mapping file from the binary, using:
** <pre>nm myprogram > mapping.txt</pre>
** Select the ''mapping.txt'' file that was just created.

(If you are dealing with C++ executables, you may want to use ''nm --demangle''
instead to get readable function names.)

The view should now update to display the function names instead. Make sure the
binary used for taking the trace is the one used for this step too (otherwise,
there is a good chance of the addresses not being the same).

=== Navigation ===

See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.

=== Marker Axis ===

See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.

== Flame Graph View ==

This is an aggregate view of the function calls from the '''Call Stack View'''. This shows a bird's eye view of what are the main
time sinks in the traced applications. Each entry in the '''Flame Graph''' represents an aggregation of all the calls to a function
in a certain depth of the call stack having the same caller. So, functions in the '''Flame Graph''' are aggregated by depth and
caller. This enables the user to find the most executed code path easily.

* In a '''Flame Graph''', each entry (box) represents a function in the stack.
* If one takes a single vertical line in the view, it represents a full call stack with parents calling children.
* The ''x-axis'' represents total duration (execution time) and not absolute time, so it is not aligned with the other views.
* The width of an entry is the total time spent in that function, including time spent calling the children.
* The total time can exceed the longest duration, if the program is pre-empted and not running during its trace time.
* Each thread traced makes its own flame graph.

The function name is visible on each Flame graph event if the size permits. Each box in the '''Flame Graph'''
has the same color as the box representing the same function in the '''Call Stack'''.

To open this view select a trace, expand it in the '''Project Explorer''' then expand the
'''Call Graph Analysis''' (the trace must be loaded) and open the '''Flame Graph'''.
It's also possible to go in '''Window''' -> '''Show View''' -> '''Tracing''' then
select '''Flame Graph''' in the list.

[[Image:images/Flame_Graph.png|Flame Graph View]]

To use the '''Flame graph''', one can navigate it and find which function is consuming the most self-time.
This can be seen as a large plateau. Then the entry can be inspected. At this point, the worst offender in
terms of CPU usage will be highlighted, however, it is not a single call to investigate, but rather the
aggregation of all the calls. Right mouse-clicking on that entry will open a context sensitive menu.
Selecting '''Go to minimum''' or '''Go to maximum''' will take the user to the minimum or maximum
recorded times in the trace. This is interesting to compare and contrast the two.

Hovering over a function will show a tooltip with the statistics on a per-function basis. One can see the total and self times
(''worst-case'', ''best-case'', ''average'', ''total time'', ''standard deviation'', ''number of calls'') for that function.

If one wishes to explore at a medium detail level between the "classic" flame graph view and the call stack view, a per-thread flame
graph view is available by selection the coarser menu and clicking on '''Content Presentation''' then '''Per-thread'''. To return to
the default mode, return to that menu and click on '''Aggregate Threads'''.

=== How to use a Flame Graph ===

Observing the time spent in each function can show where most of the time is spent and where one could optimize.
An example in the image above: one can see that ''mp_sort'' is a recursive sort function, it takes approximately
40% of the execution time of the program. That means that perfectly parallelizing it can yield a gain of 20% for 2 threads, 33% for 3
and so forth. Looking at the function '''print_current_files''', it takes about 30% of the time, and it has a child ''print_many_per_line'' that has a large
self time (above 10%). This could be another area that can be targeted for optimization. Knowing this in advance helps developers
know where to aim their efforts.

It is recommended to have a kernel trace as well as a user space trace in an experiment
while using the '''Flame Graph''' as it will show what is causing the largest delays.
When using the '''Flame Graph''' together with a call stack and a kernel trace,
an example work flow would be to find the worst offender in terms of time taken for a function
that seems to be taking too longThen, using the context menu '''Go to maximum''', one can navigate
to the maximum duration and see if the OS is, for example, preempting the function for too long,
or if the issue is in the code being executed.

=== Using the mouse ===

*'''Double-click on the duration ruler''' will zoom the graph to the selected duration range.
*'''Shift-left-click or drag''': Extend or shrink the selection range
*'''Mouse wheel up/down''': scroll up or down
* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically

When the mouse cursor is over entries (left pane):

*'''-''': Collapse the '''Flame Graph''' of the selected thread
*'''+''': Expand the '''Flame Graph''' of the selected thread

=== Using the keyboard ===

The following keyboard shortcuts are available:

*'''Down Arrow''': selects the next stack depth
*'''Up Arrow''': selects the previous stack depth
*'''Home''': selects the first thread's '''Flame Graph'''
*'''End''': selects the last thread's  '''Flame Graph''''s deepest depth
*'''Enter''': toggles the expansion state of the current thread in the tree
*'''Ctrl + +''': Zoom-in vertically
*'''Ctrl + -''': Zoom-out vertically
*'''Ctrl + 0''': Reset the vertical zoom

=== Toolbar ===

{|
| [[Image:images/sort_alpha.gif]]
| Sort by thread name
| Sort the threads by thread name. Clicking the icon a second time will sort the threads by name in reverse order and change the icon to [[Image:images/sort_alpha_rev.gif]]
|-
| [[Image:images/sort_num.gif]]
| Sort by thread id
| Sort the threads by thread ID. Clicking the icon a second time will sort the threads by ID in reverse order and change the icon to [[Image:images/sort_num_rev.gif]].
|}

=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===

See Call Stack View's '''[[#Call Stack View | Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ]]'''.

== Function Duration Density ==
The '''Function Duration Density''' view shows the function duration of function displayed by duration for the current active time window range. This is useful to find global outliers.

[[Image:images/FunctionDensityView.png|Function Duration Density View]]

Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset.

== Memory Usage ==

The Memory Usage view allows the user to visualize the active memory usage per thread over time, if the application and trace provide this information.

The view shows the memory consumption for the currently selected trace.

The time chart plots heap memory usage graphically over time. There is one line per process, unassigned memory usage is mapped to "Other".

In this implementation, the user needs to trace while hooking the ''liblttng-ust-libc-wrapper'' by running  ''LD_PRELOAD=liblttng-ust-libc-wrapper.so'' '''<exename>'''. This will add tracepoints to memory allocation and freeing to the heap, NOT shared memory or stack usage. If the contexts '''vtid''' and '''procname''' are enabled, then the view will associate the heap usage to processes. As detailed earlier, to enable the contexts, see the [[#Adding Contexts to Channels and Events of a Domain]] section. Or if using the command-line:
* <pre>lttng add-context -u -t vtid -t procname</pre>

If thread information is available the view will look like this:

[[Image:images/memoryUsage/memory-usage-multithread.png]]

If thread information is not available it will look like this:

[[Image:images/memoryUsage/memory-usage-no-thread-info.png]]

The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).

Please note this view will not show shared memory or stack memory usage.

=== Using the mouse ===

The Memory Usage chart is usable with the mouse. The following actions are set:

* '''left-click''': select a time or time range begin time
* '''Shift-left-click or drag''': Extend or shrink the selection range

* '''left-drag horizontally''': select a time range or change the time range begin or end time
* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally

=== Toolbar ===

The Memory Usage View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:

{|
| [[Image:images/link.gif]]
| Align Views
| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference
|-
|}

[[Image:images/LTTng_CpuUsageViewToolTip.png]]

Please note this view will not show shared memory or stack memory usage.

== Source Lookup (for LTTng-UST 2.8+) ==

Starting with LTTng 2.8, the tracer can now provide enough information to
associate trace events with their location in the original source code.

To make use of this feature, first make sure your binaries are compiled with
debug information (-g), so that the instruction pointers can be mapped to source
code locations. This lookup is made using the ''addr2line'' command-line utility,
which needs to be installed and on the '''$PATH''' of the system running Trace
Compass. ''addr2line'' is available in most Linux distributions, Mac OS X, Windows using Cygwin and others.

The following trace events need to be present in the trace:

* lttng_ust_statedump:start
* lttng_ust_statedump:end
* lttng_ust_statedump:bin_info
* lttng_ust_statedump:build_id

as well as the following contexts:

* vpid
* ip

For ease of use, you can simply enable all the UST events when setting up your
session:

  lttng enable-event -u -a
  lttng add-context -u -t vpid -t ip

Note that you can also create and configure your session using the [[#Control View | Control View]].

If you want to track source locations in shared libraries loaded by the
application, you also need to enable the "lttng_ust_dl:*" events, as well
as preload the UST library providing them when running your program:

  LD_PRELOAD=/path/to/liblttng-ust-dl.so ./myprogram

If all the required information is present, then the ''Source Location'' column
of the Event Table should be populated accordingly, and the ''Open Source Code''
action should be available. Refer to the section [[#Event Source Lookup]] for
more details.

The ''Binary Location'' information should be present even if the original
binaries are not available, since it only makes use of information found in the
trace. A '''+''' denotes a relative address (i.e. an offset within the object
itself), whereas a '''@''' denotes an absolute address, for
non-position-independent objects.

[[Image:images/sourceLookup/trace-with-debug-info.png]]

''Example of a trace with debug info and corresponding Source Lookup information, showing a tracepoint originating from a shared library''

=== Binary file location configuration ===

To resolve addresses to function names and source code locations, the analysis
makes use of the binary files (executables or shared libraries) present on the
system. By default, it will look for the file paths as they are found in the
trace, which means that it should work out-of-the-box if the trace was taken on
the same machine that Trace Compass is running.

It is possible to configure a ''root directory'' that will be used as a prefix
for all file path resolutions. The button to open the configuration dialog is
called '''Configure how addresses are mapped to function names''' and is
currently located in the [[#Call Stack View]]. Note that the Call Stack View
will also make use of this configuration to resolve its function names.

[[Image:images/sourceLookup/symbol-mapping-config-ust28.png]]

''The symbol configuration dialog for LTTng-UST 2.8+ traces''

This can be useful if a trace was taken on a remote target, and an image of that
target is available locally.

If a binary file is being traced on a target, the paths in the trace will refer
to the paths on the target. For example, if they are:

* /usr/bin/program
* /usr/lib/libsomething.so
* /usr/local/lib/libcustom.so

and an image of that target is copied locally on the system at
''/home/user/project/image'', which means the binaries above end up at:

* /home/user/project/image/usr/bin/program
* /home/user/project/image/usr/lib/libsomething.so
* /home/user/project/image/usr/local/lib/libcustom.so

Then selecting the ''/home/user/project/image'' directory in the configuration
dialog above will allow Trace Compass to read the debug symbols correctly.

Note that this path prefix will apply to both binary file and source file
locations, which may or may not be desirable.

= Trace synchronization =

It is possible to synchronize traces from different machines so that they have the same time reference. Events from the reference trace will have the same timestamps as usual, but the events from traces synchronized with the first one will have their timestamps transformed according to the formula obtained after synchronization.

== Obtain synchronizable traces ==

To synchronize traces from different machines, they need to exchange packets through the network and have events enabled such that the data can be matched from one trace to the other. For now, only TCP packets can be matched between two traces.

LTTng traces that can be synchronized are obtained using one of two methods (both methods are compatible):

=== LTTng-module network tracepoint with complete data ===

The tracepoints '''net_dev_queue''' and '''netif_receive_skb''' will be used for synchronization. Both tracepoints are available in lttng-modules since version 2.2, but they do not contain sufficient data to be used to synchronize traces.

An experimental branch introduces this extra data: lttng-modules will need to be compiled by hand.

Obtain the source code for the experimental lttng-modules

    # git clone git://git.dorsal.polymtl.ca/~gbastien/lttng-modules.git
    # cd lttng-modules

Checkout the ''net_data_experimental'' branch, compile and install lttng-modules as per the lttng-modules documentation

    # git checkout net_data_experimental
    # make
    # sudo make modules_install
    # sudo depmod -a

This experimental branch adds IP, IPv6 and TCP header data to the tracepoints. Packets received and sent with other protocols do not have this extra header data, but all packets are captured.

=== LTTng-modules addons kernel module with dynamic tracepoints ===

This method adds dynamic instrumentation on TCP packets via extra kernel modules. Only TCP packets are captured.

Obtain the source code, along with lttng-modules

    # git clone https://github.com/giraldeau/lttng-modules.git
    # cd lttng-modules

Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation. The ''make'' command will fail at first with a message about the unset SYSMAP variable. Instructions on how to generate a System.map are mentioned in the error message.

    # git checkout addons
    # make
    # (follow the instructions to obtain the System.map file and set the SYSMAP variable)
    # make
    # sudo make modules_install
    # sudo depmod -a

The lttng-addons modules must be inserted manually for the TCP tracepoints to be made available.

    # sudo modprobe lttng-addons
    # sudo modprobe lttng-probe-addons

The following tracepoints will be available

    # sudo lttng list -k
    Kernel events:
    -------------
      ...
      inet_sock_create (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_sock_delete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_sock_clone (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_accept (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_connect (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_sock_local_in (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      inet_sock_local_out (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      ...

The ones used for trace synchronization are '''inet_sock_local_in''' and '''inet_sock_local_out'''.

== Synchronize traces in Trace Compass ==

In order to synchronize traces, create a new experiment and select all traces that need to be synchronized. Right-click on the experiment and select '''Synchronize traces'''. For each trace whose time needs to be transformed, a new trace named as the original but followed by a '_' will be created with the transformed timestamps, and the original trace will be replaced in the experiment. The original trace can still be accessed under the '''Traces''' folder.

[[Image:images/Sync_menu.png|Right-click synchronize traces to perform the trace synchronization]]

When opening the experiment now, all the views will be synchronized. The following screenshot presents the differences in the filtered Control Flow View before and after the time synchronization.

[[Image:images/Sync_cfv.png|Example of Control Flow View before and after trace synchronization]]

Information on the quality of the synchronization, the timestamp transformation formula and some synchronization statistics can be visualized in the '''Synchronization''' view. To open the '''Synchronization''' view, use the Eclipse Show View dialog ('''Window''' -> '''Show View''' -> '''Other...'''). Then select '''Synchronization''' under '''Tracing'''.

[[Image:images/Sync_view.png|Example of Synchronization view]]

= Time offsetting =

The time offsetting feature allows the user to apply a fixed offset to all event timestamps in a trace. It can be used, for example, to adjust the start time of a trace, or to manually align the timestamp of events from different traces.

== Basic mode ==

If the time offset to apply is known, it can be applied directly to the trace. In the '''Project Explorer''' view, select a trace, right-click and select '''Apply Time Offset...'''. It is also possible to select multiple traces, experiments or trace folders. All contained traces will be selected.

[[Image:images/TimeOffsetApply.png|Apply Time Offset menu]]

The dialog opens, in '''Basic''' mode.

[[Image:images/TimeOffsetBasicDialog.png|Apply Time Offset dialog - Basic mode]]

Enter a time offset to apply in the '''Offset in seconds''' column, with or without decimals. Then press the '''OK''' button.

[[Image:images/TimeOffsetBasicDialogFilled.png|Apply Time Offset dialog - Basic mode - filled]]

The time offset is applied to the trace and can be seen in the '''time offset''' property in the '''Properties''' view when the trace is selected.

The applied time offset is added to any time offset or time transformation formula currently set for the trace, and the resulting offset replaces any previous setting.

== Advanced mode ==

The time offset can also be computed using selected trace events or manually entered timestamps. After selecting one or more traces in the '''Project Explorer''' view, right-click and select '''Apply Time Offset...'''. In the opened dialog, select the '''Advanced''' button.

[[Image:images/TimeOffsetAdvancedDialog.png|Apply Time Offset dialog - Advanced mode]]

Double-clicking a trace name will open the trace in an editor. The '''Reference Time''' will be set to the trace start time. Selecting any event in the trace editor will set the '''Reference Time''' for that trace to the event's timestamp.

Selecting an event or a time in any view or editor that supports time synchronization will set the '''Target Time''' for every trace in the dialog.

Pressing the '''<<''' button will compute the time offset that should be applied in order to make the reference time align to the target time, provided that both fields are set.

The '''Reference Time''', '''Target Time''' and '''Offset in seconds''' fields can also be edited and entered manually.

To synchronize two events from different traces, first select an event in the trace to which the time offset should be applied, which will set its '''Reference Time''' field.

[[Image:images/TimeOffsetAdvancedSetReference.png|Apply Time Offset dialog - Set Reference Time]]

Then select a corresponding event in the second trace, which will set the '''Target Time''' field for the first trace.

[[Image:images/TimeOffsetAdvancedSetTarget.png|Apply Time Offset dialog - Set Target Time]]

Finally, press the '''<<''' button, which will automatically compute the time offset that should be applied in order to make the first event's timestamp align to the second event's timestamp.

[[Image:images/TimeOffsetAdvancedComputeOffset.png|Apply Time Offset dialog - Compute Offset]]

Then press the '''OK''' button. The time offset is applied to the trace and can be seen in the '''time offset''' property in the '''Properties''' view when the trace is selected.

The applied time offset is added to any time offset or time transformation formula currently set for the trace, and the resulting offset replaces any previous setting.

[[Image:images/TimeOffsetProperty.png|Time Offset - Properties view]]

== Clearing time offset ==

The time offset previously applied can be cleared to reset the trace to its original timestamps. In the '''Project Explorer''' view, select a trace, right-click and select '''Clear Time Offset'''. It is also possible to select multiple traces, experiments or trace folders. All contained traces will be affected.

The time offset or any time transformation formula will be deleted.

= Timestamp formatting =

Most views that show timestamps are displayed in the same time format. The unified timestamp format can be changed in the Preferences page. To get to that page, click on '''Window''' -> '''Preferences''' -> '''Tracing''' -> '''Time Format'''. Then a window will show the time format preferences.

[[Image:images/TmfTimestampFormatPage.png]]

The preference page has several subsections:

* '''Current Format''' a format string generated by the page
* '''Sample Display''' an example of a timestamp formatted with the '''Current Format''' string.
* '''Time Zone''' the time zone to use when displaying the time. The value '''Local time''' corresponds to the local, system-configured, time zone.
* '''Data and Time format''' how to format the date (days/months/years) and the time (hours/minutes/seconds)
* '''Sub-second format''' how much precision is shown for the sub-second units
* '''Date delimiter''' the character used to delimit the date units such as months and years
* '''Time delimiter''' the character to separate super-second time units such as seconds and minutes
* '''Sub-Second Delimiter''' the character to separate the sub-second groups such as milliseconds and nanoseconds
* '''Restore Defaults''' restores the system settings
* '''Apply''' apply changes

This will update all the displayed timestamps.

= Data driven analysis =

It is possible to define custom trace analyses and a way to view them in an XML format. These kind of analyses allow doing more with the trace data than what the default analyses shipped with TMF offer. It can be customized to a specific problem, and fine-tuned to show exactly what you're looking for.

== Managing XML files containing analyses ==

The '''Manage XML Analyses''' dialog is used to manage the list of XML files containing analysis. To open the dialog:

* Open the '''Project Explorer''' view.
* Select '''Manage XML Analyses...''' from the '''Traces''' folder context menu.

[[Image:images/ManageXMLAnalysis.png]]

The list of currently defined XML analyses is displayed on the left side of the dialog.

The following actions can be performed from this dialog:

* Import

Click the '''Import''' button and select a file from the opened file dialog to import an XML file containing an analysis. The file will be validated before importing it and if successful, the new analysis and views will be shown under the traces for which they apply. You will need to close any already opened traces and re-open them before the new analysis can be executed. If an invalid file is selected, an error message will be displayed to the user.

* Export

Select an XML file from the list, click the '''Export''' button and enter or select a file in the opened file dialog to export the XML analysis. Note that if an existing file containing an analysis is selected, its content will be replaced with the analysis to export.

* Edit

Select an XML file from the list, click the '''Edit''' to open the XML editor. When the file is saved after being modified, it is validated and traces that are affected by this file are closed.

* Delete

Select an XML file from the list and click the '''Delete''' button to remove the XML file. Deleting an XML file will close all the traces for which this analysis applies and remove the analysis.

== Defining XML components ==

To define XML components, you need to create a new XML file and use the XSD that comes with the XML plugin.

''For now, the XSD is only available through the source code in org.eclipse.tracecompass.tmf.analysis.xml.core/src/org/eclipse/tracecompass/tmf/analysis/xml/core/module/xmlDefinition.xsd''.

An empty file, with no content yet would look like this:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="xmlDefinition.xsd">

</tmfxml>
</pre>

== Defining an XML state provider ==

The state system is a component of TMF which can track the states of different elements of the system over the duration of a trace. To build this state system, events have to go chronologically through a state provider, which defines what changes are caused by the event to the system.

The state system obtained by the state provider can then be used to populate data-driven views without having to re-read the trace, or to query specific timestamps in the trace without needing to access the trace file.

=== Definitions and example ===

Before we start, we'll define a few terms used in the following sections. The interested reader should read the [https://wiki.eclipse.org/index.php/Trace_Compass#User_Guides Tmf Developer Guide] for more complete description of the state system and state providers.

* The '''state system''' can be viewed as a model of the system, where the different elements (attributes) can be seen as a tree, and their evolution (states) is tracked through time.

* '''Attribute''': An attribute is the smallest element of the model that can be in any particular state. Since many attributes may have the same name, each attribute is represented by its full path in the attribute tree.

* '''State''': A state is a value assigned to an attribute at a given time. Each model has its own state values.

* '''Attribute tree''': Elements in the model can be placed in a tree-like structure, for logical grouping. Each element in the tree can have both children and a state. Also, the tree is just a logical structure, all elements may be top-level elements.

* '''State history''': Whereas the attribute tree may be seen as the first dimension of the state system, the state history is the second dimension, over time. It tracks the intervals at which an attribute was in a given state.

In the following sections, we'll use an example trace with the following events:

* start(number): A new task with ID 'number' just started.
* execute(number, fct_name): The task with ID 'number' is executing a critical section named 'fct_name'.
* wait(number): The task with ID 'number' cannot execute a critical section and needs to wait for it.
* exec_end(fct_name): A task finished executing the critical section named 'fct_name'.
* stop(number): The task with ID 'number' has just finished.

=== Determining the state system structure ===

The first thing to do is to determine the attribute tree we'll use to represent the model of the system. The attribute tree is like a file system with directories and files, where files are logically gathered in the same parent directory. There is no one good way to build a tree, the logic will depend on the situation and on the person defining it.

The generated state system may be used later on to populate views, so attributes of the tree could be grouped in such a way as to make it easy to reach them with a simple path. The view will then be more simple.

In our example case, we'll want to track the status of each task and, for each critical section, which task is running them.

<pre>
|- Tasks
|    |- 1
|    |- 2
|   ...
|- Critical section
     |- Crit_sect1
     |- Crit_sect2
    ...
</pre>

Then we determine how each event will affect the state of the attributes. But first, let's ask ourselves what values should each state take.

Let's see with the tree:

<pre>
|- Tasks            -> Empty
|    |- 1           -> Each task can be in one of
|    |- 2             RUNNING, CRITICAL, WAITING
|   ...
|- Critical section -> Empty
     |- Crit_sect1  -> Each critical section will hold the currently running task number
     |- Crit_sect2
    ...
</pre>

Then we determine how each event will affect the state of the attributes. In the attribute paths below, elements in {} are values coming from the trace event, while strings are constants. For the sake of simplicity, we'll say "update attribute", but if an attribute does not exist, it will be created.

* start(number): Update state value of attribute "Tasks/{number}" to "RUNNING".
* execute(number, fct_name): Update state value of attribute "Tasks/{number}" to "CRITICAL" and Update attribute "Critical section/{fct_name}" to "{number}".
* wait(number): Update state value of attribute "Tasks/{number}" to "WAITING".
* exec_end(fct_name): Update state value of attribute "Tasks/{valueOf Critical section/{fct_name}}" to RUNNING and update "Critical section/{fct_name}" to null.
* stop(number): Update state value of attribute "Tasks/{number}" to null.

=== Writing the XML state provider ===

Once the model is done at a high level, it is time to translate it to an XML data-driven analysis. For details on how to use each XML element, refer to the documentation available in the XSD files. Some elements will be commented on below.

First define the state provider element.

The "version" attribute indicates which version of the state system is defined here. Once a state provider has been defined for a trace type, it will typically be used by a team of people and it may be modified over time. This version number should be bumped each time a new version of the state provider is published. This will force a rebuild of any existing state histories (if applicable) whose version number is different from the current one.

The "id" attribute uniquely identifies this state provider, and the analysis that will contain it.

<pre>
<stateProvider version="0" id="my.test.state.provider">
</pre>

Optional header information can be added to the state provider. A "traceType" should be defined to tell TMF which trace type this analysis will apply to. If no tracetype is specified, the analysis will appear under every trace. A "label" can optionally be added to have a more user-friendly name for the analysis.

<pre>
<head>
    <traceType id="my.trace.id" />
    <label value="My test analysis" />
</head>
</pre>

If predefined values will be used in the state provider, they must be defined before the state providers. They can then be referred to in the state changes by name, preceded by the '$' sign. It is not necessary to use predefined values, the state change can use values like (100, 101, 102) directly.

<pre>
<definedValue name="RUNNING" value="100" />
<definedValue name="CRITICAL" value="101" />
<definedValue name="WAITING" value="102" />
</pre>

The following event handler shows what to do with the event named ''start''. It causes one state change. The sequence of '''stateAttribute''' elements represents the path to the attribute in the attribute tree, each element being one level of the tree. The '''stateValue''' indicates which value to assign to the attribute at the given path. The "$RUNNING" value means it will use the predefined value named RUNNING above.

Suppose the actual event is ''start(3)''. The result of this state change is that at the time of the event, the state system attribute "Tasks/3" will have value 100.

<pre>
<eventHandler eventName="start">
    <stateChange>
        <stateAttribute type="constant" value="Tasks" />
        <stateAttribute type="eventField" value="number" />
        <stateValue type="int" value="$RUNNING" />
    </stateChange>
</eventHandler>
</pre>

The full XML file for the example above would look like this:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../org.eclipse.tracecompass.tmf.analysis.xml.core/src/org/eclipse/tracecompass/tmf/analysis/xml/core/module/xmlDefinition.xsd">
    <stateProvider version="0" id="my.test.state.provider">
        <head>
            <traceType id="my.trace.id" />
            <label value="My test analysis" />
        </head>

        <definedValue name="RUNNING" value="100" />
        <definedValue name="CRITICAL" value="101" />
        <definedValue name="WAITING" value="102" />

        <eventHandler eventName="start">
            <stateChange>
                <stateAttribute type="constant" value="Tasks" />
                <stateAttribute type="eventField" value="number" />
                <stateValue type="int" value="$RUNNING" />
            </stateChange>
        </eventHandler>
        <eventHandler eventName="execute">
            <stateChange>
                <stateAttribute type="constant" value="Tasks" />
                <stateAttribute type="eventField" value="number" />
                <stateValue type="int" value="$CRITICAL" />
            </stateChange>
            <stateChange>
                <stateAttribute type="constant" value="Critical section" />
                <stateAttribute type="eventField" value="fct_name" />
                <stateValue type="eventField" value="number" />
            </stateChange>
        </eventHandler>
        <eventHandler eventName="wait">
            <stateChange>
                <stateAttribute type="constant" value="Tasks" />
                <stateAttribute type="eventField" value="number" />
                <stateValue type="int" value="$WAITING" />
            </stateChange>
        </eventHandler>
        <eventHandler eventName="exec_end">
            <stateChange>
                <stateAttribute type="constant" value="Tasks" />
                <stateAttribute type="query">
                    <stateAttribute type="constant" value="Critical section" />
                    <stateAttribute type="eventField" value="fct_name" />
                </stateAttribute>
                <stateValue type="int" value="$RUNNING" />
            </stateChange>
            <stateChange>
                <stateAttribute type="constant" value="Critical section" />
                <stateAttribute type="eventField" value="fct_name" />
                <stateValue type="null" />
            </stateChange>
        </eventHandler>
        <eventHandler eventName="stop">
            <stateChange>
                <stateAttribute type="constant" value="Tasks" />
                <stateAttribute type="eventField" value="number" />
                <stateValue type="null" />
            </stateChange>
        </eventHandler>
    </stateProvider>
</tmfxml>
</pre>

=== Debugging the XML state provider ===

To debug the state system that was generated by the XML state provider, one could use the [[#State System Explorer View | State System Explorer View]], along with the events editor. By selecting an event, you can see what changes this event caused and the states of other attributes at the time.

If there are corrections to make, you may modify the XML state provider file, and re-import it. To re-run the analysis, you must first delete the supplementary files by right-clicking on your trace, and selecting ''Delete supplementary files...''. Check you analysis's .ht file, so that the analysis will be run again when the trace is reopened. The supplementary file deletion will have closed the trace, so it needs to be opened again to use the newly imported analysis file.

If modifications are made to the XML state provider after it has been "published", the '''version''' attribute of the '''xmlStateProvider''' element should be updated. This avoids having to delete each trace's supplementary file manually. If the saved state system used a previous version, it will automatically be rebuilt from the XML file.

== Defining an XML pattern provider ==
It exists patterns within an execution trace that can provide high level details about the system execution. A '''pattern''' is a particular combination of events or states that are expected to occur within a trace. It may be composed of several state machines that inherit or communicate through a common state system.

We may have multiple instances (scenarios) of a running state machine within a pattern. Each scenario which has its own path in the state system can generate segments to populate the data-driven views

=== The state system structure ===

The pattern analysis generates a predefined attribute tree described as follows :

<pre>
|- state machines
|    |- state machine 0
|       |- scenario 0
|          |- status
|          |- state
|              |- start
|             ...
|          |- storedFields
|              |- field 1
|             ...
|          |- startTime
|             ...
|         ...
|       |- scenarios 1
|      ...
|    |- state machine 1
|   ...
</pre>

The user can add custom data in this tree or determine its own attribute tree beside of this one.

=== Writing the XML pattern provider ===
Details about the XML structure are available in the XSD files.

First define the pattern element. As the state provider element described in [[#Writing_the_XML_state_provider | Writing the XML state provider]], it has a "version" attribute and an "id" attribute.

<pre>
<pattern version="0" id="my.test.pattern">
</pre>

Optional header information as well as predefined values like described in [[#Writing_the_XML_state_provider | Writing the XML state provider]] can be added.

Stored values can be added before the pattern handler. The predefined action '''saveStoredField''' triggers the updates of the stored fields and the predefined action '''clearStoredFields''' reset the values.

<pre>
<storedField id="offset" alias="offset"/>
</pre>

The behavior of the pattern and the models it needs are described in the pattern handler element.

The structure of the state machine (FSM) is based on the SCXML structure. The following example describe an FSM that matches all the system call in an LTTng kernel trace.

<pre>
<fsm id="syscall" initial="start">
    <state id="start">
        <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
    </state>
    <state id="in_progress" >
        <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
    </state>
    <final id="end"/>
</fsm>
</pre>

The value of the target attribute corresponds to the 'id' of a test element described in the XML file and is a reference to it. Similarly, the value of the action attribute corresponds to the 'id' of an action element described in the XML file and is a reference to it.

Conditions are used in the transitions to switch between the state of an FSM. They are defined under the '''test''' element. Two types of conditions are allowed : '''Data condition''' and '''Time condition'''. It is possible to combine several conditions using a logical operator (OR, AND, ...).

Data conditions tests the ongoing event information against the data in the state system or constant values. The following condition tests whether the current thread running on the CPU is also the ongoing scenario thread.

<pre>
<test id="thread_condition">
    <if>
        <condition>
            <stateValue type="query" >
                <stateAttribute type="location" value="CurrentCPU" />
                <stateAttribute type="constant" value="Current_thread" />
            </stateValue>
            <stateValue type="query">
                <stateAttribute type="constant" value="#CurrentScenario" />
                <stateAttribute type="constant" value="thread" />
            </stateValue>
        </condition>
    </if>
</test>
</pre>

Two types of time conditions are available:
* Time range conditions tests whether the ongoing event happens between a specific range of time. The following condition tests whether the ongoing event happens between 1 nanosecond and 3 nanoseconds.

<pre>
<test id="time_condition">
    <if>
        <condition>
            <timerange unit="ns">
                <in begin="1" end="3" />
            </timerange>
        </condition>
    </if>
</test>
</pre>

* Elapsed time conditions tests the value of the time spent since a specific state of an fsm. The following condition tests whether the ongoing event happens less than 3 nanoseconds after that the scenario reaches the state "syscall_entry_x".

<pre>
<test id="time_condition">
    <if>
        <condition>
            <elapsedTime unit="ns">
                <less since="syscall_entry_x" value="3" />
            </elapsedTime>
        </condition>
    </if>
</test>
</pre>

Two types of actions are allowed :
* State changes update values of attributes into the state system. The following example set the value of the thread for the current scenario.

<pre>
    <action id="sys_x_found">
        <stateChange>
            <stateAttribute type="constant" value="#CurrentScenario" />
            <stateAttribute type="constant" value="thread" />
            <stateValue type="query">
                <stateAttribute type="location" value="CurrentCPU" />
                <stateAttribute type="constant" value="Current_thread" />
            </stateValue>
        </stateChange>
    </action>
</pre>

* Generate segments. The following example represents a system call segment.

<pre>
<action id="exit_syscall_founded">
    <segment>
        <segType>
            <segName>
                <stateValue type="query">
                    <stateAttribute type="constant" value="#CurrentScenario" />
                    <stateAttribute type="constant" value="syscall" />
                    <stateAttribute type="constant" value="name" />
                </stateValue>
            </segName>
        </segType>
    </segment>
</action>
</pre>

When existing, the stored fields will be added as fields for the generated segments.

Here is the complete XML file by combining all the examples models above:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="xmlDefinition.xsd">

<pattern version="1" id="my.test.pattern">
    <head>
        <traceType id="org.eclipse.linuxtools.lttng2.kernel.tracetype" />
        <label value="xml syscall" />
    </head>

    <storedField id="filename"/>
    <storedField id="fd"/>
    <storedField id="ret" alias="ret"/>
    <storedField id="flags" alias="flags"/>
    <storedField id="offset" alias="offset"/>
    <storedField id="fd_in" alias="fd_in"/>
    <storedField id="fd_out" alias="fd_out"/>
    <storedField id="uservaddr" alias="uservaddr"/>
    <storedField id="upeer_sockaddr" alias="upeer_sockaddr"/>

    <location id="CurrentThread">
        <stateAttribute type="constant" value="Threads" />
        <stateAttribute type="query">
        <stateAttribute type="constant" value="CPUs" />
        <stateAttribute type="eventField" value="cpu" />
        <stateAttribute type="constant" value="Current_thread" />
        </stateAttribute>
    </location>

    <location id="CurrentCPU">
        <stateAttribute type="constant" value="CPUs" />
        <stateAttribute type="eventField" value="cpu" />
    </location>

    <patternHandler>
        <test id="time_condition">
            <if>
                <or>
                    <not>
                        <condition>
                            <timerange unit="ns">
                                <in begin="1" end="3" />
                            </timerange>
                        </condition>
                    </not>
                    <condition>
                        <elapsedTime unit="ns">
                            <less since="syscall_entry_x" value="3" />
                        </elapsedTime>
                    </condition>
                </or>
            </if>
        </test>

        <test id="thread_condition">
            <if>
                <condition>
                    <stateValue type="query" >
                        <stateAttribute type="location" value="CurrentCPU" />
                        <stateAttribute type="constant" value="Current_thread" />
                    </stateValue>
                    <stateValue type="query">
                        <stateAttribute type="constant" value="#CurrentScenario" />
                        <stateAttribute type="constant" value="thread" />
                    </stateValue>
                </condition>
            </if>
        </test>

        <action id="sys_x_founded">
            <stateChange>
                <stateAttribute type="constant" value="#CurrentScenario" />
                <stateAttribute type="constant" value="syscall" />
                <stateAttribute type="constant" value="name" />
                <stateValue type="eventName"/>
            </stateChange>

            <stateChange>
                <stateAttribute type="constant" value="#CurrentScenario" />
                <stateAttribute type="constant" value="cpu" />
                <stateValue type="eventField" value="cpu"/>
            </stateChange>

            <stateChange>
                <stateAttribute type="constant" value="#CurrentScenario" />
                <stateAttribute type="constant" value="thread" />
                <stateValue type="query">
                    <stateAttribute type="location" value="CurrentCPU" />
                    <stateAttribute type="constant" value="Current_thread" />
                </stateValue>
            </stateChange>
        </action>

        <action id="exit_syscall_founded">
            <segment>
                <segType>
                    <segName>
                        <stateValue type="query">
                            <stateAttribute type="constant" value="#CurrentScenario" />
                            <stateAttribute type="constant" value="syscall" />
                            <stateAttribute type="constant" value="name" />
                        </stateValue>
                    </segName>
                </segType>
            </segment>
        </action>

        <fsm id="syscall" initial="start">
            <state id="start">
                <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
            </state>
            <state id="in_progress" >
                <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
            </state>
            <final id="end"/>
        </fsm>
    </patternHandler>
</pattern>
</tmfxml>
</pre>

=== Representing the scenarios ===

Segments generated by the pattern analysis are used to populate latency views. A description of these views can be found in [[#Latency_Analyses | Latency Analyses]].

The full XML analysis example described above will generate the following views :

* Latency Table

[[Image:images/XMLPatternAnalysis/LatencyTable.png| Latency Table example - System Call pattern]]

* Latency vs Time

[[Image:images/XMLPatternAnalysis/LatencyVSTime.png| Latency vs Time example - System Call pattern]]

* Latency Statistics

[[Image:images/XMLPatternAnalysis/LatencyStatistics.png| Latency Statistics example - System Call pattern]]

* Latency vs Count

[[Image:images/XMLPatternAnalysis/LatencyVSCount.png| Latency vs Count example - System Call pattern]]

== Defining an XML time graph view ==

A time graph view is a view divided in two, with a tree viewer on the left showing information on the different entries to display and a Gantt-like viewer on the right, showing the state of the entries over time. The [[#Control_Flow_View | Control Flow View]] is an example of a time graph view.

Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').

In the example above, suppose we want to display the status for each task. In the state system, it means the path of the entries to display is "Tasks/*". The attribute whose value should be shown in the Gantt chart is the entry attribute itself. So the XML to display these entries would be as such:

<pre>
<entry path="Tasks/*">
    <display type="self" />
</entry>
</pre>

But first, the view has to be declared. It has an ID, to uniquely identify this view among all the available XML files.

<pre>
<timeGraphView id="my.test.time.graph.view">
</pre>

Optional header information can be added to the view. '''analysis''' elements will associate the view only to the analysis identified by the "id" attribute. It can be either the ID of the state provider, like in this case, or the analysis ID of any analysis defined in Java. If no analysis is specified, the view will appear under every analysis with a state system. The '''label''' element allows to give a more user-friendly name to the view. The label does not have to be unique. As long as the ID is unique, views for different analyses can use the same name.

<pre>
<head>
    <analysis id="my.test.state.provider" />
    <label value="My Sample XML View" />
</head>
</pre>

Also, if the values of the attributes to display are known, they can be defined, along with a text to explain them and a color to draw them with. Note that the values are the same as defined in the state provider, but the name does not have to be the same. While in the state provider, a simple constant string makes sense to use in state changes. But in the view, the name will appear in the legend, so a user-friendly text is more appropriate.

<pre>
<definedValue name="The process is running" value="100" color="#118811" />
<definedValue name="Critical section" value="101" color="#881111" />
<definedValue name="Waiting for critical section" value="102" color="#AEB522" />
</pre>

Here is the full XML for the time graph view:

<pre>
<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../org.eclipse.tracecompass.tmf.analysis.xml.core/src/org/eclipse/tracecompass/tmf/analysis/xml/core/module/xmlDefinition.xsd">
    <timeGraphView id="my.test.time.graph.view">
        <head>
            <analysis id="my.test.state.provider" />
            <label value="My Sample XML View" />
        </head>

        <definedValue name="The process is running" value="100" color="#118811" />
        <definedValue name="Critical section" value="101" color="#881111" />
        <definedValue name="Waiting for critical section" value="102" color="#AEB522" />

        <entry path="Tasks/*">
            <display type="self" />
        </entry>
    </timeGraphView>
</tmfxml>
</pre>

The following screenshot shows the result of the preceding example on a test trace. The trace used, as well as the XML file are available [https://secretaire.dorsal.polymtl.ca/~gbastien/Xml4Traces/XML_documentation_example.tar.gz here].

[[Image:images/Xml_analysis_screenshot.png| XML analysis with view]]

==== Using the keyboard ====
*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])

== Defining an XML XY chart ==

An XY chart displays series as a set of numerical values over time. The X-axis represents the time and is synchronized with the trace's current time range. The Y-axis can be any numerical value.

Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').

We will use the Linux Kernel Analysis on LTTng kernel traces to show an example XY chart. In this state system, the status of each CPU is a numerical value. We will display this value as the Y axis of the series. There will be one series per CPU. The XML to display these entries would be as such:

<pre>
<entry path="CPUs/*">
	<display type="constant" value="Status" />
	<name type="self" />
</entry>
</pre>

But first, the view has to be declared. It has an ID, to uniquely identify this view among all the available XML files.

<pre>
<xyView id="my.test.xy.chart.view">
</pre>

Like for the time graph views, optional header information can be added to the view. '''analysis''' elements will associate the view only to the analysis identified by the "id" attribute. It can be either the ID of the state provider, like in this case, or the analysis ID of any analysis defined in Java. If no analysis is specified, the view will appear under every analysis with a state system. The '''label''' element allows to give a more user-friendly name to the view. The label does not have to be unique. As long as the ID is unique, views for different analyses can use the same name.

<pre>
<head>
    <analysis id="org.eclipse.tracecompass.analysis.os.linux.kernel" />
    <label value="CPU status XY view" />
</head>
</pre>

Here is the full XML for the XY Chart that displays the CPU status over time of an LTTng Kernel Trace:

<pre>
<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../org.eclipse.tracecompass.tmf.analysis.xml.core/src/org/eclipse/tracecompass/tmf/analysis/xml/core/module/xmlDefinition.xsd">
	<xyView id="my.test.xy.chart.view">
		<head>
			<analysis id="org.eclipse.tracecompass.lttng2.kernel.analysis" />
			<label value="CPU status XY view" />
		</head>

		<entry path="CPUs/*">
			<display type="constant" value="Status" />
			<name type="self" />
		</entry>
	</xyView>
</tmfxml>
</pre>

The following screenshot shows the result of the preceding example on a LTTng Kernel Trace.

[[Image:images/XML_xy_chart.png| XML XY chart]]

= Latency Analyses =

Trace Compass offers a feature called Latency analysis. This allows an analysis to return intervals and these intervals will be displayed in four different views. An example analysis is provided with kernel system call latencies being provided. The available views are: 

* System Call Latencies
A '''table''' of the raw latencies. This view is useful to inspect individual latencies.

 [[Image:images/LatenciesTable.png| Latency Table example - System Call Latencies]]


* System Call Latency vs Time
A time aligned '''scatter chart''' of the latencies with respect to the current window range. This view is useful to see the overall form of the latencies as they arrive. 

[[Image:images/LatenciesScatter.png| Latency Scatter Chart example - System Call Latency vs Time]]


* System Call Latency Statistics
A view of the '''statistics''' of the latencies. These show the ''minimum'', ''maximum'', ''average'', ''standard deviation'', ''count'' and ''Total'' of the latencies when applicable. The view shows the total statistics for the whole trace also as the local statistics for a selection range. This tool is useful for finding the outliers on a per-category basis.

Right-clicking on an entry of the table and select '''Go to minimum''' allows to select the range of the minimum latency for the selected entry and synchronize the other views to this time range.

Right-clicking on an entry of the table and select '''Go to maximum''' allows to select the range of the maximum latency for the selected entry and synchronize the other views to this time range.

[[Image:images/LatenciesStatistics.png| Latency Statistics example - System Call Latency Statistics]]

* System Call Density
A '''density''' view, analyzing the current time range. This is useful to find global outliers. Selecting a duration in the table it will synchronize other views to this time range.

[[Image:images/LatenciesDensity.png| Latency Densities example - System Call Density]]

Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset. 

= Virtual Machine Analysis =

Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.

To better understand what is happening in such an environment, it is necessary to trace all the machines involved, guests and hosts, and correlate this information in an experiment that will display a complete view of the virtualized environment.

== Virtual Machine Experiment ==

A trace has to be taken for each machine, guest and host, in the virtualized environment. The host trace is the most important to have, as missing guests will only give an incomplete view of the system, but missing hosts usually won't allow to identify the hypervisor, nor determine when a guest is preempted from the host CPUs. The virtual machine analysis only makes sense if the host trace is available.

Once all the traces are imported in Trace Compass, they can be [[#Creating a Experiment | added to an experiment]]. The type of the experiment should by set to '''Virtual Machine Experiment''' by clicking on the right mouse button over the experiment name, then selecting '''Select Experiment Type...'''.

[[Image:images/vmAnalysis/VM_experiment.png | Virtual Machine Experiment]]

Depending on the hypervisor used, traces might need to be [[#Trace synchronization | synchronized]] so that they have the same time reference and their events can be correctly correlated.

== Virtual CPU View ==

The Virtual CPU view shows the status of CPUs and threads on guests augmented with the preemption and hypervisor data we get from the host.

In the image below, we see for the virtual CPU status that it has a few more states than the CPUs in the [[#Resources View | Resources View]]: in red and purple respectively, when the virtual CPU is running hypervisor code and when the CPU is preempted on the host.

The entries for each thread of the machine corresponds to the one from the [[#Control flow | Control Flow View]], augmented with the data from the Virtual CPU, so that we see that even though it is running from the guest's point of view, it is actually not running when the Virtual CPU it runs on is in preempted or hypervisor mode.

[[Image:images/vmAnalysis/VM_CPU_view.png | Virtual CPU view]]

==== Using the keyboard ====
*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])

== Hypervisor-specific Tracing ==

In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.

The following sections describe how to obtain traces for each hypervisor.

=== Qemu/KVM ===

The Qemu/KVM hypervisor require extra tracepoints not yet shipped in LTTng for both guests and hosts, as well as compilation with the full kernel source tree on the host, to have access to kvm_entry/kvm_exit events on x86.

Obtain the source code with extra tracepoints, along with lttng-modules

    # git clone https://github.com/giraldeau/lttng-modules.git
    # cd lttng-modules

Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.

    # git checkout addons
    # make
    # sudo make modules_install
    # sudo depmod -a

On the host, to have complete kvm tracepoints support, the make command has to include the full kernel tree. So first, you'll need to obtain the kernel source tree. See your distribution's documentation on how to get it. This will compile extra modules, including lttng-probe-kvm-x86, which we need.

    # make KERNELDIR=/path/to/kernel/dir

The lttng addons modules must be inserted manually for the virtual machine extra tracepoints to be available:

    # sudo modprobe lttng-addons
    # sudo modprobe lttng-vmsync-host # on the host
    # sudo modprobe lttng-vmsync-guest # on the guest

The following tracepoints will be available

    # sudo lttng list -k
    Kernel events:
    -------------
      ...
      kvm_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      kvm_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
      vmsync_gh_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
      vmsync_hg_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
      vmsync_gh_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
      vmsync_hg_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
      ...

Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [[#Trace synchronization | trace synchronization]] for information on how to obtain synchronizable traces.

= Java Logging =

Trace Compass contains some Java Utility Logging (JUL) tracepoints in various places in the code. To diagnose issues with Trace Compass or when reporting problems with the application, a JUL trace may be useful to help pinpoint the problem. The following sections explain how to enable JUL logging in Trace Compass and use various handlers to handle the data.

== Enable JUL Logging ==

By default, all the logging of the Trace Compass namespace is disabled. To enable it, one needs to add the following property to the ''vmargs'': ''-Dorg.eclipse.tracecompass.logging=true''.

The log levels and components can be controlled via a configuration file whose path is specified also in the ''vmargs'' like this: ''-Djava.util.logging.config.file=/path/to/logger.properties''. An example configuration file can be found in the next section.

If running the RCP, these arguments can be appended at the end of the ''tracecompass.ini'' file located in the folder where the executable is located. If running from Eclipse in development mode, in the ''Run configurations...'', the arguments should be added in the ''Arguments'' tab in the ''VM args'' box.

== Configuring JUL logging ==

JUL logging can be fine-tuned to log only specific components, specific levels, but also to different log handlers, with different formats, etc. Or else, the default level is INFO and the default log handler is a ConsoleHandler which displays all log message to the Console, which can be quite cumbersome.

Here is an example ''logger.properties'' file to control what is being logged and where.

    # Specify the handlers to create in the root logger
    # (all loggers are children of the root logger)
    # These are example handlers

    # Console handler
    handlers = java.util.logging.ConsoleHandler
    # Console and file handlers
    #handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler
    # No handler
    #handlers =

    # Set the default logging level for the root logger
    # Possible values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
    .level = OFF

    # Fine tune log levels for specific components
    # Use the INFO level for all tracecompass, but FINEST for the StateSystem component
    #org.eclipse.tracecompass.internal.statesystem.core.StateSystem.level = FINEST
    org.eclipse.tracecompass.level = INFO

== LTTng JUL log handler ==

The various log handlers have an overhead on the application. The ConsoleHandler has a visible impact on Trace Compass performance. The FileHandler also has an overhead though less visible, but when logging from multiple threads at the same time, the file becomes a bottleneck, so that logging data cannot be used with accuracy for performance analysis. The [http://lttng.org/docs/#doc-java-application LTTng log handler] is much better in a multi-threads context.

LTTng-UST comes with the Java JUL agent in most distros. Otherwise, it is possible to manually compile lttng-ust with options ''--enable-java-agent-jul'' and install it.

    git clone git://git.lttng.org/lttng-ust.git
    cd lttng-ust
    ./bootstrap
    ./configure --enable-java-agent-jul
    make
    sudo make install

The necessary classes for the java agent will have been installed on the system. Since Equinox (the OSGi implementation used by Eclipse and thus Trace Compass) uses its own classpath and ignores any classpath entered on the command line for security reasons, one needs to specify the agent class path with the bootclasspath argument:

    -Xbootclasspath/a:/usr/local/share/java/lttng-ust-agent-jul.jar:/usr/local/share/java/lttng-ust-agent-common.jar

Note that unlike the -classpath argument, -Xbootsclasspath does not follow the dependencies specified by a jar's Manifest, thus it is required to list both the -jul and the -common jars here.

These classes need to load the LTTng JNI library. Because they were loaded from the boot class path by the boot ClassLoader, the library path entered on the command line is ignored. A workaround is to manually copy the library to the jvm's main library path. For example

    sudo cp /usr/local/lib/liblttng-ust-jul-jni.so /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/

Or to overwrite the JVM's library path with the following VM argument.

    -Dsun.boot.library.path=/usr/local/lib

''Disclaimer: this last method overwrites the main java library path. It may have unknown side-effects. None were found yet.''

LTTng can now be used as a handler for Trace Compass's JUL, by adding the following line to the logger.properties file

    handlers = org.lttng.ust.agent.jul.LttngLogHandler

The tracepoints will be those logged by a previously defined configuration file. Here is how to setup LTTng to handle JUL logging:

    lttng create
    lttng enable-event -j -a
    lttng start

= Limitations =

* When parsing text traces, the timestamps are assumed to be in the local time zone. This means that when combining it to CTF binary traces, there could be offsets by a few hours depending on where the traces were taken and where they were read.
* LTTng Tools v2.1.0 introduced the command line options ''--no-consumer'' and ''--disable-consumer'' for session creation as well as the commands ''enable-consumer'' and ''disable-consumer''. The LTTng Tracer Control in Eclipse doesn't support these options and commands because they will obsolete in LTTng Tools v2.2.0 and because the procedure for session creation offers already all relevant advanced parameters.

= How to use LTTng to diagnose problems =

LTTng is a tracer, it will give an enormous amount of information about the system it is running on. This means it can solve many types of problems.

The following are examples of problems that can be solved with a tracer.

== Random stutters ==

Bob is running a computer program and it stutters periodically every 2 minutes. The CPU load is relatively low and Bob isn't running low on RAM.

He decides to trace his complete system for 10 minutes. He opens the LTTng view in eclipse. From the control, he creates a session and enables all kernel tracepoints.

He now has a 10 GB trace file. He imports the trace to his viewer and loads it up.

A cursory look at the histogram bar on the bottom show relatively even event distribution, there are no interesting spikes, so he will have to dig deeper to find the issue. If he had seen a spike every 2 minutes, there would be strong chances this would be the first thing to investigate as it would imply a lot of kernel activity at the same period as his glitch, this would have been a path to investigate.

As Bob suspects that he may be having some hardware raising IRQs or some other hardware based issue and adding delays. He looks at the ressource view and doesn't see anything abnormal.

Bob did note an exact second one glitch occurred: 11:58:03. He zooms into the time range or 11:58:02-11:58:04 using the histogram. He is happy to see the time is human readable local wall clock time and no longer in "nanseconds since the last reboot". <br>In the resource view, once again, he sees many soft irqs being raised at the same time, around the time his GUI would freeze. He changes views and looks at the control flow view at that time and sees a process spending a lot of time in the kernel: FooMonitor- his temperature monitoring software.

At this point he closes FooMonitor and notices the bug disappeared. He could call it a day but he wants to see what was causing the system to freeze. He cannot justify closing a piece of software without understanding the issue. It may be a conflict that HIS software is causing after all.

The system freezes around the time this program is running. He clicks on the process in the control flow view and looks at the corresponding events in the detailed events view. He sees: open - read - close repeated hundreds of times on the same file. The file being read was /dev/HWmonitor. He sends a report to the FooMonitor team and warns his team that FooMonitor was glitching their performance.

The FooMonitor team finds that they were calling a system bus call that would halt a CPU while reading the temperature so that the core would not induce an 0.1 degree error in the reading, by disabling this feature, they improve their software and stop the glitches from occurring on their custommer's machine. They also optimize their code to open the file read and clone it once.

By using system wide kernel tracing, even without deep kernel knowledge Bob was able to isolate a bug in a rogue piece of software in his system.

== Slow I/O ==

Alice is running her server. She noticed that one of her nodes was slowing down, and wasn't sure why, upon reading the trace she noticed that her time between a block request and complete was around 10ms.

This is abnormal, normally her server handles IOs in under 100us, since they are quite local.

She walks up to the server and hears the hard drive thrashing, This prompts her to look up in the events view the sectors being read in the block complete requests. There are her requests interleaved with other ones at the opposite side of the hard drive.

She sees the tracer writing but there is another process that is writing to the server disk non stop. She looks in the control flow view and sees that there's a program from another fellow engineer, "Wally" that is writing in his home in a loop "All work and no play makes Jack a dull boy.".

Alice kills the program, and immediately the server speeds up. She then goes to discuss this with Wally and implements strict hard disk quotas on the server.

= References =

* [http://www.eclipse.org/tracecompass Trace Compass project] 
* [https://wiki.eclipse.org/index.php/Trace_Compass#User_Guides Trace Compass User Guides]
* [http://www.lttng.org/ LTTng project]
* [http://lttng.org/files/doc/man-pages/man1/lttng.1.html LTTng Tracer Control Command Line Tool User Manual]
* [http://lttng.org/files/doc/man-pages/man8/lttng-relayd.8.html LTTng relayd User Manual]