* ''Control Flow'' - to visualize processes state transitions
* ''Resources'' - to visualize system resources state transitions
-* ''CPU usage'' - to visualize the usage of the processor with respect to the time in traces
+* ''CPU Usage'' - to visualize the usage of the processor with respect to the time in traces
+* ''Kernel Memory Usage'' - to visualize the relative usage of system memory
+* ''IO Usage'' - to visualize the usage of input/output devices
+* ''System Calls'' - presents all the system calls in a table view
+* ''System Call Statistics'' - present all the system calls statistics
+* ''System Call Density'' - to visualize the system calls displayed by duration
+* ''System Call vs Time'' - to visualize when system calls occur
Also, the LTTng plug-ins supports the following User Space traces views:
* ''Memory Usage'' - to visualize the memory usage per thread with respect to time in the traces
* ''Call Stack'' - to visualize the call stack's evolution over time
+* ''Function Duration Density'' - to visualize function calls displayed by duration
+* ''Flame Graph'' - to visualize why the CPU is busy
Finally, the LTTng plug-ins supports the following Control views:
* ''Control'' - to control the tracer and configure the tracepoints
The option '''Preserve folder structure''' will create, if necessary, the structure of folders relative to (and excluding) the selected '''Root directory''' (or '''Archive file''') into the target trace folder.
+The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the '''Root directory''' name, when importing from directory, or the ''' Archive file''' name, when importing from archive. One can change the experiment name by typing a new name in the text box beside the option.
+
[[Image:images/ProjectImportTraceDialog.png]]
If a trace already exists with the same name in the target trace folder, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example smalltrace becomes smalltrace(2).
The columns of the table are defined by the fields (aspects) of the specific trace type. These are the defaults:
* '''Timestamp''': the event timestamp
-* '''Type''': the event type
+* '''Event Type''': the event type
* '''Contents''': the fields (or payload) of this event
The first row of the table is the header row a.k.a. the Search and Filter row.
The Events editor can be closed, disposing a trace. When this is done, all the views displaying the information will be updated with the trace data of the next event editor tab. If all the editor tabs are closed, then the views will display their empty states.
+Column order and size is preserved when changed. If a column is lost due to it being resized to 0 pixels, right click on the context menu and select '''Show All''', it will be restored to a visible size.
+
=== Searching and Filtering ===
Searching and filtering of events in the table can be performed by entering matching conditions in one or multiple columns in the header row (the first row below the column header).
-To toggle between searching and filtering, click on the 'search' ([[Image:images/TmfEventSearch.gif]]) or 'filter' ([[Image:images/TmfEventFilter.gif]]) icon in the header row's left margin, or right-click on the header row and select '''Show Filter Bar''' or '''Show Search Bar''' in the context menu.
+To apply a matching condition to a specific column, click on the column's header row cell, type in a [http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html regular expression]. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.
-To apply a matching condition to a specific column, click on the column's header row cell, type in a [http://download.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html regular expression] and press the '''ENTER''' key. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.
+Press the '''Enter''' key to apply the condition as a search condition. It will be added to any existing search conditions.
+
+Press the '''Ctrl+Enter''' key to immediately add the condition (and any other existing search conditions) as a filter instead.
When matching conditions are applied to two or more columns, all conditions must be met for the event to match (i.e. 'and' behavior).
-To clear all matching conditions in the header row, press the '''DEL''' key.
+A preset filter created in the [[#Filters_View | Filters]] view can also be applied by right-clicking on the table and selecting '''Apply preset filter...''' > ''filter name''
==== Searching ====
[[Image:images/TraceEditor-Search.png]]
-Pressing the '''ENTER''' key will search and select the next matching event. Pressing the '''SHIFT-ENTER''' key will search and select the previous matching event. Wrapping will occur in both directions.
+Pressing the '''Enter''' key will search and select the next matching event. Pressing the '''Shift+Enter''' key will search and select the previous matching event. Wrapping will occur in both directions.
+
+Press '''Esc''' to cancel an ongoing search.
-Press '''ESC''' to cancel an ongoing search.
+To add the currently applied search condition(s) as filter(s), click the '''Add as Filter''' [[Image:images/filter_add.gif]] button in the header row margin, or press the '''Ctrl+Enter''' key.
-Press '''DEL''' to clear the header row and reset all events to normal.
+Press '''Delete''' to clear the header row and reset all events to normal.
==== Filtering ====
-When a filtering condition is entered in the head row, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace. The characters in each column which match the regular expression will be highlighted.
+When a new filter is applied, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace. The characters in each column which match the regular expression will be highlighted.
A status row will be displayed before and after the matching events, dynamically showing how many matching events were found and how many events were processed so far. Once the filtering is completed, the status row icon in the left margin will change from a 'stop' to a 'filter' icon.
[[Image:images/TraceEditor-Filter.png]]
-Press '''ESC''' to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.
+Press '''Esc''' to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.
+
+The header bar will be displayed above the table and will show a label for each applied filter. Clicking on a label will highlight the matching strings in the events that correspond to this filter condition. Pressing the '''Delete''' key will clear this highlighting.
+
+To remove a specific filter, click on the [[Image:images/delete_button.gif]] icon on its label in the header bar. The table will be updated with the events matching the remaining filters.
+
+The header bar can be collapsed and expanded by clicking on the [[Image:images/expanded_ovr.gif]][[Image:images/collapsed_ovr.gif]] icons in the top-left corner or on its background. In collapsed mode, a minimized version of the filter labels will be shown that can also be used to highlight or remove the corresponding filter.
-Press '''DEL''' or right-click on the table and select '''Clear Filters''' from the context menu to clear the header row and remove the filtering. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filter is removed.
+Right-click on the table and select '''Clear Filters''' from the context menu to remove all filters. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filters are removed.
-You can also search on the subset of filtered events by toggling the header row to the Search Bar while a filter is applied. Searching and filtering conditions are independent of each other.
+You can also search on the subset of filtered events by entering a search condition in the header row while a filter is applied. Searching and filtering conditions are independent of each other.
==== Bookmarking ====
=== Event Source Lookup ===
-For CTF traces using specification v1.8.2 or above, information can optionally be embedded in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
+Some trace types can optionally embed information in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
==== Source Code ====
''Note'': The columns in the text file are separated by tabs.
=== Refreshing of Trace ===
+
It's possible to refresh the content of the trace and resume indexing in case the current open trace was updated on the media. To refresh the trace, right-click into the table and select menu item '''Refresh'''. Alternatively, press key '''F5'''.
=== Collapsing of Repetitive Events ===
[[Image:images/TablePostCollapse.png]]
-To clear collapsing, press the right mouse button in the table and select menu item '''Clear Filters''' in the context sensitive menu. ''Note'' that collapsing is also removed when another filter is applied to the table.
+To remove the collapse filter, press the ([[Image:images/delete_button.gif]]) icon on the '''Collapse''' label in the header bar, or press the right mouse button in the table and select menu item '''Clear Filters''' in the context sensitive menu (this will also remove any other filters).
=== Customization ===
== Statistics View ==
-The Statistics View displays the various event counters that are collected when analyzing a trace. The data is organized per trace. After opening a trace, the element '''Statistics''' is added under the '''Tmf Statistics Analysis''' tree element in the Project Explorer. To open the view, double-click the '''Statistics''' tree element. Alternatively, select '''Statistics''' under '''Tracing''' within the '''Show View''' window ('''Window''' -> '''Show View''' -> '''Other...'''). This view shows 3 columns: ''Level'' ''Events total'' and ''Events in selected time range''. After parsing a trace the view will display the number of events per event type in the second column and in the third, the currently selected time range's event type distribution is shown. The cells where the number of events are printed also contain a colored bar with a number that indicates the percentage of the event count in relation to the total number of events. The statistics is collected for the whole trace. This view is part of the '''Tracing and Monitoring Framework (TMF)''' and is generic. It will work for any trace type extensions. For the LTTng 2.0 integration the Statistics view will display statistics as shown below.:
+The Statistics View displays the various event counters that are collected when analyzing a trace. After opening a trace, the element '''Statistics''' is added under the '''Tmf Statistics Analysis''' tree element in the Project Explorer. To open the view, double-click the '''Statistics''' tree element. Alternatively, select '''Statistics''' under '''Tracing''' within the '''Show View''' window ('''Window''' -> '''Show View''' -> '''Other...'''). The statistics is collected for the whole trace. This view is part of the '''Tracing and Monitoring Framework (TMF)''' and is generic. It will work for any trace type extensions.
+
+The view is separated in two sides. The left side of the view presents the Statistics in a table. The table shows 3 columns: ''Level'' ''Events total'' and ''Events in selected time range''. The data is organized per trace. After parsing a trace the view will display the number of events per event type in the second column and in the third, the currently selected time range's event type distribution is shown. The cells where the number of events are printed also contain a colored bar with a number that indicates the percentage of the event count in relation to the total number of events.
+
+[[Image:images/LTTng2StatisticsTableView.png]]
-[[Image:images/LTTng2StatisticsView.png]]
+The right side illustrates the proportion of types of events into two pie charts. The legend of each pie chart gives the representation of each color in the chart.
+* The ''Global'' pie chart displays the general proportion of the events in the trace.
+* When there is a range selection, the ''Events in selection'' pie chart appears next to the ''Global'' pie chart and displays the proportion the event in the selected range of the trace.
+
+[[Image:images/LTTng2StatisticsPieChartView.png]]
By default, the statistics use a state system, therefore will load very quickly once the state system is written to the disk as a supplementary file.
To modify the time of attributes shown in the view, select a different current time in other views that support time synchronization (e.g. event table, histogram view). When a time range is selected, this view uses the begin time.
+== External Analyses ==
+
+Trace Compass supports the execution of '''external analyses''' conforming to the [https://github.com/lttng/lami-spec/blob/v1.0.1/lami.adoc LAMI 1.0.x specification]. This includes recent versions of the [https://github.com/lttng/lttng-analyses LTTng-Analyses project].
+
+An external analysis is a [[#Running an External Analysis|program executed by Trace Compass]]. When the program is done analyzing, Trace Compass generates a '''[[#Opening a Report|report]]''' containing its results. A report contains one or more tables which can also be viewed as bar and scatter [[#Creating a Chart from a Result Table|charts]].
+
+'''Note''': The program to execute is found by searching the directories listed in the standard <code>$PATH</code> environment variable when no path separator (<code>/</code> on Unix and OS X, <code>\</code> on Windows) is found in its command.
+
+Trace Compass ships with a default list of ''descriptors'' of external analyses (not the analyses themselves), including the descriptors of the [http://github.com/lttng/lttng-analyses LTTng analyses]. If the LTTng analyses project is installed, its analyses are available when opening or importing an LTTng kernel trace.
+
+=== Running an External Analysis ===
+
+To run an external analysis:
+
+# [[#Importing Traces to the Project|Import a trace to the project]].
+# Make sure the trace is opened by double-clicking its name in the [[#Project Explorer View]].
+# Under the trace in the [[#Project Explorer View]], expand ''External Analyses'' to view the list of available external analyses.<p>The external analyses which are either missing or not compatible with the trace are stroke and cannot be executed.</p><p>[[Image:images/externalAnalyses/external-analyses-list.png]]</p>
+# '''Optional''': If you want the external analysis to analyze a specific time range of the current trace, make a time range selection.<p>You can use views like the [[#Histogram View]] and the [[#Control Flow View]] (if it's available for this trace) to make a time range selection.</p><p>External analyses are executed on the current time range selection if there is one, or on the whole trace otherwise.</p>
+# Right-click the external analysis to run and click '''Run External Analysis'''.<p>[[Image:images/externalAnalyses/run-external-analysis.png]]</p>
+# In the opened ''External Analysis Parameters'' window, optionally enter extra parameters to pass to the program.<p>[[Image:images/externalAnalyses/external-analysis-parameters-dialog.png]]</p>
+# Click '''OK''' to start the analysis.
+
+Note that many external analyses can be started concurrently.
+
+When the external analysis is done analyzing, its results are saved as a [[#Opening a Report|report]] in Trace Compass. The tables contained in this report are also automatically opened into a new report view when the analysis is finished.
+
+=== Opening a Report ===
+
+A '''report''' is created after a successful [[#Running an External Analysis|execution of an external analysis]].
+
+To open a report:
+
+* Under ''Reports'' under a trace in the [[#Project Explorer View]], double-click the report to open.<p>Each result table generated by the external analysis is shown in its own tab in the opened report view.</p><p>[[Image:images/externalAnalyses/report-view.png]]</p>
+
+=== Creating a Chart from a Result Table ===
+
+To create a bar or a scatter chart from the data of a given result table:
+
+# [[#Opening a Report|Open the report]] containing the result table to use for creating the chart.
+# In the opened report view, click the tab of the result table to use for creating the chart.
+# Click the ''View Menu'' button, then click either '''New custom bar chart''' or '''New custom scatter chart'''.<p>[[Image:images/externalAnalyses/new-custom-scatter-chart-menu.png]]</p>
+# In the opened ''Bar chart series creation'' or ''Scatter chart series creation'' window, under ''Series creator'', select a column to use for the X axis of the chart, and one or more columns to use for the Y axis of the chart, then click '''Add''' to create a series.<p>[[Image:images/externalAnalyses/chart-configuration-dialog.png]]</p><p>Repeat this step to create more series.</p>
+# Click '''OK''' to create the chart.<p>The chart is created and shown at the right of its source result table.</p><p>[[Image:images/externalAnalyses/table-and-chart.png]]</p>
+
+=== Showing or Hiding a Result Table ===
+
+To show or hide a result table once a [[#Creating a Chart from a Result Table|chart]] has been created:
+
+* In the report view, click the ''Toggle the Table view of the results'' button.<p>[[Image:images/externalAnalyses/table-and-chart-toggle-button.png]]</p><p>If the result table was visible, it is now hidden:</p><p>[[Image:images/externalAnalyses/chart-only.png]]</p>
+
+=== Adding and Removing a User-Defined External Analysis ===
+
+You can add a user-defined external analysis to the current list of external analyses. Note that the command to invoke must conform to the machine interface of [http://github.com/lttng/lttng-analyses LTTng analyses] 0.4.
+
+'''Note''': If you want to create your own external analysis, consider following the [http://lttng.org/files/lami/lami-1.0.1.html LAMI 1.0 specification], which is supported by later versions of Trace Compass.
+
+To add a user-defined external analysis:
+
+# Under any trace in the [[#Project Explorer View]], right-click ''External Analyses'' and click '''Add External Analysis'''.<p>[[Image:images/externalAnalyses/add-external-analysis.png]]</p>
+# In the opened ''Add External Analysis'' window, enter the name of the new external analysis and the associated command to run.<p>[[Image:images/externalAnalyses/add-external-analysis-dialog.png]]</p><p>The name is the title of the external analysis as shown under ''External Analyses'' in the [[#Project Explorer View]].</p><p>The command is the complete command line to execute. You can put arguments containing spaces or other special characters in double quotes.</p><p>'''Note''': If the command is not a file system path, then it must be found in the directories listed in the <code>$PATH</code> environment variable.</p>
+# Click '''OK''' to add the user-defined external analysis.<p>A user-defined external analysis with a green icon is created under ''External Analyses'' in the [[#Project Explorer View]].</p><p>[[Image:images/externalAnalyses/user-defined-external-analysis.png]]</p>
+
+'''Note''': The new external analysis entry is saved in the workspace.
+
+To remove a user-defined external analysis:
+
+* Under ''External Analyses'' in the [[#Project Explorer View]], right-click the external analysis to remove and click '''Remove External Analysis'''.<p>[[Image:images/externalAnalyses/remove-external-analysis.png]]</p><p>'''Note''': Only user-defined (green icon) external analyses can be removed.</p>
+
== Custom Parsers ==
Custom parser wizards allow the user to define their own parsers for text or XML traces. The user defines how the input should be parsed into internal trace events and identifies the event fields that should be created and displayed. Traces created using a custom parser can be correlated with other built-in traces or traces added by plug-in extension.
Fill out the first wizard page with the following information:
* '''Category:''' Enter a category name for the trace type.
-* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser.
-* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp.<br>
-Note: information about date and time patterns can be found here: [../reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]
+* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser. This will also be the default event type name.
+* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp, or leave blank to use the default Time Format preference.<br>
+Note: information about date and time patterns can be found here: [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]
Click the '''Add next line''', '''Add child line''' or '''Remove line''' buttons to create a new line of input or delete it. For each line of input, enter the following information:
* '''Regular expression:''' Enter a regular expression that should match the input line in the log, using capturing groups to extract the data.<br>
-Note: information about date and time patterns can be found here: [http://java.sun.com/javase/6/docs/api/java/util/regex/Pattern.html]
+Note: information about regular expression patterns can be found here: [http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html]
* '''Cardinality:''' Enter the minimum and maximum number of lines matching this line's regular expression that must be found in the log. At least the minimum number of lines must be found before the parser will consider the next line. Child lines will always be considered first.
+* '''Event type:''' Optionally enable this text field to enter an event type name that will override the default (trace type) when this line matches.
+
<u>Important note:</u> The custom parsers identify a log entry when the first line's regular expression matches (Root Line n). Each subsequent text line in the log is attempted to be matched against the regular expression of the parser's input lines in the order that they are defined (Line n.*). Only the first matching input line will be used to process the captured data to be stored in the log entry. When a text line matches a Root Line's regular expression, a new log entry is started.
Click the '''Add group''' or '''Remove group''' buttons to define the data extracted from the capturing groups in the line's regular expression. For each group, enter the following information:
* '''Name combo:''' Select a name for the extracted data:
-** '''Time Stamp''': Select this option to identify the time stamp data. The input's data and time pattern must be entered in the format: text box.
+** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
+** '''Event type''': Select this option to identify the event type name. This will override the default or line-specific event type name.
** '''Message''': Select this option to identify the main log entry's message. This is usually a group which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box.
Fill out the first wizard page with the following information:
* '''Category:''' Enter a category name for the trace type.
-* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser.
-* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp.<br>
-
-Note: information about date and time patterns can be found here: [../reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]
+* '''Trace type:''' Enter a name for the trace type, which is also the name of the custom parser. This will also be the default event type name.
+* '''Time Stamp format:''' Enter the date and time pattern that will be used to output the Time Stamp, or leave blank to use the default Time Format preference.<br>
+Note: information about date and time patterns can be found here: [http://archive.eclipse.org/tracecompass/doc/stable/org.eclipse.tracecompass.doc.user/reference/api/org/eclipse/tracecompass/tmf/core/timestamp/TmfTimestampFormat.html TmfTimestampFormat]
Click the '''Add document element''' button to create a new document element and enter a name for the root-level document element of the XML file.
* '''Log entry:''' Select this checkbox to identify an element which represents a log entry. Each element with this name in the XML file will be parsed to a new log entry. At least one log entry element must be identified in the XML document. Log entry elements cannot be nested.
* '''Name combo:''' Select a name for the extracted data:
** '''Ignore''': Select this option to ignore the extracted element's data at this level. It is still possible to extract data from this element's child elements.
-** '''Time Stamp''': Select this option to identify the time stamp data. The input's data and time pattern must be entered in the format: text box.
+** '''Event type''': Select this option to identify the event type name. This will override the default or element-specific event type name.
+** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
** '''Message''': Select this option to identify the main log entry's message. This is usually an input which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box. It does not have to match the element name.
* '''Action combo:''' Select the action to be performed on the extracted data:
** '''Set''': Select this option to overwrite the data for the chosen name when there is a match for this element.
** '''Append''': Select this option to append to the data with the chosen name, if any, when there is a match for this element.
** '''Append with |''' : Select this option to append to the data with the chosen name, if any, when there is a match for this element, using a | separator between matches.
+* '''Event type:''' Optionally enable this text field to enter an event type name that will override the default (trace type) when this element is present.
Note: An element's extracted data 'value' is a parsed string representation of all its attributes, children elements and their own values. To extract more specific information from an element, ignore its data value and extract the data from one or many of its attributes and children elements.
* '''Attribute name:''' Enter a name for the attribute that must match an attribute of this element in the XML file.
* '''Name combo:''' Select a name for the extracted data:
-** '''Time Stamp''': Select this option to identify the time stamp data. The input's data and time pattern must be entered in the format: text box.
+** '''Timestamp''': Select this option to identify the timestamp data. The input's data and time pattern must be entered in the format: text box.
+** '''Event type''': Select this option to identify the event type name. This will override the default or element-specific event type name.
** '''Message''': Select this option to identify the main log entry's message. This is usually an input which could have text of greater length.
** '''Other''': Select this option to identify any non-standard data. The name must be entered in the name: text box. It does not have to match the element name.
* '''Action combo:''' Select the action to be performed on the extracted data:
[[Image:images/TimeAlignment_sash.png]]
+== Searching in Time Graph Views ==
+
+Search for an entry in a '''Time Graph view''', e.g. [[#Control_Flow_View | Control Flow View]] or [[#Resources_View | Resources View]], using the ''' Find ''' dialog. To use the dialog :
+
+* Select the time graph view you want to search in
+* Press ''' Ctrl + F '''. The following screen will be shown :
+
+[[Image:images/FindDialog.png]]
+
+* Enter the string to find in the ''' Find ''' text drop down and select the ''' Options ''' and ''' Direction ''' you need.
+* Press the ''' Find ''' button or ''' Enter ''' or ''' Alt + n '''. The next match in the selected time graph view will be selected.
+
+Various options are available in the ''' Options ''' group :
+* ''' Case sensitive ''' makes the search case sensitive.
+* ''' Wrap search ''' restarts the search from the first index, depending of the direction, when no entry were found.
+* ''' Whole word ''' allows to search for whole words, delimited by spaces or special character, that are identical to the search text.
+* ''' Regular expression ''' specifies that the search text is a regular expression or not.
+
+The ''' Direction ''' group allows to select the search direction : ''' Forward ''' or ''' Backward '''.
+
= LTTng Tracer Control =
The LTTng Tracer Control in Eclipse for the LTTng Tracer toolchain version v2.0 (or later) is done using SSH and requires an SSH server to be running on the remote host. For the SSH connection the SSH implementation of Remote Services is used. The functions to control the LTTng tracer (e.g. start and stop), either locally or remotely, are available from a dedicated Control View.
Under the '''Provider''' group all trace providers are displayed. Trace providers are '''Kernel''' and any user space application that supports UST tracing. Under each provider a corresponding list of events are displayed.
-Under the '''Sessions''' group all current sessions will be shown. The level under the sessions show the configured domains. Currently the LTTng 2.0 Tracer Toolchan supports domain '''Kernel''' and '''UST global'''. Under each domain the configured channels will be displayed. The last level is under the channels where the configured events are displayed.
+Under the '''Sessions''' group all current sessions will be shown. The level under the sessions show the configured domains. Currently the LTTng Tracer Toolchain supports domain '''Kernel''', '''UST global''', '''JUL''', '''Log4j''' and '''Python'''. Under the domains '''Kernel''' and '''UST Global''' the configured channels will be displayed. The last level is under the channels where the configured events are displayed.
Each session can be '''ACTIVE''' or '''INACTIVE'''. Active means that tracing has been started, inactive means that the tracing has been stopped. Depending on the state of a session a different icon is displayed. The icon for an active session is [[Image:images/Session_active.gif]]. The icon for an inactive session is [[Image:images/Session_inactive.gif]].
Refer to chapter [[#Recording a Snapshot | Recording a Snapshot]] for how to create a snapshot.
-=== Creating a Live Tracing Session ===
+<!--=== Creating a Live Tracing Session ===
LTTng Tools version v2.4.0 introduces the possibility to create live tracing sessions. The live mode allows you to stream the trace and view it while it's being recorded. To create such a live session, open the trace session dialog as described in chapter [[#Creating a Tracing Session | Creating a Tracing Session]].
[[Image:images/LTTng2CreateSessionDialog_Live.png]]
[[Image:images/LTTng2CreateSessionDialog_Live_Advanced.png]]
Fill in all necessary information, select the radio button for '''Live Mode''' and press '''Ok'''.
-
+-->
=== Enabling Channels - General ===
Enabling channels can be done using a session tree node when the domain hasn't be created in the session or, alternatively on a domain tree node of a session in case the domain is already available.
By default the domain '''Kernel''' is selected. To create a UST channel, select '''UST''' under the domain section. The label <Default> in any text box indicates that the default value of the tracer will be configured. To initialize the dialog box press button '''Default'''.
+'''Note''': You cannot create a channel under the '''JUL''', '''LOG4J''' and '''Python''' domain. Instead those domains uses a default channel under the '''UST global''' domain named '''lttng_jul_channel''', '''lttng_log4j_channel''' or '''lttng_python_channel'''. Those are the channels that LTTng uses to trace Java or Python application and you cannot add '''UST''' events to those channels.
+
If required update the following channel information and then press '''Ok'''.
* '''Channel Name''': The name of the channel.
By default the domain '''Kernel''' is selected and the kernel specific data sections are created. From this dialog box kernel '''Tracepoint''' events, '''System calls (Syscall)''', a '''Dynamic Probe''' or a '''Dynamic Function entry/return''' probe can be enabled. Note that events of one of these types at a time can be enabled.
-To enable '''Tracepoint''' events, first select the corresponding '''Select''' button, then select either all tracepoins (select '''All''') or select selectively one or more tracepoints in the displayed tree of tracepoints and finally press '''Ok'''.
+To enable all '''Tracepoints''' and all '''System calls (Syscall)''', select the button '''Select''' of section '''All Tracepoint Events and Syscalls''' and press '''Ok'''.
+
+[[Image:images/LTTng2EnableAllEventsDialog.png]]
+
+Upon successful operation, the domain '''Kernel''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all a wildcard event '''*''' of type '''TRACEPOINT''' under the channel and a wildcard event '''*''' of type '''SYSCALL''' . The channel and events will be '''ENABLED'''.
+
+To enable '''Tracepoint''' events, first select the corresponding '''Select''' button, then select either all tracepoins (select '''All''') or select selectively one or more tracepoints in the displayed tree of tracepoints. You can also enter directly the name of the events you want to enable (comma separated list and wildcards are supported). Finally press '''Ok'''.
[[Image:images/LTTng2TracepointEventsDialog.png]]
[[Image:images/LTTng2EnabledKernelTracepoints.png]]
-To enable all '''Syscalls''', select the corresponding '''Select''' button and press '''Ok'''.
+To enable '''Syscall''' events, first select the corresponding '''Select''' button, then select either all syscalls (select '''All''') or select selectively one or more syscalls in the displayed tree of syscalls. You can also enter directly the name of the events you want to enable (comma separated list and wildcards are supported). Finally press '''Ok'''.
[[Image:images/LTTng2SyscallsDialog.png]]
-Upon successful operation, the event with the name '''syscalls''' and event type '''SYSCALL''' will be added under the default channel (channel0). If necessary the domain '''Kernel''' and the channel '''channel0''' will be created.
+Upon successful operation, the domain '''Kernel''' will be created in the tree (if neccessary), the default channel with name "channel0" will be added under the domain (if necessary) as well as all requested events of type '''SYSCALL''' under the channel. The channel and events will be '''ENABLED'''.
[[Image:images/LTTng2EnabledKernelSyscalls.png]]
[[Image:images/LTTng2EnabledUstWildcardEvents.png]]
+When enabling '''Tracepoint''' with wildcard, it is possible to specify event(s) (comma separated list) that we want to '''exclude''' from that wildcard selection. To '''exclude''' '''Tracepoint''' events, check the corresponding '''Select''' check box, fill the '''Event Names''' field and press '''Ok'''.
+
+[[Image:images/LTTng2UstExcludeEventsDialog.png]]
+
For UST it is possible to enable '''Tracepoint''' events using log levels. To enable '''Tracepoint''' events using log levels, select first the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.
* '''Event Name''': Name to display
[[Image:images/LTTng2EnabledUstLoglevelEvents.png]]
+=== Enabling JUL Events On Session Level ===
+
+For enabling JUL loggers, first open the enable events dialog as described in section [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]] and select domain '''JUL'''.
+
+To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers and finally press '''Ok'''.
+
+[[Image:images/LTTng2JulLoggerEventsDialog.png]]
+
+Upon successful operation, the domain '''JUL''' will be created in the tree (if neccessary). With JUL loggers there is no channel, you see the enabled loggers directly under the '''JUL''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.
+
+[[Image:images/LTTng2EnabledAllJulLoggers.png]]
+
+For JUL it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.
+
+* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
+* '''loglevel-only''': To specify that only the specified log level shall be configured
+
+[[Image:images/LTTng2JulLoglevelEventsDialog.png]]
+
+=== Enabling LOG4J Events On Session Level ===
+
+For enabling LOG4J loggers, first open the enable events dialog as described in section [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]] and select domain '''LOG4J'''.
+
+To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers and finally press '''Ok'''.
+
+[[Image:images/LTTng2Log4jLoggerEventsDialog.png]]
+
+Upon successful operation, the domain '''LOG4J''' will be created in the tree (if neccessary). With LOG4J loggers there is no channel, you see the enabled loggers directly under the '''LOG4J''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.
+
+[[Image:images/LTTng2EnabledAllLog4jLoggers.png]]
+
+For LOG4J it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.
+
+* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
+* '''loglevel-only''': To specify that only the specified log level shall be configured
+
+[[Image:images/LTTng2Log4jLoglevelEventsDialog.png]]
+
+=== Enabling Python Events On Session Level ===
+
+For enabling Python loggers, first open the enable events dialog as described in section [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]] and select domain '''Python'''.
+
+To enable '''Loggers''', first select the corresponding '''Select''' button, then select either all loggers (select '''All''') or select selectively one or more loggers in the displayed tree of loggers. You can also enter the name of your logger in the text field. Finally press '''Ok'''.
+
+[[Image:images/LTTng2PythonLoggerEventsDialog.png]]
+
+Upon successful operation, the domain '''Python''' will be created in the tree (if neccessary). With Python loggers there is no channel, you see the enabled loggers directly under the '''Python''' domain. Note that for the case that '''All''' loggers were selected the wildcard '''*''' is used which will be shown in the Control View as below.
+
+[[Image:images/LTTng2EnabledAllPythonLoggers.png]]
+
+For Python it is possible to enable '''Logger''' events using log levels. To enable '''Logger''' events using log levels, check the corresponding '''Select''' button, select a log level from the drop down menu, fill in the relevant information (see below) and press '''Ok'''.
+
+* '''loglevel''': To specify if a range of log levels (0 to selected log level) shall be configured
+* '''loglevel-only''': To specify that only the specified log level shall be configured
+
+[[Image:images/LTTng2PythonLoglevelEventsDialog.png]]
+
=== Enabling Events On Domain Level ===
-Kernel events can also be enabled on the domain level. For that select the relevant domain tree node, click the right mouse button and the select '''Enable Event (default channel)...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''' or '''UST global''', the domain specifc fields are shown and the domain selector is preselected and read-only.
+Kernel events can also be enabled on the domain level. For that select the relevant domain tree node, click the right mouse button and the select '''Enable Event (default channel)...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''', '''UST global''', '''JUL''', '''LOG4J''' or '''Python''', the domain specific fields are shown and the domain selector is preselected and read-only.
[[Image:images/LTTng2EventOnDomainAction.png]]
-To enable events for domain '''Kernel''' follow the instructions in section [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]], for domain '''UST global''', see section [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]].
-The events will be add to the default channel '''channel0'''. This channel will be created by on the server side if neccessary.
+Instructions for enalbing events for a particular domain can be found here:
+* '''Kernel''' domain: [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]]
+* '''UST global''' domain: [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]]
+* '''JUL''' domain: [[#Enabling JUL Events On Session Level | Enabling JUL Events On Session Level]]
+* '''LOG4J''' domain: [[#Enabling LOG4J Events On Session Level | Enabling LOG4J Events On Session Level]]
+* '''Python''' domain: [[#Enabling Python Events On Session Level | Enabling Python Events On Session Level]]
+
+The events will be added to the default channel '''channel0'''. This channel will be created by on the server side if necessary.
=== Enabling Events On Channel Level ===
Kernel events can also be enabled on the channel level. If necessary, create a channel as described in sections [[#Enabling Channels On Session Level | Enabling Channels On Session Level]] or [[#Enabling Channels On Domain Level | Enabling Channels On Domain Level]].
-Then select the relevant channel tree node, click the right mouse button and the select '''Enable Event...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''' or '''UST global''', the domain specifc fields are shown and the domain selector is preselected and read-only.
+Then select the relevant channel tree node, click the right mouse button and the select '''Enable Event...'''. A new dialog box will open for providing information about the events to be enabled. Depending on the domain, '''Kernel''' or '''UST global''', the domain specific fields are shown and the domain selector is preselected and read-only. Since there is no channel under the '''JUL''', '''LOG4J''' or '''Python''' domain you cannot enable those loggers directly from a channel.
[[Image:images/LTTng2EventOnChannelAction.png]]
[[Image:images/LTTng2ImportDialog.png]]
-By default all traces are selected. A default project with the name '''Remote''' is selected which will be created if necessary. Update the list of traces to be imported, if necessary, by selecting and deselecting the relevant traces in the tree viewer. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Also if needed, change the tracing project from the '''Available Projects''' combo box. Then press button '''Finish'''. Upon successful import operation the selected traces will be stored in the '''Traces''' directory of the specified tracing project. A directory with the connection name will be created under the '''Traces''' directory. Underneath that, the session directory structure as well as the trace names will be preserved in the destination tracing project. For '''Kernel''' traces the trace type '''Linux Kernel Trace''' and for '''UST''' traces the trace type '''LTTng UST Trace''' will be set. From the '''Project Explorer''' view, the trace can be analyzed further.
+By default all traces are selected. A default project with the name '''Remote''' is selected which will be created if necessary. Update the list of traces to be imported, if necessary, by selecting and deselecting the relevant traces in the tree viewer. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Also if needed, change the tracing project from the '''Available Projects''' combo box. The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the session name. One can change the experiment name by typing a new name in the text box beside the option.
+
+Then press button '''Finish'''. Upon successful import operation the selected traces will be stored in the '''Traces''' directory of the specified tracing project. A directory with the connection name will be created under the '''Traces''' directory. Underneath that, the session directory structure as well as the trace names will be preserved in the destination tracing project. For '''Kernel''' traces the trace type '''Linux Kernel Trace''' and for '''UST''' traces the trace type '''LTTng UST Trace''' will be set. From the '''Project Explorer''' view, the trace can be analyzed further.
'''Note''': If a trace already exists with the same name in the destination directory, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example kernel becomes kernel(2).
** '''Event Type''': The event type ('''TRACEPOINT''' only).
** '''Fields''': Shows a list of fields defined for the selected event. (UST only, since support for LTTng Tools v2.1.0)
** '''Log Level''': The log level of the event.
+* '''Logger''' Properties (Provider)
+** '''Logger Name''': The name of the logger.
+** '''Logger Type''': The event type ('''TRACEPOINT''' only).
* '''Session''' Properties
** '''Session Name''': The name of the Session.
** '''Session Path''': The path on the remote host where the traces will be stored. (Not shown for snapshot sessions).
** '''State''': The channel state ('''ENABLED''' or '''DISABLED''')
** '''Sub Buffer size''': The size of the sub-buffers of the channel (in bytes).
** '''Switch Timer Interval''': The switch timer interval.
+** '''Number of Discarded Events''': The number of discarded events of the channel.
+** '''Number of Lost Packets''': The number of lost packets of the channel.
* '''Event''' Properties (Channel)
** '''Event Name''': The name of the event.
** '''Event Type''': The event type ('''TRACEPOINT''', '''SYSCALL''' or '''PROBE''').
** '''Log Level''': The log level of the event. (For LTTng Tools v2.4.0 or later, '''<=''' prior the log level name will indicate a range of log levels and '''==''' a single log level.)
** '''State''': The Event state ('''ENABLED''' or '''DISABLED''')
** '''Filter''': Shows '''with filter''' if a filter expression is configured else property '''Filter''' is omitted. (since support for LTTng Tools v2.1.0)
+* '''Logger''' Properties (Domain)
+** '''Logger Name''': The name of the logger.
+** '''Logger Type''': The logger type ('''TRACEPOINT''').
+** '''Log Level''': The log level of the logger. (For LTTng Tools v2.4.0 or later, '''<=''' prior the log level name will indicate a range of log levels and '''==''' a single log level.)
+** '''State''': The logger state ('''ENABLED''' or '''DISABLED''')
== LTTng Tracer Control Preferences ==
= LTTng Kernel Analysis =
-Historically, LTTng was developped to trace the Linux kernel and, over time, a number of kernel-oriented analysis views were developped and organized in a perspective.
+Historically, LTTng was developed to trace the Linux kernel and, over time, a number of kernel-oriented analysis views were developed and organized in a perspective.
+
+This section presents a description of the '''OS Tracing Overview''' perspective and the '''LTTng Kernel''' perspective.
+
+== OS Tracing Overview Perspective ==
+
+The '''OS Tracing Overview''' perspective groups the following views:
+
+* [[#Project Explorer_View | Project Explorer View]]
+* [[#Events_Editor | Events Editor]]
+* [[#Histogram_View | Histogram View]]
+* [[#LTTng CPU Usage View | CPU Usage View]]
+* [[#Disk I/O Activity View | Disk I/O Activity View]]
+* [[#Kernel Memory Usage View | Kernel Memory Usage View]]
+
+The perspective can be opened from the Eclipse Open Perspective dialog ('''Window > Open Perspective... > Other''').
+
+[[Image:images/osOverview/select_os_overview.png]]
-This section presents a description of the LTTng Kernel Perspective.
+This perspective is intended to be used to locate performance issues by observing resource usage.
+
+The perspective can show times resource usage is anomalous. This can help locating the causes of system slowdowns in throughput or response time.
+
+An example can be program that is doing a lot of processing then slows down due to a database access. The user will see a dip in CPU usage and maybe a slight rise in I/O access. The user should consider both spike and slums to be an indication of an area worth investigating.
+
+[[Image:images/osOverview/os_overview_perspective.png]]
+
+Once a performance issue has been localized, it can be further investigated with the #LTTng kernel Perspective.
== LTTng Kernel Perspective ==
==== Using the mouse ====
-The states flow is usable with the mouse. The following actions are set:
+The following mouse actions are available:
* '''left-click''': select a time or time range begin time
-* '''Shift-left-click''': select a time range end time
+* '''Shift-left-click or drag''': Extend or shrink the selection range
+
* '''left-drag horizontally''': select a time range or change the time range begin or end time
* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''click on a colored bar''': the associated process node is selected and the current time indicator is moved where the click happened
* '''mouse wheel up/down''': scroll up or down
+* '''Shift-mouse wheel up/down''': scroll left or right
* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically
* '''drag the time ruler horizontally''': zoom in or out with fixed start time
==== Using the keyboard ====
-The states flow is usable with the keyboard. The following actions are set:
+The following keyboard shortcuts are available:
+
*'''arrow-right key''': selects the next state for the selected process
*'''arrow-left key''': selects the previous state for the selected process
*'''Shift + arrow-right key''': updates the selection end time of the current selection range by selecting the next state of the current process
*'''Ctrl + +''': Zoom-in vertically
*'''Ctrl + -''': Zoom-out vertically
*'''Ctrl + 0''': Reset the vertical zoom
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+When the mouse cursor is over entries (left pane):
+*'''-''': Collapse selected entry
+*'''+''': Expand selected entry
+*'''*''': Expand selected entry to the level with at least one collapsed entry
+
+'''Please note that the behavior of some shortcuts can slightly differ based on the operating system.'''
When the selection indicators are changed, all the other views are '''synchronized'''. For example, the [[#LTTng Kernel Events Editor|Events Editor]] will show the event matching the current time indicator. The reverse behaviour is also implemented: selecting an event within the Events View will update the Control Flow View current time indicator.
| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
-| Select Previous Event
+| Select Previous State Change
| Selects the previous state for the selected process. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_event.gif]]
-| Select Next Event
+| Select Next State Change
| Selects the next state for the selected process. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/add_bookmark.gif]]
| [[Image:images/follow_arrow_fwd.gif]]
| Follow CPU Forward
| Selects the next state following CPU execution across processes. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
+|-
+| [[Image:images/shift_l_edit.gif]]
+| Go to previous event of the selected thread
+| Move to the closest previous event belonging to the selected thread. This action looks through all trace events, unlike the ''Select Previous State Change'' action which only stops at state changes.
+|-
+| [[Image:images/shift_r_edit.gif]]
+| Go to next event of the selected thread
+| Move to the closest following event belonging to the selected thread. This action looks through all trace events, unlike the ''Select Next State Change'' action which only stops at state changes.
|}
View Menu
{|
|
| Show Markers
-| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost event occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
+| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
+|-
+|
+| Thread Presentation
+| Select the threads layout. Two layouts are available. '''Flat''' layout lists the threads in a flat list per trace. '''Hierarchical''' layout shows the threads in a parent-child tree per trace.
|}
+=== Marker Axis ===
+
+The marker axis is visible only when at least one marker category with markers for the current trace is shown.
+
+The marker axis displays one row per marker category. Each marker's time range and/or label (if applicable) are drawn on the marker axis.
+
+Clicking on any marker's time range or label will set the current time selection to the marker's time or time range.
+
+Clicking on the "X" icon to the left of the marker category name will hide this marker category from the time graph. It can be shown again using the corresponding '''Show Markers''' menu item in the view menu.
+
+The marker axis can be collapsed and expanded by clicking on the icon at the top left of the marker axis. The marker axis can be completely removed by hiding all available marker categories.
== Resources View ==
This view shows the state of system resources i.e. if changes occurred during the trace either on '''CPUs''', '''IRQs''' or '''soft IRQs''', it will appear in this view. The left side of the view present a list of resources that are affected by at least one event of the trace. The right side illustrate the state in which each resource is at some point in time. For state '''USERMODE''' it also prints the process name in the state bar. For state '''SYSCALL''' the name of the system call is
displayed in the state region.
+When an '''IRQ''' is handled by a '''CPU''', its states are shown under the corresponding '''CPU''' entry. Similarly, the '''CPU''' that was handling an '''IRQ''' is shown under the handled '''IRQ'''. Therefore, the trace can be visualized from a '''CPU''' point of view or from an '''IRQ''' point of view.
+
Just like other views, according to which trace points and system calls are activated, the content of this view may change from one trace to another.
The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).
This view is also synchronized with the others : [[#Histogram_View | Histogram View]], [[#LTTng_Kernel_Events_Editor | Events Editor]], [[#Control_Flow_View | Control Flow View]], etc.
+=== Follow CPU ===
+It is possible to follow a CPU by right-clicking on its entry in the view, then selecting ''Follow CPU X'' where X is the number of the CPU. Following a CPU will filter the [[#LTTng CPU Usage View | CPU Usage View]] to display only usage for the selected CPU. To unfollow a CPU, one needs to right-click on any CPU entry and select ''Unfollow CPU''.
+
=== Navigation ===
See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.
| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
-| Select Previous Event
+| Select Previous State Change
| Selects the previous state for the selected resource. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/next_event.gif]]
-| Select Next Event
+| Select Next State Change
| Selects the next state for the selected resource. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
| [[Image:images/add_bookmark.gif]]
{|
|
| Show Markers
-| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost event occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
+| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
|}
+=== Marker Axis ===
+
+See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.
== LTTng CPU Usage View ==
=== Process Information ===
+
The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID), process name (Process), the average (%) execution time and the actual execution time (Time) during the current time range. It shows all threads that were executing on the CPUs in the current time range.
The CPU Usage chart is usable with the mouse. The following actions are set:
* '''left-click''': select a time or time range begin time
-* '''left-drag horizontally''': select a time range or change the time range begin or end time
-* '''middle-drag''': pan left or right
-* '''right-drag horizontally''': zoom region
-* '''mouse wheel up/down''': zoom in or out
+* '''Shift-left-click or drag''': Extend or shrink the selection range
+* '''left-drag horizontally''': select a time range or change the time range begin or end time
+* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
+* '''right-drag horizontally''': [[#Zoom region|zoom region]]
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
==== Tooltips ====
[[Image:images/LTTng_CpuUsageViewToolTip.png]]
+==== CPU Filtering ====
+[[#Follow CPU | Follow a CPU]] will filter the CPU Usage View and will display only usage for the followed CPU.
+
+== Kernel Memory Usage View ==
+
+The Kernel Memory Usage and view is specific to kernel traces. To open the view, double-click on the '''Kernel Memory Usage Analysis''' tree element under the '''Kernel''' tree element of the Project Explorer.
+
+[[Image:images/kernelMemoryUsage/OpenKernelMemoryUsageView.png]]
+
+Now, the Kernel memory usage view will show:
+
+[[Image:images/kernelMemoryUsage/KernelMemoryUsageView.png]]
+
+Where:
+
+* '''TID''': The ID of the thread this event belongs to
+* '''Process''': The process of the TID that belongs to it
+
+The view is divided into the following important sections: '''Process Information''' and the '''Relative Kernel Memory Usage'''. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).
+
+
+=== Process Information ===
+
+The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID) and the process name (Process).
+
+
+=== Relative Kernel Memory Chart ===
+
+The Relative Kernel Memory Chart on the right side of the view, plots the relative amount of memory that was allocated and deallocated during that period of time.
+
+
+==== Using the mouse ====
+
+The Relative Kernel Memory chart is usable with the mouse. The following actions are set:
+
+* '''left-click''': select a time or time range begin time
+* '''Shift-left-click or drag''': Extend or shrink the selection range
+
+* '''left-drag horizontally''': select a time range or change the time range begin or end time
+* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
+* '''right-drag horizontally''': [[#Zoom region|zoom region]]
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
+
+
+==== Tooltips ====
+
+Hover the cursor over a line of the chart and a tooltip will pop up with the following information:
+* '''time''': current time of mouse position
+* '''Total''': The total CPU usage
+
+[[Image:images/kernelMemoryUsage/KernelMemoryUsageChart.png]]
+
+== Process Wait Analysis ==
+
+TraceCompass can recover wait causes of local and distributed processes using operating system events. The analysis highlights the tasks and devices causing wait. Wait cause recovery is recursive, comprise all tasks running on the system and works across computers using packet trace synchronization.
+
+The analysis details are available in the paper [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7294678&isnumber=4359390 Wait analysis of distributed systems using kernel tracing].
+
+=== Prerequisites ===
+
+The analysis requires a Linux kernel trace. Additional instrumentation may be required for specific kernel version and for distributed tracing. This instrumentation is available in [https://github.com/giraldeau/lttng-modules/tree/addons LTTng modules addons] on GitHub.
+
+The required events are:
+* '''sched_switch, sched_wakeup''': Scheduling events indicate when a process is blocked and the wake-up event indicates the task or resource that unblocked the task. For kernel versions comprised between 3.8 and 4.1, the event '''sched_ttwu''' (which stands for Try To Wake-Up) is provided for backward compatibility in LTTng modules addons.
+* '''IRQ, SoftIRQ and IPI''': Interrupt events are required to distinguish the context of the wake-up. When a wake-up occurs inside an interrupt handler, it must be associated with the device causing the interrupt and not the interrupted task. For that reason, interrupt entry and exit events are required.
+* '''inet_sock_local_in, inet_sock_local_out''': The network events record a subset of TCP/IP packet header using a netfilter hook in the kernel. The send and receive events are matched to show the communication between distributed processes. Network events are mandatory for analyzing wait in TCP/IP programs, whether they are executing locally or on different computers. They also used to synchronize traces recorded on multiple computers. For further details, refer to the [[#Trace synchronization]] section.
+
+To analyze a distributed program, all computers involved in the processing must be traced simultaneously. The LTTng Tracer Control of TraceCompass can trace a remote computer, but controlling simultaneous tracing is not supported at the moment, meaning that all sessions must be started separately and interactively. TraceCompass will support this feature in the future. For now, it is suggested to use [https://github.com/giraldeau/lttng-cluster lttng-cluster] command line tool to control simultaneous tracing sessions on multiple computers. This tool is based on [http://www.fabfile.org/ Fabric] and uses SSH to start the tracing sessions, execute a workload, stop the sessions and gather traces on the local computer. For more information, refer to the lttng-cluster documentation.
+
+We use the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] as an example to demonstrate the wait analysis. [https://www.djangoproject.com/ Django] is a popular Web framework. The application is the [https://docs.djangoproject.com/en/1.9/intro/tutorial01/ Django Poll app tutorial]. The traces were recorded on three computers, namely the client (implemented with Python Mechanize), the Web server (Apache with WSGI) and the database server (PostgreSQL). The client simulates a vote in the poll.
+
+=== Running the analysis ===
+
+To open all three traces simultaneously, we first create an experiment containing these traces and then synchronize the traces, such that they have a common time base. Then, the analysis is done by selecting a task in the '''Control Flow View'''. The result is displayed in the '''Critical Flow View''', which works like the '''Control Flow View'''. The steps to load the Django example follows.
+
+# Download and extract the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] archive.
+# In TraceCompass, open the [[#LTTng Kernel Perspective]].
+# Create a new tracing project. Select '''File -> New -> Tracing -> Tracing Project''', choose a name and click '''Finish'''.
+# Under the created tracing project, right-click on '''Traces''' and select '''Import...'''. In the import dialog, select the root directory containing the extracted trace by clicking on '''Browse'''. Three traces should be listed. Select the traces and click '''Finish'''. After the import is completed, the traces should be listed below '''Traces'''.
+# Right-click on '''Experiments''', select '''New...''' and enter a name for the experiment, such as '''django'''.
+# Right-click on the '''django''' experiment and click on '''Select Traces...'''. In the dialog, check the three traces '''django-client''', '''django-httpd''' and '''django-db'''. These traces will appear below the experiment. If the experiment is opened at this stage, the traces are not synchronized and there will be a large time gap between events from different traces.
+# To synchronize the traces, right-click on the '''django''' experiment and select '''Synchronize Traces'''. In the '''Select reference trace''' dialog, select any available trace and click '''Finish'''. Once the synchronization is completed, a new entry with an underline suffix will appear for each modified trace. The created trace entries have a function which is applied to the timestamps of events in order to shift the time according to the reference trace. The '''Project Explorer''' after the import is shown below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisProjectExplorer.png]]
+# Open the experiment '''django'''. The '''Control Flow''' and the '''Resources''' views should display the three traces simultaneously.
+# In the main menu, select '''Window -> Show View -> Other...''' and under '''LTTng''' select '''Critical Flow View'''. The view is empty for the moment.
+# In the '''Critical Flow View''', right-click on the '''Process''' entry to analyze and select '''Follow''', as shown in the figure below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisFollow.png]]
+#:The analysis will execute and the result will appear in the '''Critical Flow View'''. For the Django example, use the '''View Filters''' to search for the python process with TID 2327. When zooming on the execution, the view displays the work done by the Web server and the database to process the request of the python client. Vertical arrows represent synchronization and communication between processes. The legend [[Image:images/show_legend.gif]] displays the colors associated with the processes states.
+
+[[Image:images/waitAnalysis/KernelWaitAnalysisDjango.png]]
+
+== Input/Output Analysis ==
+
+TraceCompass can analyse disk input/output through the read/write system calls to get the read/write per processes, but also with the disk request events, to get the actual reads and writes to disk.
+
+=== Get the trace ===
+
+The following tracepoints should be enabled to get the disk read/write data. Also, enabling syscalls will allow to match the reads and writes per processes.
+
+ # sudo lttng list -k
+ Kernel events:
+ -------------
+ ...
+ block_rq_complete (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ block_rq_insert (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ block_rq_issue (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ block_bio_frontmerge (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ ...
+
+For full disk request tracking, some extra tracepoints are necessary. They are not required for the I/O analysis, but make the analysis more complete. Here is the procedure to get those tracepoints that are not yet part of the mainline kernel.
+
+ # git clone https://github.com/giraldeau/lttng-modules.git
+ # cd lttng-modules
+
+Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.
+
+ # git checkout addons
+ # make
+ # sudo make modules_install
+ # sudo depmod -a
+
+The lttng addons modules must be inserted manually for the extra tracepoints to be available:
+
+ # sudo modprobe lttng-addons
+ # sudo modprobe lttng-elv
+
+And enable the following tracepoint
+
+ addons_elv_merge_requests
+
+=== Input/Output Views ===
+
+The following views are available for input/output analyses:
+
+==== Disk I/O Activity View ====
+A time aligned XY chart of the read and write speed for the different disks on the system. This view is useful to see where there was more activity on the disks and whether it was mostly reads or writes.
+
+ [[Image:images/io/diskIoActivity.png| Disk I/O Activity Example]]
+
+== System Call Latency Analysis ==
+
+The '''System Call Latency Analysis''' measures the system call latency between system call entry and exit per type of system call. The durations are visualized using the '''Latency''' views. For more information about the '''Latency''' views see chapter [[#Latency_Analyses | Latency Analyses]].
+
== LTTng Kernel Events Editor ==
The LTTng Kernel Events editor '''is''' the plain TMF [[#Events_Editor | Events Editor]], except that it provides its own specialized viewer to replace the standard one. In short, it has exactly the same behaviour but the layout is slightly different:
Double-clicking on a call stack event will zoom the time graph to the selected function's range of execution.
-Clicking the '''Select Next Event''' or '''Select Previous Event''' or using the left and right arrows will navigate to the next or previous call stack event, and select the function currently at the top of the call stack. Note that pressing the '''Shift''' key at the same time will update the selection end time of the current selection.
+Clicking the '''Select Next State Change''' or '''Select Previous State Change''' or using the left and right arrows will navigate to the next or previous call stack event, and select the function currently at the top of the call stack. Note that pressing the '''Shift''' key at the same time will update the selection end time of the current selection.
-Clicking the '''Import Mapping File''' ([[Image:images/import.gif]]) icon will open a file selection dialog, allowing you to import a text file containing mappings from function addresses to function names. If the callstack provider for the current trace type only provides function addresses, a mapping file will be required to get the function names in the view. See the following sections for an example with LTTng-UST traces.
+Clicking the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) icon will open a file selection dialog, allowing you to import a text file containing mappings from function addresses to function names. If the callstack provider for the current trace type only provides function addresses, a mapping file will be required to get the function names in the view. See the following sections for an example with LTTng-UST traces.
=== Using the Callstack View with LTTng-UST traces ===
-There is support in the LTTng-UST integration plugin to display the callstack of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see the ''liblttng-ust-cyg-profile'' man page for additional information). To do so, you need to:
+There is support in the LTTng-UST integration plugin to display the callstack
+of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see
+the ''liblttng-ust-cyg-profile'' man page for additional information). To do
+so, you need to:
* Recompile your application with "''-g -finstrument-functions''".
-* Add the ''vtid'' and ''procname'' contexts to your trace session. See the [[#Adding Contexts to Channels and Events of a Domain]] section. Or if using the command-line:
-** <pre>lttng add-context -u -t vtid -t procname</pre>
+* Set up a tracing session with the the ''vpid'', ''vtid'' and ''procname'' contexts. See the [[#Enabling UST Events On Session Level]] and [[#Adding Contexts to Channels and Events of a Domain]] sections. Or if using the command-line:
+** <pre>lttng enable-event -u -a</pre>
+** <pre>lttng add-context -u -t vpid -t vtid -t procname</pre>
* Preload the ''liblttng-ust-cyg-profile'' library when running your program:
** <pre>LD_PRELOAD=/usr/lib/liblttng-ust-cyg-profile.so ./myprogram</pre>
-Once you load the resulting trace, making sure it's set to the ''Common Trace Format - LTTng UST Trace'' type, the Callstack View should be populated with the relevant information. However, since GCC's cyg-profile instrumentation only provides function addresses, and not names, an additional step is required to get the function names showing in the view. The following section explains how to do so.
+Once you load the resulting trace, the Callstack View should be populated with
+the relevant information.
-=== Importing a function name mapping file for LTTng-UST traces ===
+Note that for non-trivial applications, ''liblttng-ust-cyg-profile'' generates a
+'''lot''' of events! You may need to increase the channel's subbuffer size to
+avoid lost events. Refer to the
+[http://lttng.org/docs/#doc-fine-tuning-channels LTTng documentation].
-If you followed the steps in the previous section, you should have a Callstack View populated with function entries and exits. However, the view will display the function addresses instead of names in the intervals, which are not very useful by themselves. To get the actual function names, you need to:
+For traces taken with LTTng-UST 2.8 or later, the Callstack View should show the
+function names automatically, since it will make use of the debug information
+statedump events (which are enabled when using ''enable-event -u -a'').
+For traces taken with prior versions of UST, you would need to set the path to
+the binary file or mapping manually:
+
+=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===
+
+''For LTTng-UST 2.8+, if it doesn't resolve symbols automatically, see the [[#Binary file location configuration | Source Lookup's Binary file location configuration]].''
+
+If you followed the steps in the previous section, you should have a Callstack
+View populated with function entries and exits. However, the view will display
+the function addresses instead of names in the intervals, which are not very
+useful by themselves. To get the actual function names, you need to:
+
+* Click the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) button in the Callstack View.
+
+Then either:
+* Point to the binary that was used for taking the trace
+OR
* Generate a mapping file from the binary, using:
** <pre>nm myprogram > mapping.txt</pre>
-* Click the '''Import Mapping File''' ([[Image:images/import.gif]]) button in the Callstack View, and select the ''mapping.txt'' file that was just created.
+** Select the ''mapping.txt'' file that was just created.
+
+(If you are dealing with C++ executables, you may want to use ''nm --demangle''
+instead to get readable function names.)
+
+The view should now update to display the function names instead. Make sure the
+binary used for taking the trace is the one used for this step too (otherwise,
+there is a good chance of the addresses not being the same).
+
+=== Navigation ===
+
+See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.
+
+=== Marker Axis ===
+
+See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.
+
+== Flame Graph View ==
+
+This is an aggregate view of the function calls from the '''Call Stack View'''. This shows a bird's eye view of what are the main
+time sinks in the traced applications. Each entry in the '''Flame Graph''' represents an aggregation of all the calls to a function
+in a certain depth of the call stack having the same caller. So, functions in the '''Flame Graph''' are aggregated by depth and
+caller. This enables the user to find the most executed code path easily.
+
+* In a '''Flame Graph''', each entry (box) represents a function in the stack.
+* If one takes a single vertical line in the view, it represents a full call stack with parents calling children.
+* The ''x-axis'' represents total duration (execution time) and not absolute time, so it is not aligned with the other views.
+* The width of an entry is the total time spent in that function, including time spent calling the children.
+* The total time can exceed the longest duration, if the program is pre-empted and not running during its trace time.
+* Each thread traced makes its own flame graph.
+
+The function name is visible on each Flame graph event if the size permits. Each box in the '''Flame Graph'''
+has the same color as the box representing the same function in the '''Call Stack'''.
+
+To open this view select a trace, expand it in the '''Project Explorer''' then expand the
+'''Call Graph Analysis''' (the trace must be loaded) and open the '''Flame Graph'''.
+It's also possible to go in '''Window''' -> '''Show View''' -> '''Tracing''' then
+select '''Flame Graph''' in the list.
+
+[[Image:images/Flame_Graph.png|Flame Graph View]]
+
+To use the '''Flame graph''', one can navigate it and find which function is consuming the most self-time.
+This can be seen as a large plateau. Then the entry can be inspected. At this point, the worst offender in
+terms of CPU usage will be highlighted, however, it is not a single call to investigate, but rather the
+aggregation of all the calls. Right mouse-clicking on that entry will open a context sensitive menu.
+Selecting '''Go to minimum''' or '''Go to maximum''' will take the user to the minimum or maximum
+recorded times in the trace. This is interesting to compare and contrast the two.
+
+Hovering over a function will show a tooltip with the statistics on a per-function basis. One can see the total and self times
+(''worst-case'', ''best-case'', ''average'', ''total time'', ''standard deviation'', ''number of calls'') for that function.
+
+=== How to use a Flame Graph ===
+
+Observing the time spent in each function can show where most of the time is spent and where one could optimize.
+An example in the image above: one can see that ''mp_sort'' is a recursive sort function, it takes approximately
+40% of the execution time of the program. That means that perfectly parallelizing it can yield a gain of 20% for 2 threads, 33% for 3
+and so forth. Looking at the function '''print_current_files''', it takes about 30% of the time, and it has a child ''print_many_per_line'' that has a large
+self time (above 10%). This could be another area that can be targeted for optimization. Knowing this in advance helps developers
+know where to aim their efforts.
+
+It is recommended to have a kernel trace as well as a user space trace in an experiment
+while using the '''Flame Graph''' as it will show what is causing the largest delays.
+When using the '''Flame Graph''' together with a call stack and a kernel trace,
+an example work flow would be to find the worst offender in terms of time taken for a function
+that seems to be taking too longThen, using the context menu '''Go to maximum''', one can navigate
+to the maximum duration and see if the OS is, for example, preempting the function for too long,
+or if the issue is in the code being executed.
+
+=== Using the mouse ===
+
+*'''Double-click on the duration ruler''' will zoom the graph to the selected duration range.
+*'''Shift-left-click or drag''': Extend or shrink the selection range
+*'''Mouse wheel up/down''': scroll up or down
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
+* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically
+
+When the mouse cursor is over entries (left pane):
+
+*'''-''': Collapse the '''Flame Graph''' of the selected thread
+*'''+''': Expand the '''Flame Graph''' of the selected thread
+
+=== Using the keyboard ===
+
+The following keyboard shortcuts are available:
+
+*'''Down Arrow''': selects the next stack depth
+*'''Up Arrow''': selects the previous stack depth
+*'''Home''': selects the first thread's '''Flame Graph'''
+*'''End''': selects the last thread's '''Flame Graph''''s deepest depth
+*'''Enter''': toggles the expansion state of the current thread in the tree
+*'''Ctrl + +''': Zoom-in vertically
+*'''Ctrl + -''': Zoom-out vertically
+*'''Ctrl + 0''': Reset the vertical zoom
+
+=== Toolbar ===
+
+{|
+| [[Image:images/sort_alpha.gif]]
+| Sort by thread name
+| Sort the threads by thread name. Clicking the icon a second time will sort the threads by name in reverse order and change the icon to [[Image:images/sort_alpha_rev.gif]]
+|-
+| [[Image:images/sort_num.gif]]
+| Sort by thread id
+| Sort the threads by thread ID. Clicking the icon a second time will sort the threads by ID in reverse order and change the icon to [[Image:images/sort_num_rev.gif]].
+|}
+
+=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===
-(If you are dealing with C++ executables, you may want to use ''nm --demangle'' instead to get readable function names.)
+See Call Stack View's '''[[#Call Stack View | Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ]]'''.
-The view should now update to display the function names instead. Make sure the binary used for taking the trace is the one used for this step too (otherwise, there is a good chance of the addresses not being the same).
+== Function Duration Density ==
+The '''Function Duration Density''' view shows the function duration of function displayed by duration for the current active time window range. This is useful to find global outliers.
+
+[[Image:images/FunctionDensityView.png|Function Duration Density View]]
+
+Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset.
== Memory Usage ==
The Memory Usage chart is usable with the mouse. The following actions are set:
* '''left-click''': select a time or time range begin time
-* '''left-drag horizontally''': select a time range or change the time range begin or end time
-* '''middle-drag''': pan left or right
-* '''right-drag horizontally''': zoom region
-* '''mouse wheel up/down''': zoom in or out
+* '''Shift-left-click or drag''': Extend or shrink the selection range
+* '''left-drag horizontally''': select a time range or change the time range begin or end time
+* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
+* '''right-drag horizontally''': [[#Zoom region|zoom region]]
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
=== Toolbar ===
Please note this view will not show shared memory or stack memory usage.
+== Source Lookup (for LTTng-UST 2.8+) ==
+
+Starting with LTTng 2.8, the tracer can now provide enough information to
+associate trace events with their location in the original source code.
+
+To make use of this feature, first make sure your binaries are compiled with
+debug information (-g), so that the instruction pointers can be mapped to source
+code locations. This lookup is made using the ''addr2line'' command-line utility,
+which needs to be installed and on the '''$PATH''' of the system running Trace
+Compass. ''addr2line'' is available in most Linux distributions, Mac OS X, Windows using Cygwin and others.
+
+The following trace events need to be present in the trace:
+
+* lttng_ust_statedump:start
+* lttng_ust_statedump:end
+* lttng_ust_statedump:bin_info
+* lttng_ust_statedump:build_id
+
+as well as the following contexts:
+
+* vpid
+* ip
+
+For ease of use, you can simply enable all the UST events when setting up your
+session:
+
+ lttng enable-event -u -a
+ lttng add-context -u -t vpid -t ip
+
+Note that you can also create and configure your session using the [[#Control View | Control View]].
+
+If you want to track source locations in shared libraries loaded by the
+application, you also need to enable the "lttng_ust_dl:*" events, as well
+as preload the UST library providing them when running your program:
+
+ LD_PRELOAD=/path/to/liblttng-ust-dl.so ./myprogram
+
+If all the required information is present, then the ''Source Location'' column
+of the Event Table should be populated accordingly, and the ''Open Source Code''
+action should be available. Refer to the section [[#Event Source Lookup]] for
+more details.
+
+The ''Binary Location'' information should be present even if the original
+binaries are not available, since it only makes use of information found in the
+trace. A '''+''' denotes a relative address (i.e. an offset within the object
+itself), whereas a '''@''' denotes an absolute address, for
+non-position-independent objects.
+
+[[Image:images/sourceLookup/trace-with-debug-info.png]]
+
+''Example of a trace with debug info and corresponding Source Lookup information, showing a tracepoint originating from a shared library''
+
+=== Binary file location configuration ===
+
+To resolve addresses to function names and source code locations, the analysis
+makes use of the binary files (executables or shared libraries) present on the
+system. By default, it will look for the file paths as they are found in the
+trace, which means that it should work out-of-the-box if the trace was taken on
+the same machine that Trace Compass is running.
+
+It is possible to configure a ''root directory'' that will be used as a prefix
+for all file path resolutions. The button to open the configuration dialog is
+called '''Configure how addresses are mapped to function names''' and is
+currently located in the [[#Call Stack View]]. Note that the Call Stack View
+will also make use of this configuration to resolve its function names.
+
+[[Image:images/sourceLookup/symbol-mapping-config-ust28.png]]
+
+''The symbol configuration dialog for LTTng-UST 2.8+ traces''
+
+This can be useful if a trace was taken on a remote target, and an image of that
+target is available locally.
+
+If a binary file is being traced on a target, the paths in the trace will refer
+to the paths on the target. For example, if they are:
+
+* /usr/bin/program
+* /usr/lib/libsomething.so
+* /usr/local/lib/libcustom.so
+
+and an image of that target is copied locally on the system at
+''/home/user/project/image'', which means the binaries above end up at:
+
+* /home/user/project/image/usr/bin/program
+* /home/user/project/image/usr/lib/libsomething.so
+* /home/user/project/image/usr/local/lib/libcustom.so
+
+Then selecting the ''/home/user/project/image'' directory in the configuration
+dialog above will allow Trace Compass to read the debug symbols correctly.
+
+Note that this path prefix will apply to both binary file and source file
+locations, which may or may not be desirable.
+
= Trace synchronization =
It is possible to synchronize traces from different machines so that they have the same time reference. Events from the reference trace will have the same timestamps as usual, but the events from traces synchronized with the first one will have their timestamps transformed according to the formula obtained after synchronization.
Select an XML file from the list, click the '''Export''' button and enter or select a file in the opened file dialog to export the XML analysis. Note that if an existing file containing an analysis is selected, its content will be replaced with the analysis to export.
+* Edit
+
+Select an XML file from the list, click the '''Edit''' to open the XML editor. When the file is saved after being modified, it is validated and traces that are affected by this file are closed.
+
* Delete
Select an XML file from the list and click the '''Delete''' button to remove the XML file. Deleting an XML file will close all the traces for which this analysis applies and remove the analysis.
</head>
</pre>
-If pre-defined values will be used in the state provider, they must be defined before the state providers. They can then be referred to in the state changes by name, preceded by the '$' sign. It is not necessary to use pre-defined values, the state change can use values like (100, 101, 102) directly.
+If predefined values will be used in the state provider, they must be defined before the state providers. They can then be referred to in the state changes by name, preceded by the '$' sign. It is not necessary to use predefined values, the state change can use values like (100, 101, 102) directly.
<pre>
<definedValue name="RUNNING" value="100" />
If modifications are made to the XML state provider after it has been "published", the '''version''' attribute of the '''xmlStateProvider''' element should be updated. This avoids having to delete each trace's supplementary file manually. If the saved state system used a previous version, it will automatically be rebuilt from the XML file.
+== Defining an XML pattern provider ==
+It exists patterns within an execution trace that can provide high level details about the system execution. A '''pattern''' is a particular combination of events or states that are expected to occur within a trace. It may be composed of several state machines that inherit or communicate through a common state system.
+
+We may have multiple instances (scenarios) of a running state machine within a pattern. Each scenario which has its own path in the state system can generate segments to populate the data-driven views
+
+=== The state system structure ===
+
+The pattern analysis generates a predefined attribute tree described as follows :
+
+<pre>
+|- state machines
+| |- state machine 0
+| |- scenario 0
+| |- status
+| |- state
+| |- start
+| ...
+| |- storedFields
+| |- field 1
+| ...
+| |- startTime
+| ...
+| ...
+| |- scenarios 1
+| ...
+| |- state machine 1
+| ...
+</pre>
+
+The user can add custom data in this tree or determine its own attribute tree beside of this one.
+
+=== Writing the XML pattern provider ===
+Details about the XML structure are available in the XSD files.
+
+First define the pattern element. As the state provider element described in [[#Writing_the_XML_state_provider | Writing the XML state provider]], it has a "version" attribute and an "id" attribute.
+
+<pre>
+<pattern version="0" id="my.test.pattern">
+</pre>
+
+Optional header information as well as predefined values like described in [[#Writing_the_XML_state_provider | Writing the XML state provider]] can be added.
+
+Stored values can be added before the pattern handler. The predefined action '''saveStoredField''' triggers the updates of the stored fields and the predefined action '''clearStoredFields''' reset the values.
+
+<pre>
+<storedField id="offset" alias="offset"/>
+</pre>
+
+The behavior of the pattern and the models it needs are described in the pattern handler element.
+
+The structure of the state machine (FSM) is based on the SCXML structure. The following example describe an FSM that matches all the system call in an LTTng kernel trace.
+
+<pre>
+<fsm id="syscall" initial="start">
+ <state id="start">
+ <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
+ </state>
+ <state id="in_progress" >
+ <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
+ </state>
+ <final id="end"/>
+</fsm>
+</pre>
+
+The value of the target attribute corresponds to the 'id' of a test element described in the XML file and is a reference to it. Similarly, the value of the action attribute corresponds to the 'id' of an action element described in the XML file and is a reference to it.
+
+Conditions are used in the transitions to switch between the state of an FSM. They are defined under the '''test''' element. Two types of conditions are allowed : '''Data condition''' and '''Time condition'''. It is possible to combine several conditions using a logical operator (OR, AND, ...).
+
+Data conditions tests the ongoing event information against the data in the state system or constant values. The following condition tests whether the current thread running on the CPU is also the ongoing scenario thread.
+
+<pre>
+<test id="thread_condition">
+ <if>
+ <condition>
+ <stateValue type="query" >
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ </stateValue>
+ </condition>
+ </if>
+</test>
+</pre>
+
+Two types of time conditions are available:
+* Time range conditions tests whether the ongoing event happens between a specific range of time. The following condition tests whether the ongoing event happens between 1 nanosecond and 3 nanoseconds.
+
+<pre>
+<test id="time_condition">
+ <if>
+ <condition>
+ <timerange unit="ns">
+ <in begin="1" end="3" />
+ </timerange>
+ </condition>
+ </if>
+</test>
+</pre>
+
+* Elapsed time conditions tests the value of the time spent since a specific state of an fsm. The following condition tests whether the ongoing event happens less than 3 nanoseconds after that the scenario reaches the state "syscall_entry_x".
+
+<pre>
+<test id="time_condition">
+ <if>
+ <condition>
+ <elapsedTime unit="ns">
+ <less since="syscall_entry_x" value="3" />
+ </elapsedTime>
+ </condition>
+ </if>
+</test>
+</pre>
+
+Two types of actions are allowed :
+* State changes update values of attributes into the state system. The following example set the value of the thread for the current scenario.
+
+<pre>
+ <action id="sys_x_found">
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ <stateValue type="query">
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ </stateChange>
+ </action>
+</pre>
+
+* Generate segments. The following example represents a system call segment.
+
+<pre>
+<action id="exit_syscall_founded">
+ <segment>
+ <segType>
+ <segName>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ </stateValue>
+ </segName>
+ </segType>
+ </segment>
+</action>
+</pre>
+
+When existing, the stored fields will be added as fields for the generated segments.
+
+Here is the complete XML file by combining all the examples models above:
+
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:noNamespaceSchemaLocation="xmlDefinition.xsd">
+
+<pattern version="1" id="my.test.pattern">
+ <head>
+ <traceType id="org.eclipse.linuxtools.lttng2.kernel.tracetype" />
+ <label value="xml syscall" />
+ </head>
+
+ <storedField id="filename"/>
+ <storedField id="fd"/>
+ <storedField id="ret" alias="ret"/>
+ <storedField id="flags" alias="flags"/>
+ <storedField id="offset" alias="offset"/>
+ <storedField id="fd_in" alias="fd_in"/>
+ <storedField id="fd_out" alias="fd_out"/>
+ <storedField id="uservaddr" alias="uservaddr"/>
+ <storedField id="upeer_sockaddr" alias="upeer_sockaddr"/>
+
+ <location id="CurrentThread">
+ <stateAttribute type="constant" value="Threads" />
+ <stateAttribute type="query">
+ <stateAttribute type="constant" value="CPUs" />
+ <stateAttribute type="eventField" value="cpu" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateAttribute>
+ </location>
+
+ <location id="CurrentCPU">
+ <stateAttribute type="constant" value="CPUs" />
+ <stateAttribute type="eventField" value="cpu" />
+ </location>
+
+ <patternHandler>
+ <test id="time_condition">
+ <if>
+ <or>
+ <not>
+ <condition>
+ <timerange unit="ns">
+ <in begin="1" end="3" />
+ </timerange>
+ </condition>
+ </not>
+ <condition>
+ <elapsedTime unit="ns">
+ <less since="syscall_entry_x" value="3" />
+ </elapsedTime>
+ </condition>
+ </or>
+ </if>
+ </test>
+
+ <test id="thread_condition">
+ <if>
+ <condition>
+ <stateValue type="query" >
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ </stateValue>
+ </condition>
+ </if>
+ </test>
+
+ <action id="sys_x_founded">
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ <stateValue type="eventName"/>
+ </stateChange>
+
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="cpu" />
+ <stateValue type="eventField" value="cpu"/>
+ </stateChange>
+
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ <stateValue type="query">
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ </stateChange>
+ </action>
+
+ <action id="exit_syscall_founded">
+ <segment>
+ <segType>
+ <segName>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ </stateValue>
+ </segName>
+ </segType>
+ </segment>
+ </action>
+
+ <fsm id="syscall" initial="start">
+ <state id="start">
+ <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
+ </state>
+ <state id="in_progress" >
+ <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
+ </state>
+ <final id="end"/>
+ </fsm>
+ </patternHandler>
+</pattern>
+</tmfxml>
+</pre>
+
+=== Representing the scenarios ===
+
+Segments generated by the pattern analysis are used to populate latency views. A description of these views can be found in [[#Latency_Analyses | Latency Analyses]].
+
+The full XML analysis example described above will generate the following views :
+
+* Latency Table
+
+[[Image:images/XMLPatternAnalysis/LatencyTable.png| Latency Table example - System Call pattern]]
+
+* Latency vs Time
+
+[[Image:images/XMLPatternAnalysis/LatencyVSTime.png| Latency vs Time example - System Call pattern]]
+
+* Latency Statistics
+
+[[Image:images/XMLPatternAnalysis/LatencyStatistics.png| Latency Statistics example - System Call pattern]]
+
+* Latency vs Count
+
+[[Image:images/XMLPatternAnalysis/LatencyVSCount.png| Latency vs Count example - System Call pattern]]
+
== Defining an XML time graph view ==
A time graph view is a view divided in two, with a tree viewer on the left showing information on the different entries to display and a Gantt-like viewer on the right, showing the state of the entries over time. The [[#Control_Flow_View | Control Flow View]] is an example of a time graph view.
-Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any pre-defined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
+Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
In the example above, suppose we want to display the status for each task. In the state system, it means the path of the entries to display is "Tasks/*". The attribute whose value should be shown in the Gantt chart is the entry attribute itself. So the XML to display these entries would be as such:
[[Image:images/Xml_analysis_screenshot.png| XML analysis with view]]
+==== Using the keyboard ====
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+
== Defining an XML XY chart ==
An XY chart displays series as a set of numerical values over time. The X-axis represents the time and is synchronized with the trace's current time range. The Y-axis can be any numerical value.
-Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any pre-defined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
+Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
We will use the Linux Kernel Analysis on LTTng kernel traces to show an example XY chart. In this state system, the status of each CPU is a numerical value. We will display this value as the Y axis of the series. There will be one series per CPU. The XML to display these entries would be as such:
<pre>
<head>
- <analysis id="org.eclipse.tracecompass.lttng2.kernel.analysis" />
+ <analysis id="org.eclipse.tracecompass.analysis.os.linux.kernel" />
<label value="CPU status XY view" />
</head>
</pre>
Trace Compass offers a feature called Latency analysis. This allows an analysis to return intervals and these intervals will be displayed in four different views. An example analysis is provided with kernel system call latencies being provided. The available views are:
-* System Call Latencies
+* System Call Latencies
A '''table''' of the raw latencies. This view is useful to inspect individual latencies.
[[Image:images/LatenciesTable.png| Latency Table example - System Call Latencies]]
* System Call Latency Statistics
-A view of the total '''statistics''' of the latencies. These show the ''minimum'', ''maximum'', ''average'' and ''standard deviation'' of the latencies when applicable. This tool is useful for finding the outliers on a per-category basis.
+A view of the total '''statistics''' of the latencies. These show the ''minimum'', ''maximum'', ''average'', ''standard deviation'', and ''count'' of the latencies when applicable. This tool is useful for finding the outliers on a per-category basis.
-[[Image:images/LatenciesStatistics.png| Latency Statistics example - System Call Latency Statistics]]
+Right-clicking on an entry of the table and select '''Go to minimum''' allows to select the range of the minimum latency for the selected entry and synchronize the other views to this time range.
+Right-clicking on an entry of the table and select '''Go to maximum''' allows to select the range of the maximum latency for the selected entry and synchronize the other views to this time range.
+
+[[Image:images/LatenciesStatistics.png| Latency Statistics example - System Call Latency Statistics]]
* System Call Density
-A '''density''' view, analyzing the current time range. This is useful to find global outliers.
+A '''density''' view, analyzing the current time range. This is useful to find global outliers. Selecting a duration in the table it will synchronize other views to this time range.
[[Image:images/LatenciesDensity.png| Latency Densities example - System Call Density]]
+Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset.
+
= Virtual Machine Analysis =
Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.
[[Image:images/vmAnalysis/VM_CPU_view.png | Virtual CPU view]]
+==== Using the keyboard ====
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+
== Hypervisor-specific Tracing ==
In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.
Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [[#Trace synchronization | trace synchronization]] for information on how to obtain synchronizable traces.
+= Java Logging =
+
+Trace Compass contains some Java Utility Logging (JUL) tracepoints in various places in the code. To diagnose issues with Trace Compass or when reporting problems with the application, a JUL trace may be useful to help pinpoint the problem. The following sections explain how to enable JUL logging in Trace Compass and use various handlers to handle the data.
+
+== Enable JUL Logging ==
+
+By default, all the logging of the Trace Compass namespace is disabled. To enable it, one needs to add the following property to the ''vmargs'': ''-Dorg.eclipse.tracecompass.logging=true''.
+
+The log levels and components can be controlled via a configuration file whose path is specified also in the ''vmargs'' like this: ''-Djava.util.logging.config.file=/path/to/logger.properties''. An example configuration file can be found in the next section.
+
+If running the RCP, these arguments can be appended at the end of the ''tracecompass.ini'' file located in the folder where the executable is located. If running from Eclipse in development mode, in the ''Run configurations...'', the arguments should be added in the ''Arguments'' tab in the ''VM args'' box.
+
+== Configuring JUL logging ==
+
+JUL logging can be fine-tuned to log only specific components, specific levels, but also to different log handlers, with different formats, etc. Or else, the default level is INFO and the default log handler is a ConsoleHandler which displays all log message to the Console, which can be quite cumbersome.
+
+Here is an example ''logger.properties'' file to control what is being logged and where.
+
+ # Specify the handlers to create in the root logger
+ # (all loggers are children of the root logger)
+ # These are example handlers
+
+ # Console handler
+ handlers = java.util.logging.ConsoleHandler
+ # Console and file handlers
+ #handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler
+ # No handler
+ #handlers =
+
+ # Set the default logging level for the root logger
+ # Possible values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
+ .level = OFF
+
+ # Fine tune log levels for specific components
+ # Use the INFO level for all tracecompass, but FINEST for the StateSystem component
+ #org.eclipse.tracecompass.internal.statesystem.core.StateSystem.level = FINEST
+ org.eclipse.tracecompass.level = INFO
+
+== LTTng JUL log handler ==
+
+The various log handlers have an overhead on the application. The ConsoleHandler has a visible impact on Trace Compass performance. The FileHandler also has an overhead though less visible, but when logging from multiple threads at the same time, the file becomes a bottleneck, so that logging data cannot be used with accuracy for performance analysis. The [http://lttng.org/docs/#doc-java-application LTTng log handler] is much better in a multi-threads context.
+
+LTTng-UST comes with the Java JUL agent in most distros. Otherwise, it is possible to manually compile lttng-ust with options ''--enable-java-agent-jul'' and install it.
+
+ git clone git://git.lttng.org/lttng-ust.git
+ cd lttng-ust
+ ./bootstrap
+ ./configure --enable-java-agent-jul
+ make
+ sudo make install
+
+The necessary classes for the java agent will have been installed on the system. Since Equinox (the OSGi implementation used by Eclipse and thus Trace Compass) uses its own classpath and ignores any classpath entered on the command line for security reasons, one needs to specify the agent class path with the bootclasspath argument:
+
+ -Xbootclasspath/a:/usr/local/share/java/lttng-ust-agent-jul.jar:/usr/local/share/java/lttng-ust-agent-common.jar
+
+Note that unlike the -classpath argument, -Xbootsclasspath does not follow the dependencies specified by a jar's Manifest, thus it is required to list both the -jul and the -common jars here.
+
+These classes need to load the LTTng JNI library. Because they were loaded from the boot class path by the boot ClassLoader, the library path entered on the command line is ignored. A workaround is to manually copy the library to the jvm's main library path. For example
+
+ sudo cp /usr/local/lib/liblttng-ust-jul-jni.so /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/
+
+Or to overwrite the JVM's library path with the following VM argument.
+
+ -Dsun.boot.library.path=/usr/local/lib
+
+''Disclaimer: this last method overwrites the main java library path. It may have unknown side-effects. None were found yet.''
+
+LTTng can now be used as a handler for Trace Compass's JUL, by adding the following line to the logger.properties file
+
+ handlers = org.lttng.ust.agent.jul.LttngLogHandler
+
+The tracepoints will be those logged by a previously defined configuration file. Here is how to setup LTTng to handle JUL logging:
+
+ lttng create
+ lttng enable-event -j -a
+ lttng start
+
= Limitations =
* When parsing text traces, the timestamps are assumed to be in the local time zone. This means that when combining it to CTF binary traces, there could be offsets by a few hours depending on where the traces were taken and where they were read.
projects / deliverable / tracecompass.git / blobdiff