The option '''Preserve folder structure''' will create, if necessary, the structure of folders relative to (and excluding) the selected '''Root directory''' (or '''Archive file''') into the target trace folder.
+The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the '''Root directory''' name, when importing from directory, or the ''' Archive file''' name, when importing from archive. One can change the experiment name by typing a new name in the text box beside the option.
+
[[Image:images/ProjectImportTraceDialog.png]]
If a trace already exists with the same name in the target trace folder, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example smalltrace becomes smalltrace(2).
When Finish is clicked, the trace is imported in the target folder. The folder structure from the trace package is restored in the target folder.
+=== Refreshing of Trace and Trace Folder ===
+Traces and trace folders in the workspace might be updated on the media. To refresh the content, right-click the trace or trace folder and select menu item '''Refresh'''. Alternatively, select the trace or trace folder and press key '''F5'''.
+
=== Remote Fetching ===
It is possible to import traces automatically from one or more remote hosts according to a predefined remote profile by using the '''Fetch Remote Traces''' wizard.
The columns of the table are defined by the fields (aspects) of the specific trace type. These are the defaults:
* '''Timestamp''': the event timestamp
-* '''Type''': the event type
+* '''Event Type''': the event type
* '''Contents''': the fields (or payload) of this event
The first row of the table is the header row a.k.a. the Search and Filter row.
Searching and filtering of events in the table can be performed by entering matching conditions in one or multiple columns in the header row (the first row below the column header).
-To toggle between searching and filtering, click on the 'search' ([[Image:images/TmfEventSearch.gif]]) or 'filter' ([[Image:images/TmfEventFilter.gif]]) icon in the header row's left margin, or right-click on the header row and select '''Show Filter Bar''' or '''Show Search Bar''' in the context menu.
+To apply a matching condition to a specific column, click on the column's header row cell, type in a [http://download.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html regular expression]. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.
+
+Press the '''Enter''' key to apply the condition as a search condition. It will be added to any existing search conditions.
-To apply a matching condition to a specific column, click on the column's header row cell, type in a [http://download.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html regular expression] and press the '''ENTER''' key. You can also enter a simple text string and it will be automatically be replaced with a 'contains' regular expression.
+Press the '''Ctrl+Enter''' key to immediately add the condition (and any other existing search conditions) as a filter instead.
When matching conditions are applied to two or more columns, all conditions must be met for the event to match (i.e. 'and' behavior).
-To clear all matching conditions in the header row, press the '''DEL''' key.
+A preset filter created in the [[#Filters_View | Filters]] view can also be applied by right-clicking on the table and selecting '''Apply preset filter...''' > ''filter name''
==== Searching ====
[[Image:images/TraceEditor-Search.png]]
-Pressing the '''ENTER''' key will search and select the next matching event. Pressing the '''SHIFT-ENTER''' key will search and select the previous matching event. Wrapping will occur in both directions.
+Pressing the '''Enter''' key will search and select the next matching event. Pressing the '''Shift+Enter''' key will search and select the previous matching event. Wrapping will occur in both directions.
-Press '''ESC''' to cancel an ongoing search.
+Press '''Esc''' to cancel an ongoing search.
-Press '''DEL''' to clear the header row and reset all events to normal.
+To add the currently applied search condition(s) as filter(s), click the '''Add as Filter''' [[Image:images/filter_add.gif]] button in the header row margin, or press the '''Ctrl+Enter''' key.
+
+Press '''Delete''' to clear the header row and reset all events to normal.
==== Filtering ====
-When a filtering condition is entered in the head row, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace. The characters in each column which match the regular expression will be highlighted.
+When a new filter is applied, the table will clear all events and fill itself with matching events as they are found from the beginning of the trace. The characters in each column which match the regular expression will be highlighted.
A status row will be displayed before and after the matching events, dynamically showing how many matching events were found and how many events were processed so far. Once the filtering is completed, the status row icon in the left margin will change from a 'stop' to a 'filter' icon.
[[Image:images/TraceEditor-Filter.png]]
-Press '''ESC''' to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.
+Press '''Esc''' to stop an ongoing filtering. In this case the status row icon will remain as a 'stop' icon to indicate that not all events were processed.
+
+The header bar will be displayed above the table and will show a label for each applied filter. Clicking on a label will highlight the matching strings in the events that correspond to this filter condition. Pressing the '''Delete''' key will clear this highlighting.
-Press '''DEL''' or right-click on the table and select '''Clear Filters''' from the context menu to clear the header row and remove the filtering. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filter is removed.
+To remove a specific filter, click on the [[Image:images/delete_button.gif]] icon on its label in the header bar. The table will be updated with the events matching the remaining filters.
-You can also search on the subset of filtered events by toggling the header row to the Search Bar while a filter is applied. Searching and filtering conditions are independent of each other.
+The header bar can be collapsed and expanded by clicking on the [[Image:images/expanded_ovr.gif]][[Image:images/collapsed_ovr.gif]] icons in the top-left corner or on its background. In collapsed mode, a minimized version of the filter labels will be shown that can also be used to highlight or remove the corresponding filter.
+
+Right-click on the table and select '''Clear Filters''' from the context menu to remove all filters. All trace events will be now shown in the table. Note that the currently selected event will remain selected even after the filters are removed.
+
+You can also search on the subset of filtered events by entering a search condition in the header row while a filter is applied. Searching and filtering conditions are independent of each other.
==== Bookmarking ====
=== Event Source Lookup ===
-For CTF traces using specification v1.8.2 or above, information can optionally be embedded in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
+Some trace types can optionally embed information in the trace to indicate the source of a trace event. This is accessed through the event context menu by right-clicking on an event in the table.
==== Source Code ====
''Note'': The columns in the text file are separated by tabs.
+=== Refreshing of Trace ===
+
+It's possible to refresh the content of the trace and resume indexing in case the current open trace was updated on the media. To refresh the trace, right-click into the table and select menu item '''Refresh'''. Alternatively, press key '''F5'''.
+
=== Collapsing of Repetitive Events ===
The implementation for collapsing of repetitive events is trace type specific and is only available for certain trace types. For example, a trace type could allow collapsing of consecutive events that have the same event content but not the same timestamp. If a trace type supports this feature then it is possible to select the '''Collapse Events''' menu item after pressing the right mouse button in the table.
[[Image:images/TablePostCollapse.png]]
-To clear collapsing, press the right mouse button in the table and select menu item '''Clear Filters''' in the context sensitive menu. ''Note'' that collapsing is also removed when another filter is applied to the table.
+To remove the collapse filter, press the ([[Image:images/delete_button.gif]]) icon on the '''Collapse''' label in the header bar, or press the right mouse button in the table and select menu item '''Clear Filters''' in the context sensitive menu (this will also remove any other filters).
=== Customization ===
The filters can be more complex than what can be achieved with the filter header row in the events table. The filter is defined in a tree node structure, where the node types can be any of '''TRACETYPE''', '''AND''', '''OR''', '''CONTAINS''', '''EQUALS''', '''MATCHES''' or '''COMPARE'''. Some nodes types have restrictions on their possible children in the tree.
-The '''TRACETYPE''' node filters against the trace type of the trace as defined in a plug-in extension or in a custom parser. When used, any child node will have its aspect combo box restricted to the possible aspects of that trace type.
+The '''TRACETYPE''' node filters against the trace type of the trace as defined in a plug-in extension or in a custom parser. When used, any child node will have its ''type'' combo box fixed and its ''aspect'' combo box restricted to the possible aspects of that trace type.
The '''AND''' node applies the logical ''and'' condition on all of its children. All children conditions must be true for the filter to match. A ''not'' operator can be applied to invert the condition.
The '''OR''' node applies the logical ''or'' condition on all of its children. At least one children condition must be true for the filter to match. A ''not'' operator can be applied to invert the condition.
-The '''CONTAINS''' node matches when the specified event ''aspect'' value contains the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive.
+The '''CONTAINS''' node matches when the specified event ''aspect'' value contains the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive. The ''type'' combo box restricts the possible aspects to those of the specified trace type.
-The '''EQUALS''' node matches when the specified event ''aspect'' value equals exactly the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive.
+The '''EQUALS''' node matches when the specified event ''aspect'' value equals exactly the specified ''value'' string. A ''not'' operator can be applied to invert the condition. The condition can be case sensitive or insensitive. The ''type'' combo box restricts the possible aspects to those of the specified trace type.
-The '''MATCHES''' node matches when the specified event ''aspect'' value matches against the specified ''regular expression''. A ''not'' operator can be applied to invert the condition.
+The '''MATCHES''' node matches when the specified event ''aspect'' value matches against the specified ''regular expression''. A ''not'' operator can be applied to invert the condition. The ''type'' combo box restricts the possible aspects to those of the specified trace type.
-The '''COMPARE''' node matches when the specified event ''aspect'' value compared with the specified ''value'' gives the specified ''result''. The result can be set to ''smaller than'', ''equal'' or ''greater than''. The type of comparison can be numerical, alphanumerical or based on time stamp. A ''not'' operator can be applied to invert the condition.
+The '''COMPARE''' node matches when the specified event ''aspect'' value compared with the specified ''value'' gives the specified ''result''. The result can be set to ''smaller than'', ''equal'' or ''greater than''. The type of comparison can be numerical, alphanumerical or based on time stamp. A ''not'' operator can be applied to invert the condition. The ''type'' combo box restricts the possible aspects to those of the specified trace type.
For numerical comparisons, strings prefixed by "0x", "0X" or "#" are treated as hexadecimal numbers and strings prefixed by "0" are treated as octal numbers.
[[Image:images/TimeAlignment_sash.png]]
+== Searching in Time Graph Views ==
+
+Search for an entry in a '''Time Graph view''', e.g. [[#Control_Flow_View | Control Flow View]] or [[#Resources_View | Resources View]], using the ''' Find ''' dialog. To use the dialog :
+
+* Select the time graph view you want to search in
+* Press ''' Ctrl + F '''. The following screen will be shown :
+
+[[Image:images/FindDialog.png]]
+
+* Enter the string to find in the ''' Find ''' text drop down and select the ''' Options ''' and ''' Direction ''' you need.
+* Press the ''' Find ''' button or ''' Enter ''' or ''' Alt + n '''. The next match in the selected time graph view will be selected.
+
+Various options are available in the ''' Options ''' group :
+* ''' Case sensitive ''' makes the search case sensitive.
+* ''' Wrap search ''' restarts the search from the first index, depending of the direction, when no entry were found.
+* ''' Whole word ''' allows to search for whole words, delimited by spaces or special character, that are identical to the search text.
+* ''' Regular expression ''' specifies that the search text is a regular expression or not.
+
+The ''' Direction ''' group allows to select the search direction : ''' Forward ''' or ''' Backward '''.
+
= LTTng Tracer Control =
The LTTng Tracer Control in Eclipse for the LTTng Tracer toolchain version v2.0 (or later) is done using SSH and requires an SSH server to be running on the remote host. For the SSH connection the SSH implementation of Remote Services is used. The functions to control the LTTng tracer (e.g. start and stop), either locally or remotely, are available from a dedicated Control View.
Refer to chapter [[#Recording a Snapshot | Recording a Snapshot]] for how to create a snapshot.
-=== Creating a Live Tracing Session ===
+<!--=== Creating a Live Tracing Session ===
LTTng Tools version v2.4.0 introduces the possibility to create live tracing sessions. The live mode allows you to stream the trace and view it while it's being recorded. To create such a live session, open the trace session dialog as described in chapter [[#Creating a Tracing Session | Creating a Tracing Session]].
[[Image:images/LTTng2CreateSessionDialog_Live.png]]
[[Image:images/LTTng2CreateSessionDialog_Live_Advanced.png]]
Fill in all necessary information, select the radio button for '''Live Mode''' and press '''Ok'''.
-
+-->
=== Enabling Channels - General ===
Enabling channels can be done using a session tree node when the domain hasn't be created in the session or, alternatively on a domain tree node of a session in case the domain is already available.
[[Image:images/LTTng2AssignedEvents.png]]
-=== Configuring Filter Expression On UST Event Fields ===
+=== Configuring Filter Expression When Enabling Events ===
-Since LTTng Tools v2.1.0 it is possible to configure a filter expression on UST event fields. To configure a filter expression on UST event fields, open the enable event dialog as described in chapters [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]], [[#Enabling Events On Domain Level | Enabling Events On Domain Level]] or [[#Enabling Events On Channel Level | Enabling Events On Channel Level]], select UST if needed, select the relevant '''Tracepoint''' event(s) and enter the filter expression in the '''Filter Expression''' text field.
+It is possible to provide a filter expression when enabling events for UST or Kernel. This feature has been available for UST since LTTng v2.1.0 and for Kernel since v2.7.0. To configure a filter expression, open the enable event dialog as described in previous chapters [[#Enabling UST Events On Session Level | Enabling UST Events On Session Level]], [[#Enabling Kernel Events On Session Level | Enabling Kernel Events On Session Level]], [[#Enabling Events On Domain Level | Enabling Events On Domain Level]] or [[#Enabling Events On Channel Level | Enabling Events On Channel Level]]. Then configure the relevant events and enter the filter expression in the '''Filter Expression''' text field.
-[[Image:images/LTTng2EnableEventWithFilter.png]]
+[[Image:images/LTTng2EnableEventWithFilter.png]] [[Image:images/LTTng2EnableEventWithKernelFilter.png]]
-Alternatively, open the dialog box for assigning events to a session and channel described in [[#Enabling Tracepoint Events From Provider | Enabling Tracepoint Events From Provider]] (for UST providers) and enter the filter expression in the '''Filter Expression''' text field.
+Alternatively, open the dialog box for assigning events to a session and channel described in [[#Enabling Tracepoint Events From Provider | Enabling Tracepoint Events From Provider]] and enter the filter expression in the '''Filter Expression''' text field.
[[Image:images/LTTng2AssignEventDialogWithFilter.png]]
Upon successful operation, the tree in the Control View will be refreshed with the remote host configuration.
-=== Quantifing LTTng overhead (Calibrate) ===
-
-The LTTng calibrate command can be used to find out the combined average overhead of the LTTng tracer and the instrumentation mechanisms used. For now, the only calibration implemented is that of the kernel function
-instrumentation (kretprobes). To run the calibrate command, select the a domain (e.g. '''Kernel'''), click the right mouse button on the domain tree node. A context-sensitive menu will show. Select the '''Calibrate''' menu item.
-
-[[Image:images/LTTng2CalibrateAction.png]]
-
-Upon successful operation, the calibrate command is executed and relevant information is stored in the trace. Note: that the trace has to be active so that to command as any effect.
-
=== Importing Session Traces to a Tracing Project ===
To import traces from a tracing session, select the relevant session and click on the '''Import''' Button. Alternatively, click the right mouse button on the session tree node and select the menu item '''Import...''' from the context-sensitive menu.
[[Image:images/LTTng2ImportDialog.png]]
-By default all traces are selected. A default project with the name '''Remote''' is selected which will be created if necessary. Update the list of traces to be imported, if necessary, by selecting and deselecting the relevant traces in the tree viewer. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Also if needed, change the tracing project from the '''Available Projects''' combo box. Select the Overwrite button ('''Overwrite existing trace without warning''') if required. Then press button '''Ok'''. Upon successful import operation the selected traces will be stored in the '''Traces''' directory of the specified tracing project. The session directory structure as well as the trace names will be preserved in the destination tracing project. For '''Kernel''' traces the trace type '''Linux Kernel Trace''' and for '''UST''' traces the trace type '''LTTng UST Trace''' will be set. From the '''Project Explorer''' view, the trace can be analyzed further.
-
-'''Note''': If the overwrite button ('''Overwrite existing trace without warning''') was not selected and a trace with the same name of a trace to be imported already exists in the destination directory of the project, then a new confirmation dialog box will open.
+By default all traces are selected. A default project with the name '''Remote''' is selected which will be created if necessary. Update the list of traces to be imported, if necessary, by selecting and deselecting the relevant traces in the tree viewer. Use buttons '''Select All''' or '''Deselect All''' to select or deselect all traces. Also if needed, change the tracing project from the '''Available Projects''' combo box. The option '''Create Experiment''' will create an experiment with all imported traces. By default, the experiment name is the session name. One can change the experiment name by typing a new name in the text box beside the option.
-[[Image:images/LTTng2ImportOverwriteConfirmationDialog.png]]
+Then press button '''Finish'''. Upon successful import operation the selected traces will be stored in the '''Traces''' directory of the specified tracing project. A directory with the connection name will be created under the '''Traces''' directory. Underneath that, the session directory structure as well as the trace names will be preserved in the destination tracing project. For '''Kernel''' traces the trace type '''Linux Kernel Trace''' and for '''UST''' traces the trace type '''LTTng UST Trace''' will be set. From the '''Project Explorer''' view, the trace can be analyzed further.
-To Overwrite select the '''Overwrite''' Button and press '''Ok'''.
+'''Note''': If a trace already exists with the same name in the destination directory, the user can choose to rename the imported trace, overwrite the original trace or skip the trace. When rename is chosen, a number is appended to the trace name, for example kernel becomes kernel(2).
-If the existing trace should not be overwritten select, then select the '''Rename''' option of the confirmation dialog box above, enter a new name and then press '''Ok'''.
+[[Image:images/LTTng2ImportOverwriteConfirmationDialog.png]]
-[[Image:images/LTTng2ImportRenameDialog.png]]
+If one selects '''Rename All''', '''Overwrite All''' or '''Skip All''' the choice will be applied for all traces with a name conflict.
=== Importing Network Traces to a Tracing Project ===
Since LTTng Tools v2.1.0 it is possible to store traces over the network. To import network traces, execute the '''Import''' action as described in chapter [[#Importing Session Traces to a Tracing Project | Importing Session Traces to a Tracing Project]]. For network traces the '''Import Trace Wizard''' will be displayed. Follow the instructions in chapter [[#Importing | Importing]] to import the network traces of the current session.
+=== Saving Tracing Sessions ===
+Since LTTng Tools v2.5.0 it is possible to save tracing sessions. The LTTng Tools command-line tool will save the sessions to XML files located by default in a subdirectory of the user's home directory. The Trace Compass '''Control''' view integration for this feature will also store this session profile file into the user's Trace Compass workspace. This will allow user's to re-use session profiles across remote nodes. To save one or more sessions, select the tree nodes of the relevant sessions and press the right mouse button. Then select the '''Save...''' entry of the context-sensitive menu.
+
+[[Image:images/LTTng2SaveAction.png]]
+
+A new display will open for saving the sessions.
+
+[[Image:images/LTTng2SaveDialog.png]]
+
+By default the '''force''' button is selected that will overwrite any conflicting session profile files on the remote node. Click on '''Ok''' to save the session(s) otherwise click on '''Cancel'''. Upon successful operation, the session profile files will be saved on the remote node and then will be downloaded to the user's Trace Compass workspace. In the case that a session XML file already exists in the workspace the user will be prompted to skip or overwrite the existing profile file.
+
+=== Loading Tracing Sessions ===
+Since LTTng Tools v2.5.0 it is possible to load tracing sessions. The Trace Compass '''Control''' view integrations for this feature will allow to load session profiles that are located in the user's Trace Compass workspace, or alternatively, that are located on the remote node. In the first case the session profiles will be uploaded to the remote node before the load command is executed.
+
+To load one or more sessions, select the tree node '''Sessions''' and press the right mouse button. Then select the '''Load...''' entry of the context-sensitive menu.
+
+[[Image:images/LTTng2LoadAction.png]]
+
+A new display will open for loading session profiles.
+
+[[Image:images/LTTng2LoadDialog.png]]
+
+By default the '''Local''' button and '''force''' buttons are selected and session profile files of the user's workspace will be listed. Select one or more profiles, update the '''force''' button if needed and then click '''Ok'''. This will upload the session profile files to the remote node. If a session profile file with the same name already exist on the remote node, it will be overwritten. If the '''force''' button is selected any existing session with a conflicting name will be destroyed and a new one will be created.
+
+Alternatively, one can select the '''Remote''' button to list all available session profile files on the remote node. To load one of the remote session profiles, select one or more profiles, update the '''force''' button if needed and then click '''Ok'''.
+
+[[Image:images/LTTng2LoadRemoteDialog.png]]
+
+Upon successful operation, the tracing sessions of the selected session profiles are created and added under the tree node '''Sessions''' the '''Control''' view.
+
+=== Managing Tracing Session Profiles ===
+The '''LTTng Remote Profiles''' preference page is used to manage the list of LTTng session profiles that are stored in the user's Trace Compass workspace. To open the preference page, select the '''Manage...''' button of the '''Load Sessions''' dialog described in chapter [[#Loading Tracing Sessions |Loading Tracing Sessions]]. Alternatively, select '''Window -> Preferences''' from the top level menu and go to '''Tracing -> LTTng Remote Profiles'''.
+
+[[Image:images/LTTng2ManageSessionConfig.png]]
+
+The following actions can be performed from this dialog:
+
+* Delete
+
+Select one or more LTTng session profiles from the list and click the '''Delete''' button to remove the profile from the Trace Compass workspace. The user will be prompted to confirm the deletion.
+
+* Import...
+
+Click the '''Import...''' button and select a file from the opened file dialog to import a session profile file. If the file name conflicts with an existing profile file, the user will be prompted to skip or overwrite the existing profile file.
+* Export...
+
+Select one or more session profile files from the list, click the '''Export...''' button and enter or select a directory in the opened directory dialog to export the profile files. If the file name conflicts with an existing profile file in the destination directory, the user will be prompted to skip or overwrite the existing profile file.
+
== Properties View ==
The Control View provides property information of selected tree component. Depending on the selected tree component different properties are displayed in the property view. For example, when selecting the node level the property view will be filled as followed:
[[Image:images/LTTng2PreferencesTimeout.png]]
+
= LTTng Kernel Analysis =
Historically, LTTng was developped to trace the Linux kernel and, over time, a number of kernel-oriented analysis views were developped and organized in a perspective.
The TID column shows the process node's '''thread ID''' and the PTID column shows its '''parent thread ID''' (nothing is shown if the process has no parent).
+It is possible to sort the columns of the tree by clicking on the column header. Subsequent clicking will change the sort order. The hierarchy, i.e. the parent-child relationship is kept. When opening a trace for the first time, the processes are sorted by '''birth time'''. The sort order and column will be preserved when switching between open traces. Note that when opening an experiment the processes will be sorted within each trace.
+
=== Control flow ===
This part of the Control Flow View is probably the most interesting one. Using the mouse, you can navigate through the trace (go left, right) and zoom on a specific region to inspect its details.
==== Using the mouse ====
-The states flow is usable with the mouse. The following actions are set:
+The following mouse actions are available:
* '''left-click''': select a time or time range begin time
* '''Shift-left-click''': select a time range end time
* '''right-drag horizontally''': [[#Zoom region|zoom region]]
* '''click on a colored bar''': the associated process node is selected and the current time indicator is moved where the click happened
* '''mouse wheel up/down''': scroll up or down
-* '''Ctrl-mouse wheel up/down''': zoom in or out
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
+* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically
* '''drag the time ruler horizontally''': zoom in or out with fixed start time
* '''double-click the time ruler''': reset zoom to full range
==== Using the keyboard ====
-The states flow is usable with the keyboard. The following actions are set:
-*'''arrow-right key''': selects the next state for the selected process.
-*'''arrow-left key''': selects the previous state for the selected process.
-*'''Shift + arrow-right key''': updates the selection end time of the current selection range by selecting the next state of the current process.
-*'''Shift + arrow-left key''': updates the selection end time of the current selection range by selecting the previous state of the current process.
+The following keyboard shortcuts are available:
+
+*'''arrow-right key''': selects the next state for the selected process
+*'''arrow-left key''': selects the previous state for the selected process
+*'''Shift + arrow-right key''': updates the selection end time of the current selection range by selecting the next state of the current process
+*'''Shift + arrow-left key''': updates the selection end time of the current selection range by selecting the previous state of the current process
+*'''.''': selects the next active marker
+*''',''': selects the previous active marker
+*'''Shift + .''': updates the selection end time of the current selection range by selecting the next active marker boundary
+*'''Shift + ,''': updates the selection end time of the current selection range by selecting the previous active marker boundary
*'''arrow-down''': selects the next process
*'''arrow-up''': selects the previous process
*'''Page Down''': selects the process down one page
*'''Home''': selects the first process
*'''End''': selects the last process
*'''Enter''': toggles the expansion state of the current process in the tree
+*'''+''': Zoom-in horizontally
+*'''-''': Zoom-out horizontally
+*'''Ctrl + +''': Zoom-in vertically
+*'''Ctrl + -''': Zoom-out vertically
+*'''Ctrl + 0''': Reset the vertical zoom
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+When the mouse cursor is over entries (left pane):
+*'''-''': Collapse selected entry
+*'''+''': Expand selected entry
+*'''*''': Expand selected entry to the level with at least one collapsed entry
+
+'''Please note that the behavior of some shortcuts can slightly differ based on the operating system.'''
When the selection indicators are changed, all the other views are '''synchronized'''. For example, the [[#LTTng Kernel Events Editor|Events Editor]] will show the event matching the current time indicator. The reverse behaviour is also implemented: selecting an event within the Events View will update the Control Flow View current time indicator.
=== Toolbar ===
+<!-- Keep in sync with ref:resource-view-toolbar -->
+
The Control Flow View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:
{|
| [[Image:images/link.gif]]
| Align Views
-| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference
+| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in this view will disable this feature across all the views because it's a workspace preference.
|-
| [[Image:images/filter_items.gif]]
| Show View Filter
-| Opens the process filter dialog
+| Opens the process filter dialog. Filter settings will be preserved when switching between open traces.
|-
| [[Image:images/show_legend.gif]]
| Show Legend
-| Displays the states legend
+| Displays the states legend.
|-
| [[Image:images/home_nav.gif]]
| Reset the Time Scale to Default
-| Resets the zoom window to the full range
+| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
| Select Previous Event
| Select Next Event
| Selects the next state for the selected process. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
+| [[Image:images/add_bookmark.gif]]
+| Add Bookmark...
+| Adds a bookmark at the current selection range. A bookmark is a user-defined interval marker. The '''Add Bookmark''' dialog is opened where the user can enter a description and choose the highlighting color and alpha (transparency) value. This button is replaced by the '''Remove Bookmark''' button if the current selection range corresponds to an existing bookmark. The bookmarks can also be managed in the '''Bookmark View'''.
+|-
+| [[Image:images/remove_bookmark.gif]]
+| Remove Bookmark
+| Removes the bookmark at the current selection range. This button replaces the '''Add Bookmark''' when the current selection range corresponds to an existing bookmark.
+|-
+| [[Image:images/prev_bookmark.gif]]
+| Previous Marker
+| Selects the previous active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
+|-
+| [[Image:images/next_bookmark.gif]]
+| Next Marker
+| Selects the next active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range. Clicking the button drop-down arrow will open a menu where marker categories can be made active or inactive for navigation. To toggle the display of a marker category, use the '''View Menu''' instead.
+|-
| [[Image:images/prev_menu.gif]]
| Select Previous Process
-| Selects the previous process
+| Selects the previous process.
|-
| [[Image:images/next_menu.gif]]
| Select Next Process
-| Selects the next process
+| Selects the next process.
|-
| [[Image:images/zoomin_nav.gif]]
| Zoom In
-| Zooms in on the selection by 50%
+| Zooms in on the selection by 50%.
|-
| [[Image:images/zoomout_nav.gif]]
| Zoom Out
-| Zooms out on the selection by 50%
+| Zooms out on the selection by 50%.
|-
| [[Image:images/hide_arrows.gif]]
| Hide Arrows
-| Toggles the display of arrows on or off
+| Toggles the display of arrows on or off.
|-
| [[Image:images/follow_arrow_bwd.gif]]
| Follow CPU Backward
| Selects the next state following CPU execution across processes. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|}
+View Menu
+
+{|
+|
+| Show Markers
+| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
+|}
+
+=== Marker Axis ===
+
+The marker axis is visible only when at least one marker category with markers for the current trace is shown.
+
+The marker axis displays one row per marker category. Each marker's time range and/or label (if applicable) are drawn on the marker axis.
+
+Clicking on any marker's time range or label will set the current time selection to the marker's time or time range.
+
+Clicking on the "X" icon to the left of the marker category name will hide this marker category from the time graph. It can be shown again using the corresponding '''Show Markers''' menu item in the view menu.
+
+The marker axis can be collapsed and expanded by clicking on the icon at the top left of the marker axis. The marker axis can be completely removed by hiding all available marker categories.
+
== Resources View ==
This view is specific to LTTng kernel traces. The Linux Kernel Analysis is executed the first time a LTTng Kernel is opened. After opening the trace, the element '''Resources''' is added under the '''Linux Kernel Analysis''' tree element of the Project Explorer. To open the view, double-click the '''Resources''' tree element.
This view shows the state of system resources i.e. if changes occurred during the trace either on '''CPUs''', '''IRQs''' or '''soft IRQs''', it will appear in this view. The left side of the view present a list of resources that are affected by at least one event of the trace. The right side illustrate the state in which each resource is at some point in time. For state '''USERMODE''' it also prints the process name in the state bar. For state '''SYSCALL''' the name of the system call is
displayed in the state region.
+When an '''IRQ''' is handled by a '''CPU''', its states are shown under the corresponding '''CPU''' entry. Similarly, the '''CPU''' that was handling an '''IRQ''' is shown under the handled '''IRQ'''. Therefore, the trace can be visualized from a '''CPU''' point of view or from an '''IRQ''' point of view.
+
Just like other views, according to which trace points and system calls are activated, the content of this view may change from one trace to another.
The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).
=== Toolbar ===
+<!-- ref:resource-view-toolbar -->
+
The Resources View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:
{|
| [[Image:images/link.gif]]
| Align Views
-| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference
+| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in this view will disable this feature across all the views because it's a workspace preference.
+|-
+| [[Image:images/filter_items.gif]]
+| Show View Filter
+| Opens the resources filter dialog. Filter settings will be preserved when switching between open traces.
|-
| [[Image:images/show_legend.gif]]
| Show Legend
-| Displays the states legend
+| Displays the states legend.
|-
| [[Image:images/home_nav.gif]]
| Reset the Time Scale to Default
-| Resets the zoom window to the full range
+| Resets the zoom window to the full range.
|-
| [[Image:images/prev_event.gif]]
| Select Previous Event
| Select Next Event
| Selects the next state for the selected resource. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
|-
+| [[Image:images/add_bookmark.gif]]
+| Add Bookmark...
+| Adds a bookmark at the current selection range. A bookmark is a user-defined interval marker. The '''Add Bookmark''' dialog is opened where the user can enter a description and choose the highlighting color and alpha (transparency) value. This button is replaced by the '''Remove Bookmark''' button if the current selection range corresponds to an existing bookmark. The bookmarks can also be managed in the '''Bookmark View'''.
+|-
+| [[Image:images/remove_bookmark.gif]]
+| Remove Bookmark
+| Removes the bookmark at the current selection range. This button replaces the '''Add Bookmark''' when the current selection range corresponds to an existing bookmark.
+|-
+| [[Image:images/prev_bookmark.gif]]
+| Previous Marker
+| Selects the previous active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range.
+|-
+| [[Image:images/next_bookmark.gif]]
+| Next Marker
+| Selects the next active marker. Pressing the '''Shift''' key at the same time will update the selection end time of the current selection range. Clicking the button drop-down arrow will open a menu where marker categories can be made active or inactive for navigation.
+|-
| [[Image:images/prev_menu.gif]]
| Select Previous Resource
| Selects the previous resource
|-
| [[Image:images/zoomin_nav.gif]]
| Zoom In
-| Zooms in on the selection by 50%
+| Zooms in on the selection by 50%.
|-
| [[Image:images/zoomout_nav.gif]]
| Zoom Out
-| Zooms out on the selection by 50%
+| Zooms out on the selection by 50%.
|}
+View Menu
+
+{|
+|
+| Show Markers
+| A marker highlights a time interval. A marker can be used for instance to indicate a time range where lost events occurred or to bookmark an interesting interval for future reference. Selecting a category name will toggle the visibility of markers of that category.
+|}
+
+=== Marker Axis ===
+
+See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.
+
== LTTng CPU Usage View ==
The CPU Usage analysis and view is specific to LTTng Kernel traces. The CPU usage is derived from a kernel trace as long as the '''sched_switch''' event was enabled during the collection of the trace. This analysis is executed the first time that the CPU Usage view is opened after opening the trace. To open the view, double-click on the '''CPU Usage''' tree element under the '''Linux Kernel Analysis''' tree element of the Project Explorer.
=== Process Information ===
+
The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID), process name (Process), the average (%) execution time and the actual execution time (Time) during the current time range. It shows all threads that were executing on the CPUs in the current time range.
[[Image:images/LTTng_CpuUsageViewToolTip.png]]
+
+== Kernel Memory Usage ==
+
+The Kernel Memory Usage and view is specific to kernel traces. To open the view, double-click on the '''Kernel Memory Usage Analysis''' tree element under the '''Kernel''' tree element of the Project Explorer.
+
+[[Image:images/kernelMemoryUsage/OpenKernelMemoryUsageView.png]]
+
+Now, the Kernel memory usage view will show:
+
+[[Image:images/kernelMemoryUsage/KernelMemoryUsageView.png]]
+
+Where:
+
+* '''TID''': The ID of the thread this event belongs to
+* '''Process''': The process of the TID that belongs to it
+
+The view is divided into the following important sections: '''Process Information''' and the '''Relative Kernel Memory Usage'''. The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).
+
+
+=== Process Information ===
+
+The Process Information is displayed on the left side of the view and shows all threads that were executing on all available CPUs in the current time range. For each process, it shows in different columns the thread ID (TID) and the process name (Process).
+
+
+=== Relative Kernel Memory Chart ===
+
+The Relative Kernel Memory Chart on the right side of the view, plots the relative amount of memory that was allocated and deallocated during that period of time.
+
+
+==== Using the mouse ====
+
+The Relative Kernel Memory chart is usable with the mouse. The following actions are set:
+
+* '''left-click''': select a time or time range begin time
+* '''left-drag horizontally''': select a time range or change the time range begin or end time
+* '''middle-drag''': pan left or right
+* '''right-drag horizontally''': zoom region
+* '''mouse wheel up/down''': zoom in or out
+
+
+==== Tooltips ====
+
+Hover the cursor over a line of the chart and a tooltip will pop up with the following information:
+* '''time''': current time of mouse position
+* '''Total''': The total CPU usage
+
+[[Image:images/kernelMemoryUsage/KernelMemoryUsageChart.png]]
+
+== Process Wait Analysis ==
+
+TraceCompass can recover wait causes of local and distributed processes using operating system events. The analysis highlights the tasks and devices causing wait. Wait cause recovery is recursive, comprise all tasks running on the system and works across computers using packet trace synchronization.
+
+The analysis details are available in the paper [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7294678&isnumber=4359390 Wait analysis of distributed systems using kernel tracing].
+
+=== Prerequisites ===
+
+The analysis requires a Linux kernel trace. Additional instrumentation may be required for specific kernel version and for distributed tracing. This instrumentation is available in [https://github.com/giraldeau/lttng-modules/tree/addons LTTng modules addons] on GitHub.
+
+The required events are:
+* '''sched_switch, sched_wakeup''': Scheduling events indicate when a process is blocked and the wake-up event indicates the task or resource that unblocked the task. For kernel versions comprised between 3.8 and 4.1, the event '''sched_ttwu''' (which stands for Try To Wake-Up) is provided for backward compatibility in LTTng modules addons.
+* '''IRQ, SoftIRQ and IPI''': Interrupt events are required to distinguish the context of the wake-up. When a wake-up occurs inside an interrupt handler, it must be associated with the device causing the interrupt and not the interrupted task. For that reason, interrupt entry and exit events are required.
+* '''inet_sock_local_in, inet_sock_local_out''': The network events record a subset of TCP/IP packet header using a netfilter hook in the kernel. The send and receive events are matched to show the communication between distributed processes. Network events are mandatory for analyzing wait in TCP/IP programs, whether they are executing locally or on different computers. They also used to synchronize traces recorded on multiple computers. For further details, refer to the [[#Trace synchronization]] section.
+
+To analyze a distributed program, all computers involved in the processing must be traced simultaneously. The LTTng Tracer Control of TraceCompass can trace a remote computer, but controlling simultaneous tracing is not supported at the moment, meaning that all sessions must be started separately and interactively. TraceCompass will support this feature in the future. For now, it is suggested to use [https://github.com/giraldeau/lttng-cluster lttng-cluster] command line tool to control simultaneous tracing sessions on multiple computers. This tool is based on [http://www.fabfile.org/ Fabric] and uses SSH to start the tracing sessions, execute a workload, stop the sessions and gather traces on the local computer. For more information, refer to the lttng-cluster documentation.
+
+We use the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] as an example to demonstrate the wait analysis. [https://www.djangoproject.com/ Django] is a popular Web framework. The application is the [https://docs.djangoproject.com/en/1.9/intro/tutorial01/ Django Poll app tutorial]. The traces were recorded on three computers, namely the client (implemented with Python Mechanize), the Web server (Apache with WSGI) and the database server (PostgreSQL). The client simulates a vote in the poll.
+
+=== Running the analysis ===
+
+To open all three traces simultaneously, we first create an experiment containing these traces and then synchronize the traces, such that they have a common time base. Then, the analysis is done by selecting a task in the '''Control Flow View'''. The result is displayed in the '''Critical Flow View''', which works like the '''Control Flow View'''. The steps to load the Django example follows.
+
+# Download and extract the [https://github.com/giraldeau/traces/blob/master/django-vote.tar.gz Django trace] archive.
+# In TraceCompass, open the [[#LTTng Kernel Perspective]].
+# Create a new tracing project. Select '''File -> New -> Tracing -> Tracing Project''', choose a name and click '''Finish'''.
+# Under the created tracing project, right-click on '''Traces''' and select '''Import...'''. In the import dialog, select the root directory containing the extracted trace by clicking on '''Browse'''. Three traces should be listed. Select the traces and click '''Finish'''. After the import is completed, the traces should be listed below '''Traces'''.
+# Right-click on '''Experiments''', select '''New...''' and enter a name for the experiment, such as '''django'''.
+# Right-click on the '''django''' experiment and click on '''Select Traces...'''. In the dialog, check the three traces '''django-client''', '''django-httpd''' and '''django-db'''. These traces will appear below the experiment. If the experiment is opened at this stage, the traces are not synchronized and there will be a large time gap between events from different traces.
+# To synchronize the traces, right-click on the '''django''' experiment and select '''Synchronize Traces'''. In the '''Select reference trace''' dialog, select any available trace and click '''Finish'''. Once the synchronization is completed, a new entry with an underline suffix will appear for each modified trace. The created trace entries have a function which is applied to the timestamps of events in order to shift the time according to the reference trace. The '''Project Explorer''' after the import is shown below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisProjectExplorer.png]]
+# Open the experiment '''django'''. The '''Control Flow''' and the '''Resources''' views should display the three traces simultaneously.
+# In the main menu, select '''Window -> Show View -> Other...''' and under '''LTTng''' select '''Critical Flow View'''. The view is empty for the moment.
+# In the '''Critical Flow View''', right-click on the '''Process''' entry to analyze and select '''Follow''', as shown in the figure below.
+#:[[Image:images/waitAnalysis/KernelWaitAnalysisFollow.png]]
+#:The analysis will execute and the result will appear in the '''Critical Flow View'''. For the Django example, use the '''View Filters''' to search for the python process with TID 2327. When zooming on the execution, the view displays the work done by the Web server and the database to process the request of the python client. Vertical arrows represent synchronization and communication between processes. The legend [[Image:images/show_legend.gif]] displays the colors associated with the processes states.
+
+[[Image:images/waitAnalysis/KernelWaitAnalysisDjango.png]]
+
+
== LTTng Kernel Events Editor ==
The LTTng Kernel Events editor '''is''' the plain TMF [[#Events_Editor | Events Editor]], except that it provides its own specialized viewer to replace the standard one. In short, it has exactly the same behaviour but the layout is slightly different:
=== Using the Callstack View with LTTng-UST traces ===
-There is support in the LTTng-UST integration plugin to display the callstack of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see the ''liblttng-ust-cyg-profile'' man page for additional information). To do so, you need to:
+There is support in the LTTng-UST integration plugin to display the callstack
+of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see
+the ''liblttng-ust-cyg-profile'' man page for additional information). To do
+so, you need to:
* Recompile your application with "''-g -finstrument-functions''".
-* Add the ''vtid'' and ''procname'' contexts to your trace session. See the [[#Adding Contexts to Channels and Events of a Domain]] section. Or if using the command-line:
-** <pre>lttng add-context -u -t vtid -t procname</pre>
+* Set up a tracing session with the the ''vpid'', ''vtid'' and ''procname'' contexts. See the [[#Enabling UST Events On Session Level]] and [[#Adding Contexts to Channels and Events of a Domain]] sections. Or if using the command-line:
+** <pre>lttng enable-event -u -a</pre>
+** <pre>lttng add-context -u -t vpid -t vtid -t procname</pre>
* Preload the ''liblttng-ust-cyg-profile'' library when running your program:
** <pre>LD_PRELOAD=/usr/lib/liblttng-ust-cyg-profile.so ./myprogram</pre>
-Once you load the resulting trace, making sure it's set to the ''Common Trace Format - LTTng UST Trace'' type, the Callstack View should be populated with the relevant information. However, since GCC's cyg-profile instrumentation only provides function addresses, and not names, an additional step is required to get the function names showing in the view. The following section explains how to do so.
+Once you load the resulting trace, the Callstack View should be populated with
+the relevant information.
-=== Importing a function name mapping file for LTTng-UST traces ===
+Note that for non-trivial applications, ''liblttng-ust-cyg-profile'' generates a
+'''lot''' of events! You may need to increase the channel's subbuffer size to
+avoid lost events. Refer to the
+[http://lttng.org/docs/#doc-fine-tuning-channels LTTng documentation].
-If you followed the steps in the previous section, you should have a Callstack View populated with function entries and exits. However, the view will display the function addresses instead of names in the intervals, which are not very useful by themselves. To get the actual function names, you need to:
+For traces taken with LTTng-UST 2.8 or later, the Callstack View should show the
+function names automatically, since it will make use of the debug information
+statedump events (which are enabled when using ''enable-event -u -a'').
+For traces taken with prior versions of UST, you would need to set the path to
+the binary file or mapping manually:
+
+=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===
+
+If you followed the steps in the previous section, you should have a Callstack
+View populated with function entries and exits. However, the view will display
+the function addresses instead of names in the intervals, which are not very
+useful by themselves. To get the actual function names, you need to:
+
+* Click the '''Import Mapping File''' ([[Image:images/import.gif]]) button in the Callstack View.
+
+Then either:
+* Point to the binary that was used for taking the trace
+OR
* Generate a mapping file from the binary, using:
** <pre>nm myprogram > mapping.txt</pre>
-* Click the '''Import Mapping File''' ([[Image:images/import.gif]]) button in the Callstack View, and select the ''mapping.txt'' file that was just created.
+** Select the ''mapping.txt'' file that was just created.
-(If you are dealing with C++ executables, you may want to use ''nm --demangle'' instead to get readable function names.)
+(If you are dealing with C++ executables, you may want to use ''nm --demangle''
+instead to get readable function names.)
-The view should now update to display the function names instead. Make sure the binary used for taking the trace is the one used for this step too (otherwise, there is a good chance of the addresses not being the same).
+The view should now update to display the function names instead. Make sure the
+binary used for taking the trace is the one used for this step too (otherwise,
+there is a good chance of the addresses not being the same).
+
+=== Navigation ===
+
+See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.
+
+=== Marker Axis ===
+
+See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.
== Memory Usage ==
* '''right-drag horizontally''': zoom region
* '''mouse wheel up/down''': zoom in or out
-
=== Toolbar ===
The Memory Usage View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:
Please note this view will not show shared memory or stack memory usage.
+== Source Lookup (for LTTng-UST 2.8+) ==
+
+Starting with LTTng 2.8, the tracer can now provide enough information to
+associate trace events with their location in the original source code.
+
+To make use of this feature, first make sure your binaries are compiled with
+debug information (-g), so that the instruction pointers can be mapped to source
+code locations. This lookup is made using the ''addr2line'' command-line utility,
+which needs to be installed and on the '''$PATH''' of the system running Trace
+Compass. ''addr2line'' is available in most Linux distributions, Mac OS X, Windows using Cygwin and others.
+
+The following trace events need to be present in the trace:
+
+* lttng_ust_statedump:start
+* lttng_ust_statedump:end
+* lttng_ust_statedump:bin_info
+* lttng_ust_statedump:build_id
+
+as well as the following contexts:
+
+* vpid
+* ip
+
+For ease of use, you can simply enable all the UST events when setting up your
+session:
+
+ lttng enable-event -u -a
+ lttng add-context -u -t vpid -t ip
+
+Note that you can also create and configure your session using the [[#Control View | Control View]].
+
+If you want to track source locations in shared libraries loaded by the
+application, you also need to enable the "lttng_ust_dl:*" events, as well
+as preload the UST library providing them when running your program:
+
+ LD_PRELOAD=/path/to/liblttng-ust-dl.so ./myprogram
+
+If all the required information is present, then the ''Source Location'' column
+of the Event Table should be populated accordingly, and the ''Open Source Code''
+action should be available. Refer to the section [[#Event Source Lookup]] for
+more details.
+
+The ''Binary Location'' information should be present even if the original
+binaries are not available, since it only makes use of information found in the
+trace. A '''+''' denotes a relative address (i.e. an offset within the object
+itself), whereas a '''@''' denotes an absolute address, for
+non-position-independent objects.
+
+[[Image:images/sourceLookup/trace-with-debug-info.png]]
+
+''Example of a trace with debug info and corresponding Source Lookup information, showing a tracepoint originating from a shared library''
+
+=== Binary file location configuration ===
+
+To resolve addresses to function names and source code locations, the analysis
+makes use of the binary files (executables or shared libraries) present on the
+system. By default, it will look for the file paths as they are found in the
+trace, which means that it should work out-of-the-box if the trace was taken on
+the same machine that Trace Compass is running.
+
+It is possible to configure a ''root directory'' that will be used as a prefix
+for all file path resolutions. The button to open the configuration dialog is
+called '''Configure how addresses are mapped to function names''' and is
+currently located in the [[#Call Stack View]]. Note that the Call Stack View
+will also make use of this configuration to resolve its function names.
+
+[[Image:images/sourceLookup/symbol-mapping-config-ust28.png]]
+
+''The symbol configuration dialog for LTTng-UST 2.8+ traces''
+
+This can be useful if a trace was taken on a remote target, and an image of that
+target is available locally.
+
+If a binary file is being traced on a target, the paths in the trace will refer
+to the paths on the target. For example, if they are:
+
+* /usr/bin/program
+* /usr/lib/libsomething.so
+* /usr/local/lib/libcustom.so
+
+and an image of that target is copied locally on the system at
+''/home/user/project/image'', which means the binaries above end up at:
+
+* /home/user/project/image/usr/bin/program
+* /home/user/project/image/usr/lib/libsomething.so
+* /home/user/project/image/usr/local/lib/libcustom.so
+
+Then selecting the ''/home/user/project/image'' directory in the configuration
+dialog above will allow Trace Compass to read the debug symbols correctly.
+
+Note that this path prefix will apply to both binary file and source file
+locations, which may or may not be desirable.
+
= Trace synchronization =
It is possible to synchronize traces from different machines so that they have the same time reference. Events from the reference trace will have the same timestamps as usual, but the events from traces synchronized with the first one will have their timestamps transformed according to the formula obtained after synchronization.
It is possible to define custom trace analyses and a way to view them in an XML format. These kind of analyses allow doing more with the trace data than what the default analyses shipped with TMF offer. It can be customized to a specific problem, and fine-tuned to show exactly what you're looking for.
-== Importing an XML file containing analysis ==
+== Managing XML files containing analyses ==
+
+The '''Manage XML Analyses''' dialog is used to manage the list of XML files containing analysis. To open the dialog:
+
+* Open the '''Project Explorer''' view.
+* Select '''Manage XML Analyses...''' from the '''Traces''' folder context menu.
+
+[[Image:images/ManageXMLAnalysis.png]]
+
+The list of currently defined XML analyses is displayed on the left side of the dialog.
-If you already have an XML file defining state providers and/or views, you can import it in your TMF workspace by right-clicking on the ''Traces'' or ''Experiments'' folder and selecting ''Import XML Analysis''.
+The following actions can be performed from this dialog:
+
+* Import
+
+Click the '''Import''' button and select a file from the opened file dialog to import an XML file containing an analysis. The file will be validated before importing it and if successful, the new analysis and views will be shown under the traces for which they apply. You will need to close any already opened traces and re-open them before the new analysis can be executed. If an invalid file is selected, an error message will be displayed to the user.
-[[Image:images/import_XML_analysis.png| Import XML analysis menu]]
+* Export
-You will be prompted to select the file. It will be validated before importing it and if successful, the new analysis and views will be shown under the traces for which they apply. You will need to close any already opened traces and re-open them before the new analysis can be executed.
+Select an XML file from the list, click the '''Export''' button and enter or select a file in the opened file dialog to export the XML analysis. Note that if an existing file containing an analysis is selected, its content will be replaced with the analysis to export.
+
+* Delete
-Right now, there is no way to "unimport" analyses from within the application. A UI to manage the imported analyses is currently being worked on. In the meantime, you can navigate to your workspace directory, and delete the files in .metadata/.plugins/org.eclipse.tracecompass.tmf.analysis.xml.core/xml_files .
+Select an XML file from the list and click the '''Delete''' button to remove the XML file. Deleting an XML file will close all the traces for which this analysis applies and remove the analysis.
== Defining XML components ==
</head>
</pre>
-If pre-defined values will be used in the state provider, they must be defined before the state providers. They can then be referred to in the state changes by name, preceded by the '$' sign. It is not necessary to use pre-defined values, the state change can use values like (100, 101, 102) directly.
+If predefined values will be used in the state provider, they must be defined before the state providers. They can then be referred to in the state changes by name, preceded by the '$' sign. It is not necessary to use predefined values, the state change can use values like (100, 101, 102) directly.
<pre>
<definedValue name="RUNNING" value="100" />
If modifications are made to the XML state provider after it has been "published", the '''version''' attribute of the '''xmlStateProvider''' element should be updated. This avoids having to delete each trace's supplementary file manually. If the saved state system used a previous version, it will automatically be rebuilt from the XML file.
+== Defining an XML pattern provider ==
+It exists patterns within an execution trace that can provide high level details about the system execution. A '''pattern''' is a particular combination of events or states that are expected to occur within a trace. It may be composed of several state machines that inherit or communicate through a common state system.
+
+We may have multiple instances (scenarios) of a running state machine within a pattern. Each scenario which has its own path in the state system can generate segments to populate the data-driven views
+
+=== The state system structure ===
+
+The pattern analysis generates a predefined attribute tree described as follows :
+
+<pre>
+|- state machines
+| |- state machine 0
+| |- scenario 0
+| |- status
+| |- state
+| |- start
+| ...
+| |- storedFields
+| |- field 1
+| ...
+| |- startTime
+| ...
+| ...
+| |- scenarios 1
+| ...
+| |- state machine 1
+| ...
+</pre>
+
+The user can add custom data in this tree or determine its own attribute tree beside of this one.
+
+=== Writing the XML pattern provider ===
+Details about the XML structure are available in the XSD files.
+
+First define the pattern element. As the state provider element described in [[#Writing_the_XML_state_provider | Writing the XML state provider]], it has a "version" attribute and an "id" attribute.
+
+<pre>
+<pattern version="0" id="my.test.pattern">
+</pre>
+
+Optional header information as well as predefined values like described in [[#Writing_the_XML_state_provider | Writing the XML state provider]] can be added.
+
+Stored values can be added before the pattern handler. The predefined action '''saveStoredField''' triggers the updates of the stored fields and the predefined action '''clearStoredFields''' reset the values.
+
+<pre>
+<storedField id="offset" alias="offset"/>
+</pre>
+
+The behavior of the pattern and the models it needs are described in the pattern handler element.
+
+The structure of the state machine (FSM) is based on the SCXML structure. The following example describe an FSM that matches all the system call in an LTTng kernel trace.
+
+<pre>
+<fsm id="syscall" initial="start">
+ <state id="start">
+ <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
+ </state>
+ <state id="in_progress" >
+ <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
+ </state>
+ <final id="end"/>
+</fsm>
+</pre>
+
+The value of the target attribute corresponds to the 'id' of a test element described in the XML file and is a reference to it. Similarly, the value of the action attribute corresponds to the 'id' of an action element described in the XML file and is a reference to it.
+
+Conditions are used in the transitions to switch between the state of an FSM. They are defined under the '''test''' element. Two types of conditions are allowed : '''Data condition''' and '''Time condition'''. It is possible to combine several conditions using a logical operator (OR, AND, ...).
+
+Data conditions tests the ongoing event information against the data in the state system or constant values. The following condition tests whether the current thread running on the CPU is also the ongoing scenario thread.
+
+<pre>
+<test id="thread_condition">
+ <if>
+ <condition>
+ <stateValue type="query" >
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ </stateValue>
+ </condition>
+ </if>
+</test>
+</pre>
+
+Two types of time conditions are available:
+* Time range conditions tests whether the ongoing event happens between a specific range of time. The following condition tests whether the ongoing event happens between 1 nanosecond and 3 nanoseconds.
+
+<pre>
+<test id="time_condition">
+ <if>
+ <condition>
+ <timerange unit="ns">
+ <in begin="1" end="3" />
+ </timerange>
+ </condition>
+ </if>
+</test>
+</pre>
+
+* Elapsed time conditions tests the value of the time spent since a specific state of an fsm. The following condition tests whether the ongoing event happens less than 3 nanoseconds after that the scenario reaches the state "syscall_entry_x".
+
+<pre>
+<test id="time_condition">
+ <if>
+ <condition>
+ <elapsedTime unit="ns">
+ <less since="syscall_entry_x" value="3" />
+ </elapsedTime>
+ </condition>
+ </if>
+</test>
+</pre>
+
+Two types of actions are allowed :
+* State changes update values of attributes into the state system. The following example set the value of the thread for the current scenario.
+
+<pre>
+ <action id="sys_x_found">
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ <stateValue type="query">
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ </stateChange>
+ </action>
+</pre>
+
+* Generate segments. The following example represents a system call segment.
+
+<pre>
+<action id="exit_syscall_founded">
+ <segment>
+ <segType>
+ <segName>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ </stateValue>
+ </segName>
+ </segType>
+ </segment>
+</action>
+</pre>
+
+When existing, the stored fields will be added as fields for the generated segments.
+
+Here is the complete XML file by combining all the examples models above:
+
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<tmfxml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:noNamespaceSchemaLocation="xmlDefinition.xsd">
+
+<pattern version="1" id="my.test.pattern">
+ <head>
+ <traceType id="org.eclipse.linuxtools.lttng2.kernel.tracetype" />
+ <label value="xml syscall" />
+ </head>
+
+ <storedField id="filename"/>
+ <storedField id="fd"/>
+ <storedField id="ret" alias="ret"/>
+ <storedField id="flags" alias="flags"/>
+ <storedField id="offset" alias="offset"/>
+ <storedField id="fd_in" alias="fd_in"/>
+ <storedField id="fd_out" alias="fd_out"/>
+ <storedField id="uservaddr" alias="uservaddr"/>
+ <storedField id="upeer_sockaddr" alias="upeer_sockaddr"/>
+
+ <location id="CurrentThread">
+ <stateAttribute type="constant" value="Threads" />
+ <stateAttribute type="query">
+ <stateAttribute type="constant" value="CPUs" />
+ <stateAttribute type="eventField" value="cpu" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateAttribute>
+ </location>
+
+ <location id="CurrentCPU">
+ <stateAttribute type="constant" value="CPUs" />
+ <stateAttribute type="eventField" value="cpu" />
+ </location>
+
+ <patternHandler>
+ <test id="time_condition">
+ <if>
+ <or>
+ <not>
+ <condition>
+ <timerange unit="ns">
+ <in begin="1" end="3" />
+ </timerange>
+ </condition>
+ </not>
+ <condition>
+ <elapsedTime unit="ns">
+ <less since="syscall_entry_x" value="3" />
+ </elapsedTime>
+ </condition>
+ </or>
+ </if>
+ </test>
+
+ <test id="thread_condition">
+ <if>
+ <condition>
+ <stateValue type="query" >
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ </stateValue>
+ </condition>
+ </if>
+ </test>
+
+ <action id="sys_x_founded">
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ <stateValue type="eventName"/>
+ </stateChange>
+
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="cpu" />
+ <stateValue type="eventField" value="cpu"/>
+ </stateChange>
+
+ <stateChange>
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="thread" />
+ <stateValue type="query">
+ <stateAttribute type="location" value="CurrentCPU" />
+ <stateAttribute type="constant" value="Current_thread" />
+ </stateValue>
+ </stateChange>
+ </action>
+
+ <action id="exit_syscall_founded">
+ <segment>
+ <segType>
+ <segName>
+ <stateValue type="query">
+ <stateAttribute type="constant" value="#CurrentScenario" />
+ <stateAttribute type="constant" value="syscall" />
+ <stateAttribute type="constant" value="name" />
+ </stateValue>
+ </segName>
+ </segType>
+ </segment>
+ </action>
+
+ <fsm id="syscall" initial="start">
+ <state id="start">
+ <transition event="syscall_entry_*" target="syscall_entry_x" action="sys_x_founded" saveStoredFields="true"/>
+ </state>
+ <state id="in_progress" >
+ <transition event="syscall_exit_*" cond="thread_condition" target="syscall_exit_x" action="exit_syscall_found" saveStoredFields="true" clearStoredFields="true"/>
+ </state>
+ <final id="end"/>
+ </fsm>
+ </patternHandler>
+</pattern>
+</tmfxml>
+</pre>
+
+=== Representing the scenarios ===
+
+Segments generated by the pattern analysis are used to populate latency views. A description of these views can be found in [[#Latency_Analyses | Latency Analyses]].
+
+The full XML analysis example described above will generate the following views :
+
+* Latency Table
+
+[[Image:images/XMLPatternAnalysis/LatencyTable.png| Latency Table example - System Call pattern]]
+
+* Latency vs Time
+
+[[Image:images/XMLPatternAnalysis/LatencyVSTime.png| Latency vs Time example - System Call pattern]]
+
+* Latency Statistics
+
+[[Image:images/XMLPatternAnalysis/LatencyStatistics.png| Latency Statistics example - System Call pattern]]
+
+* Latency vs Count
+
+[[Image:images/XMLPatternAnalysis/LatencyVSCount.png| Latency vs Count example - System Call pattern]]
+
== Defining an XML time graph view ==
A time graph view is a view divided in two, with a tree viewer on the left showing information on the different entries to display and a Gantt-like viewer on the right, showing the state of the entries over time. The [[#Control_Flow_View | Control Flow View]] is an example of a time graph view.
-Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any pre-defined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
+Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
In the example above, suppose we want to display the status for each task. In the state system, it means the path of the entries to display is "Tasks/*". The attribute whose value should be shown in the Gantt chart is the entry attribute itself. So the XML to display these entries would be as such:
[[Image:images/Xml_analysis_screenshot.png| XML analysis with view]]
+==== Using the keyboard ====
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+
== Defining an XML XY chart ==
An XY chart displays series as a set of numerical values over time. The X-axis represents the time and is synchronized with the trace's current time range. The Y-axis can be any numerical value.
-Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any pre-defined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
+Such views can be defined in XML using the data in the state system. The state system itself could have been built by an XML-defined state provider or by any predefined Java analysis. It only requires knowing the structure of the state system, which can be explored using the [[#State System Explorer View | State System Explorer View]] (or programmatically using the methods in ''ITmfStateSystem'').
We will use the Linux Kernel Analysis on LTTng kernel traces to show an example XY chart. In this state system, the status of each CPU is a numerical value. We will display this value as the Y axis of the series. There will be one series per CPU. The XML to display these entries would be as such:
<pre>
<head>
- <analysis id="org.eclipse.tracecompass.lttng2.kernel.analysis" />
+ <analysis id="org.eclipse.tracecompass.analysis.os.linux.kernel" />
<label value="CPU status XY view" />
</head>
</pre>
[[Image:images/XML_xy_chart.png| XML XY chart]]
+= Latency Analyses =
+
+Trace Compass offers a feature called Latency analysis. This allows an analysis to return intervals and these intervals will be displayed in four different views. An example analysis is provided with kernel system call latencies being provided. The available views are:
+
+* System Call Latencies
+A '''table''' of the raw latencies. This view is useful to inspect individual latencies.
+
+ [[Image:images/LatenciesTable.png| Latency Table example - System Call Latencies]]
+
+
+* System Call Latency vs Time
+A time aligned '''scatter chart''' of the latencies with respect to the current window range. This view is useful to see the overall form of the latencies as they arrive.
+
+[[Image:images/LatenciesScatter.png| Latency Scatter Chart example - System Call Latency vs Time]]
+
+
+* System Call Latency Statistics
+A view of the total '''statistics''' of the latencies. These show the ''minimum'', ''maximum'', ''average'' and ''standard deviation'' of the latencies when applicable. This tool is useful for finding the outliers on a per-category basis.
+
+[[Image:images/LatenciesStatistics.png| Latency Statistics example - System Call Latency Statistics]]
+
+
+* System Call Density
+A '''density''' view, analyzing the current time range. This is useful to find global outliers.
+
+[[Image:images/LatenciesDensity.png| Latency Densities example - System Call Density]]
+
+= Virtual Machine Analysis =
+
+Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.
+
+To better understand what is happening in such an environment, it is necessary to trace all the machines involved, guests and hosts, and correlate this information in an experiment that will display a complete view of the virtualized environment.
+
+== Virtual Machine Experiment ==
+
+A trace has to be taken for each machine, guest and host, in the virtualized environment. The host trace is the most important to have, as missing guests will only give an incomplete view of the system, but missing hosts usually won't allow to identify the hypervisor, nor determine when a guest is preempted from the host CPUs. The virtual machine analysis only makes sense if the host trace is available.
+
+Once all the traces are imported in Trace Compass, they can be [[#Creating a Experiment | added to an experiment]]. The type of the experiment should by set to '''Virtual Machine Experiment''' by clicking on the right mouse button over the experiment name, then selecting '''Select Experiment Type...'''.
+
+[[Image:images/vmAnalysis/VM_experiment.png | Virtual Machine Experiment]]
+
+Depending on the hypervisor used, traces might need to be [[#Trace synchronization | synchronized]] so that they have the same time reference and their events can be correctly correlated.
+
+== Virtual CPU View ==
+
+The Virtual CPU view shows the status of CPUs and threads on guests augmented with the preemption and hypervisor data we get from the host.
+
+In the image below, we see for the virtual CPU status that it has a few more states than the CPUs in the [[#Resources View | Resources View]]: in red and purple respectively, when the virtual CPU is running hypervisor code and when the CPU is preempted on the host.
+
+The entries for each thread of the machine corresponds to the one from the [[#Control flow | Control Flow View]], augmented with the data from the Virtual CPU, so that we see that even though it is running from the guest's point of view, it is actually not running when the Virtual CPU it runs on is in preempted or hypervisor mode.
+
+[[Image:images/vmAnalysis/VM_CPU_view.png | Virtual CPU view]]
+
+==== Using the keyboard ====
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+
+== Hypervisor-specific Tracing ==
+
+In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.
+
+The following sections describe how to obtain traces for each hypervisor.
+
+=== Qemu/KVM ===
+
+The Qemu/KVM hypervisor require extra tracepoints not yet shipped in LTTng for both guests and hosts, as well as compilation with the full kernel source tree on the host, to have access to kvm_entry/kvm_exit events on x86.
+
+Obtain the source code with extra tracepoints, along with lttng-modules
+
+ # git clone https://github.com/giraldeau/lttng-modules.git
+ # cd lttng-modules
+
+Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.
+
+ # git checkout addons
+ # make
+ # sudo make modules_install
+ # sudo depmod -a
+
+On the host, to have complete kvm tracepoints support, the make command has to include the full kernel tree. So first, you'll need to obtain the kernel source tree. See your distribution's documentation on how to get it. This will compile extra modules, including lttng-probe-kvm-x86, which we need.
+
+ # make KERNELDIR=/path/to/kernel/dir
+
+The lttng addons modules must be inserted manually for the virtual machine extra tracepoints to be available:
+
+ # sudo modprobe lttng-addons
+ # sudo modprobe lttng-vmsync-host # on the host
+ # sudo modprobe lttng-vmsync-guest # on the guest
+
+The following tracepoints will be available
+
+ # sudo lttng list -k
+ Kernel events:
+ -------------
+ ...
+ kvm_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ kvm_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ vmsync_gh_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ vmsync_hg_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ vmsync_gh_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
+ vmsync_hg_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
+ ...
+
+Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [[#Trace synchronization | trace synchronization]] for information on how to obtain synchronizable traces.
+
= Limitations =
* When parsing text traces, the timestamps are assumed to be in the local time zone. This means that when combining it to CTF binary traces, there could be offsets by a few hours depending on where the traces were taken and where they were read.
projects / deliverable / tracecompass.git / blobdiff