+ # git checkout addons
+ # make
+ # sudo make modules_install
+ # sudo depmod -a
+
+The lttng addons modules must be inserted manually for the extra tracepoints to be available:
+
+ # sudo modprobe lttng-addons
+ # sudo modprobe lttng-elv
+
+And enable the following tracepoint
+
+ addons_elv_merge_requests
+
+=== Input/Output Views ===
+
+The following views are available for input/output analyses:
+
+==== Disk I/O Activity View ====
+A time aligned XY chart of the read and write speed for the different disks on the system. This view is useful to see where there was more activity on the disks and whether it was mostly reads or writes.
+
+ [[Image:images/io/diskIoActivity.png| Disk I/O Activity Example]]
+
+== System Call Latency Analysis ==
+
+The '''System Call Latency Analysis''' measures the system call latency between system call entry and exit per type of system call. The durations are visualized using the '''Latency''' views. For more information about the '''Latency''' views see chapter [[#Latency_Analyses | Latency Analyses]].
+
+== LTTng Kernel Events Editor ==
+
+The LTTng Kernel Events editor '''is''' the plain TMF [[#Events_Editor | Events Editor]], except that it provides its own specialized viewer to replace the standard one. In short, it has exactly the same behaviour but the layout is slightly different:
+
+* '''Timestamp''': the event timestamp
+* '''Channel''': the event channel (data collector)
+* '''CPU''': the CPU on which the event was taken
+* '''Event Type''': the event type (or kernel marker)
+* '''Contents''': the fields (or payload) of this event
+* '''TID''': The ID of the thread this event belongs to
+* '''Prio''': The priority of the thread this event belongs to
+
+[[Image:images/LTTng2EventsEditor.png]]
+
+= LTTng-UST Analyses =
+
+The Userspace traces are taken on an application level. With kernel traces, you know what events you will have as the domain is known and cloistered. Userspace traces can contain pretty much anything. Some analyses are offered if certain events are enabled.
+
+== Call Stack View ==
+
+The Call Stack view allows the user to visualize the call stack per thread over time, if the application and trace provide this information.
+
+To open this view go in '''Window''' -> '''Show View''' -> '''Other...''' and select '''Tracing/Call Stack''' in the list. The view shows the call stack information for the currently selected trace. Conversely, you can select a trace and expand it in the '''Project Explorer''' then expand '''LTTng-UST CallStack Analysis''' (the trace must be loaded) and open '''Call Stack'''.
+
+The table on the left-hand side of the view shows the threads and call stack. The function name, depth, entry and exit time and duration are shown for the call stack at the selected time.
+
+Double-clicking on a function entry in the table will zoom the time graph to the selected function's range of execution.
+
+The time graph on the right-hand side of the view shows the call stack state graphically over time. The function name is visible on each call stack event if size permits. The color of each call stack event is randomly assigned based on the function name, allowing for easy identification of repeated calls to the same function.
+
+Clicking on the time graph will set the current time and consequently update the table with the current call stack information.
+
+Shift-clicking on the time graph will select a time range. When the selection is a time range, the begin time is used to update the stack information.
+
+Double-clicking on a call stack event will zoom the time graph to the selected function's range of execution.
+
+Clicking the '''Select Next State Change''' or '''Select Previous State Change''' or using the left and right arrows will navigate to the next or previous call stack event, and select the function currently at the top of the call stack. Note that pressing the '''Shift''' key at the same time will update the selection end time of the current selection.
+
+Clicking the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) icon will open a file selection dialog, allowing you to import a text file containing mappings from function addresses to function names. If the callstack provider for the current trace type only provides function addresses, a mapping file will be required to get the function names in the view. See the following sections for an example with LTTng-UST traces.
+
+=== Using the Callstack View with LTTng-UST traces ===
+
+There is support in the LTTng-UST integration plugin to display the callstack
+of applications traced with the ''liblttng-ust-cyg-profile.so'' library (see
+the ''liblttng-ust-cyg-profile'' man page for additional information). To do
+so, you need to:
+
+* Recompile your application with "''-g -finstrument-functions''".
+* Set up a tracing session with the the ''vpid'', ''vtid'' and ''procname'' contexts. See the [[#Enabling UST Events On Session Level]] and [[#Adding Contexts to Channels and Events of a Domain]] sections. Or if using the command-line:
+** <pre>lttng enable-event -u -a</pre>
+** <pre>lttng add-context -u -t vpid -t vtid -t procname</pre>
+* Preload the ''liblttng-ust-cyg-profile'' library when running your program:
+** <pre>LD_PRELOAD=/usr/lib/liblttng-ust-cyg-profile.so ./myprogram</pre>
+
+Once you load the resulting trace, the Callstack View should be populated with
+the relevant information.
+
+Note that for non-trivial applications, ''liblttng-ust-cyg-profile'' generates a
+'''lot''' of events! You may need to increase the channel's subbuffer size to
+avoid lost events. Refer to the
+[http://lttng.org/docs/#doc-fine-tuning-channels LTTng documentation].
+
+For traces taken with LTTng-UST 2.8 or later, the Callstack View should show the
+function names automatically, since it will make use of the debug information
+statedump events (which are enabled when using ''enable-event -u -a'').
+
+For traces taken with prior versions of UST, you would need to set the path to
+the binary file or mapping manually:
+
+=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===
+
+''For LTTng-UST 2.8+, if it doesn't resolve symbols automatically, see the [[#Binary file location configuration | Source Lookup's Binary file location configuration]].''
+
+If you followed the steps in the previous section, you should have a Callstack
+View populated with function entries and exits. However, the view will display
+the function addresses instead of names in the intervals, which are not very
+useful by themselves. To get the actual function names, you need to:
+
+* Click the '''Configure symbol providers''' ([[Image:images/binaries_obj.gif]]) button in the Callstack View.
+
+Then either:
+* Point to the binary that was used for taking the trace
+OR
+* Generate a mapping file from the binary, using:
+** <pre>nm myprogram > mapping.txt</pre>
+** Select the ''mapping.txt'' file that was just created.
+
+(If you are dealing with C++ executables, you may want to use ''nm --demangle''
+instead to get readable function names.)
+
+The view should now update to display the function names instead. Make sure the
+binary used for taking the trace is the one used for this step too (otherwise,
+there is a good chance of the addresses not being the same).
+
+=== Navigation ===
+
+See Control Flow View's '''[[#Using_the_mouse | Using the mouse]]''', '''[[#Using_the_keyboard | Using the keyboard]]''' and '''[[#Zoom_region | Zoom region]]'''.
+
+=== Marker Axis ===
+
+See Control Flow View's '''[[#Marker_Axis | Marker Axis]]'''.
+
+== Flame Graph View ==
+
+This is an aggregate view of the function calls from the '''Call Stack View'''. This shows a bird's eye view of what are the main
+time sinks in the traced applications. Each entry in the '''Flame Graph''' represents an aggregation of all the calls to a function
+in a certain depth of the call stack having the same caller. So, functions in the '''Flame Graph''' are aggregated by depth and
+caller. This enables the user to find the most executed code path easily.
+
+* In a '''Flame Graph''', each entry (box) represents a function in the stack.
+* If one takes a single vertical line in the view, it represents a full call stack with parents calling children.
+* The ''x-axis'' represents total duration (execution time) and not absolute time, so it is not aligned with the other views.
+* The width of an entry is the total time spent in that function, including time spent calling the children.
+* The total time can exceed the longest duration, if the program is pre-empted and not running during its trace time.
+* Each thread traced makes its own flame graph.
+
+The function name is visible on each Flame graph event if the size permits. Each box in the '''Flame Graph'''
+has the same color as the box representing the same function in the '''Call Stack'''.
+
+To open this view select a trace, expand it in the '''Project Explorer''' then expand the
+'''Call Graph Analysis''' (the trace must be loaded) and open the '''Flame Graph'''.
+It's also possible to go in '''Window''' -> '''Show View''' -> '''Tracing''' then
+select '''Flame Graph''' in the list.
+
+[[Image:images/Flame_Graph.png|Flame Graph View]]
+
+To use the '''Flame graph''', one can navigate it and find which function is consuming the most self-time.
+This can be seen as a large plateau. Then the entry can be inspected. At this point, the worst offender in
+terms of CPU usage will be highlighted, however, it is not a single call to investigate, but rather the
+aggregation of all the calls. Right mouse-clicking on that entry will open a context sensitive menu.
+Selecting '''Go to minimum''' or '''Go to maximum''' will take the user to the minimum or maximum
+recorded times in the trace. This is interesting to compare and contrast the two.
+
+Hovering over a function will show a tooltip with the statistics on a per-function basis. One can see the total and self times
+(''worst-case'', ''best-case'', ''average'', ''total time'', ''standard deviation'', ''number of calls'') for that function.
+
+=== How to use a Flame Graph ===
+
+Observing the time spent in each function can show where most of the time is spent and where one could optimize.
+An example in the image above: one can see that ''mp_sort'' is a recursive sort function, it takes approximately
+40% of the execution time of the program. That means that perfectly parallelizing it can yield a gain of 20% for 2 threads, 33% for 3
+and so forth. Looking at the function '''print_current_files''', it takes about 30% of the time, and it has a child ''print_many_per_line'' that has a large
+self time (above 10%). This could be another area that can be targeted for optimization. Knowing this in advance helps developers
+know where to aim their efforts.
+
+It is recommended to have a kernel trace as well as a user space trace in an experiment
+while using the '''Flame Graph''' as it will show what is causing the largest delays.
+When using the '''Flame Graph''' together with a call stack and a kernel trace,
+an example work flow would be to find the worst offender in terms of time taken for a function
+that seems to be taking too longThen, using the context menu '''Go to maximum''', one can navigate
+to the maximum duration and see if the OS is, for example, preempting the function for too long,
+or if the issue is in the code being executed.
+
+=== Using the mouse ===
+
+*'''Double-click on the duration ruler''' will zoom the graph to the selected duration range.
+*'''Shift-left-click or drag''': Extend or shrink the selection range
+*'''Mouse wheel up/down''': scroll up or down
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
+* '''Shift-Ctrl-mouse wheel up/down''': zoom in or out vertically
+
+When the mouse cursor is over entries (left pane):
+
+*'''-''': Collapse the '''Flame Graph''' of the selected thread
+*'''+''': Expand the '''Flame Graph''' of the selected thread
+
+=== Using the keyboard ===
+
+The following keyboard shortcuts are available:
+
+*'''Down Arrow''': selects the next stack depth
+*'''Up Arrow''': selects the previous stack depth
+*'''Home''': selects the first thread's '''Flame Graph'''
+*'''End''': selects the last thread's '''Flame Graph''''s deepest depth
+*'''Enter''': toggles the expansion state of the current thread in the tree
+*'''Ctrl + +''': Zoom-in vertically
+*'''Ctrl + -''': Zoom-out vertically
+*'''Ctrl + 0''': Reset the vertical zoom
+
+=== Toolbar ===
+
+{|
+| [[Image:images/sort_alpha.gif]]
+| Sort by thread name
+| Sort the threads by thread name. Clicking the icon a second time will sort the threads by name in reverse order and change the icon to [[Image:images/sort_alpha_rev.gif]]
+|-
+| [[Image:images/sort_num.gif]]
+| Sort by thread id
+| Sort the threads by thread ID. Clicking the icon a second time will sort the threads by ID in reverse order and change the icon to [[Image:images/sort_num_rev.gif]].
+|}
+
+=== Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ===
+
+See Call Stack View's '''[[#Call Stack View | Importing a binary or function name mapping file (for LTTng-UST <2.8 traces) ]]'''.
+
+== Function Duration Density ==
+The '''Function Duration Density''' view shows the function duration of function displayed by duration for the current active time window range. This is useful to find global outliers.
+
+[[Image:images/FunctionDensityView.png|Function Duration Density View]]
+
+Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset.
+
+== Memory Usage ==
+
+The Memory Usage view allows the user to visualize the active memory usage per thread over time, if the application and trace provide this information.
+
+The view shows the memory consumption for the currently selected trace.
+
+The time chart plots heap memory usage graphically over time. There is one line per process, unassigned memory usage is mapped to "Other".
+
+In this implementation, the user needs to trace while hooking the ''liblttng-ust-libc-wrapper'' by running ''LD_PRELOAD=liblttng-ust-libc-wrapper.so'' '''<exename>'''. This will add tracepoints to memory allocation and freeing to the heap, NOT shared memory or stack usage. If the contexts '''vtid''' and '''procname''' are enabled, then the view will associate the heap usage to processes. As detailed earlier, to enable the contexts, see the [[#Adding Contexts to Channels and Events of a Domain]] section. Or if using the command-line:
+* <pre>lttng add-context -u -t vtid -t procname</pre>
+
+If thread information is available the view will look like this:
+
+[[Image:images/memoryUsage/memory-usage-multithread.png]]
+
+If thread information is not available it will look like this:
+
+[[Image:images/memoryUsage/memory-usage-no-thread-info.png]]
+
+The time axis is aligned with other views that support automatic time axis alignment (see [[#Automatic Time Axis Alignment | Automatic Time Axis Alignment]]).
+
+Please note this view will not show shared memory or stack memory usage.
+
+=== Using the mouse ===
+
+The Memory Usage chart is usable with the mouse. The following actions are set:
+
+* '''left-click''': select a time or time range begin time
+* '''Shift-left-click or drag''': Extend or shrink the selection range
+
+* '''left-drag horizontally''': select a time range or change the time range begin or end time
+* '''middle-drag or Ctrl-left-drag horizontally''': pan left or right
+* '''right-drag horizontally''': [[#Zoom region|zoom region]]
+* '''Shift-mouse wheel up/down''': scroll left or right
+* '''Ctrl-mouse wheel up/down''': zoom in or out horizontally
+
+=== Toolbar ===
+
+The Memory Usage View '''toolbar''', located at the top right of the view, has shortcut buttons to perform common actions:
+
+{|
+| [[Image:images/link.gif]]
+| Align Views
+| Disable and enable the automatic time axis alignment of time-based views. Disabling the alignment in the this view will disable this feature across all the views because it's a workspace preference
+|-
+|}
+
+[[Image:images/LTTng_CpuUsageViewToolTip.png]]