+= Latency Analyses =
+
+Trace Compass offers a feature called Latency analysis. This allows an analysis to return intervals and these intervals will be displayed in four different views. An example analysis is provided with kernel system call latencies being provided. The available views are:
+
+* System Call Latencies
+A '''table''' of the raw latencies. This view is useful to inspect individual latencies.
+
+ [[Image:images/LatenciesTable.png| Latency Table example - System Call Latencies]]
+
+
+* System Call Latency vs Time
+A time aligned '''scatter chart''' of the latencies with respect to the current window range. This view is useful to see the overall form of the latencies as they arrive.
+
+[[Image:images/LatenciesScatter.png| Latency Scatter Chart example - System Call Latency vs Time]]
+
+
+* System Call Latency Statistics
+A view of the total '''statistics''' of the latencies. These show the ''minimum'', ''maximum'', ''average'', ''standard deviation'', and ''count'' of the latencies when applicable. This tool is useful for finding the outliers on a per-category basis.
+
+Right-clicking on an entry of the table and select '''Go to minimum''' allows to select the range of the minimum latency for the selected entry and synchronize the other views to this time range.
+
+Right-clicking on an entry of the table and select '''Go to maximum''' allows to select the range of the maximum latency for the selected entry and synchronize the other views to this time range.
+
+[[Image:images/LatenciesStatistics.png| Latency Statistics example - System Call Latency Statistics]]
+
+* System Call Density
+A '''density''' view, analyzing the current time range. This is useful to find global outliers. Selecting a duration in the table it will synchronize other views to this time range.
+
+[[Image:images/LatenciesDensity.png| Latency Densities example - System Call Density]]
+
+Using the right mouse button to drag horizontally it will update the table and graph to show only the density for the selected durations. Durations outside the selection range will be filtered out. Using the toolbar button [[Image:images/zoomout_nav.gif]] the zoom range will be reset.
+
+= Virtual Machine Analysis =
+
+Virtual environments are usually composed of host machines, who each run an hypervisor program on which one or many guests can be run. Tracing a guest machine alone can often yield some strange results as from its point of view, it has full use of the resources, but in reality, most resources are shared with the host and other guests.
+
+To better understand what is happening in such an environment, it is necessary to trace all the machines involved, guests and hosts, and correlate this information in an experiment that will display a complete view of the virtualized environment.
+
+== Virtual Machine Experiment ==
+
+A trace has to be taken for each machine, guest and host, in the virtualized environment. The host trace is the most important to have, as missing guests will only give an incomplete view of the system, but missing hosts usually won't allow to identify the hypervisor, nor determine when a guest is preempted from the host CPUs. The virtual machine analysis only makes sense if the host trace is available.
+
+Once all the traces are imported in Trace Compass, they can be [[#Creating a Experiment | added to an experiment]]. The type of the experiment should by set to '''Virtual Machine Experiment''' by clicking on the right mouse button over the experiment name, then selecting '''Select Experiment Type...'''.
+
+[[Image:images/vmAnalysis/VM_experiment.png | Virtual Machine Experiment]]
+
+Depending on the hypervisor used, traces might need to be [[#Trace synchronization | synchronized]] so that they have the same time reference and their events can be correctly correlated.
+
+== Virtual CPU View ==
+
+The Virtual CPU view shows the status of CPUs and threads on guests augmented with the preemption and hypervisor data we get from the host.
+
+In the image below, we see for the virtual CPU status that it has a few more states than the CPUs in the [[#Resources View | Resources View]]: in red and purple respectively, when the virtual CPU is running hypervisor code and when the CPU is preempted on the host.
+
+The entries for each thread of the machine corresponds to the one from the [[#Control flow | Control Flow View]], augmented with the data from the Virtual CPU, so that we see that even though it is running from the guest's point of view, it is actually not running when the Virtual CPU it runs on is in preempted or hypervisor mode.
+
+[[Image:images/vmAnalysis/VM_CPU_view.png | Virtual CPU view]]
+
+==== Using the keyboard ====
+*'''Ctrl + F''': Search in the view. (see [[#Searching in Time Graph Views | Searching in Time Graph Views]])
+
+== Hypervisor-specific Tracing ==
+
+In order to be able to correlate data from the guests and hosts traces, each hypervisor supported by Trace Compass requires some specific events, that are sometimes not available in the default installation of the tracer.
+
+The following sections describe how to obtain traces for each hypervisor.
+
+=== Qemu/KVM ===
+
+The Qemu/KVM hypervisor require extra tracepoints not yet shipped in LTTng for both guests and hosts, as well as compilation with the full kernel source tree on the host, to have access to kvm_entry/kvm_exit events on x86.
+
+Obtain the source code with extra tracepoints, along with lttng-modules
+
+ # git clone https://github.com/giraldeau/lttng-modules.git
+ # cd lttng-modules
+
+Checkout the addons branch, compile and install lttng-modules as per the lttng-modules documentation.
+
+ # git checkout addons
+ # make
+ # sudo make modules_install
+ # sudo depmod -a
+
+On the host, to have complete kvm tracepoints support, the make command has to include the full kernel tree. So first, you'll need to obtain the kernel source tree. See your distribution's documentation on how to get it. This will compile extra modules, including lttng-probe-kvm-x86, which we need.
+
+ # make KERNELDIR=/path/to/kernel/dir
+
+The lttng addons modules must be inserted manually for the virtual machine extra tracepoints to be available:
+
+ # sudo modprobe lttng-addons
+ # sudo modprobe lttng-vmsync-host # on the host
+ # sudo modprobe lttng-vmsync-guest # on the guest
+
+The following tracepoints will be available
+
+ # sudo lttng list -k
+ Kernel events:
+ -------------
+ ...
+ kvm_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ kvm_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
+ vmsync_gh_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ vmsync_hg_guest (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the guest
+ vmsync_gh_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
+ vmsync_hg_host (loglevel: TRACE_EMERG (0)) (type: tracepoint) # on the host
+ ...
+
+Host and guests can now be traced together and their traces added to an experiment. Because each guest has a different clock than the host, it is necessary to synchronize the traces together. Unfortunately, automatic synchronization with the virtual machine events is not completely implemented yet, so another kind of synchronization needs to be done, with TCP packets for instance. See section on [[#Trace synchronization | trace synchronization]] for information on how to obtain synchronizable traces.
+
+= Java Logging =
+
+Trace Compass contains some Java Utility Logging (JUL) tracepoints in various places in the code. To diagnose issues with Trace Compass or when reporting problems with the application, a JUL trace may be useful to help pinpoint the problem. The following sections explain how to enable JUL logging in Trace Compass and use various handlers to handle the data.
+
+== Enable JUL Logging ==
+
+By default, all the logging of the Trace Compass namespace is disabled. To enable it, one needs to add the following property to the ''vmargs'': ''-Dorg.eclipse.tracecompass.logging=true''.
+
+The log levels and components can be controlled via a configuration file whose path is specified also in the ''vmargs'' like this: ''-Djava.util.logging.config.file=/path/to/logger.properties''. An example configuration file can be found in the next section.
+
+If running the RCP, these arguments can be appended at the end of the ''tracecompass.ini'' file located in the folder where the executable is located. If running from Eclipse in development mode, in the ''Run configurations...'', the arguments should be added in the ''Arguments'' tab in the ''VM args'' box.
+
+== Configuring JUL logging ==
+
+JUL logging can be fine-tuned to log only specific components, specific levels, but also to different log handlers, with different formats, etc. Or else, the default level is INFO and the default log handler is a ConsoleHandler which displays all log message to the Console, which can be quite cumbersome.
+
+Here is an example ''logger.properties'' file to control what is being logged and where.
+
+ # Specify the handlers to create in the root logger
+ # (all loggers are children of the root logger)
+ # These are example handlers
+
+ # Console handler
+ handlers = java.util.logging.ConsoleHandler
+ # Console and file handlers
+ #handlers = java.util.logging.ConsoleHandler, java.util.logging.FileHandler
+ # No handler
+ #handlers =
+
+ # Set the default logging level for the root logger
+ # Possible values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
+ .level = OFF
+
+ # Fine tune log levels for specific components
+ # Use the INFO level for all tracecompass, but FINEST for the StateSystem component
+ #org.eclipse.tracecompass.internal.statesystem.core.StateSystem.level = FINEST
+ org.eclipse.tracecompass.level = INFO
+
+== LTTng JUL log handler ==
+
+The various log handlers have an overhead on the application. The ConsoleHandler has a visible impact on Trace Compass performance. The FileHandler also has an overhead though less visible, but when logging from multiple threads at the same time, the file becomes a bottleneck, so that logging data cannot be used with accuracy for performance analysis. The [http://lttng.org/docs/#doc-java-application LTTng log handler] is much better in a multi-threads context.
+
+LTTng-UST comes with the Java JUL agent in most distros. Otherwise, it is possible to manually compile lttng-ust with options ''--enable-java-agent-jul'' and install it.
+
+ git clone git://git.lttng.org/lttng-ust.git
+ cd lttng-ust
+ ./bootstrap
+ ./configure --enable-java-agent-jul
+ make
+ sudo make install
+
+The necessary classes for the java agent will have been installed on the system. Since Equinox (the OSGi implementation used by Eclipse and thus Trace Compass) uses its own classpath and ignores any classpath entered on the command line for security reasons, one needs to specify the agent class path with the bootclasspath argument:
+
+ -Xbootclasspath/a:/usr/local/share/java/lttng-ust-agent-jul.jar:/usr/local/share/java/lttng-ust-agent-common.jar
+
+Note that unlike the -classpath argument, -Xbootsclasspath does not follow the dependencies specified by a jar's Manifest, thus it is required to list both the -jul and the -common jars here.
+
+These classes need to load the LTTng JNI library. Because they were loaded from the boot class path by the boot ClassLoader, the library path entered on the command line is ignored. A workaround is to manually copy the library to the jvm's main library path. For example
+
+ sudo cp /usr/local/lib/liblttng-ust-jul-jni.so /usr/lib/jvm/java-8-openjdk/jre/lib/amd64/
+
+Or to overwrite the JVM's library path with the following VM argument.
+
+ -Dsun.boot.library.path=/usr/local/lib
+
+''Disclaimer: this last method overwrites the main java library path. It may have unknown side-effects. None were found yet.''
+
+LTTng can now be used as a handler for Trace Compass's JUL, by adding the following line to the logger.properties file
+
+ handlers = org.lttng.ust.agent.jul.LttngLogHandler
+
+The tracepoints will be those logged by a previously defined configuration file. Here is how to setup LTTng to handle JUL logging:
+
+ lttng create
+ lttng enable-event -j -a
+ lttng start
+