Once you understand how XPERF utilizes the powerful ETW trace mechanism, how can we actually control what is being collected? As you've probably already guessed, XPERF is able to collect a ton of useful information and as such has an equally powerful configuration story in the form of a command-line tool called xperf.exe. I won't go through every single option available, rather focus on some of the most common scenarios in how to enable/disable tracing.
Let's start by taking a simple look at how to use XPERF to collect diagnostics information. Start by running the following command from the MSWPT installation folder:
xperf.exe -on Base
The Base argument specifies a predefined set of commonly used providers to start collection diagnostics information. Upon successful execution, anything you do on that system will be recorded based on the enabled providers. Please note that a lot of information is traced so you may want to limit the amount of time that you spend trying to reproduce a particular problem. To give you an idea, on my machine I only ran for about 20 seconds and ended up with a trace file that was 8MB in size.
When you are ready to stop tracing, the following command can be used:
xperf.exe -d <filename>
Where filename is the name of the result file (typically ending with the extension .etl). Now, the result file is in a raw format meaning that it's nearly impossible to decipher using just a regular editor. Instead, you have to use either different parameters to the xperf.exe tool or use the xperfview.exe tool which displays the results in an easy to digest form. Before we take a look at those tools, it is also important to mention that instead of using one of the pre-canned accumulative collection arguments (such as Base) you can specify specific data providers by using the -providers switch. To find out which providers are registered on the system, use the following command:
Analyzing XPERF Diagnostics Data
Now that we all the diagnostics data collected, it's time to turn our attention to how to best look at the resulting file. Again, the xperfview.exe tool has a nice graphical user interface that makes it easy to digest the plethora of information collected by XPERF. Figure 2 shows an example of xperfview.exe when run against the generated results file:
xperf.exe -on Base
You can see in Figure 2 how xperfview.exe provides an easy way to analyze the resulting information. Each section of the output corresponds to a particular provider (such as CPU usage, I/O Counts etc.). To narrow down the investigation to a particular section of any given provider, you can highlight the section followed by right-clicking and selecting "Zoom To Selection" in the context menu. You can furthermore customize the different providers that are shown by selecting the left most arrow tab in the middle of the screen which will bring up a list of providers that you can either select or deselect.
Another useful XPERF feature is the ability to collect stack traces of the ongoing thread activity. For stack traces to work, you need to make sure that you have the correct symbols and that the symbol path is set to the symbol location. For example, if I wanted to take periodic stack traces I can use the following command:
xperf.exe -on Base -stackwalk Profile
Note that having the correct symbol path set is only required during the analysis stage and not the collection stage.
To view stack traces in xperfview.exe, select the area of interest in the CPU usage chart, right click on the selection and choose "Summary Table". In the new window, click on the Columns menu item and select Stack. Another column name "Stack" is now added and you can use the + button to expand and get call stack information.
Performance problems can be tricky to troubleshoot in an efficient matter. One of the main difficulties are in knowing which part of the system is causing the performance degradation. I've often seen developers spend weeks looking at their application logs trying to figure out why the system is running poorly when, in reality, a system-wide analysis needed to be performed to figure out a configuration problem on the server. It is a daunting task to step outside the application boundary and troubleshoot a system as a whole. Fortunately, we do have a great tool in XPERF that adds tremendous value in the troubleshooting process.