"The Anson Probability" or Locating CPU spikes when we don’t know when the occurred…

By Rob Reyes at 2/10/2013 7:05 PM
Filed Under: IT Analytics, OpsMgr, Root Cause Analysis

I was chatting with my friend Mr. Anson today and he asked about a use-case a number of his end users were asking about.  They were asking Mr. Anson when CPU spikes were occurring on their server.  Unfortunately, the end users did not know when these spikes were occurring, just that some users were mentioning that their experience was being effected through the day.

 
In this blog post, we will show you how to locate CPU spikes and patterns by investigating SCOM data with IT Analytics.


Finding the Spikes
We first start with the SCOM Performance Daily cube and filter on the Windows 2008 Server Management Pack and the % Processor Time counter. This is across all 2008 servers and across all dates and times.

 

 

We then add the server name (Host Entity – Display Name) and sort descending.  What is listed are servers that have spiked at 100% across all time.  There are others (75% or less) that have not.

 

Click to view larger image

 

The next result set is to only look at the machines that have spiked 90% or greater.  There are a number of servers that have not spiked significantly (75% or greater).

 

Click to view larger image

 

Next, we only look at data from 2012:

 

Click to view larger image

 

There is an even smaller set of computers that have spiked over 90% in 2012.  We'll filter on those next.

 

Click to view larger image

 

These are the servers we will now be concentrating on. Next add month to see the trend for all servers.

 

Click to view larger image

 

We will filter on VMDCFTP001 to look at the daily trend.

 

Click to view larger image

 

Since the highest spike is at the first of the year, we re-create this chart in the SCOM Performance Hourly cube to see the hourly trend.

 

Click to view larger image

 

We then filter on dates 01/04/2012 – 01/21/2012 and add the Hour dimension to see what hour the CPU spikes are occurring.

 

Click to view larger image

 

Now that we know when these spikes are occurring, we can investigate other data by changing (or adding) the Management Pack and associated counters to see what other information is being gathered on that server and correlate that data to investigate the root cause of the discovered CPU spikes.

 

 

Rob has over 11 years of experience in software implementation and sales. Since joining Bay Dynamics in 2003, Rob has held a number of technical, sales, and sales engineering positions that have made him a valuable asset to the organization. He continues to assist clients in effectively optimizing both their physical and virtual infrastructure through the use of IT Analytics.

Using IT Analytics KPIs in SQL Server Reporting Services Scorecards

By Rob Reyes at 1/5/2012 8:24 PM
Filed Under: IT Analytics, OpsMgr, Scorecards, SSRS

IT Analytics makes it easy to navigate System Center data and identify metrics to measure against.  While these metrics can be identified and managed within the Service Manager console, Microsoft has a number of technologies that makes it easy to share this information.  In this post, we will show you how to create a Scorecard that shows performance against a defined goal and the KPI Trend relative to that goal, all within a SQL Server Reporting Services Report.

 

 Click to view larger image

 

Business Need

Management continues to ask the team about disk space utilization and how we can continue to optimize available disk space to support our growing environment.  Since all this storage data does exist in SCOM, we will create a scorecard listing the servers of interest, the drives associated to each  server, and free space metrics.

 

New KPI

The first item we will need to create is a Key Performance Indicator.  This KPI will be based on the "Windows Server 2008 Operating System (Monitoring)" management pack counter "% Free Space."   By creating this KPI based on this single counter, we will be able to split this metric across servers and drives in our scorecard.

 

 

We highlight the grand total value and click "New KPI."  In the "Edit KPI" dialog box, we add a goal of "75" and trend this over the past 180 days.

 

 

After clicking on "Save KPI," the KPI values are saved to the cube.  We can now include these values in a SQL Server Reporting Services data set.

 

SSRS Scorecard

Following the steps outlined in this blog entry to connect to the IT Analytics SCOM Performance Daily cube, we create the dataset we are interested in.  We include KPI Trend data as well.

 

Click to view larger image

 

Next, we build out the report by adding host name and entity name to our rows, counter name as a column and the average as the main value.

 

Click to view larger image

 

The next two columns we add will make our scorecard really come to life.  We right click on the cell that contains the "average" value and select "Insert Column - Inside Group - Right." 

 

Click to view larger image

 

We will do this action twice: one for the state indicator and one for the trend indicator so our table looks like this:

 

Click to view larger image

 

The first indicator we will insert is the "state" indicator.  We will place this indicator in the first cell to the right of the average.

 

Click to view larger image

 

Once the indicator has been placed, a dialog box will appear allowing us to select what indicator we want to include.  For this first indicator, we will select Shapes:

 

 

Once in place, we right click on the icon and select "Indicator Properties" to modify the default settings.  On the first page, we name our indicator.

 

 

In the Value and States section, we set "Value" to [Sum[Average]].  While these can be adjusted to match requirements, in this example we will leave the default settings.

 

 

In the actions page, we leave the default settings of "None."

 

 

We will repeat the process above and insert a second indicator to the right.

 

Click to view larger image

 

The second indicator will be a directional indicator:

 

 

We then right click on the indicator to change the Value properties.  Since the KPI Trend is part of our data set, we can use that value to calculate the trend indicator. We set the value field, adjust the states and click OK.

 

 

After some additional formatting, we can view our scorecard detailing for each server, the percent free disk space for each associated drive and how the percent disk space has been changing over the past 180 days.

 

Click to view larger image

 

Summary

Management asked for a report based on free space data within SCOM and to present this data in an easy to understand format.  Within IT Analytics, we were able to quickly investigate the Free Disk Space data and define a Key Performance Indicator.  This data was then the basis for our SQL Server Reporting Services Scorecard.  This scorecard can now be embedded in a web page, included in a SharePoint site, or subscribed to for frequent updates.  This new report will enable the team and management to review performance and trending relative to defined goals and make more informed decisions regarding disk space utilization.

Configuring .Net Application Performance Monitoring in SCOM 2012 and Analyzing with IT Analytics

By Rob Reyes at 8/19/2011 9:17 AM
Filed Under: APM, Application Performance Monitoring, AVIcode, Install/Config, IT Analytics, OpsMgr, SCOM 2012

System Center Operations Manager (SCOM) 2012 beta is here!  The goal of this post is to demonstrate how to configure .Net Application Performance Monitoring in SCOM 2012, and then analyze that data with IT Analytics.

 

AVIcode Configuration

 

After SCOM 2012 has been installed and SCOM agents deployed, we then configure Application Performance Monitoring (APM).  All of the necessary agent functionality is now integrated within the SCOM 2012 agent.  All we need to do is configure a monitor to get up and running!

 

First, APM is .NET exclusive in 2012, so we need to identify and create a monitor for a specific .Net Application.  We click on "Add Monitoring Wizard", as shown on the following screen:

 

Click to view larger image

 

On the first page, we select the type of monitor:

 

Click to view larger image

 

Next, we name the monitor, give a brief description and select and existing Management Pack (MP) or create a new MP. In our example we created a new MP named AVIcode. Please note that the Monitor name cannot be changed once created.

 

Click to view larger image

 

On the next page, we specify what sites we are going to monitor.  The SCOM 2012 agent is able to scan and return .NET compatible applications.  This discovered data populates the component list, making it easier to select which applications to monitor.

 

 

In SCOM 2012, AVIcode monitors are specific to the application, regardless of what server it is located on.  To track performance of an application on a specific server, you specify these servers by clicking on the "Search" button in the Monitored Servers section and select any default group or user created groups.

 

Click to view larger image

 

For Server Side Monitoring, we leave the default options and performance event threshold.  We click on the checkbox so we can gather data on the end user experience as well.

 

Click to view larger image

 

Here, we have enabled client side monitoring for the site we are interested in.  We have not excluded any pages at this time.

 

Click to view larger image

 

This next page allows us to refine the monitor even more.

 

Click to view larger image

 

Last page is a summary of the settings.

 

Click to view larger image

 

Now that our monitor is created the final sep is to restart IIS on any servers the monitor applies to in order to enable monitoring.

 

Out of the box Reporting

The SE Viewer has been brought back in SCOM 2012, see:

 

Click to view larger image

 

The raw performance data about the site can bee seen at the URL http://localhost/AppDiagnostics/ on the SCOM server.

 

Click to view larger image

 

Management Pack reporting is consistent as well:

 

Click to view larger image

 

AVIcode Data Analysis with IT Analytics

To view all the available AVIcode/APM performance data, we create a pivot view where we filter on the "Operations Manager APM Library" and break this down by the available counters.

 

 

We filter on the "Avg. Request Time" counter and add "Entity - Name" to see each site we are tracking.

 

Click to view larger image

 

Next, we will convert this in to a chart.

 

Click to view larger image

 

To investigate that API further, I switch back to a table, filter on that App and look at the daily trending.

 

Click to view larger image

 

The 11th of August has an unusual spike.  I'll filter on that date and add "Performance Evaluation Time - Hour" to my table to get more detail.

 

Click to view larger image

 

Looks like this spike was an anomaly.  To see the trending for the remainder of the day, I exclude the "0" hour, and change this table to a chart.

 

Click to view larger image

 

Summary

The tighter integration of .Net Performance Monitoring in to SCOM 2012 will prove very beneficial, and the setup is fairly straight forward.  Once that data has been collected, the OpsMgr cubes in IT Analytics enable you to analyze that APM/AVIcode data in a fast and flexible manner, giving you the insight you need when you need it!