Monitoring Commvault with SCOM

A common request I get from customers is how to best monitor Commvault backups using SCOM. Commvault are one of the market leaders in enterprise backup technologies and I come across their products in customer sites on a regular basis.

As I don’t have a spare Commvault server to play around with in my demo environment and I’ve never really had the time to document the whole process during an actual customer deployment, a blog post on this topic has remained elusive until now.

A few weeks back I was working on a customer site who needed Commvault monitored and over the course of a lunch break one day, I managed to put some screenshots together to help document the process.

Overall, it’s pretty straight-forward to get up and running and unlike some other enterprise backup vendors, Commvault have made an effort to integrate their product with SCOM. The integration is made possible by initiating the integration from the Commvault CommCell Browser console – which then imports an unsealed management pack into SCOM for monitoring.

The management pack provided by Commvault is basic enough though and you’ll probably want to add some custom monitors and views to it as you see fit.

Management Pack Overview

The unsealed management pack provided contains a discovery rule which targets the Windows Computer class. This discovery rule (shown in the exported Excel sheet below) looks for the presence of the 'Commvault Server Event Manager Service' (the actual service name is GxEvMgrS).

When this service is detected, a new class named 'Commvault CommServer' is then created by the management pack. The class information in the management pack is shown in the image below.

There are three rules in the management pack that can generate Critical, Warning or Informational alerts in SCOM.

These rules target a CSV file named 'GalaxySCOM.csv' as their data source. This CSV file is created automatically by the Commvault application and is stored in the '\Program Files\Commvault\ContentStore\SCOM' directory on the Commvault server.

Getting Started

The first thing I'd recommend you do before deploying the Commvault management pack is to make a full list of all the Windows Services relating to Commvault that you wish to monitor. The reason for this is that the Commvault management pack will only monitor whether or not the 'Commvault Server Event Manager Service' (GxEvMgrS) service is up and running. This may be the only Commvault service you're interested in or most likely, you'll have a few more of them that are important to you.

Use the following line of PowerShell to export a list of all Windows Services on your Commvault server to a CSV file:

Get-Service | Sort-Object -Property DisplayName | Export-CSV -path C:\winserviceexport.csv

Once you've identified the service names you need for Commvault, check out my recent blog post here for a quick and easy way to monitor custom lists of Windows Services in SCOM.

The image below shows an example of the Commvault-specific services a customer recently requested to be monitored on all their Commvault servers:

Deploying the Management Pack

When you have all the Commvault services monitored, launch the CommCell Browser using an account with the required administrative permissions and you should be presented with a view similar to the one in the image below. From there, click the Control Panel button from the navigation bar at the top.

When the Control Panel area opens, you need to click the SCOM option from the Monitoring section as shown here....

This opens up the SCOM dialog box (shown below) and here, you need to input your SCOM server name along with a user account and password that has been assigned SCOM Administrator permissions.

When you've added your credentials, hit the Apply button to confirm and then click Test Configuration to validate communication between Commvault and your SCOM server is working as expected.

When you receive confirmation that the test was successful, hit the Import Management Pack button to begin the import of the unsealed management pack into SCOM.

When the process is complete, you should see a status message similar to the one in the image below that confirms the Commvault management pack has been configured...

A quick check of the Installed Management Packs view in the SCOM console confirms the management pack has been imported and is ready to go...

You should now see the four simple alert views under the CommVault Operations Manager folder in the Monitoring workspace as shown here....

If you want to confirm the new class has been created and discovered, scope your Discovered Inventory view to CommVault CommServer and you should then see all monitored Commvault servers that SCOM knows about.

Opening a Health Explorer view from the newly discovered CommVault CommServer class object shows how basic this management pack actually is with just the one Service Running State monitor in place to let you know the health state of the Commvault Windows Service.

A quick jump over to the Authoring workspace and we can see the three new Commvault alert rules that have been imported (these rules all target the new Commvault CommServer class).

A check of the Data Source properties for each of the rules gives us the location and CSV file name that will be used to collect alert information from the Commvault server...

Each rule's Data Source has been configured with a wildcard Expression value relevant to the type of alert that will fire (e.g. *Critical*, *Warning* or *Informational*).

If you want to change the name or alert description format of the alert response, you can do that from the Alert properties as shown here...

Configuring the Integration

Once the management pack has been imported and your Commvault servers have been discovered, launch the CommCell Browser again, click Alert from the navigation bar and click the Configure Alert option as shown in the following image...

When the Alerts window opens, you'll be presented with a list of all enabled and disabled alerts in Commvault. We'll click the Add button here to begin the process of creating an alert for SCOM.

From the Add Alert Wizard, type a name for the SCOM alert then choose a category and type. In our example we'll create an alert called Failed Backups and we'll choose the Job Management category with a type of Data Protection.

When you're ready, click Next to move on.

At the Entities Selection window, choose the client groups and/or clients that this alert will be scoped to then hit Next to continue.

From the Threshold and Notification Criteria Selection window, use the Alert Criteria section to scope the alert to the criteria that you need. In our example, we're only interested in Job Failed, Job Skipped and Job Succeeded with Errors alerts. Ignore the other options outside the Alert Criteria section and click Next to move on when you've made your criteria selections.

At the Notification Type(s) Selection window, click the SCOM tab then enable the Select [SCOM] for notification check box as shown in the following image...

Hit Next to continue.

At the Token Criteria Selection window you can optionally add rules to the alert that will dictate if the alerts are sent or not. You can get a full list and description of the alert tokens from here.

We won't specify any rules in our example and when you're ready, click Next to move on.

From the Security window, use the Add button to specify the user accounts and groups that you wish to grant permissions for the alert to (we'll configure an admin account with the Alert Owner role for this alert).

Click Next to move on and at the Summary window (shown in the image below), confirm your settings and hit Finish to end the wizard.

Back in the Alerts view of the CommCell Browser, you can check that the new alert has been created and is enabled as shown below...

That's all you should need to do to configure the integration between Commvault and SCOM and the next time an alert condition has been met, you should see the alert dropping into the Monitoring workspace of the SCOM console similar to this one...

If you've create a new distributed application model in SCOM for Commvault and you use either the Windows Computer or CommVault CommServer class in your component groups, these alerts will rollup to change the health of the model as expected.

Conclusion

Using the walk-through in this post should help people get up and running when monitoring Commvault with SCOM and with some additional distributed application service modeling, SLA planning and dashboard design, you can get some really nice visibility of your backup environments all from a single console.

SCOM 2016 - The Curious Case of the Missing Agent Patch List Property and Static Agent Version Value

Last week Microsoft released the second update rollup (UR2) for SCOM 2016 and a common trend I've noticed with these UR's is that the Patch List property is missing from the Agents by Version view in the Monitoring workspace of the console.

This is a bug with the SCOM 2016 agent and a bit of an annoyance when deploying update rollups as it's handy to know which agents need to be upgraded and which ones don't.

A quick check in the Agent Managed view of the Administration workspace will show a version for the agent but this version won't update to any new UR versions. The following image shows the default SCOM 2016 agent version even though I've deployed UR1 to this environment months ago...

Now, if you're thinking that after an update, all agents always drop into the Pending Management view of the Administration workspace and patiently wait until you're ready to upgrade them, then you'd be wrong. Unfortunately, depending on how you deploy the update rollup (e.g. non-admin permissions, manually installed etc.), there's a good chance that some if not all of these agents will not appear in Pending Management and you'll end up with something similar to this...

So, now your only option in the console to upgrade the agents is to run a series of bulk Repair jobs from the Agent Managed view on all of them and then hope for the best that all agents have been successfully upgraded. This is not a fun process and I really don't like not having a central view of all my agent versions direct in the console.

Thankfully Microsoft's Kevin Holman (SCOM Deity and all-round awesome community contributor) has created the new SCOM Agent Version Addendum Management Pack to help address this exact problem!

This management pack runs a script that disables the built in discovery for Microsoft.SystemCenter.DiscoverHealthServiceProperties (which has a display name of 'Discover Health Service Properties') and replaces it with a new discovery that attempts to retrieve the actual update rollup Agent Version value from a DLL file in the agent installation path.

Straight after I import this new MP, my agent version in the Agent Managed view changes to reflect the existing agent versions (the 8.0.10931.0 version shows the UR1 agents that I currently have running) and after I've deployed UR2,  I can select those agents for a Repair job as shown in the image below...

When the Repair job has completed, the version changes to show that my agents have now been updated to UR2 as shown here:

I love this MP as it adds some much needed functionality to the Agent Managed view within the console. An extra bonus is that this MP also works perfectly on SCOM 2012 R2 too!

If you want to know more, check out Kevin Holman's blog post here and you can download it directly from the TechNet Gallery here.

Enjoy!

Update Rollup 12 (UR12) Just Released for SCOM 2012 R2

Today, Microsoft released Update Rollup 12 for SCOM 2012 R2. This update contains a decent number of fixes along with some new enhancements for both Windows and cross-platform monitoring scenarios.

A full list of all the fixes and enhancements can be seen here:

https://support.microsoft.com/en-us/help/3209587/system-center-2012-r2-om-ur12


I've yet to deploy this update into my lab but I'm particularly intrigued by this one:

• Because of incorrect computations of configuration and overrides, some managed entities go into an unmonitored state. This behavior is accompanied by event 1215 errors that are logged in the Operations Manager log.

I've noticed managed entities going into an unmonitored state after applying overrides or changing distributed application configurations a lot over the last couple of years and it'll be interesting to see if this update sorts out the issue.

As should be the case for everyone deploying this update, test it in non-production environments first and be sure to read through Kevin Holman's excellent step-by-step guide to understand the order for which to apply the update and the additional manual steps that are needed:

https://blogs.technet.microsoft.com/kevinholman/2017/01/30/ur12-for-scom-2012-r2-step-by-step/

SCOM - The Topology Widget, Visio and a suped-up HD display!

Recently, I ran into an issue while creating some dashboards in the SCOM console for a customer and I thought it might be worth sharing.

Normally I use the Topology Widget to light up an image file that I initially put together using Visio and the end-result typically turns out something like this…

The difference this time though was that I’ve been using a new Windows 10 laptop that has some pretty awesome specs and a kick-ass HD display. The downside of having a laptop with Windows 10 and these specs is that application scaling becomes a nightmare and there’s a whole merry-go-round of custom tweaks that I needed to make when I started using it so as to deliver an experience where I don’t need a giant magnifying glass to work!

Here's how I have my Windows 10 laptop scaling settings configured (notice the 250% size setting)..

With these scaling settings in place on the new laptop, I went about my business by first creating a new dashboard image in Visio and then saving it as a PNG file before finally importing the file into SCOM.

When I worked my way through configuring the Topology Widget wizard to map my custom IT services (Distributed Applications) onto the image, the dashboard disappointingly turned out like this...

The problem with this dashboard view is that its grainy quality and tiny health state icons make it hard to read and understand. I've created hundreds of these dashboard views in the past and this was the first time that I've encountered a problem like this so it was time to dig a little deeper to find the solution.

The first thing I tried was to copy the problematic PNG file to another SCOM environment and create a new Topology Widget dashboard there. In this separate environment, the grainy image and tiny health state icons were still there so the problem pointed to an issue with the PNG file.

Another test I tried was to import a completely different dashboard PNG file that I knew worked fine in another customers environment and thankfully this displayed as expected. With this validation, I was confident that I was dealing with an issue either with the original problematic PNG or the Visio image that I created the PNG from.

As I traced back through my steps, I opened the Visio file again that I created this dashboard in and clicked the the Save As option from the File menu to save it as a new PNG. When I did this, I was presented with the following PNG Output Options window:

Notice the default Resolution and Size settings Visio 2016 selects for me when I go to save a new PNG file. I figured that due to the 250% display scaling option that my laptop was configured with, these settings were creating the PNG file at too high a resolution for SCOM to work with.

I went back to the original problematic PNG file and checked the Image Properties and I could see that it was configured to use 2044 x 1548 pixels as shown here....

When I checked the other dashboard PNG file that I knew worked (and which I created on my old laptop), I could see that it was configured to use a much lower pixel size.

So, back to the Visio diagram of my new dashboard and this time, when I clicked the Save As option from the File menu, I manually configured the PNG Output settings to use a resolution of Source and a pixel size of 1123 x 794 as shown in this image...

When I imported this new leaner version of the PNG file back into the same Topology Widget, I finally got the results I was looking for where the health state and image quality were far easier on the eye.

Hopefully this easy fix helps someone else out with their future SCOM dashboard creations!

The Most Useful SCOM Article on the Web Just Got an Update!

As anyone who's ever worked with SCOM will know, it's a fairly heavy and complex product to get your head around at first and the larger the environment to be monitored, the more administration and troubleshooting tasks you'll need to teach yourself.

Way back when I started working with SCOM, I quickly found myself lost in a myriad of blog posts and TechNet articles searching for help on how to extract information from the SQL databases to help me better understand the problems I was experiencing.

The one thing that kept coming up trumps for me in my searches time and time again was Kevin Holman's 'Useful Operations Manager 2007 SQL Queries' post. This post brought together a virtual treasure chest of SQL queries that the 'non-SQL admin' like me could easily copy and paste into my SQL Management Studio window for instant information or configuration changes in my customers SCOM environments.

It was probably the first SCOM reference on the web that I saved as a favourite into my web browser and was always a location that I'd tell new SCOM admins to go check out and bookmark.

As the title of Kevin's post suggests, it was originally put together nine years ago as a central repository of SQL queries for SCOM 2007. When System Center 2012 and ultimately 2016 came around, these queries still worked with the newer releases of SCOM but there was often some confusion from people trying to understand if they only worked with SCOM 2007.

So to address this, just recently Kevin took the time to archive his original 2007-named post and create a new one titled simply 'SCOM SQL Queries'.

Not only has he renamed the post but he has also formatted it in a way that all queries are now much easier to read from and copy/paste as required.

Check out the new location for what is most likely, the most useful SCOM article on the web here:

https://blogs.technet.microsoft.com/kevinholman/2016/11/11/scom-sql-queries/

Experts Live NL 2016

Today I've just finished up presenting my last public conference session of 2016 at the awesome Experts Live conference in the Netherlands.

This is my second year to attend Experts Live NL and it already seems like the conference attendee and speaker count has grown significantly in that short space of time.

My presentation this year was titled 'Hacking OMS with your OpsMgr Skills' and is an extension of the session that I co-presented with my good friend Cameron Fuller at System Center Universe 2016 in August.

The original idea and title for this session was all Cameron's and with his blessing, I put my own spin on the content to ensure that Experts Live attendees were treated to a significantly different version of the one we delivered previously at SCU. Also, with the vast number of changes and feature additions that we've now become accustomed to with OMS, there was much to show on the day.

My session was the first to open after the keynote and it was refreshing to see the room filled with a large number of current OpsMgr users waiting to hear how to advance their skillsets with OMS.

(Photo credit Pedro van Vliet)

When my presentation was done, I took some time to hang out with old friends and to network with the attendees and various booth vendors around the event.

All in all, Experts Live NL was a good closure for me to a hectic few months of traveling and presenting. I'm looking forward to now refocusing my attention back onto my poor neglected blog and bringing some useful posts into the community over the coming months!

Important SCOM 2016 and 2012 R2 Updates!

If like me, you've jumped aboard the SCOM 2016 bandwagon and started deploying the recently released GA version to your production environments, then you'll need to be aware of two very important updates that need to be added ASAP.

The first one is Update Rollup 1 for SCOM 2016:

https://support.microsoft.com/en-us/kb/3190029

Microsoft have recommended that people deploy this update rollup immediately after deploying the intial SCOM 2016 GA build as it contains fixes for a number of issues that were recently highlighted by users of the Technical Preview 5 release.

The next update is better identified as a patch (KB3200006) that Microsoft needed to quickly release in response to a widespread spate of console crashes on both SCOM 2016 and 2012 R2.

People are understandably frustrated at these crashes as you can read from here and here.

You can get access to the new patch that (hopefully) fixes this problem from the following link:

https://support.microsoft.com/en-us/kb/3200006

Hopefully this helps people out and feel free to use the comments section below (or add your thoughts to the TechNet forums mentioned above) if this patch doesn't solve the console crash issue for you.

Looking back on System Center Universe Europe 2016

A few weeks back I had the honour of presenting again at the annual System Center Universe Europe conference - which was held this year in Berlin, Germany.

This was my fourth year presenting at System Center Universe Europe and I can honestly say that the conference just keeps getting better and better each time. This is mainly due to the epic amount of time and effort the team over at itnetX dedicate to organising it.

I went into this conference initially with two sessions to present. One was a solo effort titled 'What's New with OpsMgr 2016' and the other was a joint presentation with the one and only Cameron Fuller titled 'Using your OpsMgr skills to hack OMS'.

A few short hours after landing in Germany, I ended up adding another presentation to my list. This third one was another joint presentation with Cameron Fuller and it came with the awesome title of 'OMS & OpsMgr: Mortal Enemies, Casual Acquaintances, Best Friends, or Inbred Cousins?' - only Cameron could think up a fun and quirky title like this!

Here's the low-down on how each day over there went for me:

Pre-Conference
The day before the conference began, a few of us (myself, Cameron, Robert Hedblom and Janaka Rangama) decided to do some sightseeing and took a cab over to the Berlin Wall Memorial. Although this is one of the top sights to see when you visit Berlin and definitely something on my bucket list to check out, the sombre historical significance of the wall was never far from our minds.

After a few hours soaking up some culture, we walked through the various meandering side-streets of Berlin until we came across the conference centre that would host this years System Center Universe event. Located right in the middle of Berlin's Alexanderplatz, it wasn't hard to miss due to the large SCU flags that greeted us on arrival at the front of the building.

After a quick recon mission of the rooms we would be presenting in and the overall venue, we all agreed that this was going to be a good week ahead.

When we left the conference centre, we decided (actually Robert decided) to get some food accompanied with some local beverages. Being in Germany, it would've been rude to order anything smaller than this as a beer to wash down the local staple dish of Currywurst...

Another great thing about this years host city is the fact that everything is so central and after catching up with my geek friends, it was only a short night-time walk back to the hotel - which was always easy to navigate back to due to the prominence of the building on the Berlin skyline!

Day 1
We had a bright and early start on the Wednesday morning as Marcel Zehner kicked off the opening keynote with a run-down of the few days ahead (including the all-important party list!)

When Marcel got the formalities out of the way, it was straight into tech with special guest Ed Wilson (aka The Scripting Guy). Ed delivered an awesome presentation on how to approach traditional IT challenges in a hybrid IT world.

Straight after the keynote, myself and Cameron headed over to our room to get ready for our first co-presented session titled 'OMS & OpsMgr: Mortal Enemies, Casual Acquaintances, Best Friends, or Inbred Cousins?'...

It wasn't long before the room filled up and in true Cameron style, he kicked off the presentation with a pre-recorded video of songs and images that represented mortal enemies, casual acquaintances, best friends and inbred cousins!

The interaction from the audience during our session was awesome and we had so many questions in the Q & A section that we ran out of time!

The rest of the day was spent watching and learning from some of the other presentations and later that night, it was time to chill at the speakers and sponsors party - which was hosted at Club Mio and included a top quality open-air barbecue dinner.

Due to some very suspect MVP dance moves, I'll keep the after-hours nightclub photo's away from the internet!

Day 2
On the second day, I took in a cool session from the dynamic Stefan duo (Stefan Roth & Stefan Koell) before heading over to Bob Cornelissen and Savision's session titled 'Prepare for Hybrid Monitoring - SCOM 2012, SCOM 2016 and OMS'.  Due to an unexpected hospital visit, Bob had initially asked me to be on standby to take over and deliver this presentation as he wasn't sure he'd even make it over to Berlin but, like a true professional, he stood up on stage and rocked it!

Following this, myself Cameron and Jan Vidar Elven headed over to the 'Ask the Experts' booth to host questions from attendees related to System Center 2016.

Later that afternoon me and Cameron were back on stage again for our  'Using your OpsMgr skills to hack OMS' session. This was another well attended presentation (especially considering the SquaredUp guys were doing an impromptu Whiskey Tasting in the room next door)!

When the final sessions of the day finished up, the conference venue played host to the SCU Networking Party where the mix of geek-talk, cocktails and music was rampant.

Day 3
With more than a few sore heads on the last day from the parties the previous night, I had my final presentation to deliver at the opening 09:15 time-slot. This session was titled 'What's New with OpsMgr 2016' and in it I covered all of the new features and enhancements that we can look forward to with the latest release of our favourite monitoring platform. Again, this was another session that had great interaction from the attendees.

An interesting method of gauging attendee feedback for each session was the 'Happy or Not' button stand that was positioned outside the door of each breakout session.

The results from this new rating system were uploaded and sent to speakers within a few days of the conference and thankfully the sessions I participated in where well received.

When I finished my last presentation, it was time to finally chill out a bit after prepping and rehearsing for most of the week and just after lunch a few of us decided to do some final sight-seeing before heading home the following day.

Here's me with my Ergo buddy Gareth checking out some of the amazing architecture around Berlin..

We also stumbled across what we initially thought was a normal Microsoft Store (similar to the ones in the US) however, when we went inside it was a strange bar/café combination so of course, we stopped here for some light refreshments!

When we were finished in the Microsoft bar, we headed on to the famous Brandeburg Gate (one of the top historical tourist attractions in Germany).

Closing Announcements
After the sight-seeing, we headed back to the conference for the closing keynote where some significant announcements were made relating to the future of SCU Europe.

The first announcement was that the conference would be back again in Berlin next year - which is a decision that has gone down very well with speakers and attendees.

The second announcement was that SCU Europe would be re-branded to Experts Live Europe...

I think this is a sensible decision as the conference has morphed into so much more than just System Center. Yes we will have a new release of System Center 2016 coming shortly but when you consider the amount of content discussed at this years conference on Azure and OMS technologies, it just makes perfect sense.

All-in-all, it was a great week and I'm really looking forward to heading back to Berlin next year to what has now turned into Europe's premier community conference for Microsoft IT pro's and geeks.