Monitoring Commvault with SCOM

A common request I get from customers is how to best monitor Commvault backups using SCOM. Commvault are one of the market leaders in enterprise backup technologies and I come across their products in customer sites on a regular basis.

As I don’t have a spare Commvault server to play around with in my demo environment and I’ve never really had the time to document the whole process during an actual customer deployment, a blog post on this topic has remained elusive until now.

A few weeks back I was working on a customer site who needed Commvault monitored and over the course of a lunch break one day, I managed to put some screenshots together to help document the process.

Overall, it’s pretty straight-forward to get up and running and unlike some other enterprise backup vendors, Commvault have made an effort to integrate their product with SCOM. The integration is made possible by initiating the integration from the Commvault CommCell Browser console – which then imports an unsealed management pack into SCOM for monitoring.

The management pack provided by Commvault is basic enough though and you’ll probably want to add some custom monitors and views to it as you see fit.

Management Pack Overview

The unsealed management pack provided contains a discovery rule which targets the Windows Computer class. This discovery rule (shown in the exported Excel sheet below) looks for the presence of the 'Commvault Server Event Manager Service' (the actual service name is GxEvMgrS).

When this service is detected, a new class named 'Commvault CommServer' is then created by the management pack. The class information in the management pack is shown in the image below.

There are three rules in the management pack that can generate Critical, Warning or Informational alerts in SCOM.

These rules target a CSV file named 'GalaxySCOM.csv' as their data source. This CSV file is created automatically by the Commvault application and is stored in the '\Program Files\Commvault\ContentStore\SCOM' directory on the Commvault server.

Getting Started

The first thing I'd recommend you do before deploying the Commvault management pack is to make a full list of all the Windows Services relating to Commvault that you wish to monitor. The reason for this is that the Commvault management pack will only monitor whether or not the 'Commvault Server Event Manager Service' (GxEvMgrS) service is up and running. This may be the only Commvault service you're interested in or most likely, you'll have a few more of them that are important to you.

Use the following line of PowerShell to export a list of all Windows Services on your Commvault server to a CSV file:

Get-Service | Sort-Object -Property DisplayName | Export-CSV -path C:\winserviceexport.csv

Once you've identified the service names you need for Commvault, check out my recent blog post here for a quick and easy way to monitor custom lists of Windows Services in SCOM.

The image below shows an example of the Commvault-specific services a customer recently requested to be monitored on all their Commvault servers:

Deploying the Management Pack

When you have all the Commvault services monitored, launch the CommCell Browser using an account with the required administrative permissions and you should be presented with a view similar to the one in the image below. From there, click the Control Panel button from the navigation bar at the top.

When the Control Panel area opens, you need to click the SCOM option from the Monitoring section as shown here....

This opens up the SCOM dialog box (shown below) and here, you need to input your SCOM server name along with a user account and password that has been assigned SCOM Administrator permissions.

When you've added your credentials, hit the Apply button to confirm and then click Test Configuration to validate communication between Commvault and your SCOM server is working as expected.

When you receive confirmation that the test was successful, hit the Import Management Pack button to begin the import of the unsealed management pack into SCOM.

When the process is complete, you should see a status message similar to the one in the image below that confirms the Commvault management pack has been configured...

A quick check of the Installed Management Packs view in the SCOM console confirms the management pack has been imported and is ready to go...

You should now see the four simple alert views under the CommVault Operations Manager folder in the Monitoring workspace as shown here....

If you want to confirm the new class has been created and discovered, scope your Discovered Inventory view to CommVault CommServer and you should then see all monitored Commvault servers that SCOM knows about.

Opening a Health Explorer view from the newly discovered CommVault CommServer class object shows how basic this management pack actually is with just the one Service Running State monitor in place to let you know the health state of the Commvault Windows Service.

A quick jump over to the Authoring workspace and we can see the three new Commvault alert rules that have been imported (these rules all target the new Commvault CommServer class).

A check of the Data Source properties for each of the rules gives us the location and CSV file name that will be used to collect alert information from the Commvault server...

Each rule's Data Source has been configured with a wildcard Expression value relevant to the type of alert that will fire (e.g. *Critical*, *Warning* or *Informational*).

If you want to change the name or alert description format of the alert response, you can do that from the Alert properties as shown here...

Configuring the Integration

Once the management pack has been imported and your Commvault servers have been discovered, launch the CommCell Browser again, click Alert from the navigation bar and click the Configure Alert option as shown in the following image...

When the Alerts window opens, you'll be presented with a list of all enabled and disabled alerts in Commvault. We'll click the Add button here to begin the process of creating an alert for SCOM.

From the Add Alert Wizard, type a name for the SCOM alert then choose a category and type. In our example we'll create an alert called Failed Backups and we'll choose the Job Management category with a type of Data Protection.

When you're ready, click Next to move on.

At the Entities Selection window, choose the client groups and/or clients that this alert will be scoped to then hit Next to continue.

From the Threshold and Notification Criteria Selection window, use the Alert Criteria section to scope the alert to the criteria that you need. In our example, we're only interested in Job Failed, Job Skipped and Job Succeeded with Errors alerts. Ignore the other options outside the Alert Criteria section and click Next to move on when you've made your criteria selections.

At the Notification Type(s) Selection window, click the SCOM tab then enable the Select [SCOM] for notification check box as shown in the following image...

Hit Next to continue.

At the Token Criteria Selection window you can optionally add rules to the alert that will dictate if the alerts are sent or not. You can get a full list and description of the alert tokens from here.

We won't specify any rules in our example and when you're ready, click Next to move on.

From the Security window, use the Add button to specify the user accounts and groups that you wish to grant permissions for the alert to (we'll configure an admin account with the Alert Owner role for this alert).

Click Next to move on and at the Summary window (shown in the image below), confirm your settings and hit Finish to end the wizard.

Back in the Alerts view of the CommCell Browser, you can check that the new alert has been created and is enabled as shown below...

That's all you should need to do to configure the integration between Commvault and SCOM and the next time an alert condition has been met, you should see the alert dropping into the Monitoring workspace of the SCOM console similar to this one...

If you've create a new distributed application model in SCOM for Commvault and you use either the Windows Computer or CommVault CommServer class in your component groups, these alerts will rollup to change the health of the model as expected.

Conclusion

Using the walk-through in this post should help people get up and running when monitoring Commvault with SCOM and with some additional distributed application service modeling, SLA planning and dashboard design, you can get some really nice visibility of your backup environments all from a single console.

SCOM - New Community MP for Monitoring Windows Services

A couple of months ago I came across a blog post from SCOM community contributor Andy Leibundgut describing a new management pack he'd authored to help monitor Windows Services using SCOM and I was keen to give it a test drive to see exactly what it could do.

Built-in Windows Service Monitoring Option

You might be thinking to yourself that this capability has to be available already out of the box with SCOM and you'd be correct. The Windows Service Monitoring Template (accessible from the Authoring workspace in the SCOM console and shown in the image below) will walk you through a wizard to help configure a custom monitor for a Windows Service that might not be automatically monitored with a vendor management pack.

The Problem

This template wizard certainly meets the requirements of bringing Windows services into SCOM but as Andy points out in his post, there's a lot of monitoring bloat that comes with each service monitoring configuration you create using the template - each service has its own class, its own discovery and comes with 3 monitors and 8 overrides!

Also, if you have a large number of custom Windows services to monitor (a common request from my customers), then using the built-in template in the console to create a monitor for each one can be fairly time-consuming.

The Solution

With these points in mind, Andy looked into a better method of spinning up monitors in SCOM for Windows services without having to deal with the extra bloat and cumbersome on-boarding process for multiple services. The solution he came up with comprises a new management pack and a PowerShell-based Service MP Editor complete with an easy-to-follow user interface (shown below).

Along with the Service MP Editor, the new management pack contains the following features for monitoring Windows Services:

• Uses the same data source for Windows Service monitoring that SCOM uses.
• Date and time filtering so you can exclude certain days/times from monitoring on a per-service or service object basis.
• Handy console tasks for starting, stopping and checking the status of the Windows Service.
• Automatic service recovery (disabled by default). Works on a 3 strikes and you’re out format (overrideable setting where after 3 failures in a 24 hour period it will stop trying to restart the service). 
• Timer reset monitor (closes itself after 24 hours and enabled by default) to watch for and alert on the 3 strike out situation.
• Monitor all service startup types with the exclusion of disabled services from alerting.
• Custom discovery which discovers and adds all the service objects to one class rather than scattering them about like the templates do.

Taking the MP for a Test Drive

I've been running this MP solution in my demo and semi-production SCOM environments (both 2012 R2 and 2016) for the last couple of months with no issues and I felt it was about time to spread the word on how much effort you can save when you use it.

The first thing you'll need to do is to download the latest version of this management pack and you can get it from the TechNet Gallery here.

Next up, you'll want to get yourself a coffee (or beer, if that's how you roll) and take a read through Andy's original blog post using the link below:

https://leibundgutski.wordpress.com/2017/02/19/scom-windows-service-monitor-management-pack/

Note: Everything you need to know is available in Andy's blog post and for clarity, I'll just blog my own experience on getting the MP up and running here.

When you download the zip file containing the MP, extract it to a location on a computer that runs the SCOM console and you should see the following three files...

The Readme.txt contains a note from the author highlighting the fact that this MP is still in it's early days of development and that you should always test it in a non-production environment first.

The WindowsServiceMonitor.xml file is the unsealed MP used for monitoring your Windows Services and the ServiceMPEditor.ps1 file is a clever PowerShell script that launches the UI-based editor to help you customize the MP for your own (and customer) environments.

Before we go any further, we need to import the WindowsServiceMonitor.xml file into SCOM using the Import Management Packs option from the Administration\Management Packs area in the console as shown here....

Next up, we'll create a temporary folder on the same computer that you've just used the console to import the management pack with. We'll create a folder in C:\Temp\SCOM (shown in the image below) but you can use whatever path you wish for this.

Once the folder has been created, launch a PowerShell window with administrative permissions and run the ServiceMPEditor.ps1 script to open the Service MP Editor similar to the following image...

Now, there's a specific order of steps that you need to follow when entering information into the Service Editor and for this part, I've borrowed the original numbered step-by-step image from Andy's blog post that should make things easy to understand...

If you want to add monitors for a small number of Windows Services, then follow these steps in order (we'll cover importing a larger list of services later):

1. Management Server – type the name of one of your SCOM management servers in this field.
2. Management Pack Location – type the location of the temporary folder that you created earlier (we'll use C:\Temp\SCOM) into this field.
3. Get MP Config – clicking this button will export a copy of the original WindowsServiceMonitor.xml management pack from your SCOM environment into the temporary folder location specified in the previous step.
4. New Service – a click of this button will ready the Service Name field to allow you start a new service configuration.
5. Service Name – it's imperative that you type the exact 'Service name' of the Windows Service in here and not the 'Display Name' (refer to the example in the previous image taken from Andy's blog where he has highlighted the Service name for the Print Spooler service - which is simply named Spooler).
6. Confirm Service Edit – clicking this after you've specified the Service name and are finished choosing all your service monitoring options.
7. Save MP Config – click this button when you're finished editing and ready to commit your changes to the management pack.

Here's a screenshot of what the editor looks like in my demo environment where I've added three services (Windows Firewall, Windows Time and Print Spooler).

After you've clicked the Save MP Config button, you can close the editor and check that the C:\Temp\SCOM\WindowsServiceMonitor.xml management pack contains the newly added services...

Now re-import the management pack from your temporary location back into SCOM using the Import Management Packs from the console. You'll get a notification stating that the management pack is already installed and you can just ignore this and hit the Install button to re-import it again as shown here...

Once the management pack completes it's discovery, you should be able to see the newly monitored services light up in the Discovered Inventory view from the Monitoring workspace of the console (make sure to change the target for this view to WindowsService as shown below).

If you're impatient like me and don't want to wait for the discovery to kick in automatically (by default it's configured to run once a day), you can either reduce the discovery time with an override or simply bounce the Microsoft Monitoring Agent service on the server(s) running the service that you want monitored and you should then see the Discovered Inventory view populating like this...

To test the management pack, stop one of your newly monitored services and after a minute or so, you should see the service roll up as a critical state to the Windows Computer object that's hosting it as shown here in this diagram view...

The nice thing with this management pack is that it comes with some custom tasks to help you manage your monitored services and clicking the Start Service task from the pane on the left (shown below), will then restart the problematic service for you without the need to logon directly to the computer!

Note: You could also enable the automatic WindowsService.ServiceStart.Recovery task option from the Diagnostic and Recovery tab of the monitor to get SCOM to restart the service automatically itself in the event of it stopping unexpectedly. This recovery task will restart the service automatically up to 3 times before giving up and alerting you to the fact that the service is constantly being stopped and started.

Importing a Custom List of Services

If you have a large list of Windows Services that you want to monitor and don't fancy having to manually enter each one into the Service Editor, then there's a handy Import Services option that allows you to import a CSV file with the list of custom services that you want monitored.

You need to understand the format the CSV file needs to be in first however as if you get it wrong, you'll end up having to either edit the XML file directly or just start the whole process again.

Here's the steps I went through in the editor to get this bulk import option working:

1. Management Server – type the name of one of your SCOM management servers in this field.
2. Management Pack Location – type the location of the temporary folder that you created earlier (we'll use C:\Temp\SCOM) into this field.
3. Get MP Config – clicking this button will export a copy of the original WindowsServiceMonitor.xml management pack from your SCOM environment into the temporary folder location specified in the previous step.
4. Import Services –  this button is used to select a CSV file that contains a list of Windows Service names that you want to monitor. The CSV file must be named WindowsServiceMonitor.csv and needs to be located in the temporary folder location specified in the Management Pack Location field.

Here's a screenshot of the steps you need to take in the editor when you want to import a list from CSV....

When you click the Import Services button, you'll be presented with the warning below stating the name and location that your CSV file needs to have...

Assuming you've named your CSV file correctly and copied it to the temporary location specified in the warning dialog, click Yes to continue and you'll be presented with the custom list of services to be monitored as specified in the CSV you previously created.

CSV Creation Tip #1 - The  CSV file needs to be formatted with four column names (Service, Start, End, DaysofWeekMask) and you need to specify values similar to the image below...

CSV Creation Tip #2 - An easy way to quickly grab a CSV file in the correct format is to just manually add one or two Windows Services to the Service Editor and then use the Export Services button to export a template CSV file that you can edit as you need.

CSV Creation Tip #3 - You can export a full list of Windows Service names to a CSV file by using the following PowerShell command (this exported CSV file won't be in the correct format for the Service Editor so you'll need to then copy/paste the names from here into the previously created WindowsServiceMonitor.csv file):

Get-Service | Sort-Object -Property DisplayName | Export-CSV -path C:\winserviceexport.csv

When you're satisfied with the imported bulk list of Windows Services, the final step you need to complete in the editor is to hit the Save MP Config button to commit the changes to the MP.

Now all that's left to do is to close the editor and re-import the newly updated MP from your temporary location back into SCOM.

After the discovery process completes, you can see the full list of monitored services from the Discovered Inventory view similar to my demo environment here...

All credit goes to Andy Leibundgut for his contribution of this management pack to the SCOM community - and make sure to leave any comments you have on bugs or suggested improvements directly on his original blog post here.

Cloud and Datacenter Conference 2017

Conference season is beginning to kick off again and in just over four weeks time (4th - 5th May), I'll be presenting a session at the Cloud and Datacenter Conference in Munich.

The conference is organised by Rachfal IT Solutions and it's the brainchild of my good friend and well-known MVP Carsten Rachfal (aka @hypervserver). It's my first time presenting at this event and it'll also be my first time in Munich so I'm really looking forward to it.

There's a long list of awesome speakers presenting over the two days that reads like a who's-who of Cloud and Datacenter specialists from across the MVP and Microsoft world.

My presentation will be in the Chicago room on the Thursday and it's titled 'What's New in OpsMgr 2016?'. In the session, I'll demonstrate new features, enhancements and administration tips to help make your OpsMgr 2016 environments rock.

Here's a snippet of the agenda for the room I'll be presenting in...

You can check out the full conference schedule here.

Browsing through the schedule, there's a really good mix of sessions delivered in either German or English and I've already bookmarked a number of them to attend so I can learn from the experts in area's like Hyper-V, Azure Stack, Azure and OMS.

If you're in Germany and haven't registered yet, head over to https://www.cdc-germany.de/ and get your tickets before it sells out as from what I've heard about the venue, food, content and networking at last years event, it's going to be epic!

SCOM 2016 Agent Crashing Legacy IIS Application Pools

SCOM 2016 has been generally available since late last year and as is usually the case with new versions of software, compatibility issues begin to rear their heads as more organizations begin to adopt it.

During one of our recent SCOM 2016 deployments we encountered an issue where the agent (referred to as the Microsoft Monitoring Agent) was deployed to an IIS server - initially without any apparent problems. However, when the IIS server was restarted some time later to accommodate some Windows updates, the IIS Application Pools began to crash regularly. A check of the Windows Event Log on the server threw up the following Event ID 1000 error:

Log Name: Application

Source: Application Error

Date: 24.02.2017 10:42:30

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: H-SPDEMO01.nimbuscorp.com

Description:

Faulting application name: w3wp.exe, version: 8.0.9200.16384, time stamp: 0x50108835

Faulting module name: PerfMon64.dll, version: 8.0.10918.0, time stamp: 0x577fd168

Exception code: 0xc0000409

Fault offset: 0x0000000000149794

Faulting process id: 0x2c38

Faulting application start time: 0x01d24405d195eb6a

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon64.dll

Identifying the Issue

The 'Faulting Module Path' in the above application error pointing to the Application Performance Monitoring (APM) component of the agent was the first give-away to us that SCOM was the culprit. A quick uninstall of the SCOM 2016 agent and recycle of the Application Pools gave us confirmation when the errors and crashes went away.

The Microsoft Monitoring Agent APM component comes bundled in the form of a Windows service as part of the initial agent installation but is disabled by default as shown below:

APM is typically enabled through the SCOM console on a server-by-server basis and delivers some really nice DevOps scenarios for monitoring .NET workloads at a code level.

I've previously blogged about SCOM APM, presented on it at conferences and even wrote a chapter in the Mastering SCOM 2012 R2 book about it. A number of our customers also use this feature and it's always been very successful.

The weird thing about this particular IIS crashing issue though was that the APM feature was never enabled.

We needed to dig deeper to see how we could continue monitoring this server using SCOM without having the IIS Application Pools crashing and as the faulting module path in the Event Log error referenced the APM component, I decided to focus here first.

If you use the command line to install your SCOM agents, you can specify a parameter that removes the APM feature from the agent installation (check out this link for command line options) and I figured this was the best place to start as the IIS server in question didn't require the APM feature.

Removing the Agent APM Feature

In the following steps, I'll walk you through a process to remove the APM feature on the SCOM 2016 agent using the command line. The first walk-through will perform an in-place repair on an existing agent to save you from having to uninstall the agent first. An added benefit of the repair option is that the agent will stay registered as Remotely Manageable in the console and you'll avoid having to follow this process to change them.

The second walk-through will show you how to use the command line to perform a new agent installation that doesn't contain the APM feature.

(Repair Agent Install Option)

Copy the SCOM 2016 Agent installation folder (amd64) from your SCOM server to the IIS server (this folder is located at "C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement")

Log on to the IIS server and open a command prompt using an administrative account. From there, browse to the location where you saved the SCOM 2016 Agent folder to and run the following command:

msiexec.exe /i momagent.msi NOAPM=1

This command will then launch the Microsoft Monitoring Agent Setup installer shown in the following image..

Click Next and if you've already installed the SCOM 2016 agent to your IIS server, then you'll be presented with the Program Maintenance window shown below.

Select the Repair option and hit Next to move on.

Hit Install at the next window and the agent repair should kick off. After a minute or so, the agent repair job will be complete and you'll be presented with the following confirmation of success...

Now if you open the Windows Services (services.msc) snap-in and check the services listed for the Microsoft Monitoring Agent, you'll see that the APM component is no longer installed as shown here..

With the agent sucessfully repaired, it'd be a good idea to check the Update Rollup version of the agent and if needs be, to re-apply the latest one (UR2 at this time). You can check the UR version of your agent by importing the awesome SCOM Agent Version Addendum Management Pack from Microsoft's Kevin Holman.

Recycle the IIS Application Pools on your IIS server to ensure the new agent changes take affect.

(New Agent Install Option)

If you've already uninstalled the SCOM 2016 agent from your IIS servers or haven't yet deployed it, then follow these steps to get it deployed without the APM feature:

Copy the SCOM 2016 Agent installation folder (amd64) from your SCOM server to the IIS server (this folder is located at "C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement")

Log on to the IIS server and open a command prompt using an administrative account. From there, browse to the location where you saved the SCOM 2016 Agent folder to and run the following command:

msiexec.exe /i momagent.msi NOAPM=1

This command will then launch the Microsoft Monitoring Agent Setup installer where you will need to click Next and then hit the I Agree button in the following window to accept the license agreement.

At the Destination Folder window, confirm the installation path for the agent and click Next to continue.

When you see the Agent Setup Options window, select Connect the agent to System Center Operations Manager (shown below), then hit Next.

At the Management Group Configuration window, fill in the information required to connect the agent to your SCOM environment and remember that the Management Group Name field is case sensitive!

Click Next to continue and at the Agent Action Account window, leave the Local System option selected, then hit Next again.

Review your installation settings at the Ready to Install window, then click Install to deploy the agent without the APM feature.

When the agent has installed, open the Windows Services (services.msc) snap-in and check the services listed for the Microsoft Monitoring Agent, you'll see that the APM component is no longer installed (see image below).

As this is a new manually installed agent, you will need to change the Remotely Manageable status of the agent back to Yes by following the steps in Kevin Holman's post here (although this post references SCOM 2007, the steps are still the same for SCOM 2016).

You will also need to install the latest update rollup - which is currently at UR3 - and if you've set the Remotely Manageable status back, you should be able to push the update rollup out from the SCOM console. Make sure to reference the SCOM Agent Version Addendum Management Pack to deliver easier visibility of your agent UR versions.

Recycle the IIS Application Pools on your IIS server to ensure the new agent changes take affect.

***Update 20th August 2017 - Handy Tip for Reinstalling Agents: Microsoft's Kevin Holman and Brian Barrington have come up with a nice workaround to easily reinstall your agents with the NOAPM switch. Check it out here.***

More Information About This Issue

This issue was the first time in years that I've encountered a scenario where the SCOM agent 'broke' something and as such, I wanted to investigate it a bit further and raise it with Microsoft. One of the massive benefits of being a Microsoft MVP for me is that I get the opportunity to interact with the SCOM Product Group on a regular basis.

After a few emails back and forth, the awesome folks on the Product Group came back to me with the following detailed information about the issue:

• Issue affects only IIS Application Pools running .NET Framework 2.0/3.5 and can be seen on any version of Windows Server or IIS that hosts these pools.
• Switching the IIS pool to .NET Framework 4.0 (or higher) will solve the issue however, this is not a suitable workaround for SharePoint as SharePoint 2010 doesn't support 4.0 pools.
• If you need to deploy to the system with IIS running pools 4.0+ - no action is needed and a default installation of the SCOM 2016 Agent will work fine.
• For now, if you need to deploy to a server running .NET Framework 2.0/3.5 application pools, then you'll need to either install the SCOM 2016 agent with the NOAPM=1 switch (following my walkthrough above) or you can continue to use the SCOM 2012 R2 agent as it’s forward compatible with SCOM 2016 and doesn't crash these application pools.
• A permanent fix for this issue should be included in Update Rollup 3 for the SCOM 2016 agent.

***Update March 21st 2017: The Product Group have also just released a blog on this issue and in their post, they have confirmed that it will be resolved in Update Rollup 3 along with a chance that a hotix may be released sooner. Check out their blog post here.***

***Update 24th May 2017: Microsoft has just released Update Rollup 3 and you can read all about it in my post here.***

***Update 31st May 2017: Microsoft has just posted that this agent crash issue is still not resolved with UR3 (not cool...). I'll update here when we have more info***

***Update 6th June 2017: Microsoft has posted more information about this issue remaining after UR3 and have mentioned a hotfix is still in the works. Check out their latest post here.***

Conclusion

Although this issue has been an annoyance, it's good to know that it only affects a small subset of legacy systems and the workaround is relatively simple to implement. Hopefully this blog post will help to serve people who've already deployed SCOM 2016 (or who are about to deploy it) and need to monitor legacy IIS application pools prior to the release of UR3.