SCOM and the Exchange Test CAS Connectivity User Account

This is just a quick post to highlight an issue that I've seen in 3 out of the last 5 SCOM installations that I have been involved with that have Exchange 2010 as their messaging environment.

A pre-requisite for the Exchange 2010 Management Pack is that a test mailbox account be setup for synthetic transactions using an existing Exchange 2010 powershell script that comes bundled with the Exchange 2010 installation called:

'New-TestCasConnectivityUser.Ps1'

Below are the instructions from the Exchange 2010 Management Pack guide on how to create the test account using the above script:
 
In this procedure you create test mailboxes for Outlook Web App, Exchange ActiveSync, and Exchange Web Services to monitor connectivity by using PowerShell to run the New-TestCasConnectivityUser.ps1 script.

1. Open the Exchange Management Shell.

2. In the Shell, change directory to the C:\ Program Files\Microsoft\Exchange Server\V14\Scripts folder by running the following command:

Set-Location “C:\Program Files\Microsoft\Exchange Server\V14\Scripts”

3. Run the test-user script using the following command:

New-TestCasConnectivityUser.ps1

4. Follow the on-screen installation instructions in the Shell to create the test mailbox. You'll be prompted to enter a temporary secure password for creating test users. You'll also be prompted to specify the Mailbox server where you want the test user created. 
5. Repeat this process on an Exchange 2010 Mailbox server in each Active Directory site that you want to test.

When following these instructions, you might get hit with the following error:

CreateTestUser : Mailbox could not be created. Verify that OU (Users) exists and that password meets complexity requirements.

The solution that I have found to this problem has been to modify the New-TestCasConnectivity.Ps1 script and remove the parameter '-OrganisationalUnit:$OrganisationalUnit'

This parameter is responsible for specifying the OU that the new account will be created into and when it is removed from the script, the script simply runs and installs the account into the default 'Users' OU as it should.

I came across this solution from the following blog post:

http://www.definit.co.uk/2011/03/exchange-2010-createtestuser-mailbox-could-not-be-created-verify-that-ou-users-exists-and-that-password-meets-complexity-requirements/

Hope this helps!

SCOM 2007 R2 and McAfee - Workflow Initialization Failed to start a workflow that runs a process or script

I've been working on a new SCOM 2007 R2 implementation this week for a large customer in Dublin that are using McAfee VirusScan Enterprise + AntiSpyware Enterprise 8.8 as their anti-virus solution and ran into an annoying problem / bug that needed attention to allow SCOM to work the way it should.
The issue started appearing shortly after I had added an additional SCOM Management Server (MS) to work alongside the SCOM Root Management Server (RMS). I hadn't got to the stage of deploying any agents to any other servers within the environment at this point as I wanted to concentrate on patching, dashboard customisation and custom configuration first.

The customer is using the McAfee EPO to auto push out their policies to all new servers and clients within the network and it was when the McAfee VirusScan Enterprise + AntiSpyware Enterprise 8.8 client went onto each of the two SCOM servers, that I started to see the error below:

Workflow Initialization: Failed to start a workflow that runs a process or script

Data was found in the output, but has been dropped because the Event Policy for the process started at 14:20:31 has detected errors.
The 'ExitCode' policy expression:
[^0]+
matched the following output:
-1073741819

Command executed: "C:\Windows\system32\cscript.exe" /nologo "MemoryUtilization.vbs" 2.5 SCOM-RMS.contoso.com 5841.
Working Directory: C:\Program Files\System Center Operations Manager 2007\Health Service State\Monitoring Host Temporary Files 15\2564\

One or more workflows were affected by this.

Workflow name: Microsoft.Windows.Server.2008.OperatingSystem.MemoryAvailableMBytes

Instance name: Microsoft Windows Server 2008 R2 Datacenter

This was one of many similar errors all referencing different workflows and indicating that any scripts that SCOM attempted to run were being blocked by the Anti-Virus client (McAfee in this case).

A quick check on the McAfee configuration indicated that the client was using both the 'On Access Scanner' and the 'Access Protection' scanner too.

The difference between these two components is that the 'On Access Scanner' is similar to all standard types of A/V on-access scanners and subject to the same exclusion lists and configurations (screenshot below)

But the 'Access Protection' scanner is more designed to block 'Anti-Spyware' applications, mass mailing worms, IRC etc. (See screenshots below)

Initially we thought that it was the 'Access Protection Anti Spyware Maximum Protection' setting that was blocking the scripts from running as it has an option of 'Prevent execution of scripts from the temp folder' and this was enabled, so we disabled this particular feature for the SCOM servers and waited to see if the errors would return (restart the SCOM health service on any of the servers to kick off the scripts again). Within 5 minutes however, the script errors came back and we needed to dig deeper!

I found a few pointers on the internet that were of similar issue but generally these just recommended turing off the 'Access Protection' option altogether for SCOM servers which isn't really an option in a high security environment!!

I came across a post on the McAfee forum that indicated this was a known issue with McAfee and in particular with version 8.8 of the engine. A quick check to confirm the customers McAfee engine again and it was indeed version 8.8!! (see below)

Most posters on the forum had to call McAfee to get the issue resolved and this resolution came around in the form of a custom SDAT file that isn't readily available on the internet but one poster who had the issue managed to resolve the problem by unregistering a DLL file that was related to a feature called 'ScriptScan' within the A/V client.

The 'ScriptScan' feature (it's part of the 'On Access Scanner') was one of the first things I had initially checked for this problem as it would be the most obvious but had found that it was disabled and the customer had never enabled this through the EPO policies so I had presumed it wasn't the cause.

The bug however lies in the fact that although the 'ScriptScan' feature may appear to be disabled on the clients, it still has it's DLL file registered and this is what was causing the problem.

Once I ran the command to unregister the DLL file on each SCOM server and rebooted the SCOM health service on each one, the error didn't come back at all!

The issue now is going to be that this DLL will have to be unregistered from each server that will have the SCOM agent installed for monitoring to allow it to work properly.

Here's the command to manually unregister the DLL on each server:

cd "C:\Program Files\Common Files\McAfee\SystemCore"

regsvr32.exe /u SCRIPTSN.dll

I've also created a really basic .bat file that you could run on a large number of servers through group policy that should automatically unregister this DLL file for you, here's the link to the file on my SkyDrive:

http://sdrv.ms/10QfLJ8

Once this DLL has been unregistered, restart the SCOM health service (if you've already installed the SCOM agent on the server) and the errors should disappear forever!!

As a reference, here are some links for some more information on SCOM Anti Virus Exclusions and recommendations although none of them are official and some quite old:

http://blogs.technet.com/b/kevinholman/archive/2007/12/12/antivirus-exclusions-for-mom-and-opsmgr.aspx

http://blogs.msdn.com/b/nickmac/archive/2008/07/18/antivirus-exclusions-for-operations-manager-2007.aspx

McAfee's Stance on SCOM Exclusions!

https://kc.mcafee.com/corporate/index?page=content&id=KB59944&cat=INTEROPERABILITY&actp=LIST

Download the Hyper V Virtual Machine Mover Tool

I played with the idea of joining the Twitter revolution for over a year now and finally took the plunge and signed up last weekend (@kgreeneit). The main purpose of joining wasn't to tell the world about what I had for breakfast, but more to do with the fast flow of information within the IT industry that kept passing me by when I was busy and didn't have a chance to read through the hundreds of RSS feeds on my laptop!

An excellent example of the power of Twitter for me was in coming across this blog post today about a utility that has been around for a while but I'm only hearing about now!

The utility is called the 'Hyper-V Virtual Machine Mover Tool' and is available for free on 'Codeplex'.

This utility is to be used as a much faster alternative to the 'Export/Import' options that come as standard with  Hyper-V and are specifically for non-clustered environments - otherwise we'd just use Live Migration!

The download link is directly below this and I've also included some links to some more detailed blog posts around the features of this cool utility:

Download
http://vmmove.codeplex.com/

Lai Yoong Seng Blog Post
http://www.ms4u.info/2011/06/moving-virtual-machine-without-using.html

Jermone Laban's Original Blog Posting and creator of this utility
http://www.jaylee.org/page/Hyper-V-Virtual-Machine-Mover.aspx


Enjoy!

Using The Active Directory Recycle Bin to Recover a Cluster Name Object (CNO) for Windows Server 2008 R2 Failover Clusters

This is another excellent post that I came across this weekend while catching up on whats been happening in the blog world!

Last year I had to recover a customers Hyper V cluster as an admin onsite in the customers premises decided to do a little 'Active Directory Tidy Up' and deleted the Cluster Name Object (CNO) for the Windows 2008 R2 Failover Cluster!!

I found this Technet article at the time and after reading through it I wanted to give it a go but unfortunatley the customer site didn't have a Windows Server 2008 R2 domain controller and I ended up just rebuilding the cluster (it was only a 2 node Hyper V with all the VM's backed up with DPM).

I remember thinking how useful and time-saving the Active Directory Recycle Bin would be if I was faced with a similar issue again but then completly forgot all about it until I read these blog posts from Chuck Timon and  John Marlin (Senior Support Escalation Engineers with Microsoft).

This is going to become a staple part of any new Windows Server 2008 R2 Domain Controller installations that I do because the amount of time that can be saved with this process is unbelievable when compared any of the alternatives.

I highly recommend anyone reading this to add it as a standard part of their Windows 2008 R2 installations in future and even would go so far as to recommend adding it in now to any Windows 2008 R2 Domain Controllers on your customer sites.

Here are the links to both blog postings, the original one being authored by Chuck Timon in 2009 when nobody really had Windows Server 2008 R2 in mainstream use and the second and more recent one authored by John Marlin, who goes deep into the Active Directory Recycle Bin utility and how to recover from it.

Chuck Timon
http://blogs.technet.com/b/askcore/archive/2009/04/27/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster.aspx

John Marlin
http://blogs.technet.com/b/askcore/archive/2011/05/18/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster-part-2.aspx

Now, no more excuses for having to spend two days onsite rebuilding a cluster due to the CNO being deleted by some 'smart' thinking sys admin!

Enjoy!

The 'Unoffical' System Center Management Pack Catalog

I came across this blog posting yesterday when trying to catch up on all of my rss feeds and thought it was worth posting as sometimes trying to find a SCOM Management Pack or Opalis (Orchestrator) Integration Pack can be difficult.
The author of the blog has compiled a list of hardware and software vendors that supply SCOM and Opalis (Orchestrator) MP's and IP's and it could be a useful tool when trying to locate that hard-to-find Management Pack.

Update Jan 2013: I'm updating the link below to reflect some updates by the author and a new link

Here is the link to the 'Unofficial' System Center Management Pack Catalog:

SCOM
http://unofficialsystemcentercatalog.wordpress.com/2012/12/27/operations-manager-is-there-a-management-pack-for/

Opalis (Orchestrator)
http://unofficialsystemcentercatalog.wordpress.com/2011/04/22/opaliscatalog/

Microsoft have just revamped their Pinpoint website to make it much easier to find the Management Pack that you need - the previous implementation looked good but was much harder to find the MP you were looking for and a lot of the 3rd party MP links were outdated too with older Management Packs of no use.

The link to the new Microsoft Pinpoint website is below:

http://systemcenter.pinpoint.microsoft.com/en-US/Default.aspx

Download System Center Operations Manager R2 Admin Resource Kit

Microsoft just yesterday release the new System Center Operations Manager R2 Admin Resource Kit - better late than never I suppose!

The feature set includes the following:

Scheduled Maintenance Mode

Ability to schedule and manage maintenance mode in the management group.

Clean Mom

Helps remove all installed R2 components.

MP Event Analyzer

MP Event Analyzer tool is designed to help a user with functional and exploratory testing and debugging of event based management pack workflows like rules and monitors.


Download the resource kit from the URL below:

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=e37ab90b-ec7b-4130-8e3f-9691a76530db&displaylang=en