Showing posts with label Clustering. Show all posts
Showing posts with label Clustering. Show all posts

Sunday, June 5, 2011

Using The Active Directory Recycle Bin to Recover a Cluster Name Object (CNO) for Windows Server 2008 R2 Failover Clusters

This is another excellent post that I came across this weekend while catching up on whats been happening in the blog world!

Last year I had to recover a customers Hyper V cluster as an admin onsite in the customers premises decided to do a little 'Active Directory Tidy Up' and deleted the Cluster Name Object (CNO) for the Windows 2008 R2 Failover Cluster!!

I found this Technet article at the time and after reading through it I wanted to give it a go but unfortunatley the customer site didn't have a Windows Server 2008 R2 domain controller and I ended up just rebuilding the cluster (it was only a 2 node Hyper V with all the VM's backed up with DPM).

I remember thinking how useful and time-saving the Active Directory Recycle Bin would be if I was faced with a similar issue again but then completly forgot all about it until I read these blog posts from Chuck Timon and  John Marlin (Senior Support Escalation Engineers with Microsoft).

This is going to become a staple part of any new Windows Server 2008 R2 Domain Controller installations that I do because the amount of time that can be saved with this process is unbelievable when compared any of the alternatives.

I highly recommend anyone reading this to add it as a standard part of their Windows 2008 R2 installations in future and even would go so far as to recommend adding it in now to any Windows 2008 R2 Domain Controllers on your customer sites.

Here are the links to both blog postings, the original one being authored by Chuck Timon in 2009 when nobody really had Windows Server 2008 R2 in mainstream use and the second and more recent one authored by John Marlin, who goes deep into the Active Directory Recycle Bin utility and how to recover from it.

Chuck Timon
http://blogs.technet.com/b/askcore/archive/2009/04/27/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster.aspx

John Marlin
http://blogs.technet.com/b/askcore/archive/2011/05/18/recovering-a-deleted-cluster-name-object-cno-in-a-windows-server-2008-failover-cluster-part-2.aspx

Now, no more excuses for having to spend two days onsite rebuilding a cluster due to the CNO being deleted by some 'smart' thinking sys admin!

Enjoy!

Friday, April 8, 2011

New SCOM Exchange 2010 SP1 Management Pack

Microsoft have just released (well 2 days ago but I didn't get around to blogging until today about it!), the updated Microsoft Exchange 2010 Management Pack which includes support for Microsoft Exchange 2010 Service Pack 1 and the associated roll-ups that SP1 comes with.

There are quite a few significant changes to this Management Pack that makes it even better than the original Exchange 2010 MP that was released last year.

Here's some of the highlights that are included in this new update:

  • Capacity planning and performance reports - New reports dig deep into the performance of individual servers and provide detailed information about how much capacity is used in each site. 
  • SMTP and remote PowerShell availability report - The management pack now includes two new availability reports for SMTP client connections and management end points. 
  • New Test-SMTPConnectivity synthetic transaction - In addition to the inbound mail connectivity tasks for protocols such as Outlook Web App, Outlook, IMAP, POP, and Exchange ActiveSync, the Management Pack now includes SMTP-connectivity monitoring for outbound mail from IMAP and POP clients.
  • New Test-ECPConnectivity view - Views for the Exchange Control Panel test task are now included in the monitoring tree. 
  • Cross-premises mail flow monitoring and reporting - The Management Pack includes new mail flow monitoring and reporting capabilities for customers who use our hosted service. 
  • Improved Content Indexing and Mailbox Disk Space monitoring - New scripts have been created to better monitor context indexing and mailbox disk space. These new scripts enable automatic repair of indexes and more accurately report of disk space issues. 
  • The ability to disable Automatic Alert Resolution in environments that include OpsMgr connectors -    When you disable Automatic Alert Resolution, the Correlation Engine won't automatically resolve alerts. This lets you use your support ticketing system to manage your environment.
  • Several other updates and improvements were also added to this version of the Management Pack, including the following. 
  • Suppression of alerts when the alerts only occur occasionally was added to many monitors. 
  • Most of the event monitors in the Exchange 2010 Management Pack are automatically reset by the Correlation Engine. Automatic reset was added to those event monitors so that issues aren't missed the next time they occur.
  • Monitoring was added for processes that crash repeatedly. 
  • Additional performance monitoring was added for Outlook Web App. 
  • Monitoring of Active Directory access was improved. 
  • Monitoring of anonymous calendar sharing was added. 
  • Reliability of database offline alerts was improved. 
  • Monitoring for the database engine (ESE) was added.

Also, Microsoft had promised that they would update their documentation guide for the original Exchange 2010 MP to include information on making the Microsoft Exchange Monitoring Correlation Service Highly Available in a cluster environment but I didn't see anything in the new and updated guide in relation to this. See my earlier blog posts on this if you want to make the Correlation Service Highly Available:


If you are one of the many thousands who have already deployed the original SCOM Exchange 2010 MP, then you will need to download the new updated one from the link below and import it into your SCOM infrastructure as you had done previously with the older MP (you can't update this MP from the SCOM Catalog automatically - it needs to be downloaded first and then have it's .MSI file run to accomodate the Correlation Service installation)



As always, don't forget to read the MP guide fully before you install the MP into your live environment and implement any recommendations that the guide gives around configuration and alert tuning.

Happy SCOMming!!!


Thursday, April 7, 2011

Clustering the SCOM Microsoft Exchange Monitoring Correlation Service Part 2

In part 1 of this guide, I explained what the Microsoft Exchange Monitoring Correlation Service was and in this part, I will demonstrate what is needed to make this service Highly Available.

Follow the steps below to begin your configuration:

  • Run the .msi installer for the Exchange 2010 MP on all cluster nodes first as this creates the ‘Microsoft Exchange Monitoring Correlation’ service
  • Go to the ‘services.msc’ snapin on each cluster node and stop the ‘Microsoft Exchange Monitoring Correlation’ service 
  • Set the startup type for this service on all nodes to ‘Manual’ 
  • Edit the CONFIG file located at 'C:\Program Files\Microsoft\Exchange Server\V14\Bin\Microsoft.Exchange.Monitoring.CorrelationEngine.exe' for the Microsoft Exchange Monitoring Correlation Service to reflect the Virtual RMS Cluster name (it is set as 'Localhost' initially and this is what needs to be changed)

  • Provision a LUN from your shared storage and present it to the cluster nodes (1GB or 2GB should be more than enough)
  • From one of the cluster nodes, open 'Failover Cluster Manager' from the 'Administrative Tools' menu, expand your cluster name, right mouse click on 'Services and Applications' and then click on 'Configure a Service or Application' to open the 'High Availability Wizard'

  • Select 'Next' and then click on 'Generic Service'

  • Click 'Next again and then select 'Microsoft Exchange Monitoring Correlation' from the 'Select Service' window

  • Click 'Next and then input in the name and IP address that you are going to assign to the newly clustered service

  • Select your shared storage that you had presented to the cluster previously

  • Don’t bother selecting any registry settings to transfer over as the service is already installed on the other nodes. 
  • Click on ‘Next’ until you come to the review page, then click on ‘Finish’ to complete the service clustering. 
  • Test service cluster failover by moving the service between nodes using ‘Failover Cluster Manager’ and keep an eye on the service state from the ‘services.msc’ snapin as this should go from started to stopped on the current node, then from stopped to started on the new node you have chosen to move the service to. 

This completes your Microsoft Exchange Server 2010 Monitoring Correlation Service clustering.

All that’s left to do now is to import the Exchange 2010 MP into your environment and then tune away those noisy alerts (if any!)



Clustering the SCOM Microsoft Exchange Monitoring Correlation Service Part 1

It's not often I get to install SCOM into RMS clusters and this week was no exception. I'm finishing off a project that involves the installation of a SCOM 2007 R2 Highly Available environment configured as an RMS cluster on an SQL 2008 R2 cluster.

See the link below for the initial SCOM cluster installation steps:

http://kevingreeneitblog.blogspot.com/2011/02/clustering-scom-2007-r2-rms-role-on.html

The part of this project I am working on this week is based around a new Highly Available Exchange 2010 messaging environment and bringing this into the existing SCOM infrastructure.

Now, for those of you that are used to installing the Exchange 2010 Management Pack, you will be familiar with a new method that Microsoft have been pursuing in relation to management packs, and this is based around the new 'Microsoft Exchange Monitoring Correlation Service'.

The 'Microsoft Exchange Monitoring Correlation Service' is basically a noise reduction utility that sits between the Exchange 2010 and SCOM environments and filters out unnecessary alerts before you even need to start thinking about tuning!

It's based on nearly 2 years of in-house deployment from the MSIT team on their own SCOM and Exchange 2010 environment.

The Microsoft Exchange Monitoring Correlation Service installs itself as a standard Windows Service on the SCOM RMS and is viewable from within the 'services.msc' snap-in.

When I install a Management Pack into SCOM, no matter how often I have installed the same MP into other SCOM environments, I will always read the MP guide that accompanies the Management Pack as it can contain valuable information on configuration and initial tuning settings that need to be deployed to get the most out of your new MP.

When I started to read through the Exchange 2010 MP guide, I was looking specifically for instructions on clustering the Correlation Service as there's no point in having a HA environment for your RMS and SQL services and then not having the main noise reduction service for your Exchange 2010 MP not failing over to your other cluster nodes in the case of an emergency. The Exchange 2010 MP guide unfortunately has no information on clustering this service at all and after searching the web for some additional info, all I found was some references to it being added to the next version of the Exchange 2010 MP guide - whenever that gets released!! I set about clustering this service using the Failover Clustering Wizard that is built into Windows Server 2008 R2 and decided to blog about it for anyone else who finds that the Microsoft documentation falls just short of this type of information.

In part 2 of this short guide, I will demonstrate how to cluster this service and make it Highly Available within your SCOM environment.

Saturday, February 26, 2011

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 6)

Welcome to the final blog posting in this 6 part series on 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster'. If you have come to this post before reading over the previous 5 posts in the series, I've included links for all 5 of them below:

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 1)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 2)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 3)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 4)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 5)

Following the instructions detailed in my previous 5 posts, you should now have your SQL 2008 R2 Cluster built and your SCOM 2007 R2 RMS Cluster built and tested. This post will outline some troubleshooting steps to take in the event of an issue with your newly built SCOM 2007 RMS Cluster and also explain how to install SCOM 2007 R2 Reporting into an SQL 2008 R2 environment.

Preparing an Inaccessible Management Server

The following procedure is required only if, in bringing the cluster group online, the output stated that you are required to run the ManagementServerConfigTool.exe tool by using the AddRMSNode action on any of the non-root management server cluster nodes. This is most likely caused by a cluster node that is not accessible when the InstallCluster action was executed or you are adding a new node to the cluster.

1. Log on to the computer that hosts the management server as a member of the Administrators group.

2. Open the services snap-in and if the startup type for the System Center Data Access Service is set to Disabled, change it to Manual.

3. As an administrator, open a Command Prompt window , change directories to the installation folder, and type the following:

ManagementServerConfigTool.exe AddRMSNode /vs:<VirtualServerNetbiosName> /Disk:<VirtualServer Disk Resource>

VirtualServerNetbiosName is the Network Name resource allocated to the same cluster group. The value you enter for VirtualServerNetbiosName must be the value that appears in the Name box on the General tab of the Properties dialog box for the Network Name Cluster resource.

VirtualServerDiskResource is the disk resource allocated to the cluster group being used to create this Virtual root management server. The Disk location can be found in the results pane of the properties for the RMS application.


Installing SCOM 2007 R2 Reporting into an SQL 2008 R2 Environment

If you are installing the SCOM 2007 R2 Reporting Services into an SQL 2008 R2 environment, ensure you have renamed the local SQL group by removing the _50 section from its name on the SCOM server as outlined near the start of this document (see below for a reminder):

  • On the SQL Server Reporting Services server, rename the local group SQLServerReportServerUserlt;hostname>$MSSRS10_50.<SQLInstanceName> to SQLServerReportServerUserlt;hostname>$MSSRS10.<SQLInstanceName> 

Once the SCOM 2007 Console installation is completed, re-run the installer again and select the ‘Install Operations Manager 2007 R2 Reporting’ option

Within the ‘Operations Manager 2007 Reporting Setup’ window, when selecting components to install, DO NOT install the ‘Data Warehouse’ component if you are installing into a Windows SQL 2008 R2 instance, if installing into an older version of SQL, then leave all components selected and continue on

When prompted, enter the credentials for the ‘Data Warehouse Write Account’ (srv_scom_datawrite) and select ‘Next’

When prompted, enter the credentials for the ‘Data Warehouse Read Account’ (srv_scom_dataread) and select ‘Next’

Move through the remaining menu’s selecting the default options until you see the ‘Install’ button and then press this to initiate the reporting services installation

When the installation of SCOM 2007 R2 Reporting Services is completed, you then need to rename the local group modified at the start of this page back to what it was beforehand (example below):

  • SQLServerReportServerUserlt;hostname>$MSSRS10.<SQLInstanceName> back to original name SQLServerReportServerUserlt;hostname>$MSSRS10_50.<SQLInstanceName
This completes the installation of SCOM 2007 R2 Reporting Services into an SQL 2008 R2 environment and also brings to an end the final chapter of my 6 part series on 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster'.

Hopefully somebody will find this series helpful and it will assist you in working through the numerous pitfalls of carrying out this type of installation.




Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 5)

We're into the penultimate post of this series on 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster'. In this part, I will detail how to create the actual RMS Cluster and once it is created, I will outline some basic tests to ensure you're configuration is working as it should do.

Before you read through this section, please make sure you've read Part 1Part 2 ,Part 3 and Part 4 of this series first to ensure you have your environment ready to create the SCOM 2007 R2 RMS cluster.

Creating the RMS Cluster 

In this procedure, you distribute the RMS Key to the secondary management servers and create the RMS cluster. At the end of this procedure, all nodes in a cluster can host the RMS.

Before you start


Ensure that the file share with the encryption key is accessible by all cluster nodes. This is used for distributing the RMS Key.

Creating the Virtual RMS


1. Log on to the computer that owns the RMS application with an account that is a member of the Administrators group.

2. If you successfully backed up the encryption key at the end of RMS setup, you can skip to step 5.

3. On the RMS owning node, as an Administrator, open a Command Prompt window and change current directory to the Operations Manager installation folder; for example, cd\Program Files\System Center Operations Manager 2007.

4. To back up the RMS Key, type the following, where <fileshare> is a share accessible to all cluster nodes:

SecureStorageBackup.exe Backup \\<fileshare>\<filename>.bin.

Note
This will launch the Encryption Key Backup or Restore Wizard. A password will be requested. It must be at least eight characters long and must include at least one symbol. You must confirm the password to create the encryption key file.


5. Log on to each secondary management server computer with an account that is a member of the Administrators group.

6. On each secondary management server, navigate to the Operations Manager installation directory and launch SecureStorageBackup.exe as an administrator.

7. This will launch the Encryption Key Backup or Restore Wizard. On the Introduction page, click Next and then select the ‘Restore’ option.

8. On the Provide a Location page, type in the path or browse to the encryption key file and click Next.

9. On the Provide a Password page type in the password that you used when you backed up the encryption key and click Next.

10. Click Finish.

11. Log on to the RMS computer with an account that is a member of the Administrators group.

12. In Failover Cluster Management, expand the cluster, and ensure that the RMS application is owned by node 1.

13. On the SQL Server-based computer that hosts the OperationsManager database, open the SQL Server Management Studio tool, open the Databases folder, and select the OperationsManager database. Right-click to open the context sensitive menu and select Tasks, Back Up to initiate a backup. On the Back Up Database - OperationsManager page, ensure that the Backup type value is set to Full, give the Backup set an appropriate name, and set the Backup set will expire value to a date in the distant future. In the Destination box, for the Back up to value, select Disk and add an appropriate disk location to hold the backup, if one is not already present, and then click OK.

Important
When you run the ManagementServerConfigTool to create the RMS cluster, you are advised to back up the OperationsManager database because irrecoverable damage can be done by creating the RMS cluster if something is done incorrectly.



14. On the RMS server, as an Administrator, browse to the installation media for SCOM 2007 R2 and in the ‘Support Tools’ folder, you will find the ‘ManagementServerConfigTool.exe’ file. Copy this file to the same location on the C drive that you ran the ‘SecureStorageBackup.exe’ tool from (this location is generally a default of ‘C:\Program Files\System Center Operations Manager 2007’)

Once you have copied the tool to the location as above, open an Elevated Command Prompt window, type cd <path to Operations Manager installation directory>, and then press ENTER.

15. To instantiate the RMS cluster group as a cluster, type the following, where G is the disk resource that is allocated to the cluster group that is being used to create this virtual Root Management Server and where <VirtualServerNetbiosName> is the network name resource allocated to the same cluster group:

ManagementServerConfigTool.exe InstallCluster /vs:<VirtualServerNetbiosName> /Disk:G

The value you enter for <VirtualServerNetbiosName> must be the value that appears in the Name box on the General tab of the Properties dialog box.

Note
ManagementServerConfigTool.exe InstallCluster will install the RMS as a clustered service on every available node in the cluster.



Note
When you run the ManagementServerConfigTool, the output might display instructions for running the SetSPN command; these instructions can be ignored.


16. In Failover Cluster Management, right-click the RMS application to open the context menu and select Bring Online to bring all the RMS applications online.

17. Open Failover Cluster Management and right-click the RMS application to open the context menu. Select Move this service or application to another node and select the next node in the cluster. Repeat this so that the RMS application is moved to each node of the cluster.

Important
The RMS application must be moved and come online successfully on each cluster node to set the state of the services correctly on each node at this time. Do not skip this step.


Clustered RMS setup is complete.


Testing the Cluster Installation

Use the following procedure to test the cluster installation.

1. In the Operations console, click Administration.

Note
When you run the Operations console on a computer that is not a management server, the Connect To Server dialog box appears. In the Server name text box, type the name of the RMS Server (the cluster virtual server name) that you want the Operations console to connect to. 

2. In the Administration pane, point to Administration, point to Device Management, and then click Management Server.

3. In the management servers pane, the RMS Server Network Name should appear with a health state of Healthy.

4. In the Administration pane, click Agentless Managed.

5. In the Agentless Managed pane, the entry for each node in the cluster should appear with a health state of Healthy.

This concludes Part 5 of my 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster' series of posts. In the final post of this series, I will explain how to install SCOM 2007 R2 Reporting into an SQL 2008 R2 environment and also some troubleshooting steps if you are having issues with your newly created RMS Cluster.

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 4)

Welcome to Part 4 of my 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster' blog series. Before you read through this section, please make sure you've read Part 1, Part 2 and Part 3 of this series first to ensure you have your environment ready to create the SCOM 2007 R2 RMS cluster resources outlined below.

In this part of the series, we will create the SCOM 2007 R2 RMS Cluster Group and RMS Cluster Group Resources.

To prepare the Cluster Nodes, the RMS Cluster Group, and the RMS Cluster Group resources: 


1. On each RMS cluster node, ensure that the domain Operations Manager Administrators security group has been added to the local Administrators group.

2. Ensure that each cluster node meets the prerequisites for the management server and User Interface components:

  • Windows Server 2003 SP1 or Windows Server 2008 
  • MDAC version 2.80.1022.0 or later 
  • .NET Framework version 2.0 
  • .NET Framework version 3.0 features 
  • WSMAN v 1.1 (this is only required if UNIX/Linux computers will be monitored in this management group). 
3. Add the Data Access and Config service account to the Local Administrators group on each node of the RMS cluster.

4. Log on to the cluster node that will be the primary owning node for the RMS with administrative rights.

5. Start the Failover Cluster Management tool from Administrative tools.

6. If this is the first time that the Failover Cluster Management tool has been run, you will be prompted to connect to a cluster. Select the Manage a cluster option from the Action drop-down box and either enter or browse for the cluster name for the Cluster or server name box.

7. In the Failover Cluster Management tool, right-click the Services and Applications folder to open the context menu and click Configure a Service or Application to start the High Availability Wizard.

8. On the Before You Begin page, click Next.

Note
If you have chosen a different port for SQL Server communications and have already configured that in SQL Server, you should enter that value here; otherwise, accept the default of 1433. If you have installed SQL Server using a named instance, type in the dynamic port value.

On the Select Service or Application page, select Other Server and click NEXT.

9. On the Client Access Point page, type in the network name that you have planned for your rms. This name will be registered in DNS as an A record.

10. Click the address box and type the IPv4 address you have planned for the rms. This is the publicly accessible address for the rms.

11. Click Next.

12. On the Select Storage page, select the disk resource that will be used for the rms. This should not be the quorum disk.

13. Click Next.

14. On the Select Resource Types page, click Next.

15. On the Confirmation page, review the information and click Next.

16. On the Summary page, optionally review the report and click Finish.

17. Right-click the application that you just created and open its properties. On the General tab, optionally select a preferred owner node and on the Failover tab, accept the default failover values and ensure that the Prevent failback option is selected.

18. Click OK.


Installing the RMS

In this procedure, you install the first management server in the management group (the RMS).

To prepare the Cluster and install RMS and User Interfaces Components:

1. Log on to the cluster node that will be the primary owning node for the RMS with administrative rights.

2. On your installation media, start SetupOM.exe. This starts the System Center Operations Manager 2007 R2 Setup Wizard on the Start page.

3. Under the Install heading, click Install Operations Manager 2007 R2. This starts the Operations Manager 2007 R2 Setup Wizard.

4. On the Welcome page, click Next.

5. On the End User License Agreement page, select the I accept the terms in the license agreement option, and then click Next.

6. On the Product Registration page, enter the appropriate values in the User Name and Organization fields. Enter your 25-digit CD Key, and then click Next.

7. On the Custom Setup page, leave the management server and User Interfaces options set to This component, and all dependent components, will be installed on the local disk drive. Set the Database, Command Shell and Web Console components to This component will not be available, accept the default installation location, and then click Next.

8. On the SQL Server Database Instance page, type the SQL Server name and database instance in the SQL Database Name box. This is in the format of SQL Server\SQL Instance. If the SQL Server database was installed in the default instance, you only need to enter the SQL Cluster name that was created when you installed SQL Server 2008 in the cluster.

9. Check that the SQL Database Name field reads OperationsManager.

10. Check that the SQL Server Port field has the value of 1433.

Note
If you have chosen a different port for SQL Server communications and have already configured that in SQL Server, you should enter that value here; otherwise, accept the default of 1433. If you have installed SQL Server using a named instance, type in the dynamic port value.

11. Click Next.

12. On the Management Server Action Account page, accept the default Domain or Local Computer Account option, enter the credentials of the MSAA, and then click Next.

Note
By using a domain-based account, it will be much easier to perform discovery and push agent installation later on than if you chose the Local System account.

13. On the SDK and Config Service Account page, select the Domain or Local Account option, enter the credentials for the Data Access and Config service account, and then click Next.

Note
In this configuration, the account must be a domain account, because reporting is installed on a separate server. This account must have permissions on the reporting system.

Note
If you receive an Account Verification Error when you click Next, it is most likely that you mistyped the credentials or the SDK and Config service account was not added to the local Administrators group.

14. On the Customer Experience Improvement Program page, optionally indicate whether you want to join this program, and then click Next.

15. On the Microsoft Update page, optionally indicate whether you want to use the Microsoft Update services to check for updates, and then click Next.

16. On the Ready to Install the Program page, click Install when you are ready for the installation to proceed.

17. On the Completing the System Center Operations Manager 2007 R2 Setup Wizard page, clear the Start the Console check box, ensure that the Back up Encryption Key check box is selected, and then click Finish. The Encryption Key Backup or Restore Wizard will now launch.

Important
Even though the Operations console has been installed, do not launch the console at this point. Clear the Launch the Operations Console check box to prevent the Operations console from launching.

Note
If setup fails, it provides you with a value to search on and a link to open the setup log.

18. On the Introduction page of the Encryption Key Backup or Restore Wizard, click Next.

19. On the Backup or Restore? page, select Backup the Encryption Key option, and then click Next.

20. On the Provide a Location page, specify a valid path and file name for the encryption key and click Next.

Important
It is critical that the location provided for backing up the encryption key be accessible to all nodes in the cluster.

21. On the Provide a Password page, enter a password to secure the encryption key backup file and click Next to start the backup process. You will be prompted for this password when you restore the RMS encryption key later in this procedure.

22. You should see the Secure Storage Backup Complete page. Click Finish.

Note
Be sure to copy the encryption key to a location that is accessible by all computers that will be management servers. Also be sure to make multiple copies and store them in separate, secure locations.


Installing the Secondary Management Servers (Additional RMS Cluster Node Members)

In this procedure, you will install secondary management servers on all other nodes in the cluster. These servers are secondary management servers until this process is complete, at which time they will be able to host the root management server.

To install secondary management servers in the RMS cluster:
1. Log on to each remaining cluster node with the Operations Manager Administrator account.

2. Follow the ‘Install RMS’ procedures to install the management server and User Interface components on each of the other nodes in the management group.

3. Do not start the console.

Note
If you choose to install any management server without the User Interfaces component and you want to run SecureStorageBackup.exe, you must copy Microsoft.EnterpriseManagement.UI.ConsoleFramework.dll, Microsoft.Mom.UI.Common.dll, and SecureStorageBackup.exe from the installation media (make sure to copy the correct platform version x86 or x64) to the installation directory on the management server. Typically this is C:\Program Files\System Center Operations Manager 2007. 

Preparing RMS Cluster Resources

In this procedure, you create cluster resources out of the System Center Management service (HealthService), the System Center Management Configuration service (OMCFG), and the System Center Data Access service (OMSDK). These are the RMS resources that can fail over between cluster nodes along with the network name, IP address, and physical disk.

1. Log on to the node that is the owner of the RMS application with an account that has administrative rights.

2. In Failover Cluster Management, right-click the RMS application in the navigation pane, and then select Add a resource 4-Generic Service.

3. On the Select Service page, select the System Center Management Service, and click Next.

4. On the Summary page, click Finish.

5. In the summary pane of your RMS application, right click the System Center Management resource and open its properties.

6. On the Dependencies tab, click Insert, and select the shared disk that you prepared for the RMS cluster from the Resource list.

7. Click Insert again and select the network name from the Resource list.

8. Click Apply.

9. On the General tab, select the Use Network Name for computer name option.

10. Click Apply.

11. Click OK

12. Repeat the same process for the System Center Management Configuration service and the System Center Data Access service.

That concludes Part 4 of this series of posts on 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster'. In Part 5 we will create the RMS Cluster and test the Cluster configuration.

Friday, February 25, 2011

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 3)

If you have followed Part 1 and Part 2 in this series, you will have your SQL 2008 R2 Cluster built and your Active Directory service accounts and SPN permissions configured correctly.

In this part of the series, we will manually create the two databases that SCOM uses into the SQL 2008 R2 cluster. We need to manually create these databases instead of letting the SCOM installation wizard create them due to the fact that SQL 2008 R2 is only recently supported by Microsoft and the SCOM installation wizard will not recognize the SQL 2008 R2 instances.


To check prerequisites for OperationsManager and Data Warehouse database installation:

1. Log on to Node 1 of the cluster that is to be used for SQL Server 2008 with the Operations Manager Administrator account credentials.

2. In the Failover Cluster Management tool navigation pane, expand the cluster that you are installing the OperationsManager and Data Warehouse databases on.

3. Expand the Services and Applications container and select your SQL Server instance.

4. In the results pane, ensure that all your SQL Server resources are owned by Node 1 and that they are online. The required resources are:

  • Server Name and IP Address 
  • Disk Drives 
  • SQL Server 
  • SQL Server Agent 

5. Ensure that the Data Transaction Coordinator is owned by and available on Node 1.

6. Close the Failover Cluster Management Tool.


Creating the SCOM Databases in SQL 2008 R2:

Follow the procedure below to manually create the databases:

How to use the DBCreateWizard tool to install the OperationsManager database:

Note 1 - You must run the DBCreateWizard tool on the server that is running SQL Server 2008 R2.

Note 2 - If User Account Control (UAC) is enabled, this needs to be turned off first, otherwise the DBCreateWizard.exe tool will give an error and not proceed

 Open the <InstallationMedia>\SupportTools\AMD64 folder

(Note <InstallationMedia> represents installation media of System Center Operations Manager 2007 R2.) 


  • Double-click the DBCreateWizard.exe file to start the Database Configuration Wizard.
(Note You may receive an error message when you start the wizard. For more information, click the following article number to view the article in the Microsoft Knowledge Base: 938997 http://support.microsoft.com/kb/938997)
  • On the Welcome page, click Next. 
On the Database Information page, perform the following operations:

  • Under Database Type, select Operations Manager Database.
  • Under Full Database Instance Name, select the instance of SQL Server where you want the database to be installed. 
(Note An instance of SQL Server has the name of <ServerName>\<InstanceName>. The default instance has the name of <ServerName>. )

  • Leave the Create New Database option selected. 
  • Leave the SQL Port box blank. 
(Note If you do not provide a port, the Database Configuration Wizard uses the port 1433 to connect to SQL Server. However, if the instance of SQL Server that you specify uses a port other than 1433 for connection, you must specify the port in the SQL Port box. )

  • By default, the database name is OperationsManager under Database Name. 
  • By default, the database size is 500 megabyte (MB) under Database Size. 
  • Click Next. 


On the Management Group page, perform the following operations:
  • Under Management Group name, specify a name for the management group. 

  • By default, the BUILTIN\Administrators user group is specified under Configuration Administrator. It is recommended that you select your ‘srv_scomadmin_group’ here and ensure you have added the domain administrator account to this group too if you want to be able to open the console using the domain admin account otherwise only Local Administrators specified on this computer will be able to open the console. 
  • If you want to specify a different user group, follow these steps: 
       Click Browse.

       In the Select Group dialog box, click Locations.

       In the Locations dialog box, select the domain and then click OK.

       Under Enter the object name to select, type the user group,then click Check Names.

       If the name that you typed becomes underlined, click OK.

       On the Error Reporting page, indicate whether you want to send error reports, click Next

       On the Summary page, review the configuration summary, and then click Finish.

How to use the DBCreateWizard tool to install the OperationsManagerDW database:

Note 1 -You must run the DBCreateWizard tool on the server that is running SQL Server 2008 R2.

Note 2 - I ran into problems when installing the DW database using the DBCreateWizard and specifying a database size larger than around 30GB. The SQL server kept timing out the connection and in the end, I installed the OpsMgr DW DB specifying a 30GB size, then used SQL Management under the DB properties/files section to increase the size of the MDF and LDF files to what I wanted to

To create the OperationsManagerDW database, simply follow the instructions from above on creating the OperationsManager database only this time, you will be specifying the db name of 'OperationsManagerDW' instead.

This concludes part 3 of my 'Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster' series. In Part 4, I will go through the creation of the RMS cluster group and resources.

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 2)

If you have followed Part 1 of this series of posts, then at this point you will have your SQL 2008 R2 cluster built and tested. This will allow us to now move on with the configuration of the SCOM 2007 R2 RMS cluster configuration.



How to Cluster the SCOM 2007 R2 RMS Role into an existing SQL 2008 R2 Cluster


  • Install Windows Server 2008 R2 Enterprise onto two nodes
  • Add at least 2 NIC’s to each node
(You can add more NIC's if you want to specify a Heartbeat network but this is optional)
  • Configure NIC 1 with a local IP subnet for the domain to be used for Management
  • Configure NIC 2 with an IP for your ISCSI / Fiber subnet
  • Present the ISCSI or Fibre Channel storage LUN and Quorum to each node (i.e. 1 x StorageLUN, 1 x Quorum)
  • Install Windows Failover Clustering on each node

Service Account Preparation:

Create the 5 SCOM user accounts in Active Directory following the guide below:       

To prepare accounts and groups in Active Directory:


·    In Active Directory Users and Computers, create five accounts: the Management Server Action account, the SDK and Configuration Service account, the Data Reader account, the Data Warehouse Write Action account, and an Operations Manager Administrator account. These can all be domain user accounts. No special privileges are required at the domain level. Try to stick to the same naming convention for each new installation of SCOM using similar account to these:

  •       srv_scom_action (SCOM Action Account)
  •       srv_scom_sdk (SCOM SDK Account)
  •       srv_scom_dataread (SCOM Data Warehouse Read Account)
  •       srv_scom_datawrite (SCOM Data Warehouse Write Account)
  •       srv_scom_admin (SCOM RunAs Admin Account)
(If you have a domain password expiration Group Policy in place and you do not want to change these service account passwords on the same schedule, select Password never expires for the individual accounts.)


  • In Active Directory Domain Services, create a Global Security group for the Operations Manager Administrators. If you plan to use of any of the other Operations Manager 2007 R2 roles, create e-mail-enabled Global Security groups for those also.


Use similar to:   srv_scomadmin_group (SCOM Administration Security Group)


  • Add the Operations Manager Administrator Account (srv_scom_admin) to the Operations Manager Administrators Global Security group.

To prepare accounts and groups on the Operations Manager server:

  • On the server that you are going to install Operations Manager on, log on with an account that has local administrator rights.
  • In the Computer Management tool, under Local Users and Groups, open the Administrators group and add the Operations Manager Administrators Global Security group that you created in step 2 of "To prepare accounts and groups in Active Directory." Also add the accounts that you created to use as the Management Server Action account, the SDK and Config account, the Data Reader account, and the Data Warehouse Write Action account.

To configure the SDK Service Account to create SPNs dynamically, follow these steps:

1. Click Start, click Run, type Adsiedit.msc, and then click OK.

2. In the ADSI Edit snap-in, expand Domain [DomainName], expand DC= RootDomainName, expand CN=Users, right-click CN= AccountName, and then click Properties.

Notes
  • DomainName is a placeholder for the name of the domain. 
  • RootDomainName is a placeholder for the name of the root domain. 
  • AccountName is a placeholder for the account that you specify to start the SDK service. 
  • If you specify the Local System account to start the SDK service, AccountName is a placeholder for the account that you use to log on to Microsoft Windows. 
  • If you specify a domain user account to start the SDK Service, AccountName is a placeholder for the domain user account. 

3. In the CN= AccountName Properties dialog box, click the Security tab.

4. On the Security tab, click Advanced.

5. In the Advanced Security Settings dialog box, make sure that SELF is listed under Permission entries.

(If SELF is not listed, click Add, and then add SELF. )

6. Under Permission entries, click SELF, and then click Edit.

7. In the Permission Entry dialog box, click the Properties tab.

8. On the Properties tab, click This object only in the Apply onto list, and then make sure that the check boxes for the following permissions are selected under Permissions:

  • Read servicePrincipalName 
  • Write servicePrincipalName 

9. Click OK three times, and then exit the ADSI Edit snap-in.

That completes Part 2 of this blog series on Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster. In Part 3 we will discuss how to manually create the SQL 2008 R2 databases SCOM requires using the 'DBCreateWizard' utility.

Thursday, February 17, 2011

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 1)

I've been quite busy for the last 3 weeks getting involved in a lot of System Center Operations Manager projects and I've decided to post  my experiences on the initial installation of SCOM for one particular client who wanted the SCOM RMS role clustered (no problem) but onto an SQL 2008 R2 failover cluster (maybe a slight problem!).

This is a fairly new configuration to have to install into a production environment due to the fact that Microsoft have only started supporting SQL 2008 R2 databases with SCOM 2007 R2 in the last few months.

I am going to be posting this installation over a number of blog posts as there is quite a lot of information to take in.

See below for the links to the other posts in this series:

Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 2)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 3)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 4)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 5)
Clustering the SCOM 2007 R2 RMS Role on a Microsoft SQL 2008 R2 Cluster (Part 6)

We'll start this first post by giving a High-Level overview of how to install an SQL 2008 R2 cluster from scratch.

How to Cluster SQL 2008 R2 in a Windows Server 2008 R2 Environment

  • Install Windows Server 2008 R2 Enterprise onto two nodes

  • Add at least 2 NIC’s to each node

  • Configure NIC 1 with a local IP subnet for the domain to be used for Management

  • Configure NIC 2 with an IP for your ISCSI / Fiber subnet

Note at this point, you will be creating 2 cluster resources (MSDTC and SQL Cluster) so you need to ensure you have provisioned enough LUN’s to present to these. One LUN for Quorum is enough.

  • Present the ISCSI or Fibre Channel storage LUN and Quorum to each node ensuring you have provisioned separate disks for the MSDTC and SQL Cluster Resources (i.e. 1 x MSDTC, 1 x SQL DB, 1 x SQL Logs, 1 x Quorum)

  • Add the .Net Framework 3.5 SP1 from the ‘Features’ section before you install SQL 2008 R2
Read the information supplied in the link below before installing any cluster components:
  • Install Windows Failover Clustering on each node
  • Create the cluster within Windows Failover Clustering ensuring you use unique IP addresses and names for each cluster resource (These names will be registered in DNS for Cluster communication)
  • Create a Cluster resource for the MSDTC component
  • Create the SQL Server service accounts within Active Directory
  • Enable SQL Server service accounts permissions to read and write SPN’s within Active Directory.

Follow this link to enable Read and Write of SPN’s on the newly created Service Accounts:

If the above step relating to SPN's is not followed, then you will need to make the registry modification below to disable LoopBack Check and stop an error occurring during setup of the SQL cluster.

Carry out the procedure below to stop an error when installing a SQL Cluster instance:

1.Disable the authentication loopback check by setting the DisableLoopbackCheck value in the
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa registry subkey to 1

To set the DisableLoopbackCheck registry entry to 1, follow below steps on all nodes of cluster.
     a. Click Start, click Run, type regedit, and then click OK.

     b. Locate the following registry path:
 
         HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
     c. Right-click Lsa, select New, and then click DWORD Value.

     d. Type DisableLoopbackCheck, and then press ENTER.
     e. Right-click DisableLoopbackCheck, and then click Modify.

     f. In the Value data box, type 1, and then click OK.

2. Restart the system.

Installing SQL Server

***VERY IMPORTANT NOTE***
Ensure when you are installing your SQL Server that you specify a Collation Setting of 'SQL_Latin1_General_CP1_CI_AS' otherwise you will end up with a problem described in this blog posting:


Install SQL Server 2008 R2  from the installation media following the step-by-step instructions in the link below and making sure that you select 'New SQL Server Failover Cluster Installation' on the 'Installation' tab of the intial SQL splash screen:



Once SQL 2008 R2 is installed and up and running, make sure you then download Cumulative Rollup 5 for SQL 2008 R2 and install it as soon as possible. This rollup resolves issues where the OperationsManager Database is housed on an SQL 2008 R2 database and it causes the CPU usage to rocket.

Here's a really good link on this to follow:

http://blogs.technet.com/b/kevinholman/archive/2011/02/04/if-you-are-using-sql-2008-r2-for-opsmgr-db-s-you-need-sql-2008r2-cu5.aspx


Uninstalling a SQL 2008 R2 Cluster
If you want to uninstall a SQL 2008 R2 Cluster, you must first run the ‘Remove Cluster Node’ wizard from the SQL 2008 R2 media and ensure that it is run initially on all nodes that are not the owner of the cluster resource(s).

  • Once all but the last cluster nodes have been removed from the SQL cluster, then ensure that the last SQL server is the owner of the SQL Cluster Resource and that no other SQL cluster nodes exist, then run the ‘Remove Cluster Node’ wizard from here and it will remove the last node from the cluster

  • When you have removed all the SQL cluster nodes, you can then run the SQL uninstaller from the ‘Add/Remove’ programs menu to fully remove all SQL components.

This concludes the first part of this blog post and you should now have your SQL 2008 R2 Failover Cluster built and tested before commencing onto the SCOM side of things.

Wednesday, December 29, 2010

Automatically Live Migrate Multiple Virtual Machines in a Hyper V Cluster

Over the Christmas break, I decided to use the time off to carry out some essential maintenance of our 3 Node Hyper V Failover Cluster that we currently have in our datacentre (I know, sad as I am working over the holidays!).

What I wanted to do was to update all of the Hyper V hosts with the latest HP Support Packs and Firmware updates, install the latest Windows Updates and Patches and also run scripts to identify which hotfixes are missing that are specific to Hyper V, Clustering and SCVMM (see this link for a really handy script that I recommend any Hyper V admin to use regularly - http://kevingreeneitblog.blogspot.com/2010/10/hyper-v-and-scvmm-missing-updates.html).

With all of these updates to be applied to the Hyper V hosts, it means in that although there will be no need for downtime of the virtual machines on each host (approx 15 VM's on each one), I will still need to 'Live Migrate' these machines to the other Hyper V hosts before I add the updates and reboot the hardware.

This 'Live Migrate' process can be a tedious process as unlike VMWare's VMotion, you cannot 'Live Migrate' any more than one virtual machine at a time from a physical host to another - you can 'Live Migrate' more than one virutal machine at a time within a Hyper V cluster but not more than one VM from each physical host. For example, I can migrate a VM from Host 1 to Host 2 and at the same time, I can migrate a machine from Host 3 to Host 4 as long as each host only has one 'Live Migration' occuring at any given time.

With a large amount of VM's to 'Live Migrate', clicking on each VM one at a time and waiting for it to migrate before you can do the next one is a time consuming process and a waste of valuable time over the Christmas holiday period!

As previously mentioned, although you cannot 'Live Migrate' more than one virtual machine at a time in a Hyper V cluster from one host to the other, there are several ways that you can automate this process with a couple of clicks of a mouse that will systematically move each virtual machine to a different host without the need for you to do each VM one by one.

My preferred method of automating this process requires that you have Microsoft's System Center Virtual Machine Manager 2008 R2 managing your Hyper V cluster (shame on you if you don't!).

I remembered reading through the SCVMM documentation a while back and coming across a feature for host management called 'Maintenance Mode'. Maintenance Mode allows you to specify if a host is going to be offline within your Hyper V cluster and when you enable Maintenance Mode on a particular Hyper V host, a wizard pops up explaining that this process will automate the 'Live Migration' of your virtual machines to another physical host. Just the solution I was looking for!

Simply browse to the 'Hosts' section from within SCVMM, right mouse click on the host, select 'Maintenance Mode' from the menu and then follow the wizard to move all machines automatically.

There is another way to automatically move all the Virtual Machines from one Hyper V host to another but that involves pulling the power cables from the back of the host and I really don't recommend to do that....................

Now, back to enjoying the time off work!

Thursday, December 16, 2010

Troubleshooting 'Redirected Access' on a Cluster Shared Volume (CSV)

Here's a really interesting post from Chuck Timon - Microsoft Enterprise Platforms Support Senior Support Escalation Engineer - surrounding the dreaded 'Redirected Access' message that can appear sometimes (rarely for me thankfully!) on a Hyper V Failover Cluster.

The post covers 4 reasons as to why this message will appear on your CSV and the solutions to diagnose and bring the CSV back online.

Here's the link:

http://blogs.technet.com/b/askcore/archive/2010/12/16/troubleshooting-redirected-access-on-a-cluster-shared-volume-csv.aspx

Tuesday, November 16, 2010

Increase Exchange 2010 DAG Failover Threshold

One of our clients has a 4 member node Exchange 2010 DAG spread across 4 different countries worldwide.

The client had reported to me that one of the sites that had a slight bandwidth issue was consistently failing it's Active Mailbox Store from the local site over to it's Dublin HQ site. When we manually moved the database back over to the original local site, it would randomly fail back over to the main Dublin HQ site presumably due to the intermittent latency on the Internet connection at that local site.

The customer requested that I find a way to increase the failover threshold or tolerance for the DAG so that it doesn't fail over as frequently without losing the functionality of High Availability.

After searching for quite a while on how to do this using Exchange Power Shell I found some information relating not to Exchange Server but to the Windows Server 2008 Cluster Service (which is essentially what the DAG uses when it is created for the first time) for it's clustering technology.

Using a standard Command Prompt (cmd), I started playing with the 'cluster' command and looking into what switches it used and what they could be applied to.

Here's what I came up with:

Type 'cluster /list' to display the name of the cluster that is present on the Server

When you run a 'cluster /prop' from the cmd line, it returns a number of values relating to the cluster, two of which are the following:

CrossSubnetDelay = 1000 (this is the default 1000 milliseconds which equals 1 second per heartbeat check)

CrossSubnetThreshold = 5 (this is the default number of heartbeats that can be missed before failover)

I changed the CrossSubnetDelay value to make the heartbeat check in every 2 seconds instead of the default 1 second by using the command below:

cluster /cluster:<ClusterName> /prop CrossSubnetDelay=2000

With this new setting along with the default value of 5 seconds for the CrossSubnetThreshold setting, this now allows the Cluster service to wait for 10 seconds before initiating a failover to a different DAG member.

This value can be increased to a maximum of 4000 milliseconds once the cluster is across subnets (it is a maximum of 2000 milliseconds if you are on the same subnet)

The CrossSubnetThreshold value can be modified with a value anywhere from 3 to 10.

This workaround / solution may need some tweaking with values until you reach the desired tolerance on your DAG.

It is also worth making sure you make a note of all changes that you make before and after the above commands and as always - make sure you have a full backup of your Exchange environment before you do anything like this!!!!