Failed Cluster While Backing Up Windows Server 2012 Hyper-V CSV's

Last week I had the opportunity to do something other than SCOM and get my hands dirty with a new Windows Server 2012 Hyper-V cluster project for one of our customers.

The build went well and I used the excellent Hyper-V Installation and Configuration Guide book along with this blog post on Windows Server 2012 Hyper-V Best Practices (In Easy Checklist Form) to double and then triple check all was deployed to best practice recommendations.

After we P2V'd a number of the customers physical machines and configured the DPM 2012 SP1 backup application to perform host-level backups (i.e. backing up the entire virtual machine while it's still running using the agent deployed to a Hyper-V host), we started seeing problems with performance on the cluster nodes and then ultimately inside each virtual machine running on the cluster.

Leaving the cluster in this state resulted in both hosts becoming unresponsive and everything grinding to a halt. We had to power cycle down the SAN (FC connected HP) and both hosts to get everything back online - not cool :(

So, after a small bit of searching for an answer we came across the following two recently released Hotfixes from Microsoft that needed to be applied to the Hyper-V hosts:

• Update improves cluster resiliency in Windows Server 2012 (KB2848344)
• Can't access a resource that is hosted on a Windows 8 or Windows Server 2012-based failover cluster (KB2838043)

When I spoke with some of my MVP buddies about this particular issue, I was pointed in the direction of this excellent script from Hans Vredevoort over at Hyper-V.nu:

Updated: Windows Server 2012 Hyper-V and Cluster Hotfixes and Updates

This script can be run against your Windows Server 2012 hosts to report on any missing hotfixes or updates that should be applied. Although we thought we had deployed all the relevant updates to our customers cluster, once we ran the script it was apparent that there were a few more that still needed to be added in.

Edit: Microsoft's Cristian Edwards has mentioned in his comment below that there are some additional and updated scripts that you can also use to ensure you keep your Hyper-V hosts up to date. Check out his updated blog post here:

http://blogs.technet.com/b/cedward/archive/2013/05/31/validating-hyper-v-2012-and-failover-clustering-hotfixes-with-powershell-part-2.aspx

Conclusion

If you're running a Hyper-V 2012 cluster and are using ANY backup application (not just specifically DPM 2012 SP1) that performs host-level backups of virtual machines located on Cluster Shared Volumes (CSV's), then I'd highly recommend you install the above two hotfixes at least and then also run Hans' script to see what else you need to get deployed.

With the hotfixes applied to the environment, it works perfectly now!

TechEd North America 2013 Kicks Off Announcing Heaps of New MSFT Releases

Microsoft's flagship conference - TechEd 2013 - is kicking off today (June 3rd) in New Orleans. The Day 1 keynote speech by Microsoft's Brad Anderson is of particular interest due to the media storm that has been generated around the so-called Windows 'Blue' wave of updates across their client and server stack.

Here's a high-level summary of the keynote and announcements that have been made:

Windows 8.1

Microsoft announced the newest release of their client Operating System - called Windows 8.1. This release will be made available as a free upgrade to existing Windows 8 customers. The preview bits will be made available on June 26th and the final release will hit the shelves before the year is out.

The following Windows 8.1 features have been demonstrated or mentioned during the keynote:

• Enhanced tile customization
• Group policy management of the Start Screen
• A new feature for Windows 8.1 and PowerPoint called Mirrorcast allows Windows 8.1 devices to easily pair with compatible streamless devices to allow seamless wireless presenting.
• Workplace Join is a new feature that allows you to easily connect your non-domain joined device to the workplace and to just as easily remove all the customization and management of that device once you're finished working for the day.
• Apps can be managed from the cloud with Windows Intune.

Windows Server 2012 R2

As the name suggests, Windows Server 2012 now has an updated release too. Here's what's been announced for it:

• Work Folders is a new feature that should most likely see usage of applications such as Dropbox getting less and less use in the enterprise.
• Active Directory on-premise now works with Azure Active Directory.
• Storage spaces are taken to the next level with enhanced performance and scalability. With Storage Tiering, you'll get approx. a 16x improvement on Windows Server 2012 RTM.
• Inbox VPN gateway for Network Virtualization

Hyper-V

Windows Server 2012 R2 Hyper-V has a number of significant enhancements, some of which are:

(Update - some of the info below has been added as features get mentioned in the breakout sessions)

• Using DeDup storage with virtual machines can now enhance the performance of the virtual machines by intelligently caching the storage.
• Live Migration compression now means that your virtual machines will migrate between hosts far quicker than with Windows Server 2012 RTM.
• Using RDMA with Live Migration will enhance the live migration of your VM's even quicker than with the compression option.
• Hyper-V replica can be managed across sites and with greater scale using the Hyper-V Recovery Manager feature of Windows Azure.
• Remote Access via VMBus - this is seriously cool. It basically means we now get similar functionality to the HP ILO or Dell DRAC features with our virtual machines in Hyper-V.
• Online grow and shrink of virtual hard disks to manage your virtual disk space allocations on the fly.
• Replication of virtual machines can now be extended to a third site (as opposed to the two site limit in Windows Server 2012 RTM)

Windows Azure

Azure retains it original name but a plethora of new features have been added:

(Update - some of the info below has been added as features get mentioned in the breakout sessions)

• Per minute billing
• No charge for stopped virtual machines
• Special rate for MSDN subscribers making dev/test scenarios much more feasible.
• Uses the same Hyper-V hypervisor as in Windows Server 2012.
• We can now see how much credit we've used directly inside the portal making it easier to keep the credit card bills down!
• Azure Active Directory now works with Windows Server 2012 on-premise Active Directory
• The Windows Azure Pack (WAP) has been announced. This is the new name for the Windows Azure Services for Windows Server (aka project Katal). Good whitepaper on this here (http://t.co/ahLGIn2zOQ)
• Hyper-V Recovery Manager delivers enterprise level replication of your virtual machines between sites.

System Center 2012 R2

Like Windows Server, System Center 2012 now gets an 'R2' release. The only mention in the keynote of System Center was around Intune integrating with on-premise ConfigMgr to manage Windows 8.1 apps but this integration is nothing new. Hopefully as the week goes on, we'll get some additional feature updates on what's new.

(Update - some of the info below has been added as features get mentioned in the breakout sessions)

• Built-in service template for Windows Server 2012 R2 Gateway functionality as part of SCVMM 2012 R2.
• New Java Application Performance Monitoring (APM) functionality in SCOM 2012 R2

Application Lifecycle Management (ALM)

Here's what's coming for all you developers out there:

• Team Foundation Server (TFS) and Visual Studio are getting new releases this coming year.
• Microsoft has acquired a company called 'InRelease' that delivers integrated release management into TFS.
• Enhanced load-test service will be delivered from the cloud.

SQL 2014

SQL has been given a new release too and we should see some preview releases towards the end of the year. Here's some of the new features:

• New enterprise-level data insights with Excel project GeoFlow
• Data Explorer helps to make use of big-data and translate it into tangible business use
• Transaction processing now comes 'in-memory'

Summary

That just about wraps up all I could gather from the keynote without reviewing the recording again later but I think it'll give people the general feel of what's been announced.

You can check out the keynote video and a few selected live sessions at TechEd NA on Microsoft's Channel 9 website here: http://channel9.msdn.com/

Also, if you're looking for up-to-the-minute updates from some of the best sessions on the 'Modern Datacenter' track over there, then check out Aidan Finn's blog here: http://www.aidanfinn.com/

I think it's fair to say now that if you're working in the IT space, there's some pretty exciting times ahead for us :)

SCOM - New MP Available for the DHCP Role on Windows Server 2012!

Microsoft have just released the new management pack (MP) for the DHCP role on Windows Server 2012. This MP has support for DHCP clustering and failover functionality that Windows Server 2012 brings to the table.

You can download this new MP here:

System Center Management Pack for Windows Server 2012 DHCP

Note: Unfortunately this new MP won't discover your older Windows Server 2003/2008 DHCP servers so you'll be stuck with the original (and badly written in my opinion). If that's what you're looking to monitor, then click here to download the DHCP 2003/2008 MP.


This new MP follows hot on the heels of the recently released MP's for:

• Windows Server Base O/S
• Exchange Server 2013 
• Windows 8 Client O/S
• DNS

These recent releases along with the change in MP quality control are coming to us thick and fast now thanks to the internal initiative in Microsoft led by Daniel Savage and his team. You can keep tabs on all of the MP's currently available as well as the ones that are soon due for release on the TechNet MP Wiki here:

http://social.technet.microsoft.com/wiki/contents/articles/16174.microsoft-management-packs.aspx

New System Center 2012 and Windows Server 2012 Downloads Available

There's been a surge in new downloads available from Microsoft on Windows Server and System Center 2012 over the past few days. Here's a sample of some of my favourites:

Tools and Whitepapers

Technical Documentation Download for Technical Scenarios for System Center 2012 SP1

Microsoft Operations Readiness Toolkit

Capacity Planner for Hyper-V Replica

Enabling Hybrid Cloud Today with Microsoft Technologies Whitepaper

Blogs

Server & Cloud Blog - Partnering Through Disruptive Change

SCVMM Engineering Blog - Logical Networks (Part III) – Network Isolation

SCOM Engineering Blog - New Video: System Center 2012 Operations Manager Dashboards: Visualizing Alert Data

SCOM Engineering Blog - KB: Operations Manager agents with teamed network adapters fail to be discovered and monitored

SCOM Engineering Blog - Local Resolution Time monitor in the OpsMgr DNS Management Pack may show unexpected results

DPM and System State Backup Explained


MVP Pro Speaker Series - (These are seriously excellent and a must-watch for anyone serious about System Center 2012)

Backup that really works using multiple members and functions of the System Center 2012 SP1 family (Level 300)

Building Workstation and Application Deployment Automation with Service Manager and Orchestrator (Level 300)

Operations Manager Evolution: Taking Your Environment to the Next Level with Operations Manager (Level 300)


Enjoy!

SCOM 2012 - Exchange 2013 Management Pack Is Now Available!

Microsoft have just made available the new Exchange 2013 MP for SCOM. The big news with this MP is that the Microsoft Exchange Correlation Service that was present in the Exchange 2010 MP is now no more!

I haven't had a chance to give this a good test run (that chance should come next week hopefully), but for now, you can take a look at the official TechNet Blog Post on the release here:

http://blogs.technet.com/b/momteam/archive/2013/05/14/exchange-2013-management-pack-released.aspx

The technical documentation for the MP can be reviewed here:

http://technet.microsoft.com/en-us/library/ee758046(v=exchg.150).aspx

Finally, download the MP here:

http://www.microsoft.com/en-us/download/details.aspx?id=39039

Microsoft are always interested in peoples feedback on their MP's so if you have any 'constructive' feedback to give, add a comment to this post and I'll pass it onto the team who've developed it.

Recordings from our Windows Server 2012 Deep Dive and Mini-MMS Event, April 22-23, Dublin

A few weeks back we delivered two days of Windows Server 2012 Deep Dive sessions out in Microsoft Ireland and I've just received a mail that the session videos are now available.

For me, I totally enjoyed listening to the other guys present and definitely learned some new tips and tricks along with gaining a better understanding of Windows Server 2012 features through the content delivered.

I did a lightning roundup of some of my favourite sessions that were presented at MMS 2013 - unfortunately I couldn't get all of the ones I wanted in there but hopefully it gives people an idea of what they can see if they go and download the videos from Channel 9.

Dave Northey has done an excellent job in co-ordinating the event and overseeing the final recording content so if you have some time to learn about some of the new enhancements that you get with Windows Server 2012, then go check out the Deep Dive videos from here:

http://blogs.technet.com/b/ieitpro/archive/2013/05/14/windows-server-2012-deep-dive-two-days-of-free-training.aspx

SCOM - Updated Windows Server Management Pack Released (version 6.0.7026.0)

Just seen an e-mail coming in notifying us that the SCOM Windows Server MP has received an update. Here's what's changed since the last release:

• Fixed a bug in Microsoft.Windows.Server.2008.Monitoring.mp where the performance information for Processor was not getting collected.

• Made monitoring of Cluster Shared Volume consistent with monitoring of Logical Disks by adding performance collection rules. (“Cluster Shared Volume - Free space / MB”,”Cluster Shared Volume - Total size / MB”,”Cluster Shared Volume - Free space / %”,”Cluster Disk - Total size / MB”,”Cluster Disk - Free space / MB”,”Cluster Disk - Free space / %”)

• Fixed bug in Microsoft.Windows.Server.ClusterSharedVolumeMonitoring.mp where the Cluster disks running on Windows Server 2008 (non R2) were not discovered.

• Fixed bug 'Cluster Disk Free Space Percent' and Cluster Disk Free Space MB' monitors generate alerts with bad descriptions when the volume label of a cluster disk is empty.

• Added feature to raise event when NTLM requests time out and customers are unable to use mailboxes, outlook stops responding, due to the low default value for Max Concurrent API registry Key (HLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters) , which is a ceiling for the maximum NTLM or Kerberos PAC password validations a server can take care of at a time. It uses the “Netlogon” performance counter to check for the issue.

You can download the new MP version and its associated guide from here:

http://www.microsoft.com/en-us/download/details.aspx?id=9296

SCOM 2012 - SP1 UR2 Web Console Error with Savision LiveMaps

Last Thursday, Marnix Wolf did up an excellent blog post about an error that started occurring after Update Rollup 2 for SCOM 2012 SP1 was deployed into environments that had Savision LiveMaps deployed. Essentially, when  SP1 UR2 and that particular release of LiveMaps don't like each other very much, you are presented with the following error details:

An Unexpected Error Has Occurred
System.Reflection.ReflectionTypeLoadException: [ReflectionTypeLoad_LoadFailed]

Check out the post by Marnix on what he did to determine the cause (take particular note of his technique for performing trace logging of SCOM 2012 as this is an invaluable way of determining root cause of issues inside your SCOM infrastructure):

http://thoughtsonopsmgr.blogspot.com/2013/05/om12-sp1-rup2-web-console-error.html

Before you dig deep into troubleshooting this yourself or thinking that Savision haven't got a solution to the problem - fear not, they have now!

I've just seen this on their website while looking for a solution to this problem myself:

http://www.savision.com/resources/news/fix-om12-sp1-ur2-web-console

They've released updated versions of the LiveMaps Summary Widget MP's that will alleviate the error above.

Hopefully this will save some people a bit of time and effort if they come across this problem.

New Downloads: Microsoft IaaS Product Line Architecture Guides

Microsoft have just published two excellent guides to kick start your Friday morning coffee read ;) that are based on their Infrastructure as a Service (Iaas) Product Line Architecture (PLA). The first guide is mostly based on Windows Server 2012 storage and virtualization and the second guide is based around System Center 2012.

• Infrastructure-as-a-Service Product Line Architecture Fabric Architecture Guide (Windows Server 2012)

http://www.microsoft.com/en-us/download/details.aspx?id=38817

• Infrastructure-as-a-Service Product Line Architecture Fabric Management Architecture Guide (System Center 2012)

http://www.microsoft.com/en-us/download/details.aspx?id=38813


Go check them out and I'm sure you'll learn something new from them. Also, the Microsoft PLA team are keen to get some feedback on these documents so if you have any 'constructive' feedback that you want me to pass on, feel free to leave a comment and I'll do just that.

Winners of the Mastering SCOM 2012 Book Competition!

A while back I ran a competition on this blog that required people to download and evaluate System Center 2012 in return for the chance to win a copy of the 'Mastering System Center 2012 - Operations Manager' book that I was involved in last year.

We had a target of 250 downloads to reach and eventually a few weeks ago we hit that target. Due to work commitments (and authoring another book), I've been 'slightly' delayed in announcing the winners of the competition!

To pick the winners, I used the same method that Hans Vredevoort had on his excellent Hyper-V.nu blog a while back  and asked my eldest son to pick two numbers from the full list of entrants that I had collated together into an Excel spread-sheet.

The result of his picks mean that Reidar Johansen...

 

and  Stanislav Zhelyazkov are the two lucky winners.

 Well Done Guys!

If you didn't win, I'd still highly recommend that you get your hands on the hard copy of the book from here or the e-copy from here. Adding it to your technical library will do you no harm at all if you're serious about System Center 2012 and want to ramp up your SCOM 2012 knowledge.

I'll be in touch shortly with the two winners and a massive THANK YOU to everyone who participated in boosting System Center 2012 awareness and evaluating it in their environments.