Phantom network adapters in 2008 R2

I thought this was a VMware issue initially and even blamed BGInfo for reporting spurious network cards on the desktop background, but this appears to be an issue with 2008 R2.

When removing and adding network adapters in 2008 R2 Virtual Machines in VMware, old network adapters are not fully removed from the Network Adapters list of devices in Device Manager.

The network adapter information remains in registry with the old IP address details. This causes issues for reporting software such as BGInfo. Really frikking annoying. Not really plug and play is it.

If you add a new adapter and remove the old adapter and try to use the same IP address, you will get an error message as per KB269155.

You can enable the option to see hidden devices in Device Manager by following these instructions:

  • Open a command prompt
  • Run ‘set devmgr_show_nonpresent_devices=1’
  • Launch device manager by running ‘Start DEVMGMT.MSC’
  • Click View, and then click Show Hidden Devices.

Once you can see the old network adapters just uninstall them.

Beware vShield Endpoint Driver

If you are like me and you like to include the latest drivers in your SOE I have a word of warning about the vShield Endpoint Driver.

I included this in our 2008 R2 SP1 SOE as I knew we were going to be rolling out Trend Micro Deep Security.

Bad idea! The vShield Endpoint Driver makes the server practically unusable. You won’t be able to map network drives, you’ll get RPC replication issues, it will behave as if the most anal anti virus software has been installed with all features enabled.

Took me weeks to figure out it was this innocuous driver which wasn’t supposed to be doing anything…

How to Re-Arm without rebooting KMS clients

If you’ve had the pleasure of rolling out KMS hosts in an environment where, previously they had been using MAK keys, you’ll notice that if you re-arm KMS clients they will insist on rebooting for the changes to come into effect. Not ideal!

The easiest way I found around this, was to use the provided keys from the Volume Activation Deployment Guide Windows 7. Shown below is the key for 2008 R2 Standard.

Windows Server 2008 R2 Standard Product Key —  YC6KT-GKW9T-YTKYR-T4X34-R7VHC

First reset the license key with ‘slmgr -ipk ProductKey’, then force the machine to activate with ‘slmgr -ato’.

Sorted! Reboot avoided.

Resilient Key Management Services, Priority and Weighting, Publishing in Multiple Domains

I recently had to implement KMS and decided on a resilient KMS design with a primary and secondary KMS server. As per Microsoft recommendation a minimum of two KMS hosts are required for fail over. KMS does not need a dedicated server, so you can install it on an existing server, such as a Domain Controller, saving license costs.

All you need to do, to convert an existing server to a KMS server, is to replace it’s existing key with ‘slmgr -ipk NEWKEY’. Wait for the confirmation and then run ‘slmgr -ato’ to activate the new KMS key. You should get a ‘product successfully activated’ reply. Now we have two KMS hosts configured in Active Directory with the same priority and weighting.

To designate between the primary and secondary KMS server, the priority and weighting must be applied to the DNS records associated with each KMS host.

The priority field will determine which host is contacted first – clients always attempt to contact the host with the lowest priority. The weighting field is a load balancing mechanism for hosts with the same priority. As the KMS hosts will be configured as a primary and secondary, I’ve left the weighting value the same.

Additionally as I was using the same KMS servers in an environment with multiple domains, I required DNS records to be published in every domain. The easiest way to manage this is to use the ‘DnsDomainPublishList’ multi-string field. This will configure the KMS servers to automatically publish in multiple DNS domains and set the priority and weighting of each server.

HKLM\Software\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\DnsDomainPublishList

Format: domain to publish in, priority, weighting

KMS Host

Registry Key

Multi String Value


Primary HKLM\Software\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\ DnsDomainPublishList, 10,, 10, 100
Secondary HKLM\Software\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\ DnsDomainPublishList, 20,, 20, 100

 And lastly dont forget to disable KMS client caching on each host with ‘slmgr.vbs /ckhc’. This will force a client to always use KMS auto discovery (query DNS) to determine which KMS host to activate with. If the primary KMS host if offline, the client will automatically activate with the secondary KMS host.

Site visit to VCE

I recently had the pleasure of spending some time at the EMC Factory in Cork, Ireland where vBlocks are assembled for the EMEA region.

The full name as shown at the entrance below is the ‘EMC Ireland Centre of Excellence’… not sure what a Centre of Excellence is but I guess it sounds better than the EMC Ireland Factory.

EMC Entrance

I cant think of many adjectives that do the size of the factory justice – gargantuan springs to mind. Visiting the EMC factory must be every storage engineers wet dream. A couple of football fields worth of every bit of EMC storage kit you can imagine.

The purpose of the visit was to spend time with the VCE team while they complete the logical configuration of the vBlock 300HX for our client’s main data centres. This involved trying not to get in their way as all the components were configured and trying to assimilate as much information as possible about the AMP Cluster, VNXe, the VNX 7500, UCS Manager, Service Profiles, IONIX UIM and Unisphere.

Once the vBlock order has been raised, the first week is usually spent with physical assembly. Weeks 2 and 3 are spent doing the logical configuration before the vBlocks get shipped to site in the 4th week, meeting the 30 day lead time. As 95% of the configuration is done onsite at EMC the time required at each client site is minimal. This usually involves DOA testing to make sure there has been no damage during transit, integration with the clients aggregation network and then a thorough test plan where every components is pulled, badgered and poked to stress test every possible failure scenario to ensure all parts are acting as required.

Not wanting to get anyone in trouble, I obviously couldn’t take any photos even though the urge was overwhelming. I did however recognise this picture on the web. All vBlocks outside the North America region were assembled in the hallowed turf shown in the picture below.

VCE Entrance

And lastly I must add that even though this was my first visit to a Centre of Excellence, I was pleasantly surprised at the local talent.

Event ID 10154 after promoting 2008 R2 Domain Controller

Fark. What a day. Another spurious event log error.

What’s worse than getting event log errors after deploying an OS… How about after promoting a domain controller. It creates far too many questions and far too few answers!

After building out a new Active Directory forest I noticed the following event log errors in the 2008 R2 Domain Controllers:

The WinRM service failed to create the following SPNs: WSMAN/servername.fqdn; WSMAN/servername 
Additional Data 
The error received was 1355: %%1355.
User Action 
The SPN can be created by an administrator using setspn.exe utility.

This appears to be caused by the NETWORK SERVICE not being given the necessary permissions on the domain controller AD object to make write changes.

I tried a number of things to get this to work and I think I have found the solution.

Adding the necessary values with SETSPN manually does not resolve the issue because the NETWORK SERVICE attempts to write the WSMAN SPN values each time the WinRM service is started.

You can apply the necessary permissions manually with ADSIEDIT or via the command line with dsacls.

The domain controller AD object resides in the Domain Controllers OU. If you apply permissions directly to the domain controller object they will (should!) be replaced by the permissions held on the AdminSDHolder object. In my testing this what not the case, however all the documentation I’ve read indicates differently.

Anyhow, the AdminSDHolder object resides in the System container in every domain and is used to secure privileged users, groups and objects from modification. A background process on the PDC emulator runs every 60 minutes and compares the ACL of the AdminSDHolder object with the ACL of the protected users, group and computers. If there are any differences it overwrites them. Read more about this here.

When I applied the necessary permissions to the AdminSDHolder object only, I could not get the permissions to overwrite on the domain controller object. I waited a couple of hours and even forced the SDPROP process to run. No joy.

I used adfind.exe as described in the Technet article above and confirmed that the \Users\Domain Controllers group is a protected group and as the domain controller objects reside in the Domain Controllers group they should be protected.

I can see the AdminSDHolder ACL being applied to the Domain Controllers group, but not the objects contained with the Domain Controllers. Weird.

I don’t yet know why the ACL of the domain controller object wasnt overwritten , but this was consistent across multiple domains I tested this on. This will need further investigation as all evidence points to the domain controller objects being protected. I also recreated the issue on a vanilla Windows 2008 R2 SP1 ISO to eliminate any other potential causes. When testing with the vanilla ISO, I also received the same error as above, just with a different error code in the message ‘The error received was 8344: %%8344.

To work around this issue I applied permissions directly to the domain controller and AdminSDHolder objects with dsacls:

dsacls “CN=AdminSDHolder,CN=System,DC=domain,DC=com” /G “NETWORK SERVICE:WS;Validated write to service principal name”

dsacls “CN=domaincontroller,OU=Domain Controllers,DC=domain,DC=com” /G “NETWORK SERVICE:WS;Validated write to service principal name”

After that just stop and start the WinRM service and check the System log. Event ID 10154 should be gone.

Event ID 10 after you install 2008 R2 SP1 — KB950375

I hate unnecessary event log errors, especially when it some unintelligible bullshit that repeatedly clogs up the event logs, making it harder to troubleshoot real issues.

If you’ve noticed the following below in your 2008 R2 SP1 servers, don’t panic, its not your fault:

Event filter with query “SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA “Win32_Processor” AND TargetInstance.LoadPercentage > 99″ could not be reactivated in namespace “//./root/CIMV2” because of error 0x80041003. Events cannot be delivered through this filter until the problem is corrected.

Some crappy coder forgets to do their job properly and we end up with event log errors that raise false alarms, finger pointing (usually!) and time wasting.

Whoever was the release manager for SP1 should be fired… Steve Jobs style.

The fix is easier enough. Documented under KB950375 – run a vbs script or install an msi as found in article 2545227. I went the vbs script route, seems to do the job.

–UPDATE 25/10/2011

I have tested this on a vanilla Windows 2008 R2 SP1 ISO and the error is also present so is indeed a SP1 issue.

VMware Tools Unattended install — Shared F**king Folders

Note to Self. When attempting to remove Shared Folders ‘ during an unattended installation of VMware Tools in Windows as per the unattended installation guide and KB2000399, you need to remove ThinPrint as well as it has a dependency. Otherwise you get a missing tpmon error during the installation. Having had experience building 8 node thinprint clusters and deploying and supporting Thinprint in an enterprise environment, I detest Thinprint. Universal drivers that support ‘every’ printer are always a compromise, reducing printer functionality and increase hostlity towards IT. Thinprint is one of the cheapest universal drivers for a reason. I digress… shown below is an unattended install example.

msiexec.exe /i “vmware tools64.msi” ADDLOCAL=ALL REMOVE=”Hgfs,ThinPrint” /qn /l*v c:\support\vmwaretools64-install.log /norestart

I’m beginning to hate shared folders.

Active Directory Domain Controllers and VMware Tools Shared Folders. Grrrrrrr

Don’t you wish there was an easier way to discover faults? Rather than (what seems like) continuously wasting hours of the day figuring out problems… Software should be more intelligent… For instance running dcdiag and getting ‘RPC replication errors’ should be reported as ‘RPC replication errors due to VMware Tools Shared Folders’ would save everyone a lot of time…. This leads me to my next post.

I had an issue recently where I was trying to create a domain trust between two domains to no avail and came across KB1012140 ‘Unable to create a trust relationships between Domain Controllers’. The error manifests itself as ‘The local security authority is unable to obtain an RPC connection to the Domain controller.’

Although I was grateful to come across a quick fix for this issue (Just remove the Shared Folders component from VMware Tools which requires a reinstall in 2008 R2), I don’t think the VMware KB article covers the full scope of the issue.

Even if you are not planning on creating any domain trusts, VMware Tools Shared Folders should be removed from all domain controllers.

I kept on getting odd RPC replication errors with Shared folders enabled, even when building new domains to eliminate any other possible causes. Active Directory is the core service of the application layer. Pretty much every other application has a dependency on AD. In an Enterprise environment a low risk approach should always be taken to avoid potential future issues that could result in down time of your core services, especially when the benefit is minimal (i.e. to help lazy support staff). This is especially true if you are virtualising all your domain controllers (a fairly common scenario nowadays) as all your domain controllers will potentially be at risk, guarenteeing a total loss of service.

I have gone a step further and standardised the unattended VMware Tools install for 2008 R2 and removed VMware Tools Shared Folders from all 2008 R2 virtual machines. By extension this needs to include the Thinprint component as well as it seems to have a dependency on Shared Folders. The unattended install is shown here.