The curious case of the non responsive Windows 2012 Hyper-V server

Author: Daniel Nikolic

Suddenly the System Center Virtual Machine Manager (SCVMM) 2012 Service Pack 1 CTP 2 version could not connect to the Windows 2012 Hyper-V server, different error messages appeared which didn’t really make sense:

Error (2916)
VMM is unable to complete the request. The connection to the agent %MACHINENAME% was lost.
Unknown error (0x80338126)

Recommended Action
Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTPS traffic.

This problem can also be caused by WMI service crash. Ensure that KB 982293 (http://support.microsoft.com/kb/982293) is installed on the machine if it is running Windows Server 2008 R2.

If the error persists, reboot %MACHINENAME% and then try the operation again.

or

Error (2910)
VMM does not have appropriate permissions to access the resource C:\Windows\system32\vmms.exe on the %MACHINENAME%server.
Access is denied (0x80070005)

Recommended Action
Ensure that Virtual Machine Manager has the appropriate rights to perform this action.

After checking firewall, WMI and DCOM for remote management, an uninstall of the VMMAgent should do the trick… Unfortanaly! Trying to install the agent fails with:

Error (410)
Agent installation failed on %MACHINENAME%.
Fatal error during installation (0x80070643)

Recommended Action
Try the operation again. If the problem persists, install the agent locally and then add the managed computer.

Disabling the firewall resulted in:

Error (406)
Access has been denied while contacting the server %MACHINENAME%.

Recommended Action
1. Verify that the specified user account has administrative privileges on %MACHINENAME%.
2. Verify that DCOM access, launch, and activation permissions are enabled on %MACHINENAME% for the Administrators group. Use dcomcnfg.exe to modify permissions, and then try the operation again.

Warning (10444)
The VMM management server was unable to impersonate the supplied credentials.

Recommended Action
To add a host in a disjointed domain namespace, ensure that the credentials are valid and of a domain account. In addition, the SCVMMService must run as the local system account or a domain account with sufficient privileges to be able to impersonate other users.

How weird! We tried to install the VMMAgent manually on the Hyper-V host itself, just to see if it’s related to remote access rights, but the installation also failed. Looking at the log file in %ProgramData%\VMMLogs we saw a strange error:

AddLocalComputerToAdministratorsGroup. Return value 3.
CustomAction AddLocalComputerToAdministratorsGroup returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)

Then we had a breakthrough! I tried to logon to the Hyper-V host via mstsc with the domain admin account, I received the following error:

The Local Security Authority cannot be contacted

Ok, so the domain could not be contacted! DNS? We looked at the IP settings and saw the DNS server was reset to the DHCP scope, but the IP Address still was static and correct… After adjusting the DNS server the installation of the VMMAgent completed successfully and adding the host in the SCVMM console also worked successfully!

Daniel Nikolic

Is interested in everything connected to technology. Has a passion for cloud, virtualization and software development. Always finds appropriate IT solutions for customers that match their needs strategically, technically and financially.


Core qualities

Quick thinker, result driven, ambitious, customer-friendly, enthusiastic

Hobbies
Running, listening to music, good food and doing fun things with family

Job description
CTO PepperByte, LoadGen, and BlueParq