Virtual Machine power on task fails with error "All required agent virtual machines are not currently deployed on host 'hostname'. Operation failed! error.mutationOperationFailed.unknownReason"

Products

• VMware vCenter Server

Issue/Introduction

• Virtual machine power on task fails with an error " vCenter Server was unable to find a suitable host to power on the following virtual machines for the reasons listed below. 
  
  "All required agent virtual machines are not currently deployed on host 'hostname' " when perform it from vCenter server HTML5 client. 
  
  
  
  
• VM power-on task will succeed when it perform via ESXi host UI client. 
  
  
• In the vpxd.log you see the errors similar to:
  
  /var/log/vmware/vpxd/vpxd.log 
  
  YYYY-MM-DDTHH:MM:SS.###-##:## error vpxd[86151] [Originator@6876 sub=Default opID=1f039c26] [VpxLRO] -- ERROR 1ro-46349377 -- 52b6a761-263f-13a6-62b3-63a47ca85aal -- SeasionManager -- vim. SessionManager. loginExtensionByCertificate: :vim. fault. InvalidLogin.
  Result:
  [vim, fault. InvalidLogin) (
  taultCause - (vmodl.MethodFault) null,
  faultMessage - <unset>
  
  

• The eam.log should have following  entries.
  
  /var/log/vmware/eam.log
  
  YYYY-MM-DDTHH:MM:SS.###-##:## | WARN | vim-async-0 | ExtensionSessionRenewer. java | 227 | [Retry:Login:com.vmware.vim.eam: f9c566f93c9b63f8] Re-login failed, due to:
  com. vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
  
  Caused by: com.vmware. vim. binding. vim. fault. InvalidLogin: Cannot complete login due to an incorrect user name or password.
  
  

• Following events are there in wcpsvc.log. 
  
  /var/log/vmware/wcp/wcpsvcs.log
  
  YYYY-MM-DDTHH:MM:SS.###-##:## error wcp [eamlib/lister.go:84] [opID=EAMAgent] Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.
  YYYY-MM-DDTHH:MM:SS.###-##:## error wep [informer/informer.go:129] [opID-EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later.
  
  
  

Environment

vCenter server 7.x

vCenter server 8.x

Cause

• This issue occurs when solution users are unable to authenticate to the extensions by certificate due to invalid login. 
• This issue may also occurs due to the certificate manager utility being unable to automatically update the EAM certificate when solution user certificates are updated. 

Resolution

VMware is aware of this issue and working to fix it in a future release.

To workaround this issue, run the updateExtensionCertInVC.py script as outlined below.

• Log in to the vCenter Server Appliance using SSH..
  
  
• Run this command to retrieve the vpxd-extension solution user certificate and key:
  
  mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key


• Run this command to update the extension's certificate with vCenter Server.
  
  python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u [email protected]


  Note: If this produces the error "Hostname mismatch, certificate is not valid for 'localhost'", change 'localhost' to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate.
   
  Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the [email protected] password.
  
  
• Restart EAM and WCP services and restart all the services with these commands:
  
  service-control --restart eam 
service-control --restart wcp


• To restart all services at once, restart all the vCenter services with below command: 
  
  service-control --stop --all && service-control --start --all


• Try to power on virtual machine from vCenter server after following the workaround steps.