WCP Service fails to start after replacing vCenter Server certificates or vCLS VMs don't deploy
search cancel

WCP Service fails to start after replacing vCenter Server certificates or vCLS VMs don't deploy

book

Article ID: 336094

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

Symptoms:

The following messages appear in /var/log/vmware/wcp/wcpsvc.log

Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.

The following messages appear in /var/log/vmware/eam/eam.log

Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

The SSL Certificates on the vCenter Appliance were recently replaced.

vCLS VMs also don't deploy correctly.

Clusters have error "vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS."

vCLS VMs don't deploy even after placing cluster in Retreat Mode as described in KB 316514

 

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue is due to the certificate manager utility being unable to automatically update the EAM certificate when solution user certificates are updated. WCP requires EAM to be functional in order to start and to deploy vCLS VMs.

You do not need to be licensed for or using WCP/vSphere 7 with Kubernetes in order to be susceptible to this issue.

Resolution

VMware is aware of this issue and working to fix it in a future release.

Workaround:
To workaround this issue, run the updateExtensionCertInVC.py script as outlined below.

  1. Log in to the vCenter Server Appliance using SSH.
  2. Run this command to enable access the Bash shell:

    shell.set --enabled true
     
  3. Type shell and press Enter.
  4. Run this command to retrieve the vpxd-extension solution user certificate and key:

    mkdir /certificate

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
     
  5. Run this command to update the extension's certificate with vCenter Server.

    python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u Administrator@domain.local
Note: If this produces the error "Hostname mismatch, certificate is not valid for 'localhost'", change 'localhost' to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate.
Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password.
  1. Restart EAM and restart all the services with these commands:
service-control --restart eam 
         service-control --restart wcp
         
         Note : To restart all services at once, restart all the vCenter services with below command: 
service-control --stop --all && service-control --start --all


Note: Please refer to the Related Information in this KB if you are receiving the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'" during extension's certificate update (step 5).

Additional Information

  • This issue is also known to cause problems with NSX solutions connected to vCenter. See Deploying NSX VIBs fails on the ESXi host after changing the vCenter Server certificates for more information.
  • In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.

Password to connect to VC server for user="[email protected]":
yyyy-MM-ddTHH:mm:ss.974Z Updating certificate for "com.vmware.vim.eam" extension
yyyy-MM-ddTHH:mm:ss.116Z Successfully updated certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
update_extension_cert_in_VC()
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
sessionMgr = si.content.sessionManager
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 589, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 43, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
conn.request('POST', self.path, req, headers)
File "/usr/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3.7/http/client.py", line 1451, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076)

 

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

  • Product: vCenter
  • Log File: wcpsvc.log
  • Log Expression Check "Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database"



Powered by Wolken Software