Service vpxd-svcs failed to pre-start with "invalidProperty = 'Invalid certificate'"
search cancel

Service vpxd-svcs failed to pre-start with "invalidProperty = 'Invalid certificate'"

book

Article ID: 390388

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Service vpxd-svcs failed to pre-start with "invalidProperty = 'Invalid certificate'"
  • vMon log file /var/log/vmware/vmon/vmon.log will show vpxd-svcs pre-start failure with "Invalid certificate" error message:

yyyy-mm-ddThh:mm:ss.076Z Wa(03) host-1712 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     ls_obj.reregister_service(service_info.serviceId, mutable_spec)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     return req_method(self, *args, **kargs)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     self.f(*(self.args + (obj,) + args), **kwargs)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     return self._stub.InvokeMethod(self, info, args)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     raise obj  # pylint: disable-msg=E0702
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712 pyVmomi.VmomiSupport.InvalidArgument: (vmodl.fault.InvalidArgument) {
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    dynamicType = <unset>,
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    dynamicProperty = (vmodl.DynamicProperty) [],
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    msg = '',
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    faultCause = <unset>,
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    faultMessage = (vmodl.LocalizableMessage) [],
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712    invalidProperty = 'Invalid certificate'
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712 }
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712 During handling of the above exception, another exception occurred:
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712 Traceback (most recent call last):
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     endpoint_registration_runner(logging_file)
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     self.update_endpoints()
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 146, in update_endpoints
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712     raise Exception("Tagging grpc reregistration failed while"
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712 Exception: Tagging grpc reregistration failed while executing vpxd-svcs prestart commands
yyyy-mm-ddThh:mm:ss.076Z Wa(03)+ host-1712
yyyy-mm-ddThh:mm:ss.167Z Er(02) host-1712 <vpxd-svcs> Service pre-start command failed with exit code 1.
yyyy-mm-ddThh:mm:ss.002Z Wa(03) host-1712 Failed to publish health status change.

  • Log file /var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log will show similar entries to: 

ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service
ERROR:tagging_grpc_registration:(vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   invalidProperty = 'Invalid certificate'
}
Traceback (most recent call last):
  File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints
    ls_obj.reregister_service(service_info.serviceId, mutable_spec)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
    return req_method(self, *args, **kargs)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
    self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
    raise obj  # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   invalidProperty = 'Invalid certificate'
}

  • Log file /var/log/vmware/vmcad/certificate-manager.log will show similar entries to: 

yyyy-mm-ddThh:mm:ss.822Z INFO certificate-manager MACHINE_SSL_CERT certificate replaced successfully. SerialNumber and Thumbprint changed.
yyyy-mm-ddThh:mm:ss.891Z INFO certificate-manager lstool command currently being executed is- : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmware-lookupsvc/lib/lookup-client.jar:/usr/lib/vmware-lookupsvc/lib/*:/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'get-site-id', '--url', 'https://<VC-FQDN>:443/lookupservice/sdk', '--no-check-cert']
yyyy-mm-ddThh:mm:ss.255Z ERROR certificate-manager 'lstool get-site-id' failed: 1
yyyy-mm-ddThh:mm:ss.255Z ERROR certificate-manager 'lstool get-site-id' failed: 1
yyyy-mm-ddThh:mm:ss.255Z ERROR certificate-manager please see /var/log/vmware/vmcad/certificate-manager.log for more information.

  • Log file /var/log/vmware/vmon/vmon.log will show similar entries to: 

File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
  self._sslobj.do_handshake()
ssl.SSLCertVerificationErrorL [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
<vpxd-svcs> Service pre-start command failed with exit code 1.

 

 

Environment

vCenter Server 7.X
vCenter Server 8.X

Cause

One or more service registrations in lookupservice are using different certificate as trust anchor than the vCenters machine SSL certificate. Additionally one of the certificates in the trust chain for this certificate has expired.

Resolution

Before taking any action, please ensure that the necessary precautions have been taken and the required backups or offline snapshots exist. For more information, see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

To resolve this issue

  1. Use the lsdoctor tool to verify and correct the trust anchors of the internal service registrations in the vCenter lookupservice. For more information refer to Using the 'lsdoctor' Tool
  2. Verify if there are any expired certificates in the VECS TRUSTED_ROOTS store as per Determining expired SSL certificates in vCenter Server and ESXi 6.x and 7.0.x
  3. Remove the expired certificates as outlined in Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS) and Verify and resolve expired vCenter Server certificates using command line interface
Powered by Wolken Software