vpxd-svcs service fails to start with "certificate verify failed: certificate has expired" after replacing Custom Certificates
search cancel

vpxd-svcs service fails to start with "certificate verify failed: certificate has expired" after replacing Custom Certificates

book

Article ID: 428959

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • After replacing a custom certificate (Machine SSL) via the vSphere Client (GUI) or Certificate Manager, the vpxd-svcs service fails to start.
  • In the Certificate Management page of vCenter, the following error may be observed: "Error occurred while fetching tls: String Index out of range: -1"

  • The following error is observed in the /var/log/vmware/vmon/vmon.log file regarding vpxd-svcs:

YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 100, in <module>
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     endpoint_registration_runner(logging_file)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 65, in endpoint_registration_runner
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     UpdateTaggingServiceGrpcEndpoint(logging_file).run()
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 54, in run
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self.update_endpoints()
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 83, in update_endpoints
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     ls_obj = LookupServiceClient(ls_url, retry_count=5)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 314, in __init__
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self._init_service_content()
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 294, in do_retry
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     return req_method(self, *args, **kargs)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 304, in _init_service_content
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self.service_content = si.RetrieveServiceContent()
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/pyVmomi/vmodlVmomisupport.py", line 595, in <lambda>
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self.f(*(self.args + (obj,) + args), **kwargs)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/pyVmomi/vmodlVmomisupport.py", line 385, in _InvokeMethod
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     return self._stub.InvokeMethod(self, info, args)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     conn.request('POST', self.path, req, headers)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/python3.7/http/client.py", line 1281, in request
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self._send_request(method, url, body, headers, encode_chunked)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/python3.7/http/client.py", line 1327, in _send_request
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self.endheaders(body, encode_chunked=encode_chunked)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/python3.7/http/client.py", line 1276, in endheaders
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self._send_output(message_body, encode_chunked=encode_chunked)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/python3.7/http/client.py", line 1036, in _send_output
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self.send(msg)
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001   File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001     self._sslobj.do_handshake()
YYYY-MM-DDTHH:MM:SS.SSSS Wa(03)+ host-39001 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
YYYY-MM-DDTHH:MM:SS.SSSS Er(02) host-39001 <vpxd-svcs> Service pre-start command failed with exit code 1.

Environment

vCenter 7x, 8.x

 

Cause

This issue occurs when an expired Root or Intermediary Certificate remains in the TRUSTED_ROOTS store in VECS (VMware Endpoint Certificate Store). Even if the Machine SSL certificate has been renewed, if the Root CA signing certificate (or an intermediate CA certificate in the chain) has expired and is still present in the trust store, the SSL verification used by vpxd-svcs will fail.

Resolution

To resolve this issue, identify and remove (or replace) the expired Trusted Root certificate that matches the Authority Key Identifier of the current Machine SSL certificate.

Prerequisites:

  • Ensure you have a snapshot or backup of the vCenter Server. In an Enhanced linked mode environment, powered off snapshots of vCenters in the SSO domain are required.

  • SSH access to the vCenter Server Appliance (VCSA) as root.

Steps:

  1. Identify the Authority Key Identifier (AKI) of the Machine SSL Certificate Run the following command to view the Machine SSL certificate details:

    • /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less

    • Scroll down to the X509v3 Authority Key Identifier section.

    • Note the hex string (e.g., KEY_ID: XX:XX:XX...). This represents the ID of the Root CA that signed this certificate.

  2. Identify the Expired Root Certificate List all certificates in the TRUSTED_ROOTS store:

    • /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

    • Look for a certificate where the X509v3 Subject Key Identifier matches the Authority Key Identifier you noted in Step 1.

    • Verify if this certificate is expired by checking the Not After date.

    • Note the Alias of this expired certificate.

  3. Remove the Expired Root Certificate Once the expired root certificate alias is identified, remove it from the store:
    • Run the vCert script:
      • ./vCert.sh
    • Select the option Manage Certificates

    • Select the option CA certificates in VMware Directory

      • Remove the Expired Certificate From the "Manage Certificates in VMware Directory" sub-menu, select Option 2:

        • Remove CA certificate(s) from VMware Directory
        • Follow the prompts to list the available certificates.
        • Identify the expired certificate (matching the Authority Key Identifier found in the logs or via vecs-cli)
        • Select the expired certificate for removal.
    • Start Services After removing the certificate, start the services to ensure vpxd-svcs initializes correctly:
      • service-control --start --all

Note: If the expired root or intermediate CA certificate is still present in the certificate chain of the machine SSL certificate, removing the expired cert will result an incomplete certificate chain among other issues. See KB: How to update expired intermediate certificate on the vCenter server for more information.

Powered by Wolken Software