vCenter certificate replacement fails at 85% with rollback using certificate-manager

Products

• VMware vCenter Server

Issue/Introduction

When replacing the Machine SSL certificate on a vCenter Server Appliance (VCSA) using the certificate-manager utility, the operation fails at 85% completion while starting services. The system then initiates an automatic rollback. Several services remain in a Stopped or StartPending state, including vpxd, sps, and wcp.

• CLI Output
  
  You are going to regenerate Machine SSL cert using VMCA
Continue operation : Option[Y/N] ? : y
Status : 85% Completed [starting services...]
Error while starting services, please see service-control log for more details
Status : 0% Completed [Operation failed, performing automatic rollback]
  
  
• Log Snippets:
  
  Based on the logs below, the failure is triggered by hostname mismatches and dependency service timeouts.
  Note: While these logs demonstrate specific failure reasons, other environmental factors may also cause this task to fail.
  
  
  • vmon.log
    
    /var/log/vmware/vmon/vmon.log:
    
    YYYY-MM-DDThh:mm:ss Wa(03) host-#### <vsm> Service pre-start command's stderr:     self._sslobj.do_handshake()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-#### ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'vCenter FQDN'. (_ssl.c:1017)
YYYY-MM-DDThh:mm:ss Wa(03)+ host-####
YYYY-MM-DDThh:mm:ss Er(02) host-#### <vsm> Service pre-start command failed with exit code 1.
YYYY-MM-DDThh:mm:ss In(05) host-#### <wcp> Running the API Health command as user wcp
YYYY-MM-DDThh:mm:ss In(05) host-#### <wcp-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n wcp -u /wcp/health -t 10
YYYY-MM-DDThh:mm:ss Wa(03) host-#### <wcp> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:8920/wcp/health. Exception: <urlopen error [Errno 111] Connection refused>
YYYY-MM-DDThh:mm:ss Wa(03)+ host-####
YYYY-MM-DDThh:mm:ss In(05) host-#### <wcp> Re-check service health since it is still initializing.
  • sps.log:
    
    /var/log/vmware/vmware-sps/sps.log
    
    YYYY-MM-DDThh:mm:ss [main] ERROR opId=sps-Main-632474-855 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - Failed to retrieve service content
YYYY-MM-DDThh:mm:ss [main] ERROR opId=sps-Main-632474-855 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error occurred while retrieving service content
  • vpxd.log:
    
    /var/log/vmware/vpxd/vpxd.log:
    
    YYYY-MM-DDThh:mm:ss info vpxd[21850] [Originator@6876 sub=vpxLro opID=sps-VICNotifier-950278-574-11803-953-61] [VpxLRO] -- BEGIN lro-84298 -- ha-certificate-manager-22 -- vim.host.CertificateManager.retrieveCertificateInfoList -- #####)
YYYY-MM-DDThh:mm:ss info vpxd[21630] [Originator@6876 sub=MoCluster opID=CSMM-domain-c#-740 req=#####] Failed to bootstrap cluster store; [vim.HostSystem:host-##,<Host FQDN>], N20DistEsxManagerClient11OpExceptionE(ClusterAlreadyBootstrapped calling 'bootstrap')
  • certificate-manager.log:
    
    /var/log/vmware/vmcad/certificate-manager.log
    
    YYYY-MM-DDThh:mm:ss ERROR certificate-manager  Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
YYYY-MM-DDThh:mm:ss ERROR certificate-manager  {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "None"
            ],
            "localized": "An error occurred while invoking external command : 'None'"
        },
        "Error while starting services, please see service-control log for more details"

Environment

VMware vCenter Server

Cause

The 85% failure occurs because the new certificate doesn't match the vCenter's FQDN, causing secure handshakes to fail and preventing essential services from restarting.

Resolution

Before proceeding, please ensure you have a valid, snapshot of the vCenter Server. For best practice refer: Snapshot Best practices for vCenter Server Virtual Machines

1. Verify Hostname and PNID Alignment:

   1. Check vCenter PNID by running /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost from vCenter SSH. 
   2. Ensure the FQDN used in your certificate configuration exactly matches the vCenter Server's PNID.
      Note: This value is strictly case-sensitive, even a mismatch between uppercase and lowercase letters will cause it to fail. To fix a hostname mismatch, you can run /opt/vmware/share/vami/vami_config_net and select option 3 to update the hostname so it perfectly aligns with the PNID.
   3. Re-run certificate-manager and select Option 3, ensuring the hostname provided is correct.
      
      

2. Use the vCert Script (Recommended for VMCA certificate replacement)

   1. Connect to the vCenter via SSH and login as root
   2. Download and transfer the vCert script vCert-6.1.1-20260401.zip to the vCenter Server /tmp directory using WinSCP.  
      Note: You may need to change the default shell using command (chsh -s /bin/bash) to BASH to allow WinSCP file transfers
   3. Navigate to /tmp and unzip the vCert-6.1.1-20260401.zip by running unzip vCert-6.1.1-20260401.zip 
   4. Navigate to newly extracted folder cd vCert-6.1.1-20260401.zip and run ./vCert.py
   5. Select Option 3 (Manage certificates) and follow the prompts to regenerate or replace the necessary certificates.
      
      

Additional Information

If the failure is due to a missing STS registration, use the lsdoctor tool to rebuild service registrations. Refer: Using the 'lsdoctor' Tool