Viewing Machine SSL and Trusted root store in vSphere web client fails with error message "Error occurred while fetching machine certificates: Service not found"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms

When attempting to manage certificates in the vCenter Server, users encounter the following issues:

• Navigating to Menu > Administration > Certificate Management results in the error: Error occurred while fetching machine certificates: Service not found: com.vmware.vcenter.certificate_management.vcenter.tls

• Critical vCenter services like vpxd, vpxd-svcs, and vmware-certificatemanagement fail to initialize using the command : service-control --start --all

• Below error message is observed in logs in following conditions:
  • A hostname mismatch in SAN field will trigger the following log entry in /var/log/vmware/vmon/vmon.log when the vpxd-svcs fails to start:

[YYYY-MM-DDTHH:MM:SS] Wa (03) host-####### <vpxd-svcs> Service pre-start command's stderr: ssl. SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: Hostname mismatch, certificate is not valid for 'vCenter_FQDN'. (_ssl.c:1007)

• • Expired Machine SSL certificate will trigger the following log entry in /var/log/vmware/applmgmt/applmgmt.log

[YYYY-MM-DDTHH:MM:SS] DEBUG: vmware. vherd. transport. authentication: Authentication Server Secret Renewed.
[YYYY-MM-DDTHH:MM:SS] INFO: vmware. vherd. transport. ssh_access_collector: [Unit Test] renewed 50 credits to post event
[YYYY-MM-DDTHH:MM:SS] INFO: vmware. vherd. transport. ssh_access_collector: [Unit Test]Start collecting from sshinfo. log
[YYYY-MM-DDTHH:MM:SS] ERROR:cis. vpxdevent_lib: Failed to get vcenter server endpoint urls. Err [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
[YYYY-MM-DDTHH:MM:SS] ERROR: vmware. vherd. transport. post_sso_events: Failed to get vcenter server endpoint urls. Err [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)

Environment

• VMware vCenter Server
• VMware vSphere ESXi

Cause

This issue typically occurs due to one of the following conditions:

1. Hostname Mismatch: The Machine SSL certificate’s Subject Alternative Name (SAN) or Common Name (CN) does not match the actual vCenter Server FQDN.

2. Certificate Expiration: The Machine SSL certificate has expired, preventing secure communication between internal service endpoints.

Resolution

Replace the Machine SSL certificate using one of the following methods:

Option 1:

• Replace the machine certificate using the vCert script.
  
  
  • Download and upload the vCert zip file to vCenter
    •  unzip -q vCert-<version>.zip
    • cd vCert-<version>
    • ./vCert.py
• Use the following menu options to replace the machine certificate:
  • Manage certificates (3)> Machine SSL certificate (1) 


Option 2:

• To Manually replace the machine SSL certificate refer the below:
  
  
  • For Self-Signed (Default) Certificates:
    Regenerate vSphere certificates using VMCA

• • For Custom CA-Signed Certificates:
     Replace vCenter Machine SSL using custom certificate