vCenter upgrade precheck fails with "Unable to run vCenter NTP server check" due to STS certificate chain inconsistencies

Products

• VMware vCenter Server

Issue/Introduction

• When upgrading vCenter Server (e.g., from version 9.0.x or 9.1), the Stage 1 pre-upgrade check may fail with the following error:

  Unable to run vCenter NTP server check

  
  

• While the NTP configuration appears valid in the vCenter Management Interface (VAMI), the installer cannot verify the status. This issue is characterized by authentication failures within the appliance management services.
• The following errors are observed in the logs of the source vCenter:
  • In /var/log/vmware/applmgmt/applmgmt.log

    ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
    Traceback (most recent call last):
      File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 301, in validate
        self.validate_certificate()
      File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 503, in validate_certificate
        raise AuthenticationError(
    vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

     

    validate_certificate
        raise AuthenticationError(
    vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

     

    INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.system.time, operation_id: get

    DEBUG:vmware.appliance.health.impl:com.vmware.appliance.version1.system.time.get()

  • In /var/log/vmware/applmgmt/update_microservices.log

    ERROR applmgmt-update-microservice ######## [vc@#### module="ntp_precheck" funcName="checkNtpServers" lineNum="57"] Unable to retrieve NTP servers. err: {{challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}}

  • In journalctl:

    ntpd[######]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

Environment

• vCenter Server 8.x
• vCenter Server 9.x

Cause

The upgrade precheck script uses the applmgmt VAPI endpoint to verify NTP synchronization. This authentication relies on the Security Token Service (STS) certificate. If the STS certificate store contains an invalid number of certificate chains the Holder-of-Key (HOK) token cannot be parsed. This results in an UNAUTHENTICATED error, causing the NTP check to fail even if the NTP service itself is running correctly.

Resolution

To resolve this issue, consolidate and renew the STS certificate chain to ensure the VAPI endpoint can authenticate successfully.

Follow the below steps

1. Download the vCert utility and upload to the vCenter Server.
2. SSH to vCenter server with root user
3. In vCert utility, Use Option# 3 (Manage certificates) > Option# 8 (STS signing certificates) > Option# 1 (Replace STS Signing certificate with a VMCA-signed certificate) to renew STS signing certificate
4. Restart vCenter server services

   service-control --stop --all && service-control --start --all

5. Log into the VAMI (https://<vcenter-ip>:5480) and ensure the Time section correctly reflects the NTP status without errors.
6. Retry the vCenter upgrade and proceed with the pre-upgrade checks.