ESXi upgrade from version 7.0 to 8.0 via vLCM fails with the cluster health check error - "Error while calling getSoftwareSolutionsBeingApplied API".

Products

• VMware NSX
• VMware NSX-T Data Center
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• On the vCenter --> Cluster --> Updates page, we may see remediation errors similar to the below:
  
  

• In the NSX manager's /var/log/upgrade-coordinator/upgrade-coordinator.log, we see messages similar to the below:
  
  2024-09-12T05:02:57.573Z  INFO http-nio-127.0.0.1-7442-exec-2 EHPFacadeImpl 77857 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [EHP] Begin execution of ReadyForApply  checks. Request body - EHPCheckRequestDto{vcenterGuid='########-####-####-####-########8a7b', entityId='domain-##5', disabledChecks='[]', targetSpec='null'}
[..]
2024-09-12T05:02:57.903Z  WARN http-nio-127.0.0.1-7442-exec-2 VcConnectionImpl 77857 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] Error occurred while getting vapi session Id for cm vcenter_hostname
com.vmware.nsx.management.lcm.common.exception.LcmRestException: org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized: "{"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","messages":[{"args":[],"default_message":"Authentication required.","id":"com.vmware.vapi.endpoint.method.authentication.required"}],"challenge":"Basic realm=\"VAPI endpoint\",SIGN realm=374cef8ab43f1253d83604388cb8f30c77fc1dad201a60e81c1957b311da3f6d,service=\"VAPI endpoint\",sts=\"https://vcenter_hostname/sts/STSService/vsphere.local\""}}"
[..]
2024-09-12T05:02:57.904Z  WARN http-nio-127.0.0.1-7442-exec-2 VcConnectionImpl 77857 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] service account is not authorized
[..]
2024-09-12T05:02:57.975Z  WARN http-nio-127.0.0.1-7442-exec-2 LcmRestClientFactory 77857 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] Failed to initialize rest template using thumbprint java.lang.NullPointerException: null
[..]
2024-09-12T05:02:57.975Z  INFO http-nio-127.0.0.1-7442-exec-2 RestConnectionConfiguration 77857 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] RestConnectionConfiguration: Host=vcenter_hostname Port=443 User=a thumbprint=null
2024-09-12T05:02:57.975Z  WARN http-nio-127.0.0.1-7442-exec-2 RestConnectionConfiguration 77857 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] RestConnectionConfiguration: Failed to enable trust java.lang.NullPointerException: null
[..]
2024-09-12T05:02:58.006Z  INFO http-nio-127.0.0.1-7442-exec-2 EHPServiceImpl 77857 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Error while calling getSoftwareSolutionsBeingApplied API. So returning NOT_OK


• In the vCenter's /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log, we see messages similar to the below:
  
  2024-09-12T06:02:57.489+01:00 info vmware-vum-server[59958] [Originator@6876 sub=EHP] Current NSX-T node: https://nsx_manager_or_vip:443
2024-09-12T06:02:57.492+01:00 info vmware-vum-server[59958] [Originator@6876 sub=EHP] NSX-T version: 4.1.0.2
2024-09-12T06:02:57.492+01:00 info vmware-vum-server[59958] [Originator@6876 sub=EHP] Calling NSX-T API /api/v1/vlcm/esx/health/cluster/perspectives/ready-for-apply/status?action=check (/external-tp/http1/nsx_manager_or_vip/443/6A4E7585345ECEF19B1D1BFFFBA972435FEA40EC/api/v1/vlcm/esx/health/cluster/perspectives/ready-for-apply/status?action=check).
[..]
2024-09-12T06:02:58.010+01:00 info vmware-vum-server[59958] [Originator@6876 sub=EHP] [domain-##5] [nsxt_rest] [com.vmware.nsxt.Cluster_tnp_or_tn_exists_check] reported issue: Error invoking getSoftwareSolutionsBeingApplied API
[..]
2024-09-12T06:02:58.010+01:00 info vmware-vum-server[59958] [Originator@6876 sub=EHP] [domain-##5] [Unknown] [com.vmware.nsxt.Cluster_tnp_or_tn_exists_check] returned status: NOT_OK
[..]
2024-09-12T06:02:58.010+01:00 info vmware-vum-server[50094] [Originator@6876 sub=Telemetry] [TelemetryManager 423] Sending telemetry data: {"@type":"pman_error_report","taskId":"########-####-####-####-########8a7b|########-####-####-####-########df41","entityId":"########-####-####-####-########8a7b|domain-##5","parentTaskId":"","errorMessageId":"30198","errorMessage":"Error invoking getSoftwareSolutionsBeingApplied API","errorTime":"2024-09-12T05:02:58.010813Z"}
[..]
2024-09-12T06:02:58.016+01:00 error vmware-vum-server[49903] [Originator@6876 sub=RemediateClusterTask] [Task, 511] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:########-####-####-####-########df41. Task Failed. Error: Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed<Health Check for 'cluster_name' failed>

Environment

VMware NSX-T Datacenter
VMware NSX
VMware vCenter Server
VMware vSphere ESXi

Cause

The issue occurs in relation to the service account created to integrate NSX with a Compute Manager for the vLCM functionality to authenticate vAPIs from NSX. Issue can occur in two scenarios:

• When the service account password is manually changed.
• When the service account password is expired and for some reason, it is not automatically renewed.

Resolution

To refresh the credentials of the service account, please follow the below steps and re-attempt the upgrade:

1. Edit the "Compute Manager" connection --> Enter or change the description field (this is just a cosmetic change).
2. In the FQDN or IP address field --> Click on Edit --> Enter the vCenter's admin username and password.
3. Click on Save.