NSX installation is failing on the host with an error 'vLCM remediation initiated'

Products

• VMware NSX

Issue/Introduction

• NSX installation failed on a host/cluster that is managed by vLCM
• NSX Installation failed at stage 'vLCM remediation initiated' with the following error

Failed to install software on host. Solution apply failed because the vSphere Lifecycle image contains either a new ESXi version or a new addons version or new components. Please proceed to the vSphere Client Lifecycle Manager to update ESXi or addons or components along with the solution 'com.vmware.nsxt' Solution apply failed on host: 'esxi1.example.com'. The vSphere Lifecycle Manager image contains a new 'ESXi' version '<vSphere ESXi version>'. Please proceed to the vSphere Client Lifecycle Manager to update ESXi and the solution 'com.vmware.nsxt'Solution apply failed on host: 'esxi1.example.com'

• Entries similar to the below are observed in the NSX Manager log /var/log/proxy/reverse-proxy.log

INFO https-jsse-nio-###.###.###.###-443-exec-11 BaseProxyDelegate 6565 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Processing request to /api/v1/vlcm/esx/health/cluster/perspectives/ready-for-apply/status (action=check) with /api/v1/vlcm -> 127.0.0.1:7442:/upgrade-coordinator/api/v1/vlcm

INFO https-jsse-nio-###.###.###.###-443-exec-11 BaseProxyDelegate 6565 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] Request executed, going to transfer response HttpResponseProxy{HTTP/1.1 404  [Content-Type: text/html;charset=utf-8, Content-Language: en, Content-Length: 682, Date: Mon, 04 Nov 2024 05:39:41 GMT, Keep-Alive: timeout=60, Connection: keep-alive] ResponseEntityProxy{[Content-Type: text/html;charset=utf-8,Content-Length: 682,Chunked: false]}}

• Entries similar to the below are observed in the NSX Manager log /var/log/upgrade-coordinator/localhost_access_log.txt

2024-11-04T05:39:42.593Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/vlcm/esx/health/cluster/perspectives/ready-for-apply/status?action=check HTTP/1.1" 404 682

• Entries similar to the following in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-#.log

error vmware-vum-server[######] [Originator@#### sub=ClusterApplySolutionTask opID=<ID>] [Task, ###] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplyNsxTask ID:<ID>. Apply (solution = com.vmware.nsxt) failed on some hosts while doing parallel remediation

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The upgrade coordinator service is not responding due to a crash or other system issue affecting it's functionality.

Resolution

To workaround this issue, restart the upgrade-coordinator:

1. SSH into a NSX manager 
2. Run get service install-upgrade to find which manager is the Upgrade Coordinator
3. If it is a different NSX manager, SSH into that NSX manager and run
   • /etc/init.d/upgrade-coordinator restart
4. Resolve the Alarm in NSX.