NSX upgrade on VCF fails with "Read timed out, httpStatus : , error

Products

VMware Cloud Foundation

Issue/Introduction

An NSX upgrade to 4.1.0 or later fails on the VCF side during the NSX manager postcheck.
The upgrade completes successfully from the NSX UI.
Retrying the NSX upgrade from the VCF UI fails in the precheck for the manager cluster with the error: Check for the version sync status of UC on all the MP nodes.
The failed task in VCF shows the following:

Message: error_message : Read timed out, httpStatus : , error_code : 0

Remediation Message: NSX upgrade failed in upgrading manager cluster. Check for errors in the LCM log files at 127.0.0.1:/var/log/vmware/vcf/lcm, and address those errors. Please run the upgrade precheck and restart the upgrade.

The Lifecycle Manager service log /var/log/vmware/vcf/lcm/lcm.log has errors similar to the following:

2024-01-09T05:52:16.564+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-########7c82,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId={NSX_FQDN}:_ParallelClusterUpgradeElement,bundleElementId=########-####-####-####-########6931] [c.v.e.s.l.p.i.n.s.NsxtUpgradeStageRunner,Upgrade-6] performPostcheckForComponent failed for MP with exception {}
com.vmware.vapi.client.exception.ConnectionException: Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out

2024-01-09T05:52:16.565+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-########7c82,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId={NSX_FQDN}:_ParallelClusterUpgradeElement,bundleElementId=########-####-####-####-########6931] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-6] Handling NSX Exception
com.vmware.vapi.client.exception.ConnectionException: Read timed out
com.vmware.vapi.client.exception.ConnectionException: Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
   
2024-01-09T05:52:16.565+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=########-####-####-####-########7c82,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId={NSX_FQDN}:_ParallelClusterUpgradeElement,bundleElementId=########-####-####-####-########6931] [c.v.e.s.l.p.i.n.s.NsxtManagerClusterUpgradeStageRunner,Upgrade-6] NSX Manager failed in Upgrade Coordinator postchecks NSXT_MANAGER_CLUSTER_UPGRADE_FAILED_POSTCHECK

Environment

VMware Cloud Foundation 5.x

Cause

Starting with NSX 4.1.0, NSX Manager blocks additional prechecks once its native upgrade is successful. If the NSX upgrade completes successfully on the NSX side but fails during the VMware Cloud Foundation (VCF) post-check phase, retrying the upgrade via SDDC Manager erroneously triggers a new NSX precheck. Because NSX has already been upgraded, this precheck is rejected, returning a UC version issue and causing the VCF upgrade workflow to fail.

Resolution

This issue is under review with Broadcom Engineering. Subscribe to this knowledge article to get future and ongoing updates.

Workaround to ignore the precheck in VCF:

Open an SSH session to the SDDC Manager and switch to the root user: su

Take a backup of the application-prod.properties file:

cp /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties application-prod.properties_backup

Open application-prod.properties for editing:

vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

Add the following line to the bottom and save the file:
```
lcm.nsxt.suppress.prechecks=true
```
Restart the LCM service:
```
systemctl restart lcm
```
Wait a minute for the service to start and verify the LCM service is running:
```
curl localhost/lcm/about | json_pp
```
Retry the NSX upgrade from the VCF UI.
One the NSX upgrade completes, then edit the parameter line entered previously.

Open application-prod.properties for editing:

vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties

Change the line parameter from true to false and save the file:
```
lcm.nsxt.suppress.prechecks=false
```
Restart the LCM service:
```
systemctl restart lcm
```