VCF 5.x Upgrade fails at NSX upgrade Coordinator stage
Article ID: 408644
Updated On:
Products
Issue/Introduction
- When applying updates to VCF environment through SDDC UI, the update may fail with similar error to below:
"NSX upgrade Coordinator - error_message: I/O error on POST request for "https://NSX-FQDN.domain.local/api/v1/upgrade/bundles": Connection reset by peer, httpStatus: , error_code: 0"
- We may see below also in UI:
- Looking at SDDC logs in /log/vmware/vcf/lcm/lcm.log we may see similar entries to below:
####-##-##T##:##:##.###+#### ERROR [vcf_lcm,0000000000000000,0000,upgradeId=#######-#######-######-#####,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=NSX-FQDN.domain.local:_ParallelClusterUpgradeElement,bundleElementId=#######-#######-######-#####] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-1] Handling NSX Exception
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://NSX-FQDN.domain.local/api/v1/upgrade/bundles": Connection reset by peer
and/or
####-##-##T##:##:##.###+#### ERROR [vcf_lcm,0000000000000000,0000,auditId=#######-#######-######-#####,resourceType=NSX_T_MANAGER,resourceId=NSX-FQDN.domain.local,name=NSX-FQDN.domain.local] [c.v.v.c.n.s.c.c.ComplexHelpers,Upgrade-1] Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
messages = [],
data = struct => {error_message=Cannot connect to server, error_code=98, module_name=common-service},
errorType = INTERNAL_SERVER_ERROR
Environment
VMware Cloud Foundation 5.x
NSX
Cause
The issue is caused by upgrade-coordinator service becoming unresponsive on NSX Manager nodes due to persistent long-running transaction exceptions.
As a result, SDDC is unable to load NSX cluster inventory, leading to a failure during the upgrade
Resolution
Please restart install-upgrade service on all the 3 NSX Manager nodes using “restart service install-upgrade” command.
Additional Information
Feedback
