NSX upgrade precheck stuck in SDDC due to Backup List call.[Request::URI:http://localhost:7440/nsxapi/api/v1/cluster/backups/overview method:GET]

Products

VMware SDDC Manager VMware NSX

Issue/Introduction

Symtoms

While performing the NSX parallel cluster upgrade from the SDDC manager, the upgrade may fail with a timeout error.
This issue may also occur if you are performing a hot patch upgrade of the NSX manager.
The NSX manager ui may report a different status in the Upgrade tab.
Upon checking the logs, one may notice similar snippets.

In the SDDC manager, checking /var/log/vmware/vcf/lcm/lcm.log you will notice the below:

YYYY-MM-DDT08:52:55.416+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=3eb4b7f8-d295-xxxx-xxxx-cbe29f5c6995,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=NSX_FQDN:_ParallelClusterUpgradeElement,bundleElementId=6771dbfc-5e66-4a15-xxxx-ff54e96982e5] [c.v.v.c.n.s.c.c.ComplexHelpers,Upgrade-8] Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.client.exception.ConnectionException: Read timed out
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.vmware.vapi.bindings.CompletionStageFuture.get(CompletionStageFuture.java:45)
at com.vmware.vcf.common.nsxt.sdk.client.connection.ComplexHelpers.invoke(ComplexHelpers.java:188)
at com.vmware.vcf.common.nsxt.sdk.client.connection.UpgradeCoordinatorOperations.continueUpgrade(UpgradeCoordinatorOperations.java:101)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtUpgradeStageRunner.performUpgrade(NsxtUpgradeStageRunner.java:814)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtUpgradeStageRunner.performUpgradeForComponent(NsxtUpgradeStageRunner.java:751)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtManagerClusterUpgradeStageRunner.performUpgradeForManagerCluster(NsxtManagerClusterUpgradeStageRunner.java:190)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtManagerClusterUpgradeStageRunner.doUpgradeStage(NsxtManagerClusterUpgradeStageRunner.java:81)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl.runUpgrade(NsxtParallelClusterPrimitiveImpl.java:571)
at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl.postUpgrade(NsxtParallelClusterPrimitiveImpl.java:215)
at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:313)
at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:165)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.vapi.client.exception.ConnectionException: Read timed out
at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:93)
at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:45)
at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
at com.vmware.nsx.upgrade.PlanStub.access$2800(PlanStub.java:19)
at com.vmware.nsx.upgrade.PlanStub$Invoker.invoke(PlanStub.java:320)
at com.vmware.nsx.upgrade.PlanStub$ContinueInvocationImpl.invoke(PlanStub.java:76)
at com.vmware.nsx.upgrade.PlanStub$ContinueInvocationImpl.invoke(PlanStub.java:71)

YYYY-MM-DDT08:52:55.417+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=3eb4b7f8-d295-xxxx-xxxx-cbe29f5c6995,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=NSX_FQDN:_ParallelClusterUpgradeElement,bundleElementId=6771dbfc-5e66-4a15-xxxx-ff54e96982e5] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-8] Handling NSX Exception
com.vmware.vapi.client.exception.ConnectionException: Read timed out
at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:93)

In the NSX manager's UpgradeCoordinator log, you will notice a similar snippet:

YYYY-MM-DDT08:42:41.956Z INFO pool-XX-thread-1 UpgradeServiceImpl XXX SYSTEM [nsx@4413 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Executing check Backup Operation Check
YYYY-MM-DDT08:42:55.312Z INFO http-nio-127.0.0.1-7442-exec-3 UpgradeCoordinatorFacadeImpl 1957971 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Invoke Continue upgrade with Parameters com.vmware.nsxapi.upgrade.dto.ContinueRequestParameters@679458de[componentType=MP,hasComponentType=true,hasSkip=true,skip=false]
YYYY-MM-DDT08:43:01.583Z INFO http-nio-127.0.0.1-7442-exec-3 LoggingRestTemplate 1957971 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Request::URI:http://localhost:7440/nsxapi/api/v1/cluster/backups/overview method:GET

Environment

SDDC Manager

VMware NSX

Cause

The problem is when the upgrade coordinator in NSX executes the check for the Backup Operation Check. In this phase, the precheck invokes the API call GET /nsxapi/api/v1/cluster/backups/overview. This is getting blocked and never returns the output.

Resolution

If you encounter the symptoms in this article, please engage Broadcom support.

Currently, there is no resolution. Below is the workaround to implement.

Workaround

1. Restart the UC service in the NSX manager.

To identify the node that runs the UC service, run this command:
Take an SSH session to one of the NSX managers.
Run the command : get service install-upgrade
The expected output will show the IP address of the node that runs the UC service.

Sample output:

nsx-mngr-01> get service install-upgrade

Service name: install-upgrade
Service state: running
Enabled on: 192.168.XX.X <--IP of the NSX node where the UC service is enabled.

d. Take an SSH session to this node and restart the UC service.

2. If this does not help, then reboot the NSX manager nodes one by one. (Reboot nodeA, wait for the boot to complete, open the NSX manager UI for this node, and verify the status under System >Service Deployments >Service Instances. Only once everything is green, reboot the other node B and so on).
3. Ensure that the Upgrade page in NSX Manager is loading successfully.
4. Ensure that the backup and restore page is loading with all details in the NSX Manager.
5. Then, trigger the upgrade of the NSX manager.