NSX upgrade precheck stuck in SDDC due to Backup List call.[Request::URI:http://localhost:7440/nsxapi/api/v1/cluster/backups/overview method:GET]
search cancel

NSX upgrade precheck stuck in SDDC due to Backup List call.[Request::URI:http://localhost:7440/nsxapi/api/v1/cluster/backups/overview method:GET]

book

Article ID: 404740

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware NSX

Issue/Introduction

Symptoms

  • While performing the NSX parallel cluster upgrade from the SDDC manager, the upgrade may fail with a timeout error. 
  • This issue may also occur if you are performing a hot patch upgrade of the NSX manager.
  • The NSX manager ui may report a different status in the Upgrade tab.
  • Upon checking the logs, one may notice similar snippets.

In the SDDC manager, checking /var/log/vmware/vcf/lcm/lcm.log you will notice the below:

YYYY-MM-DDT08:52:55.416+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=3eb4b7f8-d295-xxxx-xxxx-cbe29f5c6995,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=NSX_FQDN:_ParallelClusterUpgradeElement,bundleElementId=6771dbfc-5e66-4a15-xxxx-ff54e96982e5] [c.v.v.c.n.s.c.c.ComplexHelpers,Upgrade-8] Exception occurred during NSX API invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.client.exception.ConnectionException: Read timed out
 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
 at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
 at com.vmware.vapi.bindings.CompletionStageFuture.get(CompletionStageFuture.java:45)
 at com.vmware.vcf.common.nsxt.sdk.client.connection.ComplexHelpers.invoke(ComplexHelpers.java:188)
 at com.vmware.vcf.common.nsxt.sdk.client.connection.UpgradeCoordinatorOperations.continueUpgrade(UpgradeCoordinatorOperations.java:101)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtUpgradeStageRunner.performUpgrade(NsxtUpgradeStageRunner.java:814)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtUpgradeStageRunner.performUpgradeForComponent(NsxtUpgradeStageRunner.java:751)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtManagerClusterUpgradeStageRunner.performUpgradeForManagerCluster(NsxtManagerClusterUpgradeStageRunner.java:190)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.service.NsxtManagerClusterUpgradeStageRunner.doUpgradeStage(NsxtManagerClusterUpgradeStageRunner.java:81)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl.runUpgrade(NsxtParallelClusterPrimitiveImpl.java:571)
 at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl.postUpgrade(NsxtParallelClusterPrimitiveImpl.java:215)
 at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:313)
 at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:165)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.vapi.client.exception.ConnectionException: Read timed out
 at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:93)
 at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultRequestExecutor.execute(DefaultRequestExecutorFactory.java:45)
 at com.vmware.vapi.internal.protocol.client.rest.RestClientApiProvider.invoke(RestClientApiProvider.java:67)
 at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
 at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191)
 at com.vmware.nsx.upgrade.PlanStub.access$2800(PlanStub.java:19)
 at com.vmware.nsx.upgrade.PlanStub$Invoker.invoke(PlanStub.java:320)
 at com.vmware.nsx.upgrade.PlanStub$ContinueInvocationImpl.invoke(PlanStub.java:76)
 at com.vmware.nsx.upgrade.PlanStub$ContinueInvocationImpl.invoke(PlanStub.java:71)


YYYY-MM-DDT08:52:55.417+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=3eb4b7f8-d295-xxxx-xxxx-cbe29f5c6995,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=NSX_FQDN:_ParallelClusterUpgradeElement,bundleElementId=6771dbfc-5e66-4a15-xxxx-ff54e96982e5] [c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Upgrade-8] Handling NSX Exception
com.vmware.vapi.client.exception.ConnectionException: Read timed out
 at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheClientRestTransport.execute(ApacheClientRestTransport.java:93)

In the NSX manager's UpgradeCoordinator log, you will notice a similar snippet:


YYYY-MM-DDT08:42:41.956Z  INFO pool-XX-thread-1 UpgradeServiceImpl XXX SYSTEM [nsx@4413 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Executing check Backup Operation Check
YYYY-MM-DDT08:42:55.312Z  INFO http-nio-127.0.0.1-7442-exec-3 UpgradeCoordinatorFacadeImpl 1957971 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Invoke Continue upgrade with Parameters com.vmware.nsxapi.upgrade.dto.ContinueRequestParameters@679458de[componentType=MP,hasComponentType=true,hasSkip=true,skip=false]
YYYY-MM-DDT08:43:01.583Z  INFO http-nio-127.0.0.1-7442-exec-3 LoggingRestTemplate 1957971 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Request::URI:http://localhost:7440/nsxapi/api/v1/cluster/backups/overview method:GET

Environment

SDDC Manager

VMware NSX

Cause

The problem is when the upgrade coordinator in NSX executes the check for the Backup Operation Check. In this phase, the precheck invokes the API call GET /nsxapi/api/v1/cluster/backups/overview. This is getting blocked and never returns the output.

Resolution

  1. If you encounter the symptoms in this article, please engage Broadcom support.

    Currently, there is no resolution. Below is the workaround to implement.

    Workaround

    1. Restart the UC (upgrade coordinator) service in the NSX manager:
      1. SSH session to one of the NSX managers.
      2. Run the following command : get service install-upgrade
        1. The expected output will show the IP address of the node that runs the UC service. Sample output: 

    1. Then, SSH session to the UC node and restart the UC service.
      1. Admin: restart service install-upgrade
      2. Root: /etc/init.d/upgrade-coordinator restart
    2. If this does not help, then rolling reboot of the NSX Managers.
    3. Ensure that the Upgrade page in NSX Manager is loading successfully.
    4. Ensure that the backup and restore page is loading with all details in the NSX Manager.
    5. Then, trigger the upgrade of the NSX manager.
Powered by Wolken Software