NSXT Upgrade Pre-check from VCF Operations Failed with an error message "NSX Bundle prechecks are already running. Please wait until the existing run is complete before retrying"

Products

• VMware SDDC Manager
• VCF Operations

Issue/Introduction

• When running the NSX precheck from the VCF Operations UI (VCF Operations > Fleet Management > Lifecycle), the check fails and returns the following error:
  error_message : NSX Bundle prechecks are already running. Please wait until the existing run is complete before retrying., httpStatus : , error_code : 0
  
  
• lcm-debug.log(/var/log/vmware/vcf/lcm/lcm-debug.log)may report below entries,
  

  YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,0000000000000000,0000,precheckId=####-#####,resourceType=BUNDLE,resourceId=#####-#####]
[c.v.e.s.l.p.i.nsxt.NsxtUpgradeUtil,Precheck-10] Handling NSX Exception
com.vmware.evo.sddc.lcm.model.error.LcmException: NSX Bundle prechecks are already running. Please wait until the existing run is complete before retrying.
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtParallelClusterPrimitiveImpl.bundlePrecheck(NsxtParallelClusterPrimitiveImpl.java:1387)
        at com.vmware.evo.sddc.lcm.primitive.impl.bundle.BundlePrimitiveImpl.precheck(BundlePrimitiveImpl.java:167)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradePrecheckAsyncHandler$1.run(UpgradePrecheckAsyncHandler.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,0000000000000000,0000,precheckId=#####-#####,resourceType=BUNDLE,resourceId=#####-#####]
[c.v.e.s.l.p.i.n.NsxtParallelClusterPrimitiveImpl,Precheck-10] Pub prechecks after exception: [PrecheckTask(id=null, name=UC NSX Bundle Prechecks, description=Initiate and evaluate NSX Bundle level prechecks, startTime=######, endTime=######, healthStatus=RED, error=PrecheckError(errorCode=NSX_BUNDLE_PRECHECK_EXECUTION_FAILED_WITH_EXCEPTION, errorDescription=error_message : NSX Bundle prechecks are already running. Please wait until the existing run is complete before retrying., httpStatus : , error_code : 0, errorType=ERROR, metadata=null, impact=High: An unexpected exception occurred when running the prechecks., remediation=NSX Bundle Prechecks failed with exception. Address the errors and re-run the prechecks.), errors=[], silenceable=false)]



• lcm-debug.log may report "Upgrade initiation is not possible due to an ongoing rollback" 
  
  YYYY-MM-DDTHH:MM:SS DEBUG [vcf_lcm,#####] [c.v.evo.sddc.lcm.model.task.Stage,http-nio-127.0.0.1-7400-exec-9] Stage details: Stage(id=###, name=null, description=null, type=NSX_UPGRADE_STAGE_SET_UPGRADE_PAYLOAD, order=3, startTime=null, endTime=null, duration=null, status=FAILED, errorType=RECOVERABLE, errorCode=com.vmware.vcf.error.runtime.nsxt.payload.upload.failed, errorDescription=null, remediation=null, resources=null, errors=null, notifications=null, subtaskId=##, subtaskType=UPGRADE, taskId=######, taskType=com.vmware.vcf.lcm.upgrade

  YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm, 0000000000000000, 0000, upgradeId=######, resourceType=NSX_T_PARALLEL_CLUSTER, resourceId=##### :_ ParallelClusterUpgradeElement, bundleElementId=####] [c.v.e.s.l.p.i.nsxt. NsxtPrecheckUtil, Upgrade-5] Payload upload failed: org. springframework. web.client. HttpClientErrorException$BadRequest: 400 Bad Request: "{<EOL> "error_code" : 31521, <EOL> "module_name" : "upgrade-coordinator", <EOL> "error_message" : "Upgrade initiation is not possible due to an ongoing rollback. "<EOL>}"
YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,0000000000000000,0000,upgradeId=#####-#####,resourceType=#####-#####,resourceId=#####-#####
al :_ ParallelClusterUpgradeElement, bundleElementId=#####-#####] [c.v.e.s.l.p.i.nsxt. NsxtUpgradeUtil, Upgrade-5] Handling NSX Exceptionorg. springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{<EOL> "error_code" : 31521, <EOL> "module_name" : "upgrade-coordinator", <EOL> "error_message": "Upgrade initiation is not possible due to an ongoing rollback. "<EOL>}"

   

Environment

VCF 9.x

Cause

Issues with repository synchronization across NSX management cluster nodes may cause this behavior

Resolution

Perform Rolling Reboot of NSX manager nodes. Refer to the process in Reboot an NSX Manager