NSX Upgrade Stage Edge Precheck Failed in SDDC

Products

• VMware Cloud Foundation

Issue/Introduction

Unable to run prechecks on Edges when the check is triggered from SDDC.

Error Screenshot:

Log Snippet :

lcm-Debug.log

xxxx.xx.xxTxx:xx:xx.059+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=xxxxx-xxxx-xxxx-xxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=<NSX FQDN>:_ParallelClusterUpgradeElement,bundleElementId=xxxxx-xxxx-xxx-xx] [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner,Upgrade-1] Resetting Upgrade plan for EDGE
 to default
xxxx-xx-xxTxx:xx:xx.083+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=xxxxx-xxxx-xxxx-xxx,resourceType=NSX_T_PARALLEL_CLUSTER,resourceId=<NSX FQDN>:_ParallelClusterUpgradeElement,bundleElementId=xxxxx-xxxx-xxx-xx] [c.v.e.s.l.p.i.n.s.NsxtEdgeClusterParallelUpgradeStageRunner,Upgrade-1] Failed to trigger NSX upgrade coordinator checks for EDGE with exception {}
com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => {
    messages = [],

oordinator checks for EDGE with exception {}
com.vmware.vapi.std.errors.InvalidRequest: InvalidRequest (com.vmware.vapi.std.errors.invalid_request) => {
    messages = [],
    data = struct => {error_message=Cannot modify upgrade plan since pre-upgrade checks are in progress. Please wait and retry., httpStatus=BAD_REQUEST, error_code=30927, module_name=upgrade-coordinator},
    errorType = INVALID_REQUEST

Environment

Vmware cloud Foundation 4.x

Vmware cloud Foundation 5.x

Cause

In a large scale setup during NSX T edge pre-check Upgrade coordinator loads all the compute collections and matches it against the compute on which the edge is deployed. This workflow is executed in parallel processing to load all Edge clusters at the same time. This leads to the UC going OOM and hence the pre-check fails. 

Resolution

Skip NSX pre-checks, and upgrade from SDDC manager UI

Note : Take the snapshot before the activity 

• • Open an SSH session to the SDDC Manager VM and switch to the root user:
 su
  • Navigate to /opt/vmware/vcf/lcm/lcm-app/conf/
# cd /opt/vmware/vcf/lcm/lcm-app/conf
  • In application-prod.properties add the following the timeout value as below:
# lcm.nsxt.suppress.prechecks=true
  • Restart the LCM service.
# systemctl restart lcm
  • Once NSX Upgrade is complete, revert the changes made above and restart LCM.