NSX 4.1.2 upgrade prechecks fail with "NSX Manager upgrade dry run failed"

Products

• VMware NSX

Issue/Introduction

• NSX Upgrade Pre-Checks fail with "NSX Manager upgrade dry run failed. Do not proceed with the upgrade."
• NSX Manager logs may show an error similar to this example

/var/log/syslog

2023-09-21T05:14:46.957Z ERROR providerTaskExecutor-87 PolicyProviderUtil 4917 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM0" level="ERROR" subcomp="manager"] Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/logical-ports/infra-########-####-####-####-############-default:########-####-####-####-############-lp/alarms/########-####-####-####-############, message=TX ABORT  | Snapshot Time = Token(epoch=2164, sequence=3391905417) | Failed Transaction ID = ########-####-####-####-############ | Offending Address = 3391905445 | Conflict Key = 3E0923C968CFCDDC | Conflict Stream = ########-####-####-####-############ | Cause = CONFLICT | Time = 194 ms,errorId=PROVIDER_INVOCATION_FAILURE, path=null, apiError=null, sourceSiteId=null].
 

 

/var/log/upgrade-coordinator/logical-migration.log

2023-10-02T14:52:58.244Z ERROR ViewsGarbageCollector AbstractView 3162186 getLayoutUninterruptibly: Encountered error. Aborting layoutHelper
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

2023-10-02T14:56:05.371Z ERROR main Migration 3162186 - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"] Migration failed
java.lang.RuntimeException: java.util.concurrent.TimeoutException

 

/image contains a dump file migration_oom.hprof.gz

Environment

VMware NSX-T 4.x

NSX Upgrade to 4.1.2 from version 3.2.x, 4.0.x or 4.1.x.

Cause

This issue occurs when the NSX Manager database contains a large number of stale port entries which causes unexpectedly high transaction times.
Note upgrades from greenfield deployments of 4.1.1 are not impacted by this issue.

Resolution

This is a known issue impacting NSX, currently there is no resolution.

Workaround:
Please open a Support Request with Broadcom Support to get guidance on a workaround for this issue.