Warnings for ESXi hosts in Maintenance Mode state and empty/blank status post running NSX upgrade Post Check

Products

• VMware NSX
• VMware NSX-T Data Center
• VMware NSX Networking

Issue/Introduction

• When the LM is upgraded from 2.5 or 3.0.x or 3.1.x to 4.x and Post Check is run for the hosts from NSX GUI, below warnings and Status may be reported.

1. “Host (host_fqdn, node-id) is in maintenance-mode. Please consider taking the host out of maintenance-mode since it has been upgraded.”
2. The Post Check Status may be empty or blank.

• Post Check took longer to complete.
• Validate the below logs from the NSX Manager: 

/var/log/upgrade-coordinator/upgrade-coordinator-tomcat-wrapper.log

INFO   | jvm 1    | 2024/06/09 13:23:16 | "MessagingStreamListenerImpl-1-1" #297 daemon prio=5 os_prio=0 tid=0x00006f0088009800 nid=0x2274ce waiting on condition [0x00006f0127753000]
INFO   | jvm 1    | 2024/06/09 13:23:16 |    java.lang.Thread.State: WAITING (parking)
……
INFO   | jvm 1    | 2024/06/09 13:23:16 | JNI global references: 1800
INFO   | jvm 1    | 2024/06/09 13:23:16 |
INFO   | jvm 1    | 2024/06/09 13:23:16 | Heap
INFO   | jvm 1    | 2024/06/09 13:23:16 |  par new generation   total 39296K, used 24105K [0x00006f01ac000000, 0x00006f01aeaa0000, 0x00006f01b6aa0000)
INFO   | jvm 1    | 2024/06/09 13:23:16 |   eden space 34944K,  56% used [0x00006f01ac000000, 0x00006f01ad34a530, 0x00006f01ae220000)
INFO   | jvm 1    | 2024/06/09 13:23:16 |   from space 4352K, 100% used [0x00006f01ae660000, 0x00006f01aeaa0000, 0x00006f01aeaa0000)
INFO   | jvm 1    | 2024/06/09 13:23:16 |   to   space 4352K,   0% used [0x00006f01ae220000, 0x00006f01ae220000, 0x00006f01ae660000)
INFO   | jvm 1    | 2024/06/09 13:23:16 |  concurrent mark-sweep generation total 224800K, used 194257K [0x00006f01b6aa0000, 0x00006f01c4628000, 0x00006f01cc000000)
INFO   | jvm 1    | 2024/06/09 13:23:16 |  Metaspace       used 211702K, capacity 216207K, committed 218336K, reserved 1239040K
INFO   | jvm 1    | 2024/06/09 13:23:16 |   class space    used 27668K, capacity 28531K, committed 28896K, reserved 1048576K   <<<<<<<-- (28531K-27668K/28531K)~ 3 % free space only.
—----------------------------
INFO   | jvm 1    | 2024/06/25 14:05:21 |    java.lang.Thread.State: WAITING (parking)
INFO   | jvm 1    | 2024/06/25 14:05:21 |     at sun.misc.Unsafe.park(Native Method)
INFO   | jvm 1    | 2024/06/25 14:05:21 |     - parking to wait for  <0x000066366eb8d140> (a java.util.concurrent.ForkJoinPool)
INFO   | jvm 1    | 2024/06/25 14:05:21 |     at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
INFO   | jvm 1    | 2024/06/25 14:05:21 |     at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
INFO   | jvm 1    | 2024/06/25 14:05:21 |     at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
INFO   | jvm 1    | 2024/06/25 14:05:21 |   class space    used 28744K, capacity 29787K, committed 29896K, reserved 1048576K <<<<<<<-- (29787K-28744K/29787K)~ 3.5% free space only

 

Environment

VMware NSX-T Data Center
VMware NSX

Cause

Large host count creates excessive threads causing JVM crashes in Java's post-check process.

Resolution

• The fix is in upcoming NSX releases
• For workaround to overcome this situation, please open a case 
• Extract and upload the below needed information:

1. All Global Manager support bundles (including Active and Standby)
2. All Local Managers support bundles (including all Locations)
3. Login using root credentials to the NSX Manager node where the upgrade/post check is being triggered and collect the below Corfu Table Output:

/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t UpgradeInternal