Upgrade of NSX Manager from version 3.2.2 to 4.0 or 4.1 hangs on step Unpin UI

Products

• VMware NSX

Issue/Introduction

• NSX-T Data Center 3.2.2.* being upgraded to NSX 4.0.x or 4.1.x.
• During the upgrade, the Upgrade Coordinator's UI will hang on "Unpin UI".
• You see messages similar to the following in the manager's logs:

/etc/opensearch/log4j2.properties
org.opensearch.OpenSearchException: failed to bind service
 at org.opensearch.node.Node.<init>(Node.java:965) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]

• Same error is observed in Manager's /var/log/search/opensearch.log:

/var/log/search/opensearch.log
[2024-01-04T08:59:48,751][ERROR][o.o.b.Bootstrap          ] [nsx_node] Exception
org.opensearch.OpenSearchException: failed to bind service
..
Caused by: java.nio.file.AccessDeniedException: /nonconfig/search/nodes/0
..
[2024-01-04T08:59:48,758][WARN ][stderr                   ] [nsx_node] Caused by: org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/nonconfig/search/nodes/0];

 

• NSX Manager's UI may be inaccessible, and reporting services down:

"Some appliance components are not functioning properly."
"Component health: MANAGER:UP, SEARCH:DOWN, UI:UP, NODE_MGMT:UP","error_code":101

Environment

VMware NSX-T Data Center
VMware NSX

Cause

This issue is caused by race condition that occurs during upgrade from 3.2.2.* to 4.0.x or 4.1.x, where Elasticsearch is not able to stop in time and was eventually terminated after upgrade process tried to start Opensearch.
This update of specific user permissions, which will cause services to fail to start.

Resolution

This issue is resolved in VMware NSX 4.2.0

Workaround:
This workaround needs to be applied on all NSX Managers in the impacted cluster:

1. SSH to NSX Managers as root.
2. Change directory to /nonconfig/search:
   
   cd /nonconfig/search
   
   
3. List content of the directory:
   
   ls -l
   
   Note: The output will be similar to:
   
   # ls -l
drwxr-x--- 3 elasticsearch elasticsearch 4096 Apr 25  2023 nodes
drwxr-xr-x 2 nsx-search    nsx-search    4096 Jan  4 13:13 tmp
   
   
4. Change the ownership of the content in /nonconfig/search to owner and group "nsx-search":
   
   chown -R nsx-search:nsx-search nodes
   
   
5. SSH to the Manager as admin.
6. Start service search:
   
   start service search

Additional Information

Impact/Risks:
NSX Manager will fail to upgrade.