Upgrade of NSX Manager from version 3.2.2 to 4.0 or 4.1 hangs on step Unpin UI
search cancel

Upgrade of NSX Manager from version 3.2.2 to 4.0 or 4.1 hangs on step Unpin UI


Article ID: 323553


Updated On:


VMware NSX Networking


  • NSX-T Data Center 3.2.2.* being upgraded to NSX 4.0 or 4.1.
  • During the upgrade, the Upgrade Coordinator's UI will hang on "Unpin UI".
  • "failed to bind service" is observed in Manager's logs, e.g.:
org.opensearch.OpenSearchException: failed to bind service
 at org.opensearch.node.Node.<init>(Node.java:965) ~[opensearch-1.2.3.jar:1.2.3]
 at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
  • Same error is observed in Manager's /var/log/search/opensearch.log:
[2024-01-04T08:59:48,751][ERROR][o.o.b.Bootstrap          ] [nsx_node] Exception
org.opensearch.OpenSearchException: failed to bind service
Caused by: java.nio.file.AccessDeniedException: /nonconfig/search/nodes/0
[2024-01-04T08:59:48,758][WARN ][stderr                   ] [nsx_node] Caused by: org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/nonconfig/search/nodes/0];
  • NSX Manager's UI may be inaccessible, and reporting services down:
"Some appliance components are not functioning properly."
"Component health: MANAGER:UP, SEARCH:DOWN, UI:UP, NODE_MGMT:UP","error_code":101


VMware NSX-T Data Center


This issue is caused by race condition that occurs during upgrade from 3.2.2.* to 4.0 or 4.1, where Elasticsearch is not able to stop in time and was eventually terminated after upgrade process tried to start Opensearch.
This update of specific user permissions, which will cause services to fail to start.


This issue is resolved in NSX 4.2 and higher.

This workaround needs to be applied on all NSX Managers in the impacted cluster:
1. SSH to NSX Managers as root.
2. Change directory to /nonconfig/search:
cd /nonconfig/search
3. List content of the directory:
ls -l
The output will be similar to:
root@nsx-mgr-3:/#ls -l
drwxr-x--- 3 elasticsearch elasticsearch 4096 Apr 25  2023 nodes
drwxr-xr-x 2 nsx-search    nsx-search    4096 Jan  4 13:13 tmp
4. Change the ownership of the content in /nonconfig/search to owner and group "nsx-search":
chown -R nsx-search:nsx-search nodes
5. SSH to the Manager as admin.
6. Start service search:
start service search

Additional Information

NSX Manager will fail to upgrade.