Upgrade of NSX Manager from version 3.2.2 to 4.0 or 4.1 hangs on step Unpin UI
book
Article ID: 323553
calendar_today
Updated On:
Products
VMware NSX
Issue/Introduction
NSX-T Data Center 3.2.2.* being upgraded to NSX 4.0.x or 4.1.x.
During the upgrade, the Upgrade Coordinator's UI will hang on "Unpin UI".
You see messages similar to the following in the manager's logs:
/etc/opensearch/log4j2.properties org.opensearch.OpenSearchException: failed to bind service at org.opensearch.node.Node.<init>(Node.java:965) ~[opensearch-1.2.3.jar:1.2.3] at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.3.jar:1.2.3]
Same error is observed in Manager's /var/log/search/opensearch.log:
/var/log/search/opensearch.log [2024-01-04T08:59:48,751][ERROR][o.o.b.Bootstrap ] [nsx_node] Exception org.opensearch.OpenSearchException: failed to bind service .. Caused by: java.nio.file.AccessDeniedException: /nonconfig/search/nodes/0 .. [2024-01-04T08:59:48,758][WARN ][stderr ] [nsx_node] Caused by: org.opensearch.bootstrap.StartupException: OpenSearchException[failed to bind service]; nested: AccessDeniedException[/nonconfig/search/nodes/0];
NSX Manager's UI may be inaccessible, and reporting services down:
"Some appliance components are not functioning properly." "Component health: MANAGER:UP, SEARCH:DOWN, UI:UP, NODE_MGMT:UP","error_code":101
Environment
VMware NSX-T Data Center VMware NSX
Cause
This issue is caused by race condition that occurs during upgrade from 3.2.2.* to 4.0.x or 4.1.x, where Elasticsearch is not able to stop in time and was eventually terminated after upgrade process tried to start Opensearch. This update of specific user permissions, which will cause services to fail to start.
Resolution
This issue is resolved in VMware NSX 4.2.0
Workaround: This workaround needs to be applied on all NSX Managers in the impacted cluster: