After adding a custom role/feature/permission, NSX Manager UI is not working, with "Error Code 101"
Article ID: 323541
Updated On:
Products
VMware NSX
Issue/Introduction
Symptoms:
- Custom role/feature/permission was added.
- NSX Manager was rebooted, or Manager's service proton was restarted.
- After the reboot, NSX manager UI shows the following error:
Some appliance components are not functioning properly.
Component health: SEARCH:DOWN, MANAGER:UNKNOWN, NODE_MGMT:UP, UI:UP.
Error code: 101
Component health: SEARCH:DOWN, MANAGER:UNKNOWN, NODE_MGMT:UP, UI:UP.
Error code: 101
- "get cluster status" command shows manager and https down:
get cluster status
Mon Jan 22 2024 UTC 18:09:11.755
Cluster Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Overall Status: DEGRADED
Group Type: MANAGER
Group Status: UNAVAILABLE
Members:
UUID FQDN IP STATUS
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <nsx-manager-name> <NSX_Manager_IP_address> DOWN
Group Type: HTTPS
Group Status: UNAVAILABLE
Members:
UUID FQDN IP STATUS
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <nsx-manager-name> <NSX_Manager_IP_address> DOWN
Mon Jan 22 2024 UTC 18:09:11.755
Cluster Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Overall Status: DEGRADED
Group Type: MANAGER
Group Status: UNAVAILABLE
Members:
UUID FQDN IP STATUS
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <nsx-manager-name> <NSX_Manager_IP_address> DOWN
Group Type: HTTPS
Group Status: UNAVAILABLE
Members:
UUID FQDN IP STATUS
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx <nsx-manager-name> <NSX_Manager_IP_address> DOWN
- NSX Manager's logging may looks similar to the logs below:
/var/log/syslog:
2024-01-22T19:05:01.741Z <nsx-manager-name> NSX 25872 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Restarting application due to: org.springframework.context.ApplicationContextException/Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key none) =====
2024-01-22T19:09:27.751Z <nsx-manager-name> NSX 29553 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Restarting application due to: org.springframework.context.ApplicationContextException/Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key none) =====
2024-01-22T19:05:01.741Z <nsx-manager-name> NSX 25872 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Restarting application due to: org.springframework.context.ApplicationContextException/Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key none) =====
2024-01-22T19:09:27.751Z <nsx-manager-name> NSX 29553 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Restarting application due to: org.springframework.context.ApplicationContextException/Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key none) =====
/var/log/proton/nsxapi.log:
2024-01-11T09:34:36.434Z ERROR WrapperStartStopAppMain ApplicationContextManager 3396543 - [nsx@6876 comp="nsx-manager" errorCode="MP2112" level="ERROR" subcomp="manager"] Failed to start management application.
org.springframework.context.ApplicationContextException: Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key read
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[?:?]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[?:?]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_372]
..
Caused by: java.lang.IllegalStateException: Duplicate key read
at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_372]
at java.util.HashMap.merge(HashMap.java:1255) ~[?:1.8.0_372]
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_372]
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_372]
2024-01-11T09:34:36.434Z ERROR WrapperStartStopAppMain ApplicationContextManager 3396543 - [nsx@6876 comp="nsx-manager" errorCode="MP2112" level="ERROR" subcomp="manager"] Failed to start management application.
org.springframework.context.ApplicationContextException: Failed to start bean 'roleBindingInit'; nested exception is java.lang.IllegalStateException: Duplicate key read
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[?:?]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[?:?]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[?:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_372]
..
Caused by: java.lang.IllegalStateException: Duplicate key read
at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_372]
at java.util.HashMap.merge(HashMap.java:1255) ~[?:1.8.0_372]
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_372]
at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_372]
- In some cases, "get service http" command may return output as follows:
nsx-mngr-01> get service http
% Invalid value for argument <service-name>: http
<service-name>: Node service.
% Invalid value for argument <service-name>: http
<service-name>: Node service.
Environment
VMware NSX-T Data Center
Cause
This is caused by a duplicate custom role/feature/permission, where duplicity occurs in NSX Manager's database (in table "roles").
Resolution
A logic preventing creation of duplicate entries has been added in NSX 4.1.2.1 and higher.
A logic that handles existing duplicate entries will be added in future product release.
Workaround:
Please open a ticket with Global Support team and refer to this KB article for a workaround.
A logic that handles existing duplicate entries will be added in future product release.
Workaround:
Please open a ticket with Global Support team and refer to this KB article for a workaround.
Additional Information
Impact/Risks:
- NSX Manager's UI will not be accessible.
- Manager and http services will be down, causing API calls to the Manager to fail.
- NSX Manager's UI will not be accessible.
- Manager and http services will be down, causing API calls to the Manager to fail.
Feedback
Yes
No
Powered by
