"Local resources cannot be overridden (Error code: 500157)" error observed in NSX Manager's UI

Products

• VMware NSX

Issue/Introduction

Symptoms:

• LDAP account is used for login to NSX UI.
• In the UI, user may observe error "Local resources cannot be overridden."
• admin user will not see the same error.
• Response of REST API (run as LDAP user) "GET https://<nsx-mgr>/api/v1/sites/self" will fail with "HTTP Status 400 – Bad Request"
• Manager's logs will indicate "Request header is too large" error:

/var/log/site-manager/tanuki.log
INFO   | jvm 1    | 2024/03/15 09:18:22 | Mar 15, 2024 8:18:22 AM org.apache.coyote.http11.Http11Processor service
INFO   | jvm 1    | 2024/03/15 09:18:22 | INFO: Error parsing HTTP request header
INFO   | jvm 1    | 2024/03/15 09:18:22 |  Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
INFO   | jvm 1    | 2024/03/15 09:18:22 | java.lang.IllegalArgumentException: Request header is too large
INFO   | jvm 1    | 2024/03/15 09:18:22 |       at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:770)
INFO   | jvm 1    | 2024/03/15 09:18:22 |       at org.apache.coyote.http11.Http11InputBuffer.parseHeader(Http11InputBuffer.java:953)

• Additionally, when such user with too many AD groups may also see "Something went wrong" when accessing the Location Manager page under System. 

Environment

VMware NSX

VMware NSX-T Data Center

Cause

This issue is caused by default value for http header size limited to 8KB.

Resolution

This is a known issue affecting VMware NSX and it is resolved in NSX 4.2.

Workaround:
Workaround is to increase the header size to 32KB.
The workaround below needs to be implemented on all NSX Managers in the cluster:

1. SSH to NSX Manager as root.
2. Change directory to /opt/vmware/site-manager/conf
   cd /opt/vmware/site-manager/conf
3. Add the content below to file "application.properties". To do so, copy&paste the output in the box below into the SSH session. Please ensure there are no additional characters introduced (e.g. line breaks) during transfer of clipboard content:
   echo 'server.port=7999
server.address=127.0.0.1
server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.condition-unless=NO_LOG
server.tomcat.accesslog.directory=/var/log/site-manager
server.tomcat.accesslog.pattern=%{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}t %a %u "%r" %s %b %D %F
server.tomcat.accesslog.max-days=7
server.tomcat.accesslog.suffix=.log
server.tomcat.accesslog.prefix=access_log
server.tomcat.accesslog.file-date-format=
server.max-http-header-size=32768' > application.properties
4. Change permissions on the file:
   chmod 755 application.properties
5. Change the ownership of the file:
   chown nsx-replicator:nsx-replicator application.properties
6. Create a backup of config file:
   cp /usr/tanuki/conf/site-manager-service-wrapper.conf /usr/tanuki/conf/site-manager-service-wrapper.conf.bak
7. Add the updated properties file to CBM during startup:
   sed -i '/wrapper.java.additional.30/d' /usr/tanuki/conf/site-manager-service-wrapper.conf
sed -i '/wrapper.java.additional.29/a wrapper.java.additional.30=-Dspring.config.location=/opt/vmware/site-manager/conf/application.properties' /usr/tanuki/conf/site-manager-service-wrapper.conf
8. Restart service site-manager:
   /etc/init.d/site-manager-service restart
9. The following step is only required if LDAP user is observing UI issues related to cluster status:
   Please follow steps in KB article LDAP users in VMware NSX are not able to retrieve the management cluster status in User Interface (UI).

Please note: The workaround will be removed during the upgrade. Once the Manager is upgraded to a version which doesn't have a fix (NSX 4.0.x.x or 4.1.x.x), this workaround will need to be implemented again.

Additional Information

A similar issue for cluster status show "unavailable" is also caused by the default HTTP header size, please follow this KB for a workaround: https://knowledge.broadcom.com/external/article?articleNumber=322435