"Local resources cannot be overridden (Error code: 500157)" error observed in NSX Manager's UI

Products

• VMware NSX

Issue/Introduction

Symptoms:

• LDAP account is used for login to NSX UI.
• In the UI, user may observe error "Local resources cannot be overridden."
• admin user will not see the same error.
• Response of REST API (run as LDAP user) "GET https://<nsx-mgr>/api/v1/sites/self" will fail with HTTP Status 400 – Bad Request
• Manager's logs will indicate "Request header is too large" error:

/var/log/site-manager/tanuki.log
INFO   | jvm 1    | 2024/03/15 09:18:22 | Mar 15, 2024 8:18:22 AM org.apache.coyote.http11.Http11Processor service
INFO   | jvm 1    | 2024/03/15 09:18:22 | INFO: Error parsing HTTP request header
INFO   | jvm 1    | 2024/03/15 09:18:22 |  Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
INFO   | jvm 1    | 2024/03/15 09:18:22 | java.lang.IllegalArgumentException: Request header is too large
INFO   | jvm 1    | 2024/03/15 09:18:22 |       at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:770)
INFO   | jvm 1    | 2024/03/15 09:18:22 |       at org.apache.coyote.http11.Http11InputBuffer.parseHeader(Http11InputBuffer.java:953)

Environment

VMware NSX-T Data Center

Cause

This issue is caused by default value for http header size limited to 8KB.

Resolution

This is a known issue affecting VMware NSX. There is currently no resolution.

Workaround:
Workaround is to increase the header size to 32KB.
The workaround below needs to be implemented on all NSX Managers in the cluster.

1. SSH to NSX Manager as root.
2. Change directory to /opt/vmware/site-manager/conf
cd /opt/vmware/site-manager/conf
3. Add the content below to file "application.properties". To do so, copy&paste the output in the box below into the SSH session. Please ensure there are no additional characters introduced (e.g. line breaks) during transfer of clipboard content:

echo 'server.port=7999
server.address=127.0.0.1
server.tomcat.accesslog.enabled=true
server.tomcat.accesslog.condition-unless=NO_LOG
server.tomcat.accesslog.directory=/var/log/site-manager
server.tomcat.accesslog.pattern=%{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}t %a %u "%r" %s %b %D %F
server.tomcat.accesslog.max-days=7
server.tomcat.accesslog.suffix=.log
server.tomcat.accesslog.prefix=access_log
server.tomcat.accesslog.file-date-format=
server.max-http-header-size=32768' > application.properties

4. Change permissions on the file:
chmod 755 application.properties
5. Change the ownership of the file:
chown nsx-replicator:nsx-replicator application.properties
6. Add the updated properties file to CBM during startup:
sed -i '/wrapper.java.additional.30/d' /usr/tanuki/conf/site-manager-service-wrapper.conf
sed -i '/wrapper.java.additional.29/a wrapper.java.additional.30=-Dspring.config.location=/opt/vmware/site-manager/conf/application.properties' /usr/tanuki/conf/site-manager-service-wrapper.conf
7. Restart service site-manager:
/etc/init.d/site-manager-service restart


Please note: The workaround will be removed during the upgrade. Once the Manager is upgraded to a version which doesn't have a fix (NSX 4.0.x.x or 4.1.x.x), this workaround will need to be implemented again.