VMware NSX Manager Cluster status shows as UNAVAILABLE in the UI When logged in as LDAP user

Products

• VMware NSX

Issue/Introduction

• The environment has been upgraded from NSX-T 3.2.x to VMware NSX 4.1.x.
• LDAP has been configured and may have been working previously to allow LDAP users to login to the VMware NSX manager and/or run API calls.
• Cluster status is showing "UNAVAILABLE" under System > Appliances.
  
  
  
  
• When logged in as non LDAP user, the Appliances display correctly.
• When running get cluster status as admin in the NSX Manager CLI, the cluster and all components appear up and stable.
• Rebooting the VMware NSX Manager nodes, clearing browser cache, or changing browsers does not resolve the issue.
• In NSX Manager logs at /var/log/proton/nsxapi.log, there are ERRORs like the following example:
  
ERROR http-nio-127.0.0.1-7440-exec-154 ClusterManagerUtil 4912 - [nsx@6876 comp="nsx-manager" errorCode="MP2101" level="ERROR" reqId="########" subcomp="manager" username="########"] Request GET http://localhost:7989/api/v1/cluster-manager/status HTTP/1.1 failed, return code is 400
ERROR http-nio-127.0.0.1-7440-exec-154 ClusterManagerUtil 4912 - [nsx@6876 comp="nsx-manager" errorCode="MP2121" level="ERROR" reqId="################" subcomp="manager" username="########"] Cluster status retrieved from cluster manager is empty
  
  
• In NSX Manager logs at /var/log/cbm/cbm.log, entries like this are present:
  
  127.0.0.1 - - [{DATE/TIME}] "GET /api/v1/cluster-manager/status HTTP/1.1" 400 435
  
  
• In NSX Manager logs at /var/log/cbm/tanuki.log, there are entries stating that request header is too large such as in the following example:
  
  INFO  | jvm 1  | {DATE/TIME} | INFO: Error parsing HTTP request header
INFO  | jvm 1  | {DATE/TIME} | Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
INFO  | jvm 1  | {DATE/TIME} | java.lang.IllegalArgumentException: Request header is too large INFO  | jvm 1  |

Environment

VMware NSX 4.1.x

Cause

• The server has a maximum size for the request header (The default value for max-http-header-size for Springboot Tomcat/Jetty is 8KB), which is the part of the request that contains information about the user and the connection. If the request header is too large, the server will reject the request and return an error code 400.
• The request header size depends on the user’s identity and role.
• When a user has AD groups above a certain amount (over 100), the request header size may become too large to be accepted by the server, which is 8KB by default. This causes the request to fail and the UI to show the cluster status and other areas as unavailable.
• This issue occurs when an API Request header is too large, and the Cluster Boot Manager (CBM) is unable to process it. This is a known issue impacting VMware NSX.

Resolution

This issue is resolved in VMware NSX 4.2.0 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

Please make sure you have a backup in place before proceeding.

Repeat the below steps 1 to 4 on each manager node as root user.

1.

• Copy the original file, as a backup, before editing:

mkdir /root/jarFileBackup

cp /opt/vmware/cbm/cbm-app/libweb-server-cbm.jar /root/jarFileBackup/

• Ensure that libweb-server-cbm.jar is copied into jarFileBackup folder:

ls -l /root/jarFileBackup/

2.

• In order to edit the file, copy it again to another directory and unzip it: 

mkdir /root/jarFileUpdate

cp /opt/vmware/cbm/cbm-app/libweb-server-cbm.jar /root/jarFileUpdate/

cd /root/jarFileUpdate

unzip libweb-server-cbm.jar

• Check the contents of the file, making sure max http header size entry application.properties is not present: 

cat application.properties

• Append the value server.max-http-header-size=32KB to the file application.properties with below echo command: 

echo "server.max-http-header-size=32KB" >> application.properties

• Verify the value has been added to the end of the file:

cat application.properties

• Remove the old jar file without header size info:

rm /opt/vmware/cbm/cbm-app/libweb-server-cbm.jar

• Create a new jar file with edited file and other original files:

zip -r libweb-server-cbm.jar *

• Copy the new jar file to original location:

cp libweb-server-cbm.jar /opt/vmware/cbm/cbm-app/

3.

• Restart the CBM service to use the updated jar with increased header limit using below command 

/etc/init.d/nsx-cluster-boot-manager restart

NOTE: This step may incur some cluster downtime, due to the CBM service restarting, until the service is backup and cluster is healthy again.

4. 

• As admin user, check the cluster is healthy and all services are running with get cluster status command.
  
  
• Once the cluster is up and healthy again, repeat steps 1 to 4 on the remaining manager nodes:

5.  The VMware NSX UI should now show the Cluster information properly.