NSX Manager uproxy process showing high CPU usage.

Products

• VMware NSX

Issue/Introduction

• The NSX Manager is configured with a external monitoring tool.
• This tool is utilizing the NSX Manager API to pull data. 
• This monitoring tool may be a 3rd party or a VMware tool such as VCF Operations for Network.
• When the tool is enabled the NSX Manager uproxy CPU usage can increase from 20% to 400% and higher.
• The NSX Manager syslog shows high volume of logging (thousands): 
  /var/log/syslog
<DATE/TIME> ######### NSX 3117 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] checkServerTrusted: OU=OU,O=O,C=C,CN=CM for authType=UNKNOWN succeeded.
• The NSX Manager reverse-proxy.log, also shows high volume of logging (thousands): 
  /var/log/proxy/reverse-proxy.log
<DATE/TIME>  INFO Processing request ########-9f75-4bb8-9de0-############ NsxTrustManager 554966 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] checkServerTrusted: CN=<FQDN> for authType=UNKNOWN succeeded.
• The NSX Manager API limit is not being hit. See NSX API throws error message: "Client '######' exceeded request rate of 100 per second" when Client API rate limit is reached
• Running top as the root user on the NSX manager sgows the load as high, check processes, the uproxy shows high,. around 400/500%:
  

  top - ####### up ### days, ####,  1 user,  load average: 13.54, 13.60, 13.26
  Tasks: 366 total,   1 running, 365 sleeping,   0 stopped,   0 zombie
  %Cpu(s): 50.8 us,  4.6 sy,  0.0 ni, 38.0 id,  5.5 wa,  0.0 hi,  1.1 si,  0.0 st
  KiB Mem : 49295232 total,  2499780 free, 30990700 used, 15804752 buff/cache
  KiB Swap:        0 total,        0 free,        0 used. 17775640 avail Mem

   PID          USER      PR  NI    VIRT          RES       SHR   S    %CPU     %MEM        TIME+     TGID COMMAND
   #######  uproxy     20   0     2296832     823440  22836 S    510.8      1.7 #####    554966    /usr/lib/jvm/openjdk-java17-runtime-amd64/bin/java -Djava.util.logging.config.file=/opt/vmware/prox+

Environment

VMware NSX 4.x

Resolution

This is a known issue impacting VMware NSX.


Workaround:

Ensure a backup of the NSX managers is taken, prior to applying the workaround and the backup passphrase is known:

Repeat the below steps 1 through 5, one by one for each NSX manager.

1. Check if the manager cluster is stable, before proceeding, as admin user run: 

   get cluster status

2. Take backup of the tanuki conf file:

   cp /usr/tanuki/conf/proxy-tomcat-wrapper.conf /tmp/proxy-tomcat-wrapper_backup.conf

3. Edit this file /usr/tanuki/conf/proxy-tomcat-wrapper.conf After these lines:

       wrapper.java.additional.40=--add-opens=java.base/sun.security.x509=ALL-UNNAMED

     wrapper.java.additional.41=--add-opens=java.base/java.util=ALL-UNNAMED

     wrapper.java.additional.42=--add-opens=java.base/sun.security.internal.spec=ALL-UNNAMED

   Add below lines

      wrapper.java.additional.43=-Dcom.sun.jndi.ldap.connect.pool=true

     wrapper.java.additional.44=-Dcom.vmware.nsx.check_crl=false

4. To reduce logging for certificate checks, take backup of log4j2.xml:

   cp /opt/vmware/proxy-tomcat/conf/log4j2.xml /opt/vmware/proxy-tomcat/conf/log4j2_backup.xml

   Edit file  /opt/vmware/proxy-tomcat/conf/log4j2.xml, before the </Loggers> line, add the following entry (as part of the <Logger> block):

       <Logger name="com.vmware.nsx.management.security.NsxTrustManager" level="WARN"/>

       example:

                     </Root>

            <Logger name="com.vmware.nsx.management.rp" level="DEBUG" />

            <Logger name="com.vmware.nsx.management.api.leader" level="FATAL" />

            <Logger name="com.vmware.nsx.rpc" level="FATAL" />

            <Logger name="com.vmware.nsx.platform" level="FATAL" />

            <Logger name="com.vmware.nsx.sha" level="FATAL" />

            <Logger additivity="false" level="INFO" name="org.springframework.security.oauth2.client.token.grant.password" />

            <Logger name="com.vmware.nsx.management.security.NsxTrustManager" level="WARN"/>

           </Loggers>

         </Configuration>

5. Restart reverse proxy

   /etc/init.d/proxy restart