SDDC Manager experiencing high CPU usage

Products

• VMware Cloud Foundation

Issue/Introduction

vCenter Client shows a warning about the SDDC Manager VM using a high amount of CPU due to nfsd services

 

 

Environment

VMware Cloud Foundation 4.x

Cause

NFS daemon causing high CPU usage on the VM due to reaching the maximum amount of threads opened

When running a top command and sorting by CPU usage (shift + R twice) the nfsd is consuming more than 100% of the CPU

 

 

Resolution

The nfs server in the SDDC Manager is removed in the next upcoming release(4.5.1 and 5.0), so the problem should not appear

Follow the below steps to workaround this issue

 

1. Take snapshot of SDDC Manager VM
2. SSH to SDDC manager with vcf user and su to root
3. To see if it is matching the error run 

   journalcte -xe

   • You should see the below log entry
     

     nfsd: too many open connections; consider increasing the number of the threads

4. Run the following command to see how many threads are opened

   cat /proc/net/rpc/nfsd

   Looking at the output for example: th 8 0 2.610 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
   The first number indicates the NFS threads opened (8 in this case) . The second number is whether at any time all of the threads were running simultaneously. The remaining numbers indicate thread count time history.
5. Run the following command to increase the NFS thread count:

   rpc.nfsd 16

   the above command is increasing the thread count to 16, note that the maximum could be different from one environment to another.