ESXi hosts disconnected from vCenter due to excessive logging rates, causing dropped syslog messages and services to be unable to log.

Products

• VMware vSphere ESXi

Issue/Introduction

One or many hosts may become disconnected from vCenter Server, and/or other functionality like HA may become impaired due to excessive writes to the syslog of the host.

Log files for other services such as vpxa and FDM may no longer be updated and the services may enter a hung state waiting on log writes.  Because the services are blocked by the excessive writes to syslog, hosts may disconnect from vCenter and/or HA may not function correctly.

Cause

This issue is commonly observed after additional NSX logging is enabled, causing the dfwpktlogs.log file to exceed the sustainable rate of the syslog service to write and send all log messages.  However, this is not necessarily the only cause as any service that begins to exceed the sustainable log rate by syslog could cause this issue.

To determine if this is the cause, you should review the following:

1. Check for the presence of vmsyslogd-dropped.log files on the host.
   
   
2. Look for log messages similar to the following in the vmkernel.log file:
   
   

   2024-04-27T05:04:36.011Z cpu34:2097818)ALERT: vmsyslog logger 192.168.1.150:514 lost 1 log messages

   
   
   
3. Check the logging rate - the issue was observed in one case where the DFW logs were generating ~100,000 messages in 40 seconds, which is not sustainable.

Resolution

To resolve the issue, you must identify the service(s) that are logging excessively to syslog and either reduce the log level or investigate the cause of the excessive logging.  For NSX, DFW logs are a typical cause and the logging must be disabled.

At this time, the syslog service is not able to support the amount of logging required for NSX DFW logs.

To determine if the excessive logging is the cause for the host disconnect, you can disable syslog for the vpxa service and see if the host will then connect to vCenter again.

To disable vpxa from writing so syslog, follow the below steps:

1. For versions prior to 7.0 U3, the configuration can be changed by editing the file
   
   

   /etc/vmware/vpxa/vpxa.cfg

   
   
   
2. For version 7.0 U3 and later, the configuration is located in the configstore and can be changed with the following procedure:
   
   
   1. Extract the current configuration and save it to a .json file:
      
      

      /bin/configstorecli config current get -c esx -g services -k vpxa -outfile tmp.json

      
      
      
   2. Edit the tmp.json file and look for the following section:
      
      

        "log": {
            "directory": "/var/run/log",
            "level": "info",
            "max_file_num": 50,
            "max_file_size": 1048576,
            "output_to_console": false,
            "output_to_files": false,
            "output_to_syslog": true,

      
      
      
   3. Change "output_to_syslog" to false
      
      
   4. Change "level" from "info" to "none"
      
      
   5. Import the changes into the configstore:
      
      

      /bin/configstorecli config current set -c esx -g services -k vpxa -infile tmp.json

      
      
      
   6. Reload the syslog service:
      
      

      esxcli system syslog reload

Once these changes are made, you can restart the vpxa service and attempt to connect the host.  To restore defaults, revert the changes made above to the .json file to their default values and re-import the file as in step 2-5.