Configuring syslog on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Syslog:

VMware ESXi hosts run a Syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk. To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a Syslog server.

Retention, rotation, and splitting of logs received and managed by a Syslog server are fully controlled by that Syslog server. ESXi cannot configure or control log management on a remote Syslog server. For more information, see the documentation for the Syslog server.

 

Environment

VMware ESXi 6.x 

VMware ESXi 7.x 

VMware ESXi 8.x

Resolution

There are five configurable options:

• Syslog.global.logDir - Location on a local or remote datastore (VMFS, NFS, FAT) and path where logs should be saved to. Has the format [DatastoreName] DirectoryName which maps to /vmfs/volumes/DatastoreName/DirectoryName/.
  • For ESXi 6.7 and above versions, the log directory must exist before configuring the global logging dir parameter. If the specified DirectoryName does not exist, configuring Syslog.global.logDir parameter will fail with "Internal error" in UI and "Logdir must exist and be a directory" error message in log file /var/run/log/hostd.log.
  • If /scratch is defined, the default is []/scratch/log. For more information on scratch, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x
• Syslog.global.logHost - Comma-delimited list of remote servers where logs are sent using the syslog protocol. If the logHost field is blank, no logs are forwarded. Include the protocol and port, similar to tcp://hostname:514 or udp://hostname:514 or ssl://hostname:1514.
• Syslog.global.logDirUnique - A boolean option which controls whether a host-specific directory is created within the configured logDir. The directory name is the hostname of the ESXi host. A unique directory is useful if the same shared directory is used by multiple ESXi hosts. Defaults to false.
• Syslog.global.defaultRotate - The maximum number of log files to keep locally on the ESXi host in the configured logDir. Does not affect remote syslog server retention. Defaults to 8.
• Syslog.global.defaultSize - The maximum size, in kilobytes, of each local log file before it is rotated. Does not affect remote syslog server retention. Defaults to 1024 KB. For more information on sizing, see Providing Sufficient Space for System Logging.
  
  Note: To individually set the logging level for system components such as auth, hostd etc., you may select the loggers under syslog from vSphere client GUI and set to desired value.

Configuring Local and Remote Logging using the esxcli command

Local and Remote syslog functionality can be configured for a host using the esxcli command line utility, which can be used at the console of an ESXi host, in the vCLI, or in the vMA.

1. Open a ESXi Shell console session where the esxcli command is available, such as the vCLI or on the ESXi host directly.
2. Display the existing five configuration options on the host by running this command:
   esxcli system syslog config get
3. Set new host configuration, specifying options to change, by running a command:
   esxcli system syslog config set --logdir=/path/to/vmfs/directory/ --loghost=RemoteHostname --logdir-unique=true|false --default-rotate=NNN --default-size=NNN
   For example:
   To configure remote syslog using TCP on port 514: esxcli system syslog config set --loghost='tcp://10.11.12.13:514'
   
   
4. After making configuration changes, load the new configuration by running this command:
   esxcli system syslog reload
5. Run this command to test if the port is reachable from the ESXi host:
   nc -z RemoteHostname 514
   For example:
   nc -z 10.11.12.13 514

Configuring Local and Remote logging using Host Profiles

Local and Remote syslog functionality can be configured for a cluster of similar hosts using Host Profiles. For more information, see the Set Up Syslog from the Host Profiles Interface document 

1. Connect to vCenter Server using the vSphere Client.
2. Click Home.
3. Under the Management section, click Host Profiles.
4. Create a new profile or edit an existing profile.
5. In the Edit Profile dialog, set one or more of the five configuration options.
    
   • If you configured syslog using esxcli or advanced configuration options and captured this as a reference host, the 5 configuration options are already visible under the Advanced Configuration option section.
   • If syslog has not been previously configured, right-click the Advanced Configuration options section and add a profile for each of the five configuration options.
6. Save the profile and assign it to hosts.

Configuring Local and Remote logging using Host Profiles using vSphere Web Client

1. Connect to vCenter Server using vSphere Web Client.
2. Click Home.
3. Under Operations and Policies section, click Host Profiles.
4. Create a new profile or edit an existing profile.
5. In the Edit Profile dialog, set one or more of the five configuration options.
6. Save the profile and assign it hosts.

 

Configuring Local and Remote Logging using Advanced Configuration options

Local and Remote syslog functionality can be configured for a host using advanced configuration options, which can be set using the vSphere Client, vSphere Web Client, PowerCLI, or vCLI.

 

This configuration cannot be performed using the local console's esxcfg-advcfg command. For more information on setting advanced configuration options using each method, see Configuring advanced options for ESXi/ESX.

Note: If the ESXi host loses communication with the remote syslog server, logging Logging stops being pushed to the syslog server. You see the "failed to write log" error in the /var/log/.vmsyslogd.err file. Nothing is sent to the remote syslog server until the syslogd service is restarted.

 

Additional Information

Configuring ESXi Firewall Exception using the esxcli command/syslog port:

Note: You may need to manually open the Firewall rule set for syslog when redirecting logs. 

To open outbound traffic through the ESXi Firewall on UDP port 514 and TCP ports 514 and 1514, run these commands:

Additionally, you can review Port requirements for ESXi Port requirements for ESXi.

 

esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli network firewall refresh

 

With ESXi 8.0 Update 2b, when you configure syslog remote hosts, or loghosts, with non-standard ports, the vmsyslogd service automatically creates persistent dynamic firewall rules. You no longer need to manually open the firewall at ports that are different from the default 514 for TCP/UDP and 1514 for SSL protocols respectively. When configuring remote hosts with the standard ports, you still need to enable the syslog firewall ruleset.

Refer to: VMware ESXi 8.0 Update 2b Release Notes

For example:

1. esxcli system syslog config set --loghost=tcp://xxxx:12345

2. esxcli network firewall ruleset rule list | grep dynamic