The /var/log partition of the ESX host runs out of space due to vmkiscsid.log growth

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When running VMware ESX 4.x with a software iSCSI initiator, you see these symptoms:

• Running the df command in the host terminal shows that the partition /var/log is 100% full. For more information, see Investigating disk space on an ESX or ESXi host (1003564).
  
• The log file /var/log/vmkernel does not update.
• The log file /var/log/vmkiscsid.log or /var/log/vmkiscsid.log is very large.

Environment

VMware ESX 4.0.x
VMware ESX 4.1.x

Cause

On VMware ESX 4.x, the software iSCSI daemon log file at /var/log/vmkiscsid.log is not rotated or cleared, unlike other syslog-generated log files. Therefore, the vmkiscsid.log file grows to a very large size if the host has experienced, or is experiencing, ongoing communication issues with the iSCSI storage.

Resolution

Solution

This issue is resolved by ESX 4.0 Update 3, released 2011-05-05. After installing this update, 600KB of vmkiscsid logs are retained, across 6 files. For more information, see the release notes at https://www.vmware.com/support/vsphere4/doc/vsp_esx40_u3_rel_notes.html.

Workaround for ESX 4.0

1. If you need the software iSCSI client logs for troubleshooting purposes, archive them prior to reboot. See the Additional Information section.
2. Reboot the ESX 4.0 host.
3. The file /var/log/vmkiscsid.log is erased during startup and the space used by the file is released.

Workaround for ESX 4.1

1. Reboot the ESX 4.1 host.
2. The file /var/log/vmkiscsid.log is rotated to /var/log/vmkiscsid.log.previous during startup.
3. If you need the software iSCSI client logs for troubleshooting purposes, archive them following the reboot. See the Additional Information section.
4. Delete the older file using a command similar to:

rm /var/log/vmkiscsid.log.previous

Additional Information

If there is rapid logging to the /var/log/vmkiscsid.log file, investigate underlying iSCSI storage connectivity issue by reviewing your storage target configuration and system logs.

Make a compressed backup of the /var/log/vmkiscsid.log or vmkiscsid.log.previous files before deleting them or rebooting the host, in case this information is needed for troubleshooting purposes. Archive the file to a location with sufficient space, such as a datastore. For example, use a command similar to:

tar czvf /vmfs/volumes/DatastoreName/vmkiscsid.tgz /var/log/vmkiscsid.log*


Note:

• To view this file, you can extract the file using an archive manager.
• The archive can be relocated back to /var/log/ after the system reboots to keep the system logs in a single location.

Investigating disk space on an ESX or ESXi host