vSAN health service fails to start due to no space left on device

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• In /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log.stderr you see errors similar to:

Starting service process with pid: xxxx.
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 990, in emit
    self.flush()
  File "/usr/lib/python3.5/logging/__init__.py", line 970, in flush
    self.stream.flush()
OSError: [Errno 28] No space left on device
 

• df -h shows full /storage/log partition on vCenter Server:

root@vcsa [ ~ ]# df -h
Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  7.9G     0  7.9G   0% /dev
tmpfs                                     7.9G  688K  7.9G   1% /dev/shm
tmpfs                                     7.9G  680K  7.9G   1% /run
tmpfs                                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda3                                  11G  6.0G  4.1G  60% /
tmpfs                                     7.9G  1.5M  7.9G   1% /tmp
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   23M  9.2G   1% /storage/imagebuilder
/dev/mapper/netdump_vg-netdump            985M  1.3M  916M   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy      9.8G   23M  9.2G   1% /storage/autodeploy
/dev/sda1                                 120M   34M   78M  31% /boot
/dev/mapper/archive_vg-archive             50G   31G   17G  65% /storage/archive
/dev/mapper/core_vg-core                   50G  409M   47G   1% /storage/core
/dev/mapper/updatemgr_vg-updatemgr         99G   84M   94G   1% /storage/updatemgr
/dev/mapper/dblog_vg-dblog                 15G  118M   14G   1% /storage/dblog
/dev/mapper/log_vg-log                    9.8G  9.8G    0G 100% /storage/log
/dev/mapper/db_vg-db                      9.8G  315M  8.9G   4% /storage/db
/dev/mapper/seat_vg-seat                   25G  249M   23G   2% /storage/seat
root@vcsa [ ~ ]#

Environment

VMware vSAN 8.x

VMware vSAN 7.x

VMware vSAN 6.x

Resolution

To resolve this issue, check what is causing full /storage/log partition on vCenter Server. You may need to delete some older logs, dumps, etc.