The vmware-vpxd service fails to start on the VMware vCenter Server Appliance with the vpxd.log error: FileIO error: No space left

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Starting the vmware-vpxd service with the command service vmware-vpxd start fails with the error:
  
  Waiting for vpxd to initialize: .failed
  
  
• In the /var/log/vmware/vpx/vpxd.log file, you see entries similar to:
  
  ------ In-memory logs start --------
  mem> yyyy-mm-ddTHH:MM:SS [7F5681F6D740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed
  mem> yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed
  mem> yyyy-mm-ddTHH:MM:SS [7F5681F6D740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed
  mem> yyyy-mm-ddTHH:MM:SS [7F5681F6D740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed
  mem> yyyy-mm-ddTHH:MM:SS [7F5681F6D740 info 'Hooks'] Hooks Initialized
  
  ------ In-memory logs end --------
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Logging uses fast path: true
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Handling bora/lib logs with VmaCore facilities
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Initialized channel manager
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Current working directory: /storage/log/vmware/vpx
  yyyy-mm-ddTHH:MM:SSZ [7F567823D700 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F56782BE700 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F56781BC700 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Log path: /var/log/vmware/vpx
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Initializing SSL
  yyyy-mm-ddTHH:MM:SSZ [7F567013B700 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
  yyyy-mm-ddTHH:MM:SSZ [7F567813B700 info 'ThreadPool'] Thread enlisted
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 error 'vpxdvpxdDaemon'] Unhandled exception:
  --> FileIO error: No space left : /var/log/vmware/vpx/1440594937560943.vpxduptime
  yyyy-mm-ddTHH:MM:SSZ [7F5681F6D740 error 'vpxdvpxdDaemon'] Backtrace:
  --> backtrace[00] rip 00007f567bed2084 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
  --> backtrace[01] rip 00007f567bdbc132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
  --> backtrace[02] rip 00007f567bd1d8a5 Vmacore::Throwable::Throwable(std::string const&)
  --> backtrace[03] rip 00007f567bdb6429 /usr/lib/vmware-vpx/libvmacore.so(+0x2d7429) [0x7f567bdb6429]
  --> backtrace[04] rip 00007f56832cc25a /usr/lib/vmware-vpx/vpxd(+0x11de25a) [0x7f56832cc25a]
  --> backtrace[05] rip 00007f5682f9e2f8 /usr/lib/vmware-vpx/vpxd(+0xeb02f8) [0x7f5682f9e2f8]
  --> backtrace[06] rip 00007f5682f81032 /usr/lib/vmware-vpx/vpxd(+0xe93032) [0x7f5682f81032]
  --> backtrace[07] rip 00007f5679e41c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f5679e41c16]
  --> backtrace[08] rip 00007f5682f80299 /usr/lib/vmware-vpx/vpxd(+0xe92299) [0x7f5682f80299]
  
  
• Running the command df -h on the vCenter Server Appliance displays free space on all partitions:
  
  For example:
  
  df -h
  
  Filesystem Size Used Avail Use% Mounted on
  /dev/sda3 9.8G 3.7G 5.6G 41% /
  udev 4.0G 104K 4.0G 1% /dev
  tmpfs 4.0G 0 4.0G 0% /dev/shm
  /dev/sda1 128M 21M 101M 18% /boot
  /dev/sdb1 20G 173M 19G 1% /storage/core
  /dev/sdb2 20G 9.2G 9.6G 49% /storage/log
  /dev/sdb3 60G 296M 56G 1% /storage/db
  
  
• Running the command df -i on the vCenter Server Appliance displays no free inodes on /storage/log/.
  
  For example:
  
  df -i
  Filesystem Inodes IUsed IFree IUse% Mounted on
  /dev/sda3 647168 63922 583246 10% /
  udev 0 0 0 - /dev
  tmpfs 1024280 2 1024278 1% /dev/shm
  /dev/sda1 33864 40 33824 1% /boot
  /dev/sdb1 1310720 12 1310708 1% /storage/core
  /dev/sdb2 1310720 1310720 0 100% /storage/log
  /dev/sdb3 3932160 3717 3928443 1% /storage/db

Environment

VMware vCenter Server Appliance 5.1.x
VMware vCenter Server Appliance 5.5.x

Cause

This issue occurs when the the vCenter Server Appliance is configured as a syslog server. Currently there is no log rotation causing /var/log/remote to run out of space.

Resolution

This is a known issue affecting the vCenter Server Appliance 5.1 and 5.5.

Currently there is no resolution.

To work around this issue, manually remove old log files from /storage/log/.