ESXi host stops saving logs to log partition

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi logs for a host are not saved to the  configured log partition 

However the host remains responsive and all services are running and healthy

Environment

ESXi 7.x

ESXi 8.x

Cause

When using VOMA to analyze the volume that logs are configured to write too we see the following events

Detected VMFS-6 file system (labeled:'xxx_Datastore_xxx') with UUID:xxxxxx-xxxxx-xxx-xxxxx, Version 6:82
ON-DISK ERROR: <FD c6 r17> corrupted address <INVALID c24448 r307>
ON-DISK ERROR: <FD c6 r17> invalid signature 175
ON-DISK ERROR: <FD c6 r17> : invalid FD type 0x6a
ON-DISK ERROR: <FD c6 r22> corrupted address <FD c17 r13>
ON-DISK ERROR: <FD c6 r23> corrupted address <FD c35 r6>
ON-DISK ERROR: Invalid direntry <vpxa.log, 6, 17>
ON-DISK ERROR: Invalid directory entry (2, 2)
ON-DISK ERROR: Direntry hostd.log : Invalid generation 0x18
ON-DISK ERROR: Direntry fdm.log : Invalid generation 0x19
ON-DISK ERROR: Invalid total entries 334 in directory <FD c46 r46>
ON-DISK ERROR: Invalid hash idx 9433 type 1, bn 2 sn 2
ON-DISK ERROR: invalid <FD c6 r17>
ON-DISK ERROR: FB inconsistency found: (88,120) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (88,498) allocated in bitmap, but never used
ON-DISK ERROR: Cluster 88 affinityFD <FD c46 r46> inUseResources in slot (7) cannot be more than actual (5)
ON-DISK ERROR: FB inconsistency found: (97,12) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,24) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,27) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,34) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,36) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,38) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,46) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,47) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,48) allocated in bitmap, but never used
ON-DISK ERROR: FB inconsistency found: (97,50) allocated in bitmap, but never used
ON-DISK ERROR: Cluster 97 affinityFD <FD c46 r46> inUseResources in slot (31) cannot be more than actual (21)

 

In this example we see that vpxa , hostd and fdm all have on-disk metadata inconsistencies related to file descriptors

This results in the host not being able to write logs to the configured logging volume correctly 

 

Resolution

Please open a case with Broadcom support and provide a voma dumpfile of the volume you have found this corruption on 

Please use the comnmand bellow to generate the dumpfile 

voma -l -f dump -d /vmfs/devices/disks/<DEVICE>[:<PARTITION>] -D <DMP FILE>

example:

voma -l -f dump -d /vmfs/devices/disks/naa.6000d31000724b00000000000000006e:1 -D /vmfs/volumes/[VOLUME]/naa.6000d31000724b00000000000000006e.dmp

 

( Please note that the location you write the dumpfile on cannot be the same volume that the voma dump is analyzing ) 

 

 

Additional Information

Voma documentation