Failed to create journal for Vmotion-SRC: Failed to open "/var/lib/vmware/hostd/journal/**********.****" for write: There is no space left on the device

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion task fails with the below error:

A general system error occurred: Failed to create journal for Vmotion-SRC: Failed to open "/var/lib/vmware/hostd/journal/**********.****" for write: There is no space left on the device

• Datastore where scratch location is configured is removed.
• When the ESXi Shell and SSH services are disabled in the host, enabling them is not viable as the var needs to have free space in order for those services to create a temp file during the startup.
• Attempting to enable SSH and ESXi Shell for the host through the server hardware management console (such as iDRAC or iLO) does not successfully enable the service.

 

Validation:

• The following messages are seen in the /var/log/hostd.log file:

YYYY-MM-DDTHH:MM:SSZ: [VisorfsCorrelator] 2184901822us: [vob.visorfs.ramdisk.inodetable.full] Cannot create file /var/log/.vmsyslogd.err for process vmsyslogd because the inode table of its ramdisk (var) is full.

YYYY-MM-DDTHH:MM:SSZ: [VisorfsCorrelator] 2184884819us: [esx.problem.visorfs.ramdisk.inodetable.full] The file table of the ramdisk 'var' is full.  As a result, the file /var/log/.vmsyslogd.err could not be created by the application 'vmsyslogd'.


Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

When audit logging is enabled to local storage, the audit record storage directory is created by default at /scratch/auditLog. If the scratch location is changed, a host reboot is necessary for the new configuration to take effect. Upon reboot, the syslog daemon attempts to locate the audit directory at the default path. However, since the scratch partition now points to a different location, the vmsyslogd process is unable to find the audit directory. This leads to a failure in initializing the audit record storage, causing an exception and resulting in the crash of the syslog daemon.

 

Resolution

To resolve the issue, follow the steps:

• If SSH and ESXi Shell access to the host is functional, execute the following command to create a ramdisk for the location "/var/lib/vmware/hostd/journal": esxcli system visorfs ramdisk add -M 20 -m 20 -p 755 -name "hostd-Journal" -t "/var/lib/vmware/hostd/journal"
• This will enable the hostd to create the temporary file for the vMotion process.
• If remote access (SSH and ESXi Shell) is not working, PowerCLI commands can be used as an alternative to resolve the issue.

Powercli script:

• • Connect to host using PowerCLI: 

    Connect-VIServer -Server Hostname -Username root

  • Get the ESXCLI object: 

    $a = Get-EsxCli -VMHost 'Hostname'

  • To list ramdisk: 

     $a.system.visorfs.ramdisk.list.Invoke()

  • Create the RAM disk using the system visorfs ramdisk add command: 

    $esxcli = Get-ESXCLI -VMHost Hostname -V2
    $args_cmd = @{
    "maxsize" = 20
    "minsize" = 20
    "name" = "hostd-journal"
    "permissions" = "0755"
    "target" = "/var/lib/vmware/hostd/journal"}
    $esxcli.system.visorfs.ramdisk.add.Invoke($args_cmd)

• • Get the ESXCLI object: 

    $a = Get-EsxCli -VMHost 'Hostname'

  • To list ramdisk:

     $a.system.visorfs.ramdisk.list.Invoke() 

• The SSH service on host was incorrectly displayed as enabled in vSphere client. Manually enable the SSH service via Direct console to access the SSH: Enable SSH Access in the Direct Console

Additional Information

Configuring Scratch configuration and Syslog.global.auditRecord.storageEnable causes the esxi host to go unresponsive or hostd not logging