Agent unable to save configuration to disk: Error syncing firmware configuration: Fault cause: vmodl.fault.SystemError.

Products

• VMware vSphere ESXi

Issue/Introduction

Alarm message is witnessed on the Event logs on the host:

"Alarm alarm.HostErrorAlarm on Host XX.XX.XX.XX because Error detected on datacenter: Agent unable to save configuration to disk: Error syncing firmware configuration: Fault cause: vmodl.fault.SystemError."

The hostd logs and vmkernel logs may show the errors similar to:

Log Location: var/run/log/hostd.log:

<YYYY-MM-DD>T<time>Z In(166) Hostd[2098852]: [Originator@6876 sub=SysCommandPosix opID=9e0a784-4b-6d4f sid=52db5ffb user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/bin/backup.sh) 2099910
<YYYY-MM-DD>T<time>Z Er(163) Hostd[2098849]: [Originator@6876 sub=SysCommandPosix] Failed to ForkExec /bin/backup.sh: No space left on device
<YYYY-MM-DD>T<time>Z Er(163) Hostd[2098849]: [Originator@6876 sub=SysCommandPosix] Invocation of process: '/bin/backup.sh' failed: Resource error - Process Creation - Error: VmkuserCompat_ForkExec - Error No space left on device
<YYYY-MM-DD>T<time>Z Er(163) Hostd[2098849]: [Originator@6876 sub=Default] SysCmd [/bin/backup.sh ++group=host/vim/vmvisor/backup.sh,mem=70 0] invocation failed: Resource error - Process Creation - Error: VmkuserCompat_ForkExec - Error No space left on device
<YYYY-MM-DD>T<time>Z Er(163) Hostd[2098849]: [Originator@6876 sub=Hostsvc.HostConfigSyncManager] Backup failed
<YYYY-MM-DD>T<time>Z In(166) Hostd[2098844]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=vim-cmd-60-6d53 sid=52f2c2ef] Accepted password for user root from XX.XX.XX.XX - session=52f2c2ef-f894-fa0a-0c67-95b6833dbab2

Log Location: var/run/log/vmkernel.log:

<YYYY-MM-DD>T<time>Z In(182) vmkernel: cpu24:2098849)Admission failure in path: host/vim/vmvisor/backup.sh:sh.2099948
<YYYY-MM-DD>T<time>Z In(182) vmkernel: cpu24:2098849)sh.2099948 (14945) requires 71680 KB, asked 71680 KB from backup.sh (767) which has 71680 KB occupied and 0 KB available.
<YYYY-MM-DD>T<time>Z In(182) vmkernel: cpu24:2098849)Admission failure in path: host/vim/vmvisor/backup.sh:sh.2099948
<YYYY-MM-DD>T<time>Z In(182) vmkernel: cpu24:2098849)sh.2099948 (14945) requires 71680 KB, asked 71680 KB from backup.sh (767) which has 71680 KB occupied and 0 KB available.
<YYYY-MM-DD>T<time>Z Wa(180) vmkwarning: cpu24:2098849)WARNING: Sched: vm 2099948: 6396: could not create container group, status: Admission check failed for memory resource

Environment

VMware ESXi 8.0 U1

VMware ESXi 8.0 U2

Cause

Known issue with ESXi 8.0 U1 and U2 as hostd is unable to synchronize with all the callers of backup.sh.

Resolution

The issue is fixed in ESXi version 8.0 U3.