ESXi Host Upgrade is Failing due to Ramdisk Full
search cancel

ESXi Host Upgrade is Failing due to Ramdisk Full

book

Article ID: 395531

calendar_today

Updated On:

Products

VMware vSphere ESXi 7.0 VMware vCenter Server 7.0 VMware Cloud Foundation 4.x VMware Cloud Foundation 5.x VMware vCenter Server 8.0 VMware vSphere ESXi 8.0

Issue/Introduction

  • Host Upgrade Remediation fails from the vSphere Client with error:

    Cannot copy files to host data.host. Check the management service status, network connectivity with vCenter and disk space on the host.



  • vCenter logs contains error similar to the excerpt below:

    /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
    yyyy-mm-ddThh:mm:ss.xxxZ error vmware-vum-server[xxxxx] [Originator@6876 sub=HU-Upgrader] [upgraderImpl 336] Error copying files over to host: <esxi_hostname> error: Error sending file (error code 3) : The operation experienced a network error
    yyyy-mm-ddThh:mm:ss.xxxZ warning vmware-vum-server[xxxxx] [Originator@6876 sub=Libs] SSL: Unknown SSL Error
    yyyy-mm-ddThh:mm:ss.xxxZ info vmware-vum-server[xxxxx] [Originator@6876 sub=Libs] SSL Error: error:1409E10F:SSL routines:ssl3_write_bytes:bad length
    yyyy-mm-ddThh:mm:ss.xxxZ warning vmware-vum-server[xxxxx] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpSetError: Success
    yyyy-mm-ddThh:mm:ss.xxxZ warning vmware-vum-server[xxxxx] [Originator@6876 sub=Libs] [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 0.
    yyyy-mm-ddThh:mm:ss.xxxZ warning vmware-vum-server[xxxxx] [Originator@6876 sub=Libs] [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR

  • Host logs contains errors similar to the excerpt below:

    /var/log/vmkernel.log
    yyyy-mm-ddThh:mm:ss.xxxZ cpu60:xxxxxxxx opID=xxxxxxxx)upgradescratch (xxxxxxxx) extraMin/extraFromParent: 1/1, upgradescratch (xxxxxxxx) childEmin/eMinLimit: xxxxxxxx/xxxxxxxx
    yyyy-mm-ddThh:mm:ss.xxxZ cpu60:xxxxxxxx opID=xxxxxxxx)WARNING: VisorFSRam: 203: Cannot extend visorfs file /upgrade_scratch/RESVIBS.TGZ because its ramdisk (upgradescratch) is full.

    /var/log/hostd.log
    yyyy-mm-ddThh:mm:ss.xxxZ warning hostd[xxxxxxxx] [Originator@6876 sub=Libs opID=UnknownOpId-xxxxxxxxxxxxxxxxxx] [NFC ERROR]NfcFileRawWrite: Failed to write 262144 bytes in file '/upgrade_scratch/RESVIBS.TGZ': There is no space left on the device
    yyyy-mm-ddThh:mm:ss.xxxZ warning hostd[xxxxxxxx] [Originator@6876 sub=Libs opID=UnknownOpId-xxxxxxxxxxxxxxxxxx] [NFC ERROR]NfcFile_ContinueReceive: write failed. expected 262144, got -1
    yyyy-mm-ddThh:mm:ss.xxxZ info hostd[xxxxxxxx] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
    -->    key = 77,
    -->    chainId = 3471,
    -->    createdTime = "1970-01-01T00:00:00Z",
    -->    userName = "",
    -->    host = (vim.event.HostEventArgument) {
    -->       name = "<esxi_hostname>",
    -->       host = 'vim.HostSystem:ha-host'
    -->    },
    -->    eventTypeId = "esx.problem.visorfs.ramdisk.full",
    -->    arguments = (vmodl.KeyAnyValue) [
    -->       (vmodl.KeyAnyValue) {
    -->          key = "1",
    -->          value = "upgradescratch"
    -->       },
    -->       (vmodl.KeyAnyValue) {
    -->          key = "2",
    -->          value = "/upgrade_scratch/RESVIBS.TGZ"
    -->       }
    -->    ],
    -->    objectId = "ha-host",
    -->    objectType = "vim.HostSystem",
    --> }
    yyyy-mm-ddThh:mm:ss.xxxZ info hostd[xxxxxxxx] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 3484 : The ramdisk 'upgradescratch' is full.  As a result, the file /upgrade_scratch/RESVIBS.TGZ could not be written.
    yyyy-mm-ddThh:mm:ss.xxxZ info hostd[xxxxxxxx] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=UnknownOpId-xxxxxxxxxxxxxxxxxx] Event 3485 : NFC operation 'Upload' for path '/upgrade_scratch/RESVIBS.TGZ' was initiated from '<ip_address>' and completed with status 'Failure'
    yyyy-mm-ddThh:mm:ss.xxxZ warning hostd[xxxxxxxx] [Originator@6876 sub=Libs opID=UnknownOpId-xxxxxxxxxxxxxxxxxx] [NFC ERROR]NfcProcessFileDataMsg: NfcContinueFileReceive failed: There is no disk space available (NFC_NO_DISKSPACE)

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 4.x
VMware vCenter Server 8.x
VMware vCenter Server 7.x
VMware vSphere Esxi 8.x
VMware vSphere Esxi 7.x

Cause

This is a known issue related to the files generated and pushed by the Lifecycle Manager to the host for the upgrade workflow. 
When the space created on the ESXi Host Ramdisk is less than expected to accommodate all the files that need to be copied, the task will fail. 
In this situation, the issue is caused by the reserved VIB file (RESVIBS.TGZ) not being accounted for in the ramdisk allocation (upgrade scratch). 
As a result, if an image has a high number of reserved VIBs, copying the upgrade files to the host fails because the allocated ramdisk space becomes exhausted.

Resolution

A fix for this issue has been applied in vCenter and ESXi versions 8.0.2b and later.


As a workaround, the size of the reserved VIBs file must be reduced to the minimum.
Each VIB that is added, removed or modified compared to the original Standard ISO target build (e.g., custom ISO add-ons or driver version modifications) will cause the reserved VIBs file to grow.
To avoid this, use a Standard ISO build with tools (vanilla) for the upgrade.
If you are using baselines, attach only the two predefined "Host Security Patches" and "Critical Host Patches."

Additional Information

Another approach to overcome the issue is to install the patch directly on the ESXi host via the CLI using Image Profiles, following the steps in KB article 343840.

Important note: This solution should be used with caution. Out-of-band updates applied directly on the ESXi hosts may result in sync issues on the LCM images and the SDDC Manager inventory.

Powered by Wolken Software