vSphere Lifecycle Manager tasks fail with Error: Cannot allocate memory

Products

• VMware vCenter Server

Issue/Introduction

• Lifecycle Management (LCM) operations in VMware vSphere Lifecycle Manager fail with the error:

“VMware vSphere Lifecycle Manager had an unknown error. Check the events and log files for details.”

• The following errors may appear in vum-server.log:

YYYY-MM-DDTHH:MM:SSZ info vmware-vum-server[12975] [Originator@6876 sub=VUMConfig] [vumConfigImpl 125] Patch depot url: http://vcsa_fqdn:9084/vum/repository/
YYYY-MM-DDTHH:MM:SSZ verbose vmware-vum-server[12975] [Originator@6876 sub=miscUtil] [miscUtilPosix 724] Executing command: ./unzip -l /storage/updatemgr/patch-store/hostupdate/CIS/CIS-ESXi_metadata.zip
YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[12975] [Originator@6876 sub=SysCommandPosix] Failed to fork './unzip': Cannot allocate memory
YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[12975] [Originator@6876 sub=SysCommandPosix] Invocation of process: './unzip' failed: Resource error - Process Creation - Error: Cannot allocate memory
YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[12975] [Originator@6876 sub=miscUtil] [miscUtilPosix 734] Error in invoking unzip command: ./unzip -l /storage/updatemgr/patch-store/hostupdate/CIS/CIS-ESXi_metadata.zip . Error: Resource error - Process Creation - Error: Cannot allocate memory

• The issue is commonly observed when performing the following actions:
  
  • Staging baselines
  • Importing ESXi ISO images
  • Importing offline depots
  • Host remediation or upgrades

Environment

VMware vCenter Server

Cause

The issue occurs when the vmware-updatemgr service does not have sufficient memory allocated.

When Lifecycle Manager processes patch metadata or imports ESXi images, it uses internal extraction utilities (unzip, 7z). If the service memory is below recommended levels, the operating system may fail to spawn these processes, causing LCM operations to fail.

Resolution

Increase the allocated memory for the vmware-updatemgr service and restart the service.

Note: Service names may vary between vCenter Server versions 7.x and 8.x. As such, see Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services below for service names and how to stop and start services in vCenter Server.

IMPORTANT! Snapshot the vCenter before doing any changes. See "Snapshot Best practices for vCenter Server Virtual Machines" and "VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice" for more information.

• Step 1: Check current memory allocation
  • Log in to the vCenter Server Appliance shell and run:

  • cloudvm-ram-size -l

  • Review the memory currently assigned to vmware-updatemgr

• Step 2: Increase the amount of memory assigned to the services by running the following command, 

  • cloudvm-ram-size -C  #### vmware-updatemgr

Note: #### should be replaced with the desired amount of memory in MB.
For example, this command would increase the memory of the vmware-updatemgr service from 314 MB to 640 MB.

• • cloudvm-ram-size -C 640 vmware-updatemgr

    There is no "one size fits all" memory value to set in this scenario, however - as a guide, double the size of currently assigned memory of the service that has an issue. This may need to be repeated.
• Step 3: Restart the service

  • service-control --restart vmware-updatemgr

  • Verify service status:

  • service-control --status vmware-updatemgr

• Step 4: Confirm that the service has now been allocated the increased memory by using this command:
  

  • cloudvm-ram-size -l