Image compliance check failed with error "vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>"

Products

• VMware vSphere ESXi

Issue/Introduction

• Attempting to check image compliance in vCenter VCLM fails with the following error:

A failure occurred when starting a host compliance check operation on host <ESXi_Hostname> : Error: system_error Messages: vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable>

 

 

• /var/run/log/lifecycle.log of impacted host reported this exception:          

  [YYYY-MM-DDTHH:MM:SS] Db(15) lifecycle[4813724] Misc:84 VMW_bootbank_lpfc_14.2.641.5-32vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMware_bootbank_gc_8.0.2-0.40.23825572 cannot be removed live.', 'VIB VMW_bootbank_bnxtroce_225.0.4.0-8vmw.802.0.0.22380479 cannot be removed live.', 'VIB Cisco_bootbank_nenic-ens_1.0.6.0-1OEM.700.1.0.15843807 cannot be removed live.', 'VIB VMware_bootbank_vsan_8.0.2-0.40.23825572 cannot be removed live.', 'VIB VMW_bootbank_nmlx5-rdma_4.23.0.66-2vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMW_bootbank_nmlx5-core_4.23.0.66-2vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMW_bootbank_nhpsa_70.0051.0.100-4vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMW_bootbank_elxiscsi_12.0.1200.0-11vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMW_bootbank_ionic-en_20.0.0-49vmw.802.0.0.22380479
[YYYY-MM-DDTHH:MM:SS] Db(15) lifecycle[4813724] Misc:84 cannot be removed live.', 'VIB VMW_bootbank_cndi-igc_1.2.10.0-1vmw.802.0.0.22380479 cannot be removed live.', 'VIB VMW_bootbank_nvmetcp_1.0.1.8-1vmw.802.0.0.22380479 cannot be removed live.']
[YYYY-MM-DDTHH:MM:SS] In(14) lifecycle[4813724] runcommand:199 runcommand called with: args = '/bin/localcli system visorfs ramdisk remove -t /tmp/esx-update-4813724', outfile = None, returnoutput = True, timeout = 0.0.
[YYYY-MM-DDTHH:MM:SS] In(14) lifecycle[4813724] Ramdisk:245 Unmounting manual tardisk /tardisks.noauto/loadesx-4813724
[YYYY-MM-DDTHH:MM:SS] In(14) lifecycle[4813724] Ramdisk:245 Unmounting manual tardisk /tardisks.noauto/esxupdt-4813724
[YYYY-MM-DDTHH:MM:SS] In(14) lifecycle[4813724] Ramdisk:245 Unmounting manual tardisk /tardisks.noauto/tpmesxup-4813724
[YYYY-MM-DDTHH:MM:SS] In(14) lifecycle[4813724] Ramdisk:245 Unmounting manual tardisk /tardisks.noauto/weaselin-4813724
[YYYY-MM-DDTHH:MM:SS] Er(11) lifecycle[4813724] imagemanagerctl:96 [TaskError]
[YYYY-MM-DDTHH:MM:SS] Er(11)[+] lifecycle[4813724] Cannot fail task in status Status.SUCCEEDED
[YYYY-MM-DDTHH:MM:SS] Er(11) lifecycle[4813724] imagemanagerctl:101 Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] Er(11) lifecycle[4813724] imagemanagerctl:101   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1956, in ScanImage
[YYYY-MM-DDTHH:MM:SS] Er(11) lifecycle[4813724] imagemanagerctl:101     NewScanner.HostScanner(imageSpec, depotSpec, task).scan()
[YYYY-MM-DDTHH:MM:SS] Er(11) lifecycle[4813724] imagemanagerctl:101   File "/tmp/esx-update-4813724/lib64/python3.8/site-packages/vmware/esximage/ImageManager/Scanner.py", line 3496, in scan

Environment

7.x
8.x

Cause

• Scratch partition not configured correctly due to which the logs exception shows errors related to VIBs cannot be removed live.

  • In some situations it can be related to multiple hosts configured with the same location for "ScratchConfig.ConfiguredScratchLocation"

• If persistent scratch space is not available, ESXi stores this temporary data on a ramdisk, which is constrained in space. This might be problematic in low-memory situations, but is not critical to the operation of ESXi.

Note: Pointing the scratch location to a partition on an SD card or USB stick is not recommended. These devices only have a limited amount of read/write cycles. Please refer to:- SD card/USB boot device revised guidance

Resolution

• Configure the scratch location to a unique datastore location for "ScratchConfig.ConfiguredScratchLocation". Please refer to: Creating a persistent scratch location for ESXi 8.x/7.x/6.x

Configuring a persistent scratch location using the vSphere Client:

1. Connect to vCenter Server or the ESXi host using the vSphere Client.
2. Click the ESXi hostin the inventory.
3. Click the Configurationtab > Storage.
4. Right-click a Datastore and click Browse.
5. Create a uniquely-named directory for this ESXi host.
   
   For example:
   .locker-ESXHostname
6. Close the Datastore Browser.
7. Click Software > Advanced Setting
8. Select the ScratchConfig
9. Change the ConfiguredScratchLocation configuration option, specifying the full path to the directory.
   
   For example:
   /vmfs/volumes/DatastoreUUID/.locker-ESXHostname
   
   Note: To determine the Datastore UUID: 
   1. Navigate to Configuration > Storage.
   2. Select the Datastore.
   3. View location under Datastore Details.
      The Datastore UUID appears similar to /vmfs/volumes/xxxxxxxxxxxxxxxx where xxxxxxxxxxxxxxxis the UUID.
      
      
10. Click OK.
11. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.