VMware vSphere Lifecycle Manager fails to download patches after vCenter Server upgrade from 5.5.x up through to 7.0.x

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
After upgrading vCenter Server from 5.5.x or 6.0.x to 6.5.x/6.7.x to 7.0.x, you experience these symptoms:

• Sync Updates for vShere Lifecycle Manager fails.
• Attempting to sync updates with vSphere Lifecycle Manager generates an error similar to:
  
  An error occurred while downloading depot metadata from file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip.
   
• In the vmware-vum-server-log4cpp.log file, you seen entries similar to:
  
  [2020-06-17 16:47:01:314 'PM.AsyncTask.SyncDepotsTask{9}' 140440415545088 INFO]  [vciTaskBase, 1343] SerializeToVimFault fault: (vmodl.fault.SystemError) {    faultCause = (vmodl.MethodFault) null,     faultMessage = (vmodl.LocalizableMessage) [       (vmodl.LocalizableMessage) {          key = "com.vmware.vcIntegrity.lifecycle.EsxImage.MetadataDownloadError",           arg = (vmodl.KeyAnyValue) [             (vmodl.KeyAnyValue) {                key = "1",                 value = "file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip"             }          ],           message = <unset>       }    ],     reason = "vLCM Task failed, see Error Stack for details."    msg = "{     "data": null,     "error_type": "ERROR",     "messages": [         {             "args": [                 "file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip"             ],             "default_message": "An error occurred while downloading depot metadata from file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip.",             "id": "com.vmware.vcIntegrity.lifecycle.EsxImage.MetadataDownloadError"         }     ] }" } Converted fault: (vim.fault.ExtendedFault) {    faultCause = (vmodl.MethodFault) null,     faultMessage = (vmodl.LocalizableMessage) [       (vmodl.LocalizableMessage) {          key = "com.vmware.vcIntegrity.lifecycle.EsxImage.MetadataDownloadError",           arg = (vmodl.KeyAnyValue) [             (vmodl.KeyAnyValue) {                key = "1",                 value = "file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip"             }          ],           message = <unset>       }    ],     faultTypeId = "SystemError",     data = (vim.KeyValue) [       (vim.KeyValue) {          key = "faultCause",           value = ""       },        (vim.KeyValue) {          key = "reason",           value = "vLCM Task failed, see Error Stack for details."       }    ]    msg = "{     "data": null,     "error_type": "ERROR",     "messages": [         {             "args": [                 "file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip"             ],             "default_message": "An error occurred while downloading depot metadata from file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip.",             "id": "com.vmware.vcIntegrity.lifecycle.EsxImage.MetadataDownloadError"         }     ] }" }
  [2020-06-17 16:47:01:315 'PM.AsyncTask.SyncDepotsTask{9}' 140440415545088 INFO]  [vciTaskBase, 1615] Description updated
   
• In the lifecycle.log file, you see entries similar to:
  
  ImageManagerCTL: 19557: 2020-06-17 16:47:01,082 DepotCollection:327 INFO Could not download metadata.zip from file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip, skipping (('/storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip', '/tmp/tmpo6hi5ehn', "unknown url type: '/storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip'"))
  ImageManagerCTL: 19557: 2020-06-17 16:47:01,083 DepotMgr:137 ERROR Unable to connect to depot(s) https://hostupdate.vmware.com/software/VUM/PRODUCTION/vmtools-main/vmw-depot-index.xml,https://hostupdate.vmware.com/software/VUM/PRODUCTION/addon-main/vmw-depot-index.xml,https://hostupdate.vmware.com/software/VUM/PRODUCTION/iovp-main/vmw-depot-index.xml,/storage/updatemgr/patch-store/hostupdate/__hostupdate20-consolidated-index__.xml,https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml: 
  ('file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip', '', 'Could not download metadata.zip from file:///storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip, skipping ((\'/storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip\', \'/tmp/tmpo6hi5ehn\', "unknown url type: \'/storage/updatemgr/patch-store/hostupdate/vmw/vmw-ESXi-5.0.0-metadata.zip\'"))
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue occurs when upgrading from earlier releases, the Update Manager depots are preserved, including all of the items downloaded or used with those prior releases. After upgrading to vSphere 7.0, some of the downloaded items are no long useable with vSphere Lifecycle Manager, causing vSphere Lifecycle Manager to be unable to complete its synchronization process successfully.

Resolution

To resolve this issue, download the attached script and follow these steps:

1. Download and extract the scripts from the attached 80227_cleanup_unsupported_metadata_files.zip.
2. Copy and execute cleanup_patch_store.sh script on your vCenter Server.
3. Execute by running "./cleanup_patch_store.sh" from the copied file location
4. Run Sync Updates.

To highlight what the script is doing:

1. Cleanup any rows in vci_platform_metadata table for unsupported metadata files
2. Cleanup unsupported metadata file entries in vci_metadata_files table.
3. Delete unsupported metadata files from patch store.
4. Cleanup depot index and vendor index xml files.