Sync updates task fails on vCenter 9.0 after upgrading and ESX 9 image is not available in LCM.

Products

• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

After upgrading vCenter to 9.0, lifecycle manager (LCM) does not show an available image for ESX 9.0

Sync updates task fails on a brownfield 9.0GA vCenter with the following error. This results in users not able to see 9.x ESX content on the vCenter.

A general system error occurred: Download patch definitions task failed while syncing depots. Error" 'integrity.fault.VcintegrityFault: VMware vSphere Lifecycle Manager had an unknown error.

When uploading an image to the vCenter or creating a new Workload Domain or cluster in VCF. You are also unable to see any images.

Error: Confirm importing offline patches task failed while creating offline depot.
Error: 'integrity.fault.FileUploadInvalidPackage: Cannot upload file, because it is an invalid package.
Error: Bulletin: Cisco-nfnic_5.0.0.44-2vmw.900.0.24755229 with same name but different set of vibs already exists inside VUM. Re-importing in this scenario is not supported.'.

Errors in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log.

error vmware-vum-server[58900] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "uk_vci_updates"
--> DETAIL: Key (meta_uid)=(ESXi70U1-16850804) already exists.

error vmware-vum-server[63160] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "uk_vci_updates"
--> DETAIL: Key (meta_uid)=(Broadcom-ELX-lpfc_900.14.4.390.14-36vmw.900.0.24755229) already exists.;

error vmware-vum-server[58900] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed: INSERT INTO VCI_UPDATES(bulletin_category, bulletin_severity, bulletin_urgency, bundle_type, category, code_version, component, component_name, deleted, description, downloadtime, exec_platform, hidden, hostupdate_category, hyperlink, impact, installation_impacts, locale, meta_uid, patchconturl, recalled, releasedate, run, security_fix, service_pack, signatureurl, title, type, vendor, id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

error vmware-vum-server[11832] [Originator@6876 sub=VciSigUpdateTask.SigUpdateTask{7}] [vciSigUpdateTask 439] Error downloading third party patches from :https://<sddc-manager_fqdn>/vmware/vcf/umds/patch-store/hostupdate/__hostupdate20-consolidated-index__.xml: Error: Bulletin: Cisco-nfnic_5.0.0.44-2vmw.900.0.24755229 with same name but different set of vibs already exists inside VUM. Re-importing in this scenario is not supported.

error vmware-vum-server[13124] [Originator@6876 sub=VciSigUpdateTask.SigUpdateTask{7}] [vciSigUpdateTask 766] Task failed with fault: integrity.fault.MetadataDownloadFailure
error vmware-vum-server[12942] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.SyncDepotsTask] [SyncDepotsTask 465] VciSigUpdateTask failed. Error: integrity.fault.MetadataDownloadFailure

error vmware-vum-server[11832] [Originator@6876 sub=VciSigUpdateTask.SigUpdateTask{7}] [vciSigUpdateTask 559] 
[backtrace begin] product: VMware Update Manager, version: 9.0.0, build: build-24755230, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0048395D]
--> backtrace[01] libvmacore.so[0x003730D8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x003855E5]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref
<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x008D2A76]: Integrity::VciSigUpdateTask::DownloadHostUpdate20Met
adata()
--> backtrace[04] libvci-vcIntegrity.so[0x008D4ADC]: Integrity::VciSigUpdateTask::DoRun()
--> backtrace[05] libvci-vcIntegrity.so[0x008DDE7D]: Integrity::VciTaskBase::Run()
--> backtrace[06] libvmacore.so[0x002CFC04]
--> backtrace[07] libvmacore.so[0x002D550F]
--> backtrace[08] libvmacore.so[0x00462AEB]
--> backtrace[09] libc.so.6[0x000890C4]
--> backtrace[10] libc.so.6[0x0010916C]
--> backtrace[11] (no module)
--> [backtrace end]

 

Environment

vCenter 9.0 which has been upgraded from 8.x.

Cause

This is a known issue caused by duplicate database entries created after the upgrade to vCenter 9.0.

Resolution

Broadcom is aware of this issue and working to fix it in an upcoming patch.

Workaround:

1. Login to VC as root.
2. Stop updatemgr service.
   
   service-control --stop updatemgr
   
   
3. Change to 'updatemgr' user.
   
   su updatemgr -s /bin/bash
   
   
4. Run psql command to patch the DB.
   
   psql -U vumuser -d VCDB -c "UPDATE vci_updates set deleted = 0, hidden = 0 where meta_uid like 'ESXi7%' or meta_uid like 'esxi7%' or meta_uid like 'ESXi_7%' or meta_uid like 'esxi_7%'; DELETE FROM vci_updates where id not in (select update_id from vci_update_packages);"

Note: The psql command will show below. 'Permission denied' message is not relevant and can be ignored.
              
               could not change directory to "/root": Permission denied
               UPDATE #
               DELETE #
   
   
5.  Logout as updatemgr user
   
   exit
6. As root, start updatemgr service.
   
   service-control --start updatemgr
   
   
7. Depot syncs should now work.