VMware vSphere ESXi upgrades from the SDDC Manager failing on stage "ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP"
Article ID: 318228
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
- ESXi upgrade failing with error: ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP, and successfully complete the upgrades.
- From the SDDC Manager - ESXi upgrades fail on stage: ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP.
- The task does not show up in the vCenter task and events.
- LCM logs on the SDDC Manager: /var/log/vmware/vcf/lcm/lcm-debug.log
INFO [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-####-####-########7a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-####-####-########4f4] [c.v.e.s.l.p.i.e.EsxVumUpdateStageRunnerImpl,pool-3-th read-22] The stage: ESX_UPGRADE_VUM_STAGE_ATTACH_BASELINE_GROUP returned status: ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH, status msg: Exception when assigning baseline group to cluster: integrity.fault.NoVcConnection.summary
INFO [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-####-####-########7a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-####-####-########4f4] [c.v.e.s.l.p.i.e.EsxVumUpdateStageRunnerImpl,pool-3-th read-22] esxHostUpgradeStatus :ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH YYYY-MM-DDTHH:MM:SS+0000 ERROR [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-####-####-########7a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-####-####-########4f4] [c.v.e.s.l.p.i.e.EsxVumPrimitiveImpl,pool-3-thread-22] Error occurred performing the bundle precheck for cluster: e1f7c485-281c-####-####-########761 and bundle: aa7b16b1-d719-####-####-########4f4 java.lang.IllegalArgumentException: No enum constant com.vmware.evo.sddc.lcm.primitive.model.esx.PrecheckErrorCode.ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH at java.lang.Enum.valueOf(Enum.java:238) at com.vmware.evo.sddc.lcm.primitive.model.esx.PrecheckErrorCode.valueOf(PrecheckErrorCode.java:8) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.getPrecheckError(EsxUtils.java:1428) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumUpdateStageRunnerImpl.precheckUploadAndScan(EsxVumUpdateStageRunnerImpl.java:755) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumUpdateStageRunnerImpl.doBundlePrecheck(EsxVumUpdateStageRunnerImpl.java:666) ... ...
INFO [vcf_lcm,651a8746a99c92f1,d931] [c.v.e.sddc.lcm.audit.AuditScheduler,pool-6-thread-8] Last upgrade status for software type ESX_HOST and resourceId 173796b2-1f4d-####-####-########b9f is: {"id":"91","bundleId":"aa7b16b1-d719-####-####-########4f4","bundleElementId":"583db34c-2981-####-####-########8c7","bundleElementOrder":1,"resourceType":"ESX_HOST","resourceId":"173796b2-1f4d-####-####-########b9f","upgradeId":"30084d8d-e9ee-####-####-########c8d","upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP","errorCode":"com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception","errorDescription":"Failed to create baseline group: Database temporarily unavailable or has network problems.","errorDescriptionCodes":["com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception"],"errorDescriptionAttributes":{"0":"Database temporarily unavailable or has network problems."},"metadata":"\nManual intervention needed as upgrade failed during install stage.\nCheck for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm.\n Please retry the upgrade once the upgrade is available again.","metadataCodes":["com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception.remedy","MANUAL_INTERVENTION","LCM_LOG","RETRY_NOTE"],"metadataAttributes":{"LCM_LOG_LOCATION":"/var/log/vmware/vcf/lcm","LCM_HOST_ADDRESS":"127.0.0.1","BACKUP_REPO_USERNAME":"backup","BACKUP_REPO_SSH_ADDRESS":"localhost","UPGRADE_TARGET_USERNAME":"svc-vcf-wh-0000002941","UPGRADE_TARGET_SSH_ADDRESS":"esx.example.net"}},"startTime":1646145500980,"endTime":1646145532882,"numRetriesLeft":2,"upgradeStage":"[\n \"ESX_UPGRADE_VUM_STAGE_PRECHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_CUSTOM_IMAGE_SPEC\",\n \"ESX_UPGRADE_VUM_STAGE_UPLOAD_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\"\n]","backupInfoList":[],"upgradeStages":"[\n \"ESX_UPGRADE_VUM_STAGE_PRECHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_CUSTOM_IMAGE_SPEC\",\n \"ESX_UPGRADE_VUM_STAGE_ADD_HOST_TO_VC_INVENTORY\",\n \"ESX_UPGRADE_VUM_STAGE_UPLOAD_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_REGISTER_UPLOADED_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\",\n \"ESX_UPGRADE_VUM_STAGE_ATTACH_BASELINE_GROUP\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_HOST\",\n \"ESX_UPGRADE_VUM_STAGE_TAKE_BACKUP\",\n \"ESX_UPGRADE_VUM_STAGE_DISABLE_DRS_RULES\",\n \"ESX_UPGRADE_VUM_STAGE_INSTALL_UPDATE\",\n \"ESX_UPGRADE_VUM_STAGE_ENABLE_DRS_RULES\",\n \"ESX_UPGRADE_VUM_STAGE_CLEANUP\",\n \"ESX_UPGRADE_VUM_STAGE_REMOVE_HOST_FROM_VC_INVENTORY\",\n \"ESX_UPGRADE_VUM_STAGE_POSTCHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SUCCESS\"\n]","metadata":"{\n \"stage\": \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\",\n \"primitiveSpec\": {\n \"upgradeId\": \"30084d8d-e9ee-####-####-########c8d\",\n \"resourceId\": \"173796b2-1f4d-####-####-########b9f\",\n \"resourceType\": \"ESX_HOST\",\n \"bundleElementOrder\": 1,\n \"patchElement\": {\n \"id\": \"583db34c-2981-####-####-########8c7\",\n \"type\": \"ESX_HOST\",\n \"vendor\": \"VMware\",\n \"version\": \"7.0.1-18150133\",\n \"previousVersion\": \"7.0.1-17551050\",\n \"file\": \"aa7b16b1-d719-####-####-########4f4/bundle-44636/VMware-VMvisor-Installer-7.0hp-18150133.x86_64.iso\",\n \"checksum\": \"11ac4d99500c56fb9cdc20d912a777b52fd4fbc0daa21a6f30bd42a4e482f3c5\"\n },\n \"patchElements\": [],\n \"slaType\": \"SLOW\",\n \"endTime\": 1646159900972\n },\n \"esxVumUpdateStateObjects\": {\n \"upgradeId\": \"30084d8d-e9ee-####-####-########c8d\",\n \"updateFiles\": [\n \"/nfs/vmware/vcf/nfs-mount/vendor-isos/RKN-LVO-ESXI-7.0.1_18150133-LVO.701.10.3.iso\"\n ],\n \"remoteFilePathToLcmLocalFileCheckSumMap\": {},\n \"desiredVersion\": \"7.0.1-18150133\",\n \"upgradeBaselineId\": -1,\n \"patchBaselineIds\": [],\n \"baselineGroupId\": -1,\n \"esxMOID\": \"host-21\",\n \"patchKeys\": [],\n \"upgradeKey\": 2,\n \"upgradeIsoCheckSum\": \"d9094441c262e30060308d437aa0cb223eb5bbac8eaf6d081539a95bd016eff4\",\n \"drsRules\": [],\n \"isStandaloneHost\": false,\n \"shutdownVMs\": false,\n \"totalDisksCount\": 0,\n \"esxCustomImageSpec\": {\n \"bundleId\": \"aa7b16b1-d719-####-####-########4f4\",\n \"targetEsxVersion\": \"7.0.1-18150133\",\n \"useVcfBundle\": false,\n \"customIsoAbsolutePath\": \"/nfs/vmware/vcf/nfs-mount/vendor-isos/RKN-LVO-ESXI-7.0.1_18150133-LVO.701.10.3.iso\"\n },\n \"vumBundleUploadLockHeld\": true\n }\n}","stages":[]}
- VMware Update Manager logs on the vCenter Server: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
[VdbStatement] SQLError was thrown: "ODBC error: (23503) - ERROR: insert or update on table "vci_baseline_group_baselines" violates foreign key constraint "fk_vci_blgp_bl_ref_bl"; [#132] --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_BASELINE_GROUP_BASELINES (baseline_group_id, baseline_group_version, baseline_id) (SELECT ? AS baseline_group_id, ? AS baseline_group_ version, ? AS baseline_id FROM VCI_BASELINE_GROUP_BASELINES WHERE baseline_group_id = ? AND baseline_group_version = ? AND baseline_id = ? HAVING COUNT(*) = 0)" [#133] info vmware-vum-server[59216] [Originator@6876 sub=VcIntegrity] Session: [sessionId masked]; user VSPHERE.LOCAL\Administrator [#133]info vmware-vum-server[59216] [Originator@6876 sub=VcIntegrity] Active? true info vmware-vum-server[58938] [Originator@6876 sub=VcIntegrity] Connecting to host <esxi_host_fqdn> on port 80 using protocol http
Environment
VMware Cloud Foundation 4.x
Cause
The LCM Baseline is not detached or cleaned up after a host scan during precheck or after a remediation task.
Resolution
This issue is resolved in VMware Cloud Foundation 4.3.
Workaround:
1.Take a snapshot of both the SDDC Manager and the vCenter.
2. Reset the VUM DB on the vCenter using the steps provided in https://knowledge.broadcom.com/external/article?legacyId=2147284
3. Login to SDDC Manager via SSH using the vcf user and then su root.
4. Run following command to truncate the "vum_bundle_upload" table on the SDDC:
psql -U postgres -h localhost lcm -c "truncate vum_bundle_upload"
5. Retry upgrade from the SDDC Manager UI.
Additional Information
Impact/Risks:
MODERATE: The workaround involves resetting the VUM DB on the vCenter Server, as well as truncating a table on the SDDC DB. Offline snapshots of all VCs in the SSO, as well as a snapshot of the SDDC Manager are highly recommended.
MODERATE: The workaround involves resetting the VUM DB on the vCenter Server, as well as truncating a table on the SDDC DB. Offline snapshots of all VCs in the SSO, as well as a snapshot of the SDDC Manager are highly recommended.
Feedback
Yes
No
Powered by
