Unable to start VxRail manager upgrade from SDDC Manager. Error - "Scheduling immediate update of bundle failed. A problem has occurred on the server."

Products

VMware SDDC Manager VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

Unable to start VxRail manager upgrade from SDDC Manager. Error is as below

Scheduling immediate update of bundle failed. A problem has occurred on the server.

"Update Now" button is missing when initiating the VxRail upgrade from SDDC Manager

Attempting to schedule the upgrade using the "Schedule Upgrade" option results in an error message

Scheduling immediate update of bundle failed. A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.

Error in /var/log/vmware/vcf/lcm/lcm-debug.log

ERROR [vcf_lcm,98eee1bbb8314e66,f448] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7400-exec-1] [######] VCF_ERROR_INTERNAL_SERVER_ERROR A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.
com.vmware.evo.sddc.common.core.error.InternalServerErrorException: A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.

ERROR [vcf_lcm,98eee1bbb8314e66,f448] [c.v.v.l.r.a.c.v.u.UpgradeController,http-nio-127.0.0.1-7400-exec-1] Scheduling Upgrade Failed
com.vmware.evo.sddc.lcm.model.error.LcmException: Error fetching hardware support inventory data.
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.HardwareSupportClientImpl.isVlcmEnabledOnDomain(HardwareSupportClientImpl.java:342)
        at com.vmware.vcf.lcm.rest.api.controller.v1.upgrade.UpgradeDissemblers.validateUpgradeSpec(UpgradeDissemblers.java:674)

ERROR [vcf_lcm,67d64614e42f9ed2fe0cb80d271c1ca3,c60f] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,vac-scheduler-1] Exception occurred while fetching the Hardware Support Info for the cluster example-c1 ########-####-####-####-########d7a7
ERROR [vcf_lcm,67d6474a4023f006334a5c771048d5a6,869c] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,vac-scheduler-1] Exception occurred while fetching the Hardware Support Info for the cluster example-c1 ########-####-####-####-########d7a7
ERROR [vcf_lcm,67d648771eca7b33615de5779d8a98e7,5961] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,vac-scheduler-1] Exception occurred while fetching the Hardware Support Info for the cluster example-c1 ########-####-####-####-########d7a7

WARN  [vcf_lcm,67d648771eca7b33615de5779d8a98e7,5961] [c.v.e.s.l.a.i.i.HardwareSupportClientImpl,vac-scheduler-1] Exception occurred while setting Hardware Support Managers ########-####-####-####-########d833
com.vmware.vapi.std.errors.ServiceUnavailable: ServiceUnavailable (com.vmware.vapi.std.errors.service_unavailable) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vapi.endpoint.cis.ServiceUnavailable,
    defaultMessage = Service unavailable.,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = SERVICE_UNAVAILABLE
}

Update Manager service is stopped in vCenter

root@vcsa01 [ ~ ]# service-control --status vmware-updatemgr
Stopped:
 vmware-updatemgr

Attempt to start Update manager service fails

Error in /var/log/vmware/vmware-updatemgr/vmware-vci-vciInstallUtils-xx.log

Error in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
info vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] [installerRunDBCommand 683]  [DATABASE] [DoDbUpdateVumOptions] vum option registry.servicePorts.WebSSLPort matches. Nothing to do here.
info vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] [installerRunDBCommand 101]  [DATABASE] Running no-version SQL scripts
info vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] [installerRunDBCommand 117]  [DATABASE] Running no version script: /usr/lib/vmware-updatemgr/bin///..//share/VCI_noversion_postgresql_configuration.sql
verbose vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] Load script: /usr/lib/vmware-updatemgr/bin///..//share/VCI_noversion_postgresql_configuration.sql
verbose vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] [VdbODBCError::VdbODBCError] SQLGetDiagRec SQLState='42501' NativeError=1 MessageText='ERROR: must be owner of table pm_config_plugin_requests;
--> Error while executing the query' rc=0
error vmware-vci-vciInstallUtils[394056] [Originator@6876 sub=Default] [installerRunDBCommand 122]  [DATABASE] Vdb::RunScript, SQL Exec Error: "ODBC error: (42501) - ERROR: must be owner of table pm_config_plugin_requests;
--> Error while executing the query" is returned when executing SQL statement "/* *************************************************************************
-->  * Copyright (c) 2022-2024 Broadcom. All Rights Reserved.
-->  * Broadcom Confidential. The term "Broadcom" refers to Broadcom Inc.
-->  * and/or its subsidiaries.
-->  * *************************************************************************/
-->
--> /**
-->  * The files starting with VCI_noversion_postgresql_.*.sql are run on every
-->  * service startup. As such the SQL in these files should be written in such a
-->  * way that if the table, triggers, seque"
info vmware-vci-vciInstallUtils[394136] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 7, idle: 0
2025-03-16T17:34:38.614Z info vmware-vci-vciInstallUtils[394246] [Originator@6876 sub=ThreadPool] Entering worker thread loop
}
ERROR] Failed to register VC: Failed to add VUM options in Database.
ERROR] Exception raised while refreshing the certs. err: Failed to add VUM options in Database.

Cause

Ownership of table(s) in schema name vumdbc changed from vumuser to postgres

Resolution

Change ownership of database tables to vumuser with schema name vumdbc in VCDB.

Caution: It is recommended to take offline snapshot of vCenter prior to executing the below steps

Follow the below steps:

Download the script "update_vumdbc_ownership.sh" attached to this KB and transfer this to the /tmp directory in vCenter VM using WinSCP.
SSH to vCenter appliance with root user

Make the script from Step # 1 executable

chmod +x /tmp/update_vumdbc_ownership.sh

Execute the script
```
cd /tmp
```
```
./update_vumdbc_ownership.sh
```

Start Update Manager service

service-control --start vmware-updatemgr

Retry the upgrade from SDDC Manager

Attachments

update_vumdbc_ownership.sh get_app