Error "VCENTER_UPGRADE_SWITCHOVER com.vmware.vcf.error.update.vc.rdu.failed" when upgrading vCenter Server component to version 9.1.0 via VCF Operations
search cancel

Error "VCENTER_UPGRADE_SWITCHOVER com.vmware.vcf.error.update.vc.rdu.failed" when upgrading vCenter Server component to version 9.1.0 via VCF Operations

book

Article ID: 445084

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Upgrading vCenter Server to version 9.1.0 fails during the switchover phase.

  • Below log entries are seen in /var/log/vmware/vlcm/vlcm.log of the source vCenter Server Appliance.

    INFO vlcm  [vc@] [service/service.go:164] [ObjectID=] Start all VMware services
    ERROR vlcm  [vc@] [exec/exec.go:126] [ObjectID=116] Run command [/usr/bin/service-control --start --all] failed (duration:)
    ERROR vlcm  [vc@] [logger/multiline.go:19] [ObjectID=] service-control failed
    ERROR vlcm  [vc@] [logger/multiline.go:45] [ObjectID=] ERR:001 exit status 1
    ERROR vlcm  [vc@] [logger/multiline.go:53] [ObjectID=] ERR:002-
    ERROR vlcm  [vc@] [logger/multiline.go:45] [ObjectID=] OUT:001 Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start updatemgr services. Error: A system error occurred. Check logs for details
    ERROR vlcm [vc@] [vmomivm/find.go:34] [ObjectID=10[1].1] Find VM (uuid) failed: find(uuid:<ID>): POST "/sdk": 503 Service Unavailable
    INFO vlcm 487289 [vc@] [vmomiclient/client.go:87] [ObjectID=10[1].1] Release VMOMI client: client(active:0, conn:[*(token), <vCenter Server FQDN>:443], https://<vCenter Server FQDN>:443/sdk)
    ERROR vlcm [vc@] [vm/guest.go:92] [ObjectID=10[1].1] Run command [/bin/bash] in ref:[50394e54-fc22-b9b8-9cc7-553e38d820c6:vcenter-upgrade-target-appliance_ <UUID>@ conn:[target(token), <vCenter Server fqdn>:443] failed: startGuestProgram: find(uuid:<UUID>): POST "/sdk": 503 Service Unavailable
    ERROR vlcm [vc@] [engine/engine.go:1029] [ObjectID=10[1].1] wfengine: execute [<10[1].1:A> vm.StartProgram(target,SupportBundle:Cmd)] error: com.vmware.vapi.std.errors.error
    INFO vlcm [vc@] [locerrors/vapierrors.go:39] [opID=<ID>] Parsing vAPI error json to real object
    INFO vlcm [vc@] [locerrors/vapierrors.go:48] [opID=<ID>] Error type: map[Data:null ErrorType:"ERROR" Messages:[{"Id":"vcenter.deployment.migration_upgrade.unable_to_start_services_error","DefaultMessage":"Unable to start all services during the upgrade","Args":[],"Params":null,"Localized":"Unable to start all services during the upgrade"},{"Id":"com.vmware.vcenter.vlcm.general_resolution","DefaultMessage":"For more information check the VMware logs. Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.","Args":[],"Params":null,"Localized":"For more information check the VMware logs. Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request."}]]
    INFO vlcm [vc@] [locerrors/vapierrors.go:66] [opID=<ID>] Error type: ERROR

  • Below log entries are seen in /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log of the vCenter Server Appliance.

    ERROR vmware-vum-server [vc@sub="VumVapiEndpoint"] [plugin ] Loading Cli metadata file: /usr/lib/vmware-updatemgr/etc/vapi_metadata.json failed with error Error:
    -->    com.vmware.vapi.std.errors.not_found
    --> Messages:
    -->    vapi.metada.cli.source.file.not.found<Cli source file not found: /usr/lib/vmware-updatemgr/etc/vapi_metadata.json>
    --> .
    INFO vmware-vum-server [vc@sub="VumVapiEndpoint"] [plugin ] Loading Metamodel metadata file: /usr/lib/vmware-updatemgr/etc/vapi_metadata.json.
    ERROR vmware-vum-server [vc@ sub="VumVapiEndpoint"] [plugin 3218] Loading Metamodel metadata file: /usr/lib/vmware-updatemgr/etc/vapi_metadata.json failed with error Error:
    -->    com.vmware.vapi.std.errors.not_found
    --> Messages:
    -->    vapi.metada.metamodel.source.file.not.found<Metamodel source file not found: /usr/lib/vmware-updatemgr/etc/vapi_metadata.json>

  • Below log entries are seen in /var/log/vmware/upgrade/Import_com.vmware.vcIntegrity_####_##_##_##_##.log of the vCenter Server Appliance.

    DEBUG vmware-vci-vciInstallUtils [vc@ sub="Default"] [VdbODBCError::VdbODBCError] SQLGetDiagRec SQLState='' NativeError=1 MessageText='ERROR: insert or update on table "pm_recommendation_spec" viol
    ates foreign key constraint "fk_pm_recommendation_spec"
    --> DETAIL: Key (entity_id, spec_id)=(domain-c<ID>, -1000) is not present in table "pm_recommendation_info".;
    --> Error while executing the query' rc=0
    ERROR vmware-vci-vciInstallUtils [vc@sub="Default"] [installerRunDBCommand ]  [DATABASE] Vdb::RunScript, SQL Exec Error: "ODBC error: (23503) - ERROR: insert or update on table "pm_recommendation_sp
    ec" violates foreign key constraint "fk_pm_recommendation_spec"
    --> DETAIL: Key (entity_id, spec_id)=(domain-c<ID>, -1000) is not present in table "pm_recommendation_info".;
    --> Error while executing the query" is returned when executing SQL statement "ALTER TABLE IF EXISTS PM_RECOMMENDATION_SPEC
    -->     ADD CONSTRAINT FK_PM_RECOMMENDATION_SPEC
    -->         FOREIGN KEY (ENTITY_ID, SPEC_ID)
    -->         REFERENCES PM_RECOMMENDATION_INFO (ENTITY_ID, SPEC_ID)
    -->         ON DELETE CASCADE"

Environment

VMware vCenter server 8.0.3

Cause

The failure is caused by orphaned database records in the vcIntegrity database. Specifically, the pm_recommendation_spec table contains stale entries that lack corresponding parent records in pm_recommendation_info.

vCenter 9.1.0.0 introduces a strict foreign key constraint (FK_PM_RECOMMENDATION_SPEC) to harden database integrity. The Postgres engine blocks this schema change because existing orphaned data violates the constraint, throwing a fatal error.

Resolution

Refer the Resolution and Workaround steps detailed in following kb to fix this issue: "Error attempting Upgrade Database Schema" during vCenter 8.0 U3x or 9.0.x to 9.1.0.0 upgrade

Powered by Wolken Software