VCF RDU Upgrade Fails at VCENTER UPGRADE PREPARATION Stage Due to Timeout During Target vCenter Appliance Configuration
search cancel

VCF RDU Upgrade Fails at VCENTER UPGRADE PREPARATION Stage Due to Timeout During Target vCenter Appliance Configuration

book

Article ID: 419090

calendar_today

Updated On:

Products

VMware vCenter Server VMware Cloud Foundation VCF Operations VMware SDDC Manager

Issue/Introduction

  • During a vCenter RDU (Reduced Downtime Upgrade) upgrade initiated from VCF Ops, the process fails at the VCENTER_UPGRADE_PREPARATION stage.
  • The error explicitly states a timeout occurred while waiting for the newly deployed vCenter appliance (e.g., vcenter-upgrade-target-appliance_...) to be fully deployed and configured for the upgrade, leading to an automatic rollback of the process.
  • VCF Ops UI Error 
    The upgrade process timed out while waiting for the new vCenter appliance (VM vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3) to be deployed and configured for upgrade. vCenter upgrade is rolled back.
Message: VCenter upgrade failed at the VCENTER_UPGRADE_PREPARATION
stage.
Remediation Message: Please retry the upgrade once the upgrade is available again. Contact GSS for assistance, if the retry fails, Note: Check if there are any errors related to vCenter upgrade rollback failure. If so, perform the remediation steps for the rollback failure before performing the upgrade remediation steps.
Reference Token: L####V
Cause:

Message: The upgrade process timed out while waiting for the new vCenter appliance (VM vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3) to be deployed and configured for upgrade.
Remediation Message: Ensure the the upgrade process is successfully
cancelled. Once done, restart the upgrade.
Reference Token: S####T
Cause:

Message: vCenter upgrade is rolled back.
Remediation Message: Please retry the upgrade. Contact GSS for assistance, if
the retry fails.
Reference Token: 5####P
Cause

     

  • SDDC Manager /var/log/vmware/vcf/lcm/lcm-debug.log: Confirms upgradeStatus: "COMPLETED_WITH_FAILURE" and stage: "VCENTER_UPGRADE_PREPARATION". 
    INFO  [vcf_lcm,69####################cb,380b] [c.v.e.sddc.lcm.audit.AuditScheduler,Scheduled-5] Last upgrade status for software type VCENTER and resourceId e19a####-####-####-####-########7363 is: {"id":"10","bundleId":"e833####-####-####-####-########876b","bundleElementId":"75aa####-####-####-####-########d010","bundleElementOrder":1,"resourceType":"VCENTER","resourceId":"e19a####-####-####-####-########7363","upgradeId":"54c7####-####-####-####-########00bc","upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"VCENTER_UPGRADE_PREPARATION","errorCode":"com.vmware.vcf.error.update.vc.rdu.failed","errorDescription":"vCenter upgrade failed at the VCENTER_UPGRADE_PREPARATION stage.","metadata":"\n Please retry the upgrade once the upgrade is available again.\nContact GSS for assistance, if the retry fails.Note: Check if there are any errors related to vCenter upgrade rollback failure. If so, perform the remediation steps for the rollback failure before performing the upgrade remediation steps.","referenceToken":"B###K","notifications":[{"severity":"ERROR","timestamp":17#######78,"message":{"id":"com.vmware.vcf.error.update.vc.rdu.failed","defaultMessage":"vCenter upgrade failed at the VCENTER_UPGRADE_PREPARATION stage.","messageAttributes":{"0":"VCENTER_UPGRADE_PREPARATION"},"localizedMessage":"vCenter upgrade failed at the VCENTER_UPGRADE_PREPARATION stage."},

     

  • vCenter Server /var/log/vmware/vlcm/vlcm.log: Shows a timeout error: 
    info vlcm [locerrors/vapierrors.go:46] [opID=69####a6] Error type: map[Data:null ErrorType:"ERROR" Messages:[{"Id":"vcenter.deployment.migration_upgrade.deployment.operation_time_out","DefaultMessage":"The upgrade process timed out while waiting for the new vCenter appliance (VM vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3) to be deployed and configured for upgrade.","Args":[],"Params":{"0":{"S":"vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3","Dt":null,"I":null,"D":null,"L":null,"Format":null,"Precision":null}},"Localized":"The upgrade process timed out while waiting for the new vCenter appliance (VM vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3) to be deployed and configured for upgrade."},{"Id":"vcenter.deployment.migration_upgrade.deployment.operation_problem_auto_cancel.resolution","DefaultMessage":"Ensure the the upgrade process is successfully cancelled. Once done, restart the upgrade.","Args":[],"Params":null,"Localized":"Ensure the the upgrade process is successfully cancelled. Once done, restart the upgrade."}]]

Environment

VMware Cloud Foundation
VCF Ops

Cause

The failure is immediately caused by a timeout during the configuration steps of the newly deployed target vCenter appliance, which occurs after the appliance has been successfully deployed and powered on.

While the target VM was deleted, preventing definitive root cause analysis of the configuration failure, the vCenter Server vlcm.log sequence strongly suggests the failure occurred during the configuration phase, possibly around the os.SetFirewall(target) step, which prevented the VCF upgrade process from proceeding and ultimately resulted in the timeout error.

info vlcm [state/print_report.go:37] 889   * <8:A> os.SetFirewall(target)
info vlcm [state/print_report.go:37] 890   * <9:A> Value(Param:LinkLocal)
info vlcm [state/print_report.go:37] 891     sequence [ outcome:yes ]
info vlcm [state/print_report.go:37] 892       * <9[1].1:A> net.UnlockConfiguration, Next:10
info vlcm [state/print_report.go:37] 893   * <10> #end
error vlcm [ndu/executor.go:466] Got failure while working on Stage state: com.vmware.vapi.std.errors.error

Target VM successfully deployed and powered on.

info vlcm [vm/deploy.go:94] [ObjectID=54[1].7] Deploy ref:[501a####-####-####-####-########56fa:vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3] @ conn:[target(token), vcenter.example.com:443] completed

info vlcm [vm/power.go:44] [ObjectID=59[1].3] Power on VM ref:[501a####-####-####-####-########56fa:vcenter-upgrade-target-appliance_7fef####-####-####-####-########43e3] @ conn:[target(token), vcenter.example.com:443] completed

Resolution

The issue can be resolved by bypassing the VCF RDU workflow and performing a regular vCenter Server upgrade from VCF Ops.

 

Note: If the goal was specifically to retry the VCF RDU upgrade, the recommended troubleshooting would have been to implement the steps in KB 374320 to prevent the target vCenter VM from being deleted on failure. This would allow a second upgrade attempt, or enable log collection from the failed target appliance to determine the exact configuration failure.

Powered by Wolken Software