VCF 9.1 upgrade fails with Error - Backup for component vidb timed out - VCFMS-BACKUP-COMPONENT-006

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• VIDB Upgrade failure during Perform Component Upgrade Stage - Source Cluster Full Backup Status: Failed - VCFMS-BACKUP-COMPONENT-006

Backup for component vidb timed out. Please retry the operation or contact support if the issue persists. [VCFMS-BACKUP-COMPONENT-006]  && Step vmsp_upgrade_vidb failed. Reference Code:####
  
  
  
  
• Fleet upgrade service pod logs shows below error,
  
  Command:                     kubectl logs vcf-fleet-upgrade-service-fleetupgrade-#######-###### -n vcf-fleet-lcm | less 
Log location on VSP worker node: /var/log/pods/vcf-fleet-lcm_vcf-fleet-upgrade-service-fleetupgrade-#####-########/fleetupgrade/##.log 
  Log location in SOS bundle:             /sos-###/VCFMgmtComponents/vcf-management-services/<vsp fqdn>/services-logs/vcf-fleet-lcm/<node name>/vcf-fleet-upgrade-service-fleetupgrade-####/fleetupgrade/console-log-*.log


{timestamp=YYYY-MM-DDTHH:MM:SS.###Z, level=INFO, message={id=com.broadcom.vcf.lcm.upgrade.VmspStageProgress, defaultMessage=Source Cluster Full Backup. Status: Failed, localizedMessage=Source Cluster Full Backup. Status: Failed, args={stageStatus=Failed, stageDescription=Source Cluster Full Backup}}, stageId=<id>}, {timestamp=YYYY-MM-DDTHH:MM:SS.###Z, level=ERROR, message={id=com.vmware.vcfms.backup.BackupTimeout, defaultMessage=Backup for component vidb timed out. Please retry the operation or contact support if the issue persists. [VCFMS-BACKUP-COMPONENT-006], localizedMessage=Backup for component vidb timed out. Please retry the operation or contact support if the issue persists. [VCFMS-BACKUP-COMPONENT-006], args={referenceCode=######}}, stageId=id}, {timestamp=YYYY-MM-DDTHH:MM:SS.###Z, level=ERROR, message={id=com.broadcom.vcf.lcm.task.orchestrator.AsyncStepFailed, defaultMessage=Step vmsp_upgrade_vidb failed. Reference Code: #####. Check logs for details., localizedMessage=Step vmsp_upgrade_vidb failed. Reference Code: #######. Check logs for details., args={stepName=vmsp_upgrade_vidb, referenceCode=#######}}, stageId=<id>}]}]
  
  
• SDDC Build Service pod logs shows the following error message:
  
  Command:              kubectl logs vcf-sddc-build-service-sddcbuild-########-#### -n vcf-sddc-lcm | less
  Log location is SOS bundle: /var/log/pods/vcf-sddc-lcm_vcf-sddc-build-service-sddcbuild-########-####-####/sddcbuild/##.log


{"timestamp":"YYYY-MM-DDTHH:MM:SS.###Z","level":"ERROR","message":{"id":"com.broadcom.vcf.lcm.VspTaskMessage","defaultMessage":"Backup Configuration Validation Precheck. Status: FAILED Failed to copy test file to server: server=server.example.com, error=ERROR : testfile: Failed to copy: Update Create failed: sftp: \"Failure\" (SSH_FX_FAILURE);ERROR : Attempt 1/2 failed with 1 errors and: Update Create failed: sftp: \"Failure\" (SSH_FX_FAILURE);ERROR : testfile: Failed to copy: Update Create failed: sftp: \"Failure\" (SSH_FX_FAILURE);ERROR : Attempt 2/2 failed with 1 errors and: Update Create failed: sftp: \"Failure\" (SSH_FX_FAILURE);Failed to copyto: Update Create failed: sftp: \"Failure\" (SSH_FX_FAILURE);.. [VCFMS-PRECHECK-BACKUP-CONFIG-002]. Failure during copying of the test file can be caused by network connectivity or firewall issues, server out of space issues, quota issues etc. If possible, check for such issues and try again. If the failure persists, please contact support.","localizedMessage":"Backup Configuration Validation Precheck. Status: FAILED Failed to copy test file to server: server=server.example.com

Environment

• VCF 9.1

Cause

This issue occurs when there is no free space on the configured Backup Server.

Resolution

Clear the space on the respective Backup/SFTP Server and retry the upgrade.