SFTP Authentication Failure Causes "Backup for component vidb timed out" During VCF 9.1 Upgrade

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

1. During a VMware Cloud Foundation (VCF) upgrade, the Identity Broker (vidb) component fails at the backup phase. This occurs specifically during the vmsp_upgrade_vidb step, where the process reaches a 120-minute timeout limit without successfully completing the data transfer to the configured SFTP destination.

2. VCF Fleet LCM pod logs report "timed out"
   
   Command: kubectl logs vcf-fleet-upgrade-service-fleetupgrade-#######-###### -n vcf-fleet-lcm | less
   Log location on VSP worker node: /var/log/pods/vcf-fleet-lcm_vcf-fleet-upgrade-service-fleetupgrade-#####-########/fleetupgrade/##.log
   Log location in SOS bundle: /sos-###/VCFMgmtComponents/vcf-management-services/<vsp fqdn>/services-logs/vcf-fleet-lcm/<node name>/vcf-fleet-upgrade-service-fleetupgrade-####/fleetupgrade/console-log-*.log

   [timestamp] level=ERROR, message={id=com.vmware.vcfms.backup.BackupTimeout, defaultMessage=Backup for component vidb timed out. Please retry the operation or contact support if the issue persists. [VCFMS-BACKUP-COMPONENT-006], localizedMessage=Backup for component vidb timed out. Please retry the operation or contact support if the issue persists. [VCFMS-BACKUP-COMPONENT-006], args={referenceCode=#######}}, stageId=########-####-####-####-############}, {timestamp=[timestamp], level=ERROR, message={id=com.broadcom.vcf.lcm.task.orchestrator.AsyncStepFailed, defaultMessage=Step vmsp_upgrade_vidb failed. Reference Code: #######. Check logs for details., localizedMessage=Step vmsp_upgrade_vidb failed. Reference Code: #######

3. VMSP Platform upgrade-component logs report "statusCode=218"
   
   Command: kubectl logs upgrade-component-############### -n vmsp-platform -c execute-script | less
   Log location on VSP worker node: /var/log/pods/vmsp-platform_upgrade-component-#####/execute-script/##.log
   Log location in SOS bundle: /sos-###/VCFMgmtComponents/vcf-management-services/<vsp fqdn>/services-logs/vmsp-platform/<node name>/upgrade-component-####/main/console-log-*.log.gz

[timestamp] stderr F [timestamp] INFO : ############### UPGRADE: Attempt 239: statusCode=218, running=false, elapsed=7140s

4. VMSP Platform backup-hook logs report "unable to authenticate"

   Command: kubectl logs br-vmsp-platform-backup-hook-##### -n vmsp-platform | less
   Log location on VSP worker node: /var/log/pods/vmsp-platform_br-vmsp-platform-backup-hook-#####/backup-hook/##.log
   Log location in SOS bundle: /sos-###/VCFMgmtComponents/vcf-management-services/<vsp fqdn>/services-logs/vmsp-platform/<node name>/br-vmsp-platform-backup-hook-####/backup-hook/console-log-*.log

[timestamp] INFO - Starting backup job with inputs: component=vidb, timeout=120m[timestamp] INFO - Verifying SFTP configuration
[timestamp] Failed to create file system for destination "sftp:/example-directory/": NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain[timestamp] ERROR - SFTP is configured but not available.

Environment

VCFA 9.0.x

VCF Operations 9.1.x

Cause

The vmsp-platform-backup-hook pod cannot authenticate with the SFTP server. This is caused by invalid or expired credentials stored in the sftp-password-secret or an expired SOURCE_TOKEN, which prevents the backup from writing to the destination.

Resolution

Please open a case with Broadcom Support to update the SFTP password on VIDB node. For more information, see Creating and managing Broadcom support cases