VCF management component upgrade fails due to backup precheck failure

Products

• VCF Operations

Issue/Introduction

• During the management component upgrade in VCF Operations 9.1.x, the vmsp_upgrade task fails for components such as Identity Broker, Log Management, and Software Depot with the following respective errors:

  #Identity Broker:
  Database backup failed for component vidb. Please retry the operation or contact support if the issue persists.
  Step vmsp_upgrade failed. Reference Code: #######. Check logs for details.
  
  #Log Management:
  Unexpected error occurred while checking task: Error while extracting response for type [com.#####.###.###.####.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]
  An unexpected error occurred in step vmsp_upgrade. Reference Code: ########. Please contact support with this reference code. Detail: Unexpected error occurred while checking VMSP task: Error while extracting response for type [com.#######.###.###.####.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]
  
  #Software Depot:
  Unexpected error occurred while checking task: Error while extracting response for type [com.########.###.###.###.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]
  An unexpected error occurred in step vmsp_upgrade. Reference Code: #######. Please contact support with this reference code. Detail: Unexpected error occurred while checking VMSP task: Error while extracting response for type [com.#######.###.###.#####.upgrade.sdk.client.vmsp.model.Task] and content type [application/json]

• The upgrade workflow is halted because the required pre-upgrade backup fails to complete successfully.
• The following error messages are observed in the /services-logs/vcf-sddc-lcm//vcf-sddc-upgrade-service-sddcupgrade-####/sddcupgrade/console.log:

  "id": "VCFMS-PRECHECK-BACKUP-CONFIG-002"
  "id": "com.vmware.vcfms.system.precheck.backup.FailedCopyTestFile"
  "message": "Failed to create file system for destination \"sftp:/xxxxxxx//\":
               NewFs: couldn't connect SSH: ssh: handshake failed:
               knownhosts: key is unknown;"

•  

Environment

• VCF Operations 9.1.x

Cause

When the backup server's Fully Qualified Domain Name (FQDN) is configured in the VCF Management console using uppercase or mix of upper and lower case letters. The system's internal known hosts validation is strictly case-sensitive. If the configured FQDN contains uppercase characters, the SSH handshake fails to validate the key, resulting in a connection rejection and subsequent backup failure.

Resolution

To resolve the issue configure the backup FQDN using an all-lowercase hostname:

1. Login to the VCF Operations UI (https://<VCF_Operations_FQDN>/ui) as local user admin account.
2. Go to Build ➔ Lifecycle ➔ VCF Management ➔ Backup & Restore.
3. Remove the current Backup & Restore location configuration.
4. Add the backup location, using the all-lowercase FQDN.
5. Retry the upgrade task.

Once the backup is successfully completed, restart with the components upgrade.