Error: "Failed to get backup configuration..." when creating a Workload Domain in VCF Operations 9.0.x

Products

• VCF Operations

Issue/Introduction

 

• The Workload Domain creation workflow fails in VCF Operations 9.0.2.

• The deployment is blocked by an error stating: Failed to get backup configuration. Error while retrieving credential BACKUP for device null. More than 1 credential found, don't have exactly one password.

• You might also see this same error when trying to edit the Backup Settings

• The /var/log/vmware/vcf/operationsmanager/operationsmanager.log records an IllegalArgumentException, confirming the system found more than one BACKUP credential.
  
  
  RETRIEVE_BACKUP_CONFIGURATION_FROM_DB_FAILED Failed to retrieve backup configuration.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to retrieve backup configuration.
    at com.vmware.vcf.common.fsm.plugins.backup.restore.action.RetrieveBackupConfigurationAction.execute(RetrieveBackupConfigurationAction.java:59)
    at com.vmware.vcf.common.fsm.plugins.backup.restore.action.RetrieveBackupConfigurationAction.execute(RetrieveBackupConfigurationAction.java:23)
...
Caused by: com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Failed to get backup configuration.
    at com.vmware.evo.sddc.common.services.adapters.backuprestoreservice.BackupRestoreServiceAdapterImpl.getBackupConfiguration(BackupRestoreServiceAdapterImpl.java:110)
...
Caused by: com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Error while retrieving credential BACKUP for device null.
    at com.vmware.evo.sddc.common.services.adapters.credentialservice.CredentialServiceAdapterImpl.getCredential(CredentialServiceAdapterImpl.java:166)
...
Caused by: java.lang.IllegalArgumentException: More than 1 credential found, don't have exactly one password.
  
  

Environment

VCF Operations 9.0.2

Cause

The failure is triggered by a possible database collision. While only a single backup entry appears in the Fleet Management and SDDC Manager UIs, the underlying system database contains two discrete JSON credential blocks assigned to "entityType":"BACKUP":

1. One record utilizing "credentialType":"FTP"

2. An orphaned record utilizing "credentialType":"FTP_SERVER"

During the RetrieveBackupConfigurationAction phase of the Workload Domain deployment, the SDDC Manager queries the database for the backup configuration. The workflow encounters an IllegalArgumentException and fails because it strictly requires exactly one credential payload, but retrieves two.

 

To validate the database collision, open an SSH session to the SDDC Manager as the vcf user, elevate to root via su -, and execute the following API queries:

1. Generate an Access Token: (Replace <SSO_PW> with the [email protected] password)

   ACCESS_TOKEN=$(curl -vd '{"username" : "[email protected]", "password" : "<SSO_PW>"}' -H "Content-Type: application/json" -X POST https://localhost/v1/tokens -k | jq -r '.accessToken')
   

2. Dump and Parse the Credentials Database:

   curl -k -X GET https://localhost/v1/system/credentials -H "Authorization: Bearer $ACCESS_TOKEN" -H "Content-Type: application/json" > /tmp/sys_credentials.json
   grep -i "backup" -B 5 -A 10 /tmp/sys_credentials.json
   

   
   
   
3. Review Output: Examine the /tmp/sys_credentials.json dump. Look for the "entityType":"BACKUP" entries to confirm that both the valid FTP and the duplicate FTP_SERVER types are present.

Resolution

1. Take a snapshot of the SDDC Manager virtual machine.

2. Execute a guest OS reboot of the SDDC Manager appliance.

3. Once the appliance is back online, retry the Workload Domain creation workflow within VCF Operations.