VIDB Restore Fails From Fleet Management With Error "LCMVMSP10011 Resource reconciliation failed"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When attempting to restore the VMware Identity Database (VIDB) from Fleet Management, the task fails with error code LCMVMSP10011.

The vmware_vrlcm.log on the Aria Suite Lifecycle appliance contains the following error strings:

• Handling InvalidEnvironmentException with error message: Environment with ID or Name : globalenvironment does not exist.
• com.vmware.vrealize.lcm.vmsp.common.exception.RestClientException: API failed with response: {"statusCode":218,"responseMessage":"status code 218","outputData":"{\"error\":\"Resource reconciliation failed after restore\"}

Reviewing logs for vmsp-incremental-backup-XXXXX pod shows below error:

 

To resolve this issue, verify and restore connectivity to the SFTP server using the following steps:

1. SSH into the VIDB appliance using the vmware-system-user credentials and switch to the root user: 

   sudo su -

2. Export the Kubernetes configuration to enable CLI command below:

   export KUBECONFIG=/etc/kubernetes/admin.conf

3. Search for pods associated with the restoration or incremental backup tasks:
   
   

   kubectl get pods -A | grep -i restore
   kubectl get pods -A | grep vmsp-incremental-backup

    
   Example:
   
   
   
   Note: Validate the timestamp of the pods. An "Error" state on an old pod may be superseded by a more recent "Completed" pod.

4. For pods in an "Error" status, retrieve the logs to confirm the root cause:
   
   

   kubectl logs <errored_pod_name> -n vmsp-platform

    
   Example:
   
          ERROR : testfile: Failed to copy: Update Create failed: sftp: "Failure" (SSH FX FAILURE)
          ERROR : Attempt 1/3 failed with 1 errors and: Update Create failed: sftp: "Failure" (SSH FX FAILURE)
          ERROR :
          ERROR
          ERROR : testfile: Failed to copy: Update Create failed: sftp: "Failure" (SSH FX FAILURE)
          RDDAD . Attamnt 2/2 failed uith
          Failed to copyto: Update Create failed: sftp: "Failure" (SSH FX FAILURE)
          ERROR - SFTP is configured but not available.
          INFO - Hook values URL is not set, not setting hook output values
       
   Note: In example above, it shows: "SFTP is configured but not available." Similarly, other errors/failure causes are recorded.

Environment

VCF Operations 9.X
VMware Identity Broker 9.X

Cause

The SFTP server configured for VIDB backups is unavailable or unreachable, leading to a failure in resource reconciliation during the restoration process.

Resolution

• Validate that the SFTP server is powered on and reachable from the management network.
• Verify that firewall rules and Access Control Lists (ACLs) permit bidirectional traffic between the VIDB appliance and the SFTP server.
• Confirm the SFTP service is active on the backup host.
• Validate that the service account credentials provided for the restore are current and have not expired.
• Once connectivity and service availability are verified, restart the VIDB restore process within the Fleet Management interface.