Replication and Snapshot Creation fail in VMware Live Cyber Recovery 9.0.0.11

Products

• VMware Live Recovery

Issue/Introduction

• After upgrading to VMware Live Cyber Recovery (vLCR), snapshot creation fail
• You will notice the following events in the vCenter Server:

“Failed to snapshot VM '<vm name>' during stage: performing backup. Backup failed. Could not prepare VM (). There may be another backup or vMotion in progress.”
“Failed to snapshot VM '<vm name>' during stage: performing backup. VM <vm name> endAccess failed VixDiskLib_EndAccess failed. Internal error: Insufficient permissions in the host operating system [3014]”

Environment

• VMware Live Cyber Recovery 9.0.0.11
• VMware Cloud Foundation 9.x
  
  

Cause

 

• In the 9.0.0.11 release, VMware Live Cyber Recovery(vLCR) switched to a new mechanism of using a service account to authenticate with on-premises vCenter.
• The password for this account rotates every 90 days and the new password is updated for the service account.
• Under certain conditions the old password may get cached.
• This will cause certain operations to use the old cached password.
• The session for that operation gets invalidated because of the wrong password and after a few retries, the service account may get locked on the vCenter server
• This will eventually effect the communication between vLCR and the vCenter Server services causing replication and snapshot creation failures

 

Note: User interaction with vCenter is not impacted in any way.

Resolution

Broadcom Engineering is aware of the issue and is working on a permanent fix

Workaround:

 

1. Ensure no backups are in progress for VMs in the problematic Protected Site.
2. Ensure that no failback operations are in progress.  
3. Reboot/reset all connector VMs.  
4. Wait 10 minutes and attempt to take a snapshot.