VCF Operations loses connectivity to VCFA after the Redeploy and Restore procedure

Products

• VCF Automation
• VCF Operations

Issue/Introduction

After completing the Redeploy and Restore VCF Automation procedure, VCF Operations 9.1.0 (OPS) fails to authenticate to VCF Automation 9.1.0 (VCFA), causing VCFA monitoring to break. This issue typically occurs when a new VCFA instance is installed and then subsequently restored from a backup (for example, performing a fresh VCFA installation followed by a backup restoration at a later time).

The following symptoms may be observed:

• Within Operate > Administration > Integrations > Accounts, the VCF Automation for All Apps Organization adapter instance displays a Warning status.

• The Protect and Discovery view shows clear authentication or connectivity errors for VCFA.

•  VCF Operations reports VCFA as unhealthy or entirely stops receiving monitoring data from it.

• The SDDC Build Service (SBS) cannot authenticate to VCFA.

Environment

• VCF Automation 9.1.0
• VCF Operations 9.1.0

Cause

The documented Redeploy and Restore procedure causes a credential mismatch between VCFA and OPS.

The sequence of events that leads to this state is:

1. VCFA is installed or upgraded. Fleet LCM runs Day 0 account registration workflows, creating account-set-1 in the VCFA TM database and registering its opssr account with OPS.
2. A VCFA backup is taken. The backup includes the VCFA database holding account-set-1.
3. The VCFA component is deleted following the Redeploy and Restore documented procedure.
4. A new VCFA component is installed with new IP addresses and SSL certificates. Fleet LCM runs Day 0 again, creating account-set-2 in the new VCFA TM database and re-registering VCFA in OPS with account-set-2. OPS now holds account-set-2.
5. The VCFA component is restored from the backup taken in step (b). The VCFA database reverts to account-set-1. The restore does not replay Day 0 operations, so OPS is not updated and still expects account-set-2.
   
   

After the restore, VCF Operations holds account-set-2 but VCFA only knows account-set-1. This credential mismatch causes all OPS-to-VCFA and SBS-to-VCFA authentication to fail.

Resolution

There is no resolution built in the product yet and is expected in a future release. The workaround is as follows:

1. Take a backup of VCFA
   
   On-Demand Backup of VCF Automation and VCF Management Services
   
   
2. Login to https://<ops_primary_fqdn>/admin using local admin and enable SSH on the primary node
3. Login to the SDDC Manager's shell via SSH as vcf account then run command su root to elevate to root
4. Set the required environment variables:
   
   export VCFA_FQDN='<VCFA-Tenant-Manager-FQDN>'
export VCFA_SYSTEM_PASSWORD='<password-for-admin@System>'
export VCFA_PLATFORM_FQDN='<VCFA-cluster-node-FQDN>'
export VCFA_PLATFORM_PASSWORD='<SSH-password-for-VCFA-cluster-node>'
export MGMT_PLATFORM_FQDN='<Management-Platform-FQDN>'
export MGMT_PLATFORM_PASSWORD='<Management-Platform-password>'
   export MGMT_PLATFORM_CTLP_PASSWORD='<SSH-and-sudo-password-for-control-plane-node>'
export FLEET_LCM_FQDN='<Fleet-FBS-FQDN>'
export FLEET_LCM_USER='admin'
export FLEET_LCM_PASSWORD='<Fleet-SBS-admin-password>'
export OPS_FQDN='<VCF-Operations-FQDN>'
export OPS_USER='<OPS-admin-username>'
export OPS_PASSWORD='<OPS-admin-password>'
export OPS_SSH_PASSWORD='<OPS-SSH-password>'
   
   NOTE:
   
   

   Environment Variable	Description
   VCFA_FQDN	VCFA Tenant Manager FQDN
   VCFA_SYSTEM_PASSWORD	Password for admin@System
   VCFA_PLATFORM_FQDN	FQDN/IP of a VCFA cluster node with kubectl access
   VCFA_PLATFORM_PASSWORD	SSH password for VCFA_PLATFORM_FQDN (uses sshpass)
   MGMT_PLATFORM_FQDN	Management platform FQDN (FBS token endpoint + inventory)
   MGMT_PLATFORM_PASSWORD	Management platform password (VMSP password grant)
   MGMT_PLATFORM_CTLP_PASSWORD	SSH password for MANAGEMENT_PLATFORM_USER (uses sshpass); also used as the escalation password for sudo/su on the node.
   FLEET_LCM_FQDN	Fleet Build Service (FBS) FQDN
   FLEET_LCM_USER	Fleet Build Service (FBS) username - (typically "admin")
   FLEET_LCM_PASSWORD	Fleet Build Service (FBS) password
   OPS_FQDN	VCF Operations (OPS) FQDN - for cluster, use Primary node
   OPS_USER	OPS admin username - (typically "admin")
   OPS_PASSWORD	OPS admin password
   OPS_SSH_PASSWORD	OPS appliance SSH root password

   
   ** There are also optional variables that can be set, review the script details once downloaded
   
   
   
5. Download the vcfa-reregister.sh script attached to the bottom of this KB article.
6. Transfer the script via WinSCP (login as vcf user) to the SDDC node into the /root directory
7. Run /root/vcfa-reregister.sh
8. Review the summary printed at the end of the script output to confirm that:
   1. Two VCFA TM service accounts were created (vcf-sddc-lcm and opssr).
   2. VCFA is now listed in the VCF Operations inventory.
   3. Fleet LCM refresh task was submitted. Check Build > Lifecycle > VCF Management > Tasks to monitor progress
      
      
9. After the script and task completes, navigate in VCF Operations to Operate > Administration > Integrations > Accounts, click the 3 dots next to VCF Automaton for All Apps Organization adapter and select Edit, then click Validate Connection