"Forbidden: Error occurred in the backing network provider: The credentials were incorrect or the account specified has been locked., error code 403" in VCF Automation 9.x

Products

• VCF Automation

Issue/Introduction

• "Forbidden: Error occurred in the backing network provider: The credentials were incorrect or the account specified has been locked., error code 403" appears on the UI in VCF Automation Provider Management portal when navigating to Infrastructure > Networking > IP Spaces > <IPSpace> or when performing sync from Infrastructure > Networking > Edge Clusters
• Health status and type of edge cluster is displayed as 'Unknown' under Infrastructure > Networking > Edge Clusters
• On /var/log/services-logs/prelude/tenant-manager-X/file-logs/vcloud-container-info.log we see the below error message :
  

  YYYY-MM-DD HH:MM:SS,xxx | WARN     | auto-pool-networking-nsx-edge-cluster-query-xx-thread-x | NsxTResources                  | Unable to retrieve policy edge cluster list from NSX |
  com.vmware.vcloud.common.network.VsmException: Forbidden: Error occurred in the backing network provider: The credentials were incorrect or the account specified has been locked., error code 403
          at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.processException(NetworkSecurityErrorHandler.java:xxx)
          at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.handleError(NetworkSecurityErrorHandler.java:xx)
          at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:xx)

Environment

VCF Automation 9.x

Cause

VCF component can lose connectivity to another VCF component due to synchronization issues (for example, the password for the service account changes on the server side, while the client side is not updated)

Resolution

To manually remediate the VCF Automation service account in vCenter/NSX :
1. Log in to the VCF Automation interface at https://<vcf_automation_fqdn>/provider with a user assigned the Administrator role.
2. In the left pane, navigate to VCF Instances.
3. On the VCF Instances page, click the vertical ellipsis next to the VCF instance, and, from the drop-down menu, select Rotate Service Account.

Rotating a service account for a VCF Instance deletes and recreates the new service accounts for vCenter and NSX that the VCF Instance manages.

Additional Information

Remediate Passwords for VMware Cloud Foundation Components