Cluster formation fails with "Node failed to establish trust with leader"

Products

• VMware Avi Load Balancer

Issue/Introduction

• When trying to configure a 3 node cluster from the UI, the cluster configuration fails to save with the error seen below:
  

Environment

• All deployments where there is a custom secure channel certificate with a non-default common name(name other than node.controller.local).
• Avi Version 22.1.x, 30.1.x. 

Cause

• As part of the cluster configuration, the Avi admin adds the follower node details to the leader UI. 
• Ref: https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-1/vmware-nsx-advanced-load-balancer-installation-guide/installing-nsx-alb-in-vmware-vsphere-environments/deploying-avi-load-balancer-controller-in-vmware-vcenter/modes-of-deployment/deploying-avi-vantage-in-write-access-mode/configuring-controller-cluster.html
• When inviting followers to the cluster, the leader sends an invitation(securetoken-import) to followers including its secure channel certificate Root CA and Common Name of the leaf certificate.
• The follower handles the invitation by issuing Secure Key Exchange GRPC back to the leader with the Root CA cert and Common Name(received in the invitation) used for validation during TLS handshake.
• During this step, the follower nodes may use a cached default common name for validation during TLS handshake, thus, causing the secure key exchange between the leader and itself to fail.

Resolution

• This issue has been fixed in 30.2.1 and later releases. 
• The recommendation is to upgrade the leader controller as a single node controller to the latest recommended release and then try to form the cluster again.
  • Please note that the followers will also have to be brought to the same version and patch version as the leader BEFORE the cluster formation is tried again.
• For the latest recommended release, refer to: https://knowledge.broadcom.com/external/article/312808/vmware-avi-load-balancer-release-notific.html
• If upgrading the controllers is not an option, please reach out to support for the best way to resolve the issue.