VCF components failed to login with "VCF Identity Broker encountered an issue during authentication" intermittently

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• The Identity Broker was deployed with 3-node cluster structure;
• Sometimes, when trying to login VCF component, such as VCF Operations or Fleet Management, during redirecting the login request to Identity Broker, the web page was redirected to an error page with "Error: VCF Identity Broker encountered an issue during authentication. Please contact your VCF Admin with the below details for resolution. Message: You can try again. If this error persists, contact your administrator."
  
  
  
  
• When synchronizing with domain controller, it may return "Sync Failed" intermittently. On the title bar, there is error as "LDAP server is not reachable. This could be due to network issues, firewall blocks, an incorrect hostname, or invalid BIND credentials. Please verify the configuration and retry the sync".
  
  
  
  
  
• In vidb-service log, there is entry like:
  
  YYYY-MM-DDTHH:MM:SS stdout F YYYY-MM-DDTHH:MM:SS ERROR vidb-service-<vidb pod id>:usergroup (usergroup-business-pool-0) [CUSTOMER;-;127.0.0.1;xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;-] com.vmware.vidm.usergroup.service.broker.connector.ActiveDirectoryServiceImpl - Failed to connect to Active Directory socket (<Domain Controller IP or FQDN>:389)


• /vidb-service/usergroup-service/file-log-#####.log:

YYYY-MM-DDTHH:MM:SS INFO  .... [client 1 @###] opening connection to <Domain Controller IP or FQDN>:636 
YYYY-MM-DDTHH:MM:SS INFO  .... established connection with <Domain Controller IP or FQDN>:636

YYYY-MM-DDTHH:MM:SS INFO  .... [client 2 @###] opening connection to <Domain Controller IP or FQDN>:636 
YYYY-MM-DDTHH:MM:SS ERROR ... Could not connect to the Directory javax.naming.CommunicationException: <Domain Controller IP or FQDN> [Root exception is java.net.SocketTimeoutException: Connect timed out]

Environment

VMware Cloud Foundation 9.x

VMware Identity Broker 9.x

Cause

At least one of the identity broker node failed to connect to TCP port 389 (LDAP) or 636 (LDAPS) of the domain controller. 

Resolution

Ensure sure all the identity broker nodes can access TCP port of the domain controller 

• Port 389 for LDAP
• Port 636 for LDAPS