REPO_SYNC is in Fail state on all NSX Managers after replacing NSX Manager certificates

Products

• VMware NSX

Issue/Introduction

• The following logs are observed in the NSX Manager /var/log/proton/nsxapi.log:
  

  <timestamp>  INFO task-executor-1-12 ApplianceDeploymentHelper 4444 - [nsx@6876 comp="global-manager" level="INFO" subcomp="manager"] Got token for triggering repo sync on node <NSX Node IP/FQDN>
  <timestamp>  INFO task-executor-1-12 ApplianceDeploymentHelper 4444 - [nsx@6876 comp="global-manager" level="INFO" subcomp="manager"] Calling trigger repo sync API on node <NSX Node IP/FQDN>, url : api/v1/cluster/node?action=repo_sync
  <timestamp>  INFO task-executor-1-12 CertificateThumbprintServiceImpl 4444 - [nsx@6876 comp="global-manager" level="INFO" subcomp="manager"] Certificate for manager: [<NSX Node IP/FQDN>] is :
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
  <timestamp>  WARN task-executor-1-12 CertificateThumbprintServiceImpl 4444 - [nsx@6876 comp="global-manager" level="WARNING" subcomp="manager"] [CTS] Failed to get certificate thumbprint 
  java.lang.IllegalArgumentException: Illegal base64 character <ASCII hex-code>
  ...
  <timestamp> ERROR task-executor-1-12 LcmRestClient 4444 FABRIC [nsx@6876 comp="global-manager" errorCode="MP31815" level="ERROR" subcomp="manager"] TLS Error in rest call url= /api/v1/cluster/node?action=repo_sync , method= POST , response= null , error= [{"errorMessage":"TLS handshake failed","errorData":{"errorCode":"503"}}]
  org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://<Node IP/FQDN>:443/api/v1/cluster/node": <Certificate ID>; nested exception is javax.net.ssl.SSLHandshakeException: <Certificate ID>
  

   
• The NSX Manager cluster utilizes Custom CA certificates for VIP and API service types.

Cause

This issue is due to certificate string that is supposed to be pure Base64 encoded data contains a character that is not part of the standard Base64 alphabet. The error reports the character in hex code which can be identified in the ASCII list

Resolution

This issue is resolved in VMware NSX 9, available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

• Replace the current custom certificate with a new, valid certificate. Refer to Import a Self-signed or CA-signed Certificate