NSX UI inaccessible post CBM cert replacement

Products

• VMware NSX

Issue/Introduction

• CBM certs have been replaced due to the following issue - KB 324175
• Certificates are replaced on multiple NSX Manager nodes in quick succession (whilst cluster status is degraded).
  • e.g. NSX Manager A certs are replaced. As expected, this results in service restart and cluster is degraded for a period. During this period NSX Manager B and/or C certs are replaced.
• NSX Manager logs shows that the certs were applied successfully with POST request returning 200 for the CBM certs

/var/log/proxy/reverse-proxy.log

[TIMESTAMP] <IP> <IP> "POST" "/api/v1/trust-management/certificates/<UUID>?action=apply_certificate&service_type=CBM_[CERT_TYPE]&node_id=<UUID>" "HTTP/1.1" 200 - 0 0 1738 842 "<IP>"  "<UUID>" "<FQDN" "127.###.###.###:7440"

• NSX Manager logs show bad cert

/var/log/nsxapi.log 

[TIMESTAMP]  WARN netty-1 NettyClientRouter 493572 userEventTriggered: unhandled event SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate)io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

• Cluster stays in a degraded state
• NSX UI is inaccessible

Environment

NSX 4.1.x

Cause

A race condition exists between CBM certificate replacement task and the periodic sync task. As a result, the Corfu DB public trust store is not updated. Due to this, services fail to connect to Corfu DB.

Resolution

There's currently no resolution for this issue. If this issue is encountered please contact VMware NSX GSS by opening a service request and referencing this KB article.