NSX UI inaccessible post CBM cert replacement

Products

• VMware NSX

Issue/Introduction

• CBM certs have been replaced due to the following issue - NSX alarms indicating certificates have expired or are expiring
•  
• Certificates are replaced on multiple NSX Manager nodes in quick succession (whilst cluster status is degraded).
  • e.g. NSX Manager A certs are replaced. As expected, this results in service restart and cluster is degraded for a period. During this period NSX Manager B and/or C certs are replaced.
• NSX Manager logs shows that the certs were applied successfully with POST request returning 200 for the CBM certs

/var/log/proxy/reverse-proxy.log

[TIMESTAMP] <IP> <IP> "POST" "/api/v1/trust-management/certificates/<UUID>?action=apply_certificate&service_type=CBM_[CERT_TYPE]&node_id=<UUID>" "HTTP/1.1" 200 - 0 0 1738 842 "<IP>"  "<UUID>" "<FQDN" "127.###.###.###:7440"

• NSX Manager logs show bad cert

/var/log/nsxapi.log 

[TIMESTAMP]  WARN netty-1 NettyClientRouter 493572 userEventTriggered: unhandled event SslHandshakeCompletionEvent(javax.net.ssl.SSLHandshakeException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate)io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

• Cluster stays in a degraded state
• NSX UI is inaccessible

Environment

VMware NSX 4.1.x

Cause

A race condition exists between CBM certificate replacement task and the periodic sync task. As a result, the Corfu DB public trust store is not updated. Due to this, services fail to connect to Corfu DB.

Resolution

This issue is resolved in VMware NSX 4.2.1 available at Broadcom Downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

If this issue is encountered please contact VMware NSX GSS by opening a service request and referencing this KB article Creating and managing Broadcom support cases