Monitoring service shows DOWN in NSX UI

Products

• VMware NSX

Issue/Introduction

• You've recently replaced certificates and you see that the Cluster status is degraded and Monitoring service is showing down for one or 2 of the NSX Manager appliances.
• You also confirm the issue by running get cluster status command and you see the monitoring service as down and cluster status degraded confirming the issue.
  
  Below is the screenshot of 'get cluster status' command where monitoring service is shown down:
  
  
• You've checked the phonehome-coordinator service and it is up and running fine
• You've tried restarting the problem manager node and makes no difference
• When checked the phonehome-coordinator logs, you see below logs stating below error
  'io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate'
  
  

Environment

VMware NSX 
VMware NSX-T Datacenter

Cause

CBM_Monitoring certificate in keystore of the problem node does not match with corfu server trust store

Resolution

• Run the Certificate Analyzer, Results and Recovery (CARR) Script from our KB 369034 on the affected NSX Manager and it should show you the keystore mismatch issue as in below screenshot
  
  
  
  
  
• When prompted after the analysis, proceed to type 'yes' and let the script run to resolve the issue
• Observe the state of the monitoring service and it should come up after the script has run
• If the issue persists, please open a case with Broadcom Support Team troubleshoot the issue further