Malware prevention service VM instance shows status DOWN due to Certificate Expired issue.

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• Service VM deployment instance(s) show status as Down in the NSX manager UI.
• Under System → Service Deployments → Service Instances tab, all Service instances under a deployment cluster report Health Status Down
• Details for the Service Instance Health, show Solution Status Down.
• One or more Service instances under this cluster may also show an alarm. Alarm details indicate "EPP Partner Channel Down" which indicates loss of connectivity between GI Host module and NSX Malware Prevention Service
• Service VM logs (/var/log/syslog) display messages similar to: 

 "Failed to register SVM to NSX"
 "<Date>T<Time>Z localhost NSX 7194 - [nsx@6876 comp=“nsx-mps-svm” subcomp=“python” username=“root” level=“ERROR” errorCode=“(‘CLI110’,)“] POST /napp/api/v1/platform/trust-management/certificates returned status: 403#012b’{“module_name”:“common-services”,“error_message”:“Certificate expired“}'

• Inside the service VM, security hub service and RAPID containers are not running, hence the Malware Prevention Service feature is not functional on the given host.

Environment

NAPP 4.1.2

Cause

When Malware Prevention Service (MPS) service VM boots up for first time on the ESX host, an openssl certificate is generated that is registered with the NSX Application Platform (NAPP) trust manager during service VM startup. The validity of the certificate is set from the time it is created for 10 years.

It is expected that the NTP server on the service VM (or the ESX host where the service VM is deployed) is in sync with the NTP server on NSX Application platform (NAPP). However, if the NTP details are not set on the service VM,  or if the NTP time on the service VM os out of sync with the NTP time on NAPP (even if it is off by a few seconds), it is possible that the certificate validity start time is out of sync with the NSX Application Platform NTP settings, and hence when such a certificate is registered, NAPP trust manager throws an error that the certificate is invalid.

Resolution

Prior to Service VM deployment, ensure that NTP settings are correct. 

Update the ESX host network settings (for all hosts in the cluster) with appropriate NTP server details.
Ensure the time shown on the service VM or the corresponding ESXi host matches with the time on the trust-manager pod running on NAPP. (Note that even an offset in excess of 10 seconds could cause the issue)
Redeploy the MPS service on the ESX host cluster.

The recommended resolution is to upgrade the SVM to 4.2 in which the issue is addressed. 

 

If you have queries , please contact Broadcom Support for further assistance