Recovery steps for Malware Prevention Service (MPS) VM deployments when hosts are added in a cluster between NSX-T backup and restore operations

Products

• VMware NSX
• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

If a host is added to a cluster where MPS is deployed, post taking an NSX-T backup operation; after restore of NSX-T from the backup, the new SVM deployment on the newly added host is not recognised, and UI shows the SVM in failed/error state 

This is expected behaviour as the backup is old and does not have information regarding the added hosts in the cluster. In this document, we will cover steps that can be followed to recover SVM from a failed/error state after the restore

Timelines of the scenario described are as follows:

1. Malware Prevention Service has been deployed (enabled) on a cluster and SVMs have been deployed on all hosts of that cluster.
2. NSX-T backup is taken in this state.
3. After the backup, at a later point of time, a new host is added to the MPS enabled cluster. This triggers auto-deployment of SVM on the newly added host.
4. Now at a later date, NSX-T is restored from the previous backup (backup where the cluster did not have the new host added)
5. In this scenario, after restore, NSX does not recognise the SVM which was deployed post the backup and hence the SVM remains in an error/failed state.

 

 Deployment status showing Malware Prevention Status on the newly added host in a failed state. 

Environment

NSX 3.2.0 and above, all versions of NAPP and SSP

Cause

When Malware Prevention SVM is deployed on a cluster, the deployment is taken care of by the ESX Agent Manager (EAM) of the VC to deploy the MPS SVMs on each host in the cluster. After successfully deploying the Service VMs on the host, the EAM informs NSX and NSX starts recognising and configuring the SVM which has been deployed. 

Whenever a new host is added to a cluster, the EAM deploys the SVM on the newly added host and again, informs the NSX manager which starts recognising and configuring the new SVM. 

In this workflow, after the NSX is restored from the backup, the NSX only recognises the SVMs which were deployed before the backup was taken. So even if the new SVMs are present on the newly added host, the state is lost as it was not present in the backup. 

Resolution

To restore the deployment to a healthy state, we need to follow the below steps:

1. Login to vCenter

• Identify the SVMs in error/failed state on the newly added hosts (i.e., those added after the backup).
• Manually delete these SVMs from vCenter.

2. Login to NSX Manager UI

•    Navigate to: Security --> IDS/IPS & Malware Prevention --> Settings --> Shared --> Activate Hosts & Clusters for East-West Traffic

3. Click on the deployment status

• This shows the Malware Prevention Status page.

4. Locate the deployment errors in the list.

5. Choose one of the following:

• Click "Resolve All" to automatically attempt to redeploy SVMs across all failed hosts.

or 

• Click "Resolve" next to each failed SVM individually.

 

    6. Wait for NSX to redeploy the SVMs to the affected hosts.

• Once completed, the deployment status should reflect "Success" for all hosts.