NSX Manager connectivity is lost when powered off and migrated to another host prepared cluster

Products

• VMware NSX

Issue/Introduction

NSX Manager connectivity is lost when powered off and migrated to another host prepared cluster.

Cause

This issue occurs because the NSX Manager is in the list of excluded VMs for Distributed Firewall by default. The NSX Manager sends the list of excluded VMs that reside on a given cluster to hosts of that cluster. NSX Manager keeps track of VMs that are moved from one cluster to another and sends this information to hosts upon VM migration.

But if the NSX Manager is powered off, there is no communication between the hosts and the NSX Manager to send the updated exclude list down to the migrated cluster. When the NSX VM is migrated, it ends up having a DFW filter and existing DFW rules will be applied disrupting the NSX Manager communication.

VMware recommends NOT to deploy the NSX Manager inside a host prepared cluster.

To avoid getting into this issue before migrating the NSX Manager VM to another host prepared cluster, one of the these workaround applies.

1. Change the default rule to Allow.
2. Add a couple of temporary rules allowing traffic from/to NSX Manager IP (applied to ANY). Migrate the VM from one cluster to another. Once the VM is up and running, delete the temporary rules.

Resolution

This issue is resolved in VMware NSX for vSphere 6.4.0.

Workaround:

1. Clear the rules for the NSX Manager DFW filter on the host where the VM was migrated.
   1. SSH into the ESXi host that has the NSX Manager VM.
   2. Run the command summarize-dvfilter or vsipioctl getfilters to find out the filter name that is protecting the NSX VM. The filter  name starts with `nic-', such as nic-#####-eth0-vmware-sfw.2.
   3. Run the command vsipioctl getrules -f <filter-name> to get the ruleset of the firewall. There should be two rulesets, such as domain-c7 and domain-c7_L2. The first ruleset like domain-c7 is the one you need to remove.
   4. Run the command vsipioctl vsipfwcli -f <filter-name> -c "create ruleset <ruleset-name>;" to clear the ruleset.
   5. Run the command vsipioctl getrules -f <filter-name> to get the ruleset of the firewall. The first ruleset should be empty without any rules. NSX VM should be online now.
2. Reboot the NSX Manager.