Publishing Distributed Firewall rule fails in NSX Data Center for vSphere 6.4.x.

Products

• VMware NSX

Issue/Introduction

• Publishing Distributed Firewall rule fails.
• In the /home/secureall/secureall/logs/vsm.log.file on the NSX appliance, you see entries similar to:
  
  ERROR messagingTaskExecutor-5 ConfigurationPublisher:279 - - [nsxv@6876 comp="nsx- manager" errorCode="MP100" level="ERROR" subcomp="manager"] Firewall provisioning failed on Host host-5####2 for reason 301034:apply config to all vnics. Host is at generation 1579513963331. 
   
• In the /var/log/vsfwd.log file of the affected host, you see entries similar to:
  
  vsfwd: [INFO] Applying firewall config to vnic list on host host-5####2
  vsfwd: [WARN] Failed to applied RuleSet 15########331 for all vnics
  vsfwd: [INFO] Config is already saved file /etc/vmware/vsfwd/vsipfw_ruleset.dat. Skip now
  vsfwd: [WARN] Updating container securitygroup-1#9 that is not used by rules
  vsfwd: [WARN] Updating container securitygroup-1# that is not used by rules
  vsfwd: [WARN] Updating container securitygroup-4# that is not used by rules
  vsfwd: [WARN] Updating container virtualwire-1#6 that is not used by rules
  vsfwd: [WARN] Updating container virtualwire-1#9 that is not used by rules
  vsfwd: [WARN] Updating container virtualwire-1#9 that is not used by rules
  vsfwd: [WARN] Updating container virtualwire-1#0 that is not used by rules
  vsfwd: [WARN] Updating container virtualwire-##6 that is not used by rules
  vsfwd: [INFO] loaded addrset
  vsfwd: [INFO] Compressed config data from 280462 to 126525 bytes
  vsfwd: [INFO] Successfully saved config to file /etc/vmware/vsfwd/vsipfw_ruleset.dat_update
  vsfwd: [INFO] cleanup protobuf
  vsfwd: [WARN] error 33: 5# 3# 6# f# 6# a# a# 9# a# 0# 6# 7# 4# 0# 8# 0#  <<<<<<<<
  vsfwd: [WARN] error 33: 5# 3# 6# f# 6# a# a# 9# a# 0# 6# 7# 4# 0# 8# 0#  <<<<<<<<
  vsfwd: [WARN] error 33: 5# 3# 6# f# 6# a# a# 9# a# 0# 6# 7# 4# 0# 8# 0#  <<<<<<<<
  vsfwd: [INFO] vmkLinkMsgHandler: received notification for filter nic-97###46-eth0-vmware-sfw.2
  vsfwd: [INFO] vmkLinkMsgHandler: received notification for filter nic-97###46-eth0-vmware-sfw.2
  vsfwd: [INFO] Added filter nic-97####46-eth0-vmware-sfw.2
  vsfwd: [INFO] Created vmware-sfw filter for vnic 503#####-####-####-####-#########a1.000
  vsfwd: [INFO] Applying firewall config to vnic list on host host-5####2
  vsfwd: [INFO] Applied TimeoutSet 1530306171795 on vnic 503#####-####-####-####-#########a1.000
  vsfwd: [INFO] Applied TimeoutSet 1530306171795 for all vnics
  vsfwd: [INFO] Applied shared addrsets of gen number 15########331
  vsfwd: [INFO] Applying firewall config to vnic list on host host-5####2
    
  or
  
  vsfwd: [WARN] VM UUID (500#####-####-####-####-########cd8) is different than expected.
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

This issue occurs due to an invalid (non-VMX) VM UUID in the affected virtual machine.

Resolution

This is a known issue affecting VMware NSX Data Center for vSphere 6.4.x.
Currently, there is no resolution.

Workaround:

To work around this issue, edit the .vmx file entry on the affected virtual machines with these steps:

Note: This issue appears when a virtual machine has an abnormal UUID in its vmx file.The expected VM UUID is in format  "50 ## ## ## ## ## ## ##-## ## ## ## ## ## ## d8". But a user can modify this ID in the virtual machine's .vmx file and the vCenter Server accepts such abnormal ID. This causes other anomalies when applying Distributed Firewall configuration in the virtual machine.

1. On the ESXi host console, run this script vsip_vm_list.sh to fetch all the VM UUIDs. The VC UUIDs which are in the format "<world_id>:50######-####-####-####-##########d8" should be fixed manually by following the next steps.
2. Get the VM name using the VC UUID by running this command:
   
   summarize-dvfilter | grep -i "50######-####-####-####-##########d8"
    
3. Locate the virtual machine in the vCenter Server. Once located, power off the virtual machine.
4. Find the path of the VMX file by running this command and grep for the VM name in it:
   
   vim-cmd vmsvc/getallvms | grep <VM_NAME>
    
5. Navigate to the directory folder path of the VMX file.
6. Create a backup copy of the current .vmx file.
7. Edit the file and search for the line starting with vc.uuid.
8. Follow either one of the following:
   
   a. Change the format of the UUID string from "503#####-####-####-####-##########cd8" to "50 ## ## ## ## ## ## ##-## ## ## ## ## ## ## d8" and Save the file.
   or
   b. Remove the line starting with vc.uuid and Save the file.
    
9. Power on the virtual machine from the vCenter Server.
10. Run this command vsip_vm_list.sh and make sure that the UUID is now in the format of "<world_id>:50 50 ## ## ## ## ## ##-## ## ## ## ## ## ## d8" for the affected virtual machine.