NSX for vSphere stuck in publishing DFW rules

Products

• VMware NSX for vSphere

Issue/Introduction

NSX for vSphere stuck in publishing DFW rules.   Error code 301503 was generated:  Event Message:'Failed to publish firewall configuration version XXXX to cluster domain-cXXX. Refer logs for details

Environment

NSX for vSphere 6.4.x

Cause

Distributed Firewall ( DFW ) rules are modified and the publish attempted.  In the UI, the publish continued to spin and never completes.

vsm.log shows the following messages

2025-01-01 17:00:41.152 GMT ERROR TaskFrameworkExecutor-6 FirewallMessagingManager:179 - - [nsxv@6876 comp="nsx-manager" errorCode="MP100" level="ERROR" subcomp="manager"] Exception while publishing rule set to cluster: domain-c###

 

The java dump in vsm.log shows publishing is working on translation and working on virtual wires (Logical Switches) when the exception occurs

        at com.vmware.vshield.vsm.translation.service.TranslationServiceImpl.translateNoCache_aroundBody2(TranslationServiceImpl.java:258) ~[vsm-core-1.0.jar:?]
        at com.vmware.vshield.vsm.translation.service.TranslationServiceImpl$AjcClosure3.run(TranslationServiceImpl.java:1) ~[vsm-core-1.0.jar:?]
        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:67) ~[spring-aspects-5.3.22.jar:5.3.22]

        at com.vmware.vshield.vsm.translation.service.TranslationServiceImpl.translate(TranslationServiceImpl.java:224) ~[vsm-core-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl.translateInternal_aroundBody0(VirtualWireServiceImpl.java:405) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl$AjcClosure1.run(VirtualWireServiceImpl.java:1) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl.translateInternal(VirtualWireServiceImpl.java:398) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl.access$1(VirtualWireServiceImpl.java:398) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl$1.translate_aroundBody0(VirtualWireServiceImpl.java:280) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl$1$AjcClosure1.run(VirtualWireServiceImpl.java:1) ~[vsphere-1.0.jar:?]


        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:67) ~[spring-aspects-5.3.22.jar:5.3.22]
        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:73) ~[spring-aspects-5.3.22.jar:5.3.22]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.22.jar:5.3.22]
        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:71) ~[spring-aspects-5.3.22.jar:5.3.22]

        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl$1.translate(VirtualWireServiceImpl.java:279) ~[vsphere-1.0.jar:?]
        at com.vmware.vshield.vsm.vdn.service.VirtualWireServiceImpl$1.translate(VirtualWireServiceImpl.java:1) ~[vsphere-1.0.jar:?]

 

There are errors in the UI on a variety of logical switches, which means the vCenter DVS does not match the configuration in NSX Manager.   

Resolution

Run the repair API to fix the Logical Switch errors.

POST https://<nsx-manager-ip>/api/2.0/vdn/scopes/vdnscope-#?action=repair

The rule publish will succeed a few minutes later.