NSX Edge Nodes dataplane service is down after exiting maintenance mode when using Service Insertion

Products

• VMware NSX

Issue/Introduction

• North-South service insertion is deployed and in use.
• The Edge Node was placed into maintenance mode and then exited.
• Management plane connectivity from the NSX Manager to the Edge Node is not working.
• Traffic flowing through the edge node on logical routers are impacted.
• The 'dataplane' service on the edge node is down:
  

  get service dataplane
  Service name:      dataplane
  Service state:     stopped

• NSX Edge logs display errors similar to the below:
  
  /var/log/syslog
  

  <Timestamp> nsxedge-21193684-1-2301281756346299142024 NSX 81301 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="dpc-pb" tname="dp-ipc23" level="ERROR" errorCode="EDG0400536"] Service port <Service Port UUID> not found
  <Timestamp> nsxedge-21193684-1-2301281756346299142024 7bb730031118 79853 - - <Timestamp> datapathd 81301 dpc-pb tname="dp-ipc23" [ERROR] Service port <Service Port UUID>not found errorCode="EDG0400536"

Environment

VMware NSX

Cause

This issue occurs when CCP's (Central Control Plane) internal deletion of Service Insertion Redirection Policies does not work as expected, leading to stale 'SiRedirectionPolicyMsg' objects on the Edge Node.

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround

 

1.  Ensure that a recent backup of NSX Manger is available as a precaution.

2.  Ensure that the NSX Manager cluster state is healthy by running the following command from any NSX Manager (admin mode):
     get cluster status

3.  Perform a rolling reboot of the NSX Manager nodes.

• After each manager node reboot, restart the MPA service by running the following command from NSX Manager (root mode):
  /etc/init.d/nsx-mpa restart
• This step is required tin addition to the NSX Manager reboot o remove the stale 'SiRedirectionPolicyMsg' policy.
• Ensure that the NSX Manager cluster state is healthy before proceeding to the next NSX Manager node.