Dataplane service becomes in error state when the uplink config is changed in NSX Edge Bare Metal

Products

• VMware NSX

Issue/Introduction

• Impact dataplane service in NSX Edge Bare Metal after changing uplink config.
  
  
• You see messages similar to the following in /var/log/syslog.log:
  

  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 1000 ms waiting for dp-learning3 to quiesce
  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 2000 ms waiting for dp-learning3 to quiesce
  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 4000 ms waiting for dp-learning3 to quiesce
  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 8000 ms waiting for dp-learning3 to quiesce
  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 16000 ms waiting for dp-learning3 to quiesce
  ####-##-##T##:##:##.###Z BMEdge NSX 5859 SYSTEM [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="ovs-rcu" tname="urcu2" level="WARN"] blocked 32000 ms waiting for dp-learning3 to quiesce

• Edge CLI commands fail and you see similar to the following in /var/log/nsx-cli/nsxcli.log:
  

  ####-##-##T##:##:##.###Z 50985 cli.server.cli_command_service INFO {10000} CMD: get logical-router
  ####-##-##T##:##:##.###Z 50985 cli.utils.edge_utils ERROR Failed to invoke edge-appctl command: ####-##-##T##:##:##.###Z edge-appctl 51093 unixctl [WARN] failed to connect to /var/run/vmware/edge/dpd.ctl
  edge-appctl: cannot connect to "/var/run/vmware/edge/dpd.ctl" (Protocol error)
  ####-##-##T##:##:##.###Z 50985 cli.server.cli_command_service WARNING An unexpected error occurred: The dataplane service is in error state, has failed, is disabled, or Maintenance Mode is Enabled
  ####-##-##T##:##:##.###Z 50985 cli.audit INFO CMD: get logical-router (duration: 0.195s) (command: get logical-routers), Operation status: CMD_EXECUTED_WITH_ERROR_RESULT

Environment

VMware NSX

Cause

Multiple dataplane threads might be in deadlock due to the bug when the uplink config is changed in NSX Edge Bare Metal.

Resolution

There is currently no fixed version.

Workaround:
Restart dataplane using below command.

BMEdge> restart service dataplane