messaging manager is down after network partition or storage issue was recovered

Products

• VMware NSX

Issue/Introduction

• NSX Manager nodes experienced network partition between other Manager nodes or storage issue.
• The network or storage issue has been recovered but messaging manager is still down.
  • You can check the status of the cluster and each service on GUI [System] - [Appliances] or NSX CLI get cluster status.

Environment

VMware NSX-T Datacenter 3.2

VMware NSX 4.x

Cause

In case of network partition or storage issue, corfu DB is down and messaging manager can not establish a connection to corfu DB.

Eventually it gives up retry and messaging manager does not establish a connection to corfu DB even after corfu DB started working fine.

Resolution

Manually restart messaging manager service.

systemctl restart messaging-manager