Messaging manager service is down after network partition or when storage issue was recovered

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• NSX Manager nodes experienced network partition between other Manager nodes or storage issue.
• The network or storage issue has been recovered but messaging manager is still down.
  • You can check the status of the cluster and each service on GUI [System] >> [Appliances] or through NSX CLI command "get cluster status".

Environment

VMware NSX-T Data Center

VMware NSX

Cause

In case of network partition or storage issue, corfu DB is down and messaging manager can not establish a connection to corfu DB.

Eventually it gives up retry process and messaging manager does not re-establish a connection with corfu DB even after corfu DB started to work fine.

Resolution

Manually restart messaging manager service.

systemctl restart messaging-manager