NSX-v Controllers show status "disconnected"

Products

• VMware NSX for vSphere

Issue/Introduction

• The cli shows that roles are changing frequently:
  
  > show control-cluster roles

Listen-IP Master? Last-Changed Count
api_provider Not configured Yes 02/10 11:25:51 26
persistence_server N/A Yes 02/10 11:25:51 25
switch_manager 127.0.0.1 Yes 02/10 11:25:51 26
logical_manager N/A Yes 02/10 11:25:51 26
directory_server N/A Yes 02/10 11:25:51 26
   
• Running the command "show log cloudnet/cloudnet_java-zookeeper.log filtered-by fsync", you see output similar to:
  
  <Date> 20:17:44,070 667152532 [SyncThread:13] WARN org.apache.zookeeper.server.persistence.FileTxnLog - fsync-ing the write ahead log in SyncThread:13 took 1037ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide

Environment

VMware NSX for vSphere 6.4.x

Cause

The NSX-v Controller is being impacted by underlying storage issues.

Resolution

Underlying storage issues at the infrastructure layer should be investigated and addressed.