Controller service may be down in environments with high storage latency.
search cancel

Controller service may be down in environments with high storage latency.

book

Article ID: 384356

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The UI reports an alarm of cluster unavailability due to service controller outage.
  • From '/var/log/syslog' in NSX-T Manager, we can see that CBM is reporting CONTROLLER services down:

    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### - [nsx@6876 comp="nsx-manager" level="INFO" s2comp="notification-service" subcomp="ccp"] GROUP_UPDATE event generated for member <Manager#1 ID> of group <Group UUID> and maintenance mode members {<Manager#1 ID>=GroupMemberInfo{memberId=<Manager#1 ID>, memberName=<Manager#1 name>, memberStatus=DOWN, heartbeatCycleId=<UUID>}, <Manager#2 ID>=GroupMemberInfo{memberId=<Manager#2 ID>, memberName=<Manager#2 name>, memberStatus=DOWN, heartbeatCycleId=<UUID>}, <Manager#3 ID>=GroupMemberInfo{memberId=<Manager#3 ID>, memberName=<Manager#3 name>, memberStatus=DOWN, heartbeatCycleId=<UUID>}}: {<Manager#1 name>=MAINTENANCE_MODE_OFF, <Manager#2 name>=MAINTENANCE_MODE_OFF, <Manager#3 name>=MAINTENANCE_MODE_OFF}
    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="corfu-cluster"] Membership update begin
    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="corfu-cluster"] Member: <Manager#1 ID>, name: <Manager#1 name>, status: DOWN
    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="corfu-cluster"] Member: <Manager#2 ID>, name: <Manager#2 name>, status: DOWN
    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="corfu-cluster"] Member: <Manager#3 ID>, name: <Manager#3 name>, status: DOWN
    ::
    YYYY-MM-DDTHH:MM:SS <hostname> NSX ##### MONITORING [nsx@6876 comp="nsx-manager" entId="42e074e8-564d-3d90-b4ef-033c6a6cf8b6" eventFeatureName="clustering" eventSev="error" eventState="On" eventType="cluster_unavailable" level="ERROR" subcomp="cbm"] All group members <Manager#1 ID>,<Manager#2 ID>,<Manager#3 ID> of service CONTROLLER are down.
     
  • From '/var/log/cloudnet/nsx-ccp-<timestamp>.log' in NSX-T Manager, we can see that it failed to get the layout.

    YYYY-MM-DDTHH:MM:SS  WARN CorfuRuntime-0 CorfuRuntime 1640 Tried to get layout from <Manager IP>:9000 but failed with exception:
    java.util.concurrent.ExecutionException: org.corfudb.runtime.exceptions.NetworkException: Disconnected (LAYOUT_REQUEST) [endpoint=<Manager IP>:9000]
            at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_372]
            at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_372]
     
  • From '/var/run/log/vmkernel.log' of the ESXi host, we can see that the storage on which the Manager node is running has high latency.

    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu61:2098797)WARNING: NFS: 5506: NFS volume <VolumeName> performance has deteriorated. I/O latency increased from average value of 10105(us) to 911796(us).
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu33:2098796)WARNING: NFS: 5506: NFS volume <VolumeName> performance has deteriorated. I/O latency increased from average value of 10106(us) to 1828251(us).
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu34:2098796)WARNING: NFS: 5506: NFS volume <VolumeName> performance has deteriorated. I/O latency increased from average value of 10106(us) to 3744780(us).
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu55:2098797)WARNING: NFS: 5506: NFS volume <VolumeName> performance has deteriorated. I/O latency increased from average value of 10109(us) to 1631028(us).
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu55:2098797)WARNING: NFS: 5506: NFS volume <VolumeName> performance has deteriorated. I/O latency increased from average value of 10109(us) to 3352043(us).

Environment

VMware NSX

Cause

If the underlying storage I/O is slow, the increased load on the Manager cluster may cause the cluster to become unstable, resulting in Corfu layout changes and write failures.
In this condition, Corfu may be determined to be System Down, and the controller service will be restarted when Corfu is determined to be System Down.

Resolution

The storage latency on which NSX Manager is running needs to be resolved.
Storage vMotion can be used to specify low-latency storage and change the NSX Manager datastore.

Additional Information

NSX Manager tolerates a maximum disk latency of less than 10 milliseconds, as described in the "Storage Requirements" section of the following document.

NSX Installation Guide - NSX Manager VM and Host Transport Node System Requirements

Logging on ESXi storage latency is also discussed in KB 318927.

KB 318927 - "performance has deteriorated" messages in ESXi host logs

Powered by Wolken Software