NSX-T cluster is unavailable due to datastore outage and deactivate cluster
search cancel

NSX-T cluster is unavailable due to datastore outage and deactivate cluster

book

Article ID: 317152

calendar_today

Updated On:

Products

VMware NSX VMware NSX-T Data Center

Issue/Introduction

You have had a datastore outage on the ESXi's hosts where the NSX-T managers reside.

  • To recover the cluster you used the 'deactivate cluster' command and now you have a single node cluster.
  • The command 'get cluster status' shows everything up except: CORFU_NONCONFIG which is in an UNKOWN state:
Group Type: CORFU_NONCONFIG
Group Status: UNAVAILABLE
Members:
    UUID FQDN IP STATUS
    206dea88-####-####-####-###########de nsx-03.example.local ##.##.##.203 UNKNOWN
/nonconfig/corfu/corfu/LAYOUT_CURRENT.ds - shows the other 2 nodes are still present.
  "layoutServers": [
    "##.##.##.201:9040", ##########this node was removed
    "##.##.##.202:9040", ##########this node was removed
    "##.##.##.203:9040"
  ],
  "sequencers": [
    "##.##.##.201:9040", ##########this node was removed
    "##.##.##.202:9040", ##########this node was removed
    "##.##.##.203:9040"
  ],
  "segments": [
    {
      "replicationMode": "CHAIN_REPLICATION",
      "start": 0,
      "end": -1,
      "stripes": [
        {
          "logServers": [
            "##.##.##.202:9040", ##########this node was removed
            "##.##.##.201:9040" ##########this node was removed
          ]
       }
      ]
    }
  ],
  "unresponsiveServers": [
    "##.##.##.203:9040"
  ],
  "epoch": 1825,
  "clusterId": "ff44209a-####-####-####-##########86"
  • When you run the command 'get cluster config' on the remaining node, it correctly shows only the single remaining node.
  • In the following log we can see a loop looking for the now detached nodes: /var/log/corfu-nonconfig/corfu.9040.log
2022-05-31T14:59:51.264Z | DEBUG | CorfuRuntime-0 | o.corfudb.runtime.CorfuRuntime | Layout server ##.##.##.203:9040 responded with layout Layout(layoutServers=[##.##.##.201:9040, ##.##.##.202:9040, ##.##.##.203:9040], sequencers=[##.##.##.202:9040, ##.##.##.201:9040, ##.##.##.203:9040], segments=[Layout.LayoutSegment(replicationMode=CHAIN_REPLICATION, start=0, end=-1, stripes=[Layout.LayoutStripe(logServers=[##.##.##.202:9040, ##.##.##.201:9040])])], unresponsiveServers=[##.##.##.203:9040], epoch=1725, clusterId=ff44209a-####-####-####-##########86)
2022-05-31T14:59:51.264Z | INFO | initializationTaskThread | o.c.i.RecoveryHandler | Recovery layout epoch:1725, Cluster epoch: 1725
2022-05-31T14:59:51.264Z | ERROR | initializationTaskThread | o.c.i.ManagementAgent | initializationTask: Recovery failed 1364 times. Retrying in PT1Ss.
2022-05-31T14:59:52.262Z | DEBUG | client-5 | o.c.r.c.NettyClientRouter | connectAsync[##.##.##.202:9040]: Channel connection failed, reconnecting...
2022-05-31T14:59:52.265Z | DEBUG | initializationTaskThread | o.c.runtime.view.RuntimeLayout | Requested move of servers to new epoch 1726 servers are [##.##.##.203:9040, ##.##.##.202:9040, ##.##.##.201:9040]
2022-05-31T14:59:52.265Z | INFO | initializationTaskThread | o.c.runtime.clients.BaseClient | sealRemoteServer: send SEAL from me(clientId=null) to new epoch 1726
...
2022-05-31T14:59:52.464Z | DEBUG | client-6 | o.c.r.c.NettyClientRouter | connectAsync[##.##.##.201:9040]: Channel connection failed, reconnecting...
...
2022-05-31T14:59:53.265Z | DEBUG | initializationTaskThread | o.c.r.v.QuorumFuturesFactory | QuorumGet: Exception TimeoutException
2022-05-31T14:59:53.265Z | ERROR | initializationTaskThread | o.c.r.v.LayoutManagementView | Error: recovery: {}
org.corfudb.runtime.exceptions.QuorumUnreachableException: Couldn't reach quorum, reachable=1, required=2
        at 
...
2022-05-31T14:59:53.265Z | INFO | initializationTaskThread | o.c.i.RecoveryHandler | Recovery reconfiguration attempt result: false
2022-05-31T14:59:53.763Z | DEBUG | client-7 | o.c.r.c.NettyClientRouter | connectAsync[##.##.##.202:9040]: Channel connection failed, reconnecting...
2022-05-31T14:59:53.766Z | WARN | CorfuRuntime-0 | o.corfudb.runtime.CorfuRuntime | Tried to get layout from ##.##.##.202:9040 but failed by timeout
  • In the following log we also see timeout requests to the now detached nodes: /var/log/corfu-nonconfig/nonconfig-corfu-compactor-audit.log
2022-05-31T14:11:10.601Z WARN CorfuRuntime-0 CorfuRuntime - Tried to get layout from ##.##.##.201:9040 but failed by timeout
2022-05-31T14:11:11.102Z WARN CorfuRuntime-0 CorfuRuntime - Tried to get layout from ##.##.##.202:9040 but failed by timeout
2022-05-31T14:11:31.203Z ERROR main UfoCompactor - - [nsx@6876 comp="nsx-manager" errorCode="MP2" level="ERROR" subcomp="corfu-compactor"] UFO: Trim failed for ufo data in namespace ufo
org.corfudb.runtime.exceptions.UnreachableClusterException: Cluster is unavailable
        at com.vmware.nsx.platform.ufo.CorfuRuntimeHelper$1.run(CorfuRuntimeHelper.java:43) ...

Environment

VMware NSX-T Data Center
VMware NSX

Cause

This issue happens due to the underlying datastore issue and then issuing the 'deactivate cluster' command to attempt to recover the cluster.
The node where the deactivate cluster command was executed, was an unresponsive nodes at the time.

As this node was in an unresponsive state when the deactivate cluster command was issued, it was unhealthy even before the deactivate cluster command was issued.
This node needs to cure itself. However, to cure itself, it needs information from the other two nodes.
As the other two nodes no longer exist, this prevents the remaining node from being able to cure itself and therefore the cluster is down.

The loop is caused by this remaining unhealthy node trying to continually connect to the other 2 nodes to cure itself.

Best practice when a datastore issue causes impact to all 3 managers, is to restore from backup to a point in time before the datastore issue occurred.

Resolution

This is a known issue impacting NSX-T Datacenter when all three managers nodes are on the same datastore.

Workaround:

Restore the NSX-T cluster from backup, to a point before the datastore outage occurred: Restore NSX Manager from Backup
Powered by Wolken Software