Service corfu-nonconfig-server shows as UNAVAILABLE on all three NSX Manager nodes

Products

• VMware NSX

Issue/Introduction

• The NSX Manager UI reports a degraded status where the corfu-nonconfig service is DOWN or UNAVAILABLE. This typically occurs following storage latency or unexpected power events.
• NSX Manager cluster status shows CORFU_NONCONFIG as UNAVAILABLE or UNKNOWN.
• The /nonconfig partition is missing from df -h output.
• corfu-nonconfig logs show UnrecoverableCorfuError related to lock files.
• Log signature: java.lang.IllegalArgumentException: Resource limit has to be positive: ####.

Environment

VMware NSX

Cause

Filesystem corruption or database inconsistency in the Corfu non-configuration layer caused by underlying storage infrastructure issues.

Resolution

1. Verify the cluster status using the command get cluster status.
2. If only one or two nodes are impacted in a three-node cluster, detach the failed node/s.
3. Deploy new NSX Manager node/s to replace the detached node/s.
4. If the entire cluster is impacted, restore the NSX Manager from a known good backup.
5. See Download Broadcom Products and Software to obtain the relevant OVA for redeployment.
6. See KB: NSX Manager is not working properly after experiencing storage issues affecting datastores related to the NSX appliance VMs for details and tech doc links on how to perform each resolution step.