ESXi Host Configuration State Failed status on host prepared for NSX.

Products

• VMware NSX
• VMware NSX Networking

Issue/Introduction

• ESXi hosts are prepared for NSX as Host Transport Nodes.
• Status of one or more Host Transport Node in NSX UI is:
  NSX Configuration: "Failed"
  Status: "Down"
  
• On the impacted Host Transport Node, NSX_NESTDB service is "Down".
• In ESXi host's /var/log/vmkernel.log, you may see logs similar to the sample below:
  2024-09-03T07:42:09.112Z cpu64:38520490)DLX: 2651: vol '<volume-name>', lock at 35241984: Lock type: 10C00001. owner(s) MAC: <MAC-address-of-a-host-holding-exclusive-lock>:
2024-09-03T07:33:59.108Z nestdb-server[2100335]: NSX 2100335 - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" s2comp="nsx-core" tid="2100335" level="ERROR" errorCode="CMN50"] FileStore[/scratch/log/nsx-nestdb/] Trim - failed to remove file 'metrics_1725346200.log': 16-Device or resource busy

Environment

VMware NSX-T Data Center
VMware NSX

Cause

NSX services require access to host's scratch partition to persist logs, core files, and metrics. 

This issue will be observed when scratch location is not configured correctly, or the scratch partition is not used exclusively on per host basis.

Resolution

To mitigate this issue, please make sure that the ESXi host's scratch partition is correctly configured on per host basis. This will avoid file lock conflicts. 

Additional Information

For more information related to scratch location, please see KB article Creating a persistent scratch location for ESXi 8.x/7.x/6.x.