NSX Agents Down on ESXi Hosts Due to Inaccessible Scratch Partition

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vSphere ESXi

Issue/Introduction

• One or more NSX agents show down on the ESXi hosts, most commonly nestdb:
  
  
  
  
• DFW rules/policies have an "Unknown" status
• Any new DFW rules that are published do not take effect
• Any ESXi shell commands accessing /var/run/log fail with "Input/output error"
  
  
  
  
• 'esxcli system syslog config get' returns "Failed to access path /scratch/log"
  
  
  
  
• "No filesystem on the device" errors seen for various LUNs when issuing the command 'dmesg | grep -i failed'
  
  

Environment

VMware NSX - Any version

VMware vSphere ESXi - Any version

Cause

• The ESXi scratch partition is not mounted or has become inaccessible
• NSX agents rely on writing to the scratch partition for runtime data. The inability to write to the scratch partition results in the NSX agents going down. 
• This is a filesystem issue and not an NSX issue. 

Resolution

• Remount the scratch partition and then reboot the affected ESXi hosts
• Verify that all NSX agents are back up