vSAN Health Service reports "Root filesystem not responsive" for File Service in L3 Stretched Clusters

Products

• VMware vSAN

Issue/Introduction

• vSAN Skyline Health reports "Root filesystem not responsive" for vSAN File Service on newly added hosts in a stretched cluster repeatly.
• The issue may only impact hosts in one site (e.g., primary site) attempting to communicate with the volume owner in the remote site.
• Standard vmkping may succeed (potentially using the management default gateway), but TCP connectivity to port 1564 (vDFS) fails.
• Log files (/var/run/log/vdfsd-server.log) on the remote vdfs server host show accepted connections that are immediately terminated:
  

  ... 9PSRVR: ... accepted connection from 172.29.2.7 w/ fd=105
  ... VDFSSKT: ... fd=105 -- readEOF
  ... 9PSRVR: ... Shutting down 9p conn to 172.29.2.7 w/ fd=105: 0 cmds

• This error puts the client proxy into a retry loop.

Environment

• VCF-VSAN-8 (vSAN 8.x)

Cause

This issue occurs when vSAN traffic spans multiple L3 subnets (typical in stretched clusters) and the ESXi hosts lack explicit static routes for the remote vSAN subnets.

While general ICMP traffic (vmkping) might function via the default management gateway, the vDFS (vSAN Distributed File System)  requires symmetric routing on port 1564. If a static route is missing, the host may attempt to return traffic via an incorrect interface (asymmetry) or the traffic is dropped by network security policies, resulting in a readEOF error and a non-responsive filesystem.

Resolution

1. Add Static Routes: On each ESXi host, add a route for the remote vSAN subnet: esxcli network ip route ipv4 add -n [Remote_Subnet/Mask] -g [Local_vSAN_Gateway]
2. Verify Port Connectivity: nc -z [Remote_IP] 1564
3. Clear Stale State (Optional): If the health check remains Red, restart the vSAN management daemon: /etc/init.d/vsanmgmtd restart
4. Retest: Run the vSAN Skyline Health check to verify the status is Green.