vSAN File Service NFS Mount Fails with RPC Timeout due to Hung Ganesha Daemon

Products

• VMware vSAN

Issue/Introduction

NFS clients experience mount failures on specific ESXi hosts within a vSAN cluster. Diagnostic symptoms include:

• NFS mount commands from clients fail with an RPC timeout.
• Running showmount -e localhost from within the vsfs container on the affected File Service Virtual Machine (FSVM) returns: RPC: Timed out.
• The ganesha.nfsd process appears in an Sl+ (Interruptible Sleep) state when checked via ps or top inside the container.
• Other ESXi hosts in the same cluster may be able to mount the same shares successfully.

Environment

vSAN 8.x

vSAN File Services

Cause

The Ganesha NFS daemon (ganesha.nfsd) enters a hung state while processing RPC requests. This is typically characterized by a broken pipe in the transport layer, preventing the daemon from responding to new mount requests or I/O operations from clients.

In the ganesha.log (located inside the FSVM container at /var/log/ganesha.log), the following error repeats: rpc :TIRPC :svc_ioq_flushv() writev failed (32) (Broken Pipe)

This indicates that the daemon's internal I/O queue for RPC services has encountered a terminal failure on a socket, causing the service thread to hang vSAN FSVM Mount Failure – “No such file or directory” Error Observed When Mounting File Share

Resolution

To resolve the hung state and restore NFS services on the affected host, the vSAN File Service (vsfs) container must be restarted.

1. Identify the affected FSVM: Use the vSphere Client to determine which FSVM is hosting the problematic share or residing on the ESXi host reporting the timeout.
2. Access the FSVM console: Open a console session (SSH or Web Console) to the FSVM.
3. Identify the container: Run the following command to find the vsfs container ID:

   docker ps

   Review this command before running it.
4. Restart the container: Execute a restart of the vsfs container to reinitialize the Ganesha daemon:

   docker restart <container_id>

   This command will make changes to your system. Review it carefully before running.
5. Verify Service: Once restarted, verify that showmount -e localhost returns the list of exports successfully.