Troubleshooting NFS Datastore "File System Timeout" and Hung VMs due to Expired Storage Certificates

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple Virtual Machines (VMs) become unresponsive or "hung" across multiple NFS datastores.

• New VMs cannot be powered on if residing on the affected NFS datastore.

• The vmkernel.log on ESXi hosts repeatedly reports synchronous I/O timeouts: NFS: 7038: Status: File system timeout (Ok to retry). Retrying synchronous read I/O [N] of 25 times

• Network connectivity tests (vmkping, nc -vz <IP> 2049) succeed, showing the network transport is functional, but file-level operations are stalling.

Environment

VMware vSphere ESXi 8.0.x

Cause

An expired CA (Certificate Authority) or identity certificate on the storage array. ESXi does not require a valid certificate to communicate with an NFS server; however, the expired certificate might cause internal I/O process issues on the NFS server itself. This can cause the storage array to throttle or drop NFS I/O requests.

Note: The symptoms described (NFS timeouts and hung VMs) can have other causes. This article specifically addresses the scenario caused by expired certificates on the storage array.

Resolution

1. Check Storage Certificates:

   • Engage the storage vendor or internal storage team to inspect the health of the storage array.

   • Specifically check for any expired CA or identity certificates on the array.

2. Renew Certificates: If an expired certificate is found, renew it according.

3. Recovery:

   • Once the certificate is renewed, monitor the ESXi logs to ensure the "File system timeout" messages have stopped.

   • If VMs remain unresponsive, they may require a hard restart to clear the I/O stun.

Additional Information

Troubleshooting NFS datastore connectivity issues

VM hangs when NFS datastore loses connection