Error: SunRPC Synchronous RPC cancel during NFS All Paths Down (APD) in VMware vSphere ESXi
Article ID: 373878
Updated On:
Products
Issue/Introduction
Symptoms:
vMotion fails with the error: "
Operation not allowed in current state."When trying to vMotion a VM to an ESXi host this compatibility issue is encountered "
Unable to access virtual machine disk from the host"Migrating VMs to a specific host fails while others in the cluster succeed.
Provisioning new VMs fails with the error: "
fault cause cannot create DvPort ## of VDS."NFS datastores intermittently enter an All Paths Down (APD) state.
Datastore capacity occasionally displays as 0 B in the vSphere Client.
In
/var/run/log/vmkernel.log, the following entries are visible :SunRPC: ####: Synchronous RPC cancel for client 0x################ IP ##.##.##.## proc # xid 0x######## attempt # of 3 NFS: ####: Status:No connection. Retrying synchronous write I/O
Environment
- VMware vSphere ESXi 8.x
- VMware vSphere ESXi 9.x
Cause
This issue occurs when there is an MTU mismatch or packet fragmentation in the physical network path. If the ESXi host is configured for Jumbo Frames (MTU 9000) but an upstream switch or the storage controller is limited to a smaller frame size, the ESXi host cancels the Synchronous RPC request after a timeout.
Resolution
- Log in to the ESXi host via SSH.
- Verify that the VMkernel port used for NFS is configured for MTU 9000:
esxcfg-vmknic -l - Test connectivity to the NFS server using standard frames (1500 MTU):
vmkping -I vmk# <NFS_IP> - Test connectivity using Jumbo Frames (9000 MTU) with the "do not fragment" bit:
vmkping -I vmk# -d -s 8972 <NFS_IP> - If the standard ping succeeds but the Jumbo Frame ping fails, audit the physical switch path (including ToR switches and core) to ensure MTU 9000 is enabled end-to-end.
- Verify that port 111 (RPC) and 2049 (NFS) are not being blocked by a firewall.
- Check the NFS server configuration for any issues like incorrect export paths, permissions, or quotas
- Once the network path is consistent, the APD state should clear. If the datastore remains inaccessible, a host reboot may be required to clear stale I/O handles.
Additional Information
RPC (Remote Procedure Call): A protocol that allows a program on one computer to execute a program on another computer.
Synchronous RPC: A type of RPC where the client waits for a response from the server before proceeding.
Feedback
