NFS datastores enters All Paths Down condition with error 'SunRPC Synchronous RPC cancel for client'

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• NFS datastores experience intermittent APD with retrying synchronous write operations

Issue Validation:

• /var/run/log/vmkernel.log:

YYYY-MM-DDTHH:MM.SSSZ: cpu0:2098751)NFS: 6284: Status:No connection. Retrying synchronous write I/O 1 of 25 times
YYYY-MM-DDTHH:MM.SSSZ: cpu34:2100050 opID=dc6bca1b)World: 12077: VC opID lxxewe5x-1457912-auto-v8xl-h5:70078357-a5-02-8f-c2d6 maps to vmkernel opID dc6bca1b
YYYY-MM-DDTHH:MM.SSSZ: cpu34:2100050 opID=dc6bca1b)SunRPC: 3291: Synchronous RPC cancel for client 0x430eddc01540 IP ##.##.##.##.## proc 1 xid 0xa9446181 attempt 1 of 3
YYYY-MM-DDTHH:MM.SSSZ: cpu12:13085698)NFS: 6284: Status:No connection. Retrying synchronous write I/O 1 of 25 times
YYYY-MM-DDTHH:MM.SSSZ: cpu54:2098211)StorageApdHandler: 1191: APD start for 0x431acd202b50 [b773ed96-8a62270e]
YYYY-MM-DDTHH:MM.SSSZ: cpu11:2097542)StorageApdHandler: 408: APD start event for 0x431acd202b50 [b773ed96-8a62270e]
YYYY-MM-DDTHH:MM.SSSZ: cpu11:2097542)StorageApdHandlerEv: 110: Device or filesystem with identifier [b773ed96-8a62270e] has entered the All Paths Down state.

or

YYYY-MM-DDTHH:MM.SSSZ: Wa(180) vmkwarning: cpu10:2098493)WARNING: SunRPC: 4847: fail all pending calls for client 0x4319a1801f90 IP ##.##.##.##.## (socket half closed)
YYYY-MM-DDTHH:MM.SSSZ: Wa(180) vmkwarning: cpu2:2098367)WARNING: NFS41: NFS41FSAPDNotify:6856: Lost connection to the server ##.##.##.##.## mount point <Datastorename>, mounted as ########-########-####-###########

 

• /var/run/log/vobd.log:

YYYY-MM-DDTHH:MM.SSSZ: [APDCorrelator] 11051063865764us: [vob.storage.apd.start] Device or filesystem with identifier [b773ed96-8a62270e] has entered the All Paths Down state.
YYYY-MM-DDTHH:MM.SSSZ: [APDCorrelator] 11051208970053us: [esx.problem.storage.apd.start] Device or filesystem with identifier [b773ed96-8a62270e] has entered the All Paths Down state.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The error "Synchronous RPC cancel for client" in the context of NFS typically indicates an issue with the communication between the NFS client and server.

This often occurs when the client abruptly terminates a request or the server encounters an unexpected error during the RPC call.

VMware ESXi host is unable to establish a connection to the NFS server for a write operation.

Resolution

1. Verify network connectivity between the client and server using ping and other network diagnostic tools.

2. Check for network congestion that might be causing packet loss or delays.

3. Ensure that necessary ports (typically port 111 for RPC and 2049 for NFS) are open in firewalls on both the client and server.

4. Check the NFS server configuration for any issues like incorrect export paths, permissions, or quotas.

5. Monitor the NFS server for any errors or instability that could be affecting RPC calls.

Additional Information

RPC (Remote Procedure Call): A protocol that allows a program on one computer to execute a program on another computer.
Synchronous RPC: A type of RPC where the client waits for a response from the server before proceeding.

NFS Storage Guidelines and Requirements