Unable to mount NFS datastore from Veem Backup server to ESXi host
search cancel

Unable to mount NFS datastore from Veem Backup server to ESXi host

book

Article ID: 406270

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Attempts to mount Veeam Backup NFS datastore fails with the following error:
    "An error occurred during host configuration: . Operation failed, diagnostics report: Mount failed: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Unable to connect to NFS server"
  • Attempts to mount the datastore via CLI also failed with the error:
    esxcli storage nfs add -H <nfsserver> -s <sharename> -v <volumename>
    "Mount failed: Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Unable to connect to NFS server: VSI node (5001:)"

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The mount requests from the ESXi host are being retransmitted, specifically at port 1063. These retransmissions suggest a potential network-level issue, such as the port being blocked or unavailable.

Log snips:

  • var/run/log/vobd.log file reports that ESXi host is "Unable to connect to NFS server"
    YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2097765]:  [vmfsCorrelator] 1825460240241us: [vob.vmfs.nfs.mount.failed] NFS mount failed for ##.##.##.##:/VeeamBackup_######### volume VeeamBackup_#########. Status: Unable to connect to NFS server.
    YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2097765]:  [vmfsCorrelator] 1825526915407us: [esx.problem.vmfs.nfs.mount.failed] ##.##.##.##:/VeeamBackup_######### volume VeeamBackup_######### Unable to connect to NFS server.

  • var/run/log/hostd.log file lists below events:
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2100007]: [Originator@6876 sub=Solo.Vmomi opID=########-########-auto-#####-h5:########-##-##-#### sid=52110330 user=vpxuser:INTERNAL\#######] Arg spec:
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: --> (vim.host.NasVolume.Specification) {
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: -->    remoteHost = "##.##.##.##",
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: -->    remotePath = "/VeeamBackup_#########",
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: -->    localPath = "VeeamBackup_#########",
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: -->    accessMode = "readWrite",
    YYYY-MM-DDTHH:MM.SSSZZ Db(167) Hostd[2099969]: -->    type = "NFS",
    YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099969]: --> }
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2100007]: [Originator@6876 sub=Solo.Vmomi opID=########-########-auto-#####-h5:########-##-##-#### sid=52110330 user=vpxuser:INTERNAL\#######] Throw vim.fault.PlatformConfigFault
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2100007]: [Originator@6876 sub=Solo.Vmomi opID=########-########-auto-#####-h5:########-##-##-#### sid=52110330 user=vpxuser:INTERNAL\#######] Result:
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: --> (vim.fault.PlatformConfigFault) {
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->    faultMessage = (vmodl.LocalizableMessage) [
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->       (vmodl.LocalizableMessage) {
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->          key = "com.vmware.esx.hostctl.default",
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->          arg = (vmodl.KeyAnyValue) [
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->             (vmodl.KeyAnyValue) {
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->                key = "reason",
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->                value = "Mount failed: Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Unable to connect to NFS server: VSI node (5001:) "
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->             }
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->          ],
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->       }
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->    ],
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->    text = "",
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: -->    msg = ""
    YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2099969]: --> }

  • var/run/log/vmkernel.log file report errors "Synchronous RPC cancel for client"

    YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu37:2100007 opID=14b05373)SunRPC: 3615: Synchronous RPC cancel for client 0x43132ea01f80 IP ##.##.##.##.#.## proc 1 xid 0xa81eea75 attempt 1 of 3
    YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu37:2100007 opID=14b05373)SunRPC: 3615: Synchronous RPC cancel for client 0x43132ea01f80 IP ##.##.##.##.#.## proc 1 xid 0xa81eea75 attempt 2 of 3
    YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu42:2100007 opID=14b05373)SunRPC: 3615: Synchronous RPC cancel for client 0x43132ea01f80 IP ##.##.##.##.#.## proc 1 xid 0xa81eea75 attempt 3 of 3

    The error "Synchronous RPC cancel for client" in the context of NFS typically indicates an issue with the communication between the NFS client and server. This often occurs when the client abruptly terminates a request or the server encounters an unexpected error during the RPC call. VMware ESXi host is unable to establish a connection to the NFS server for a write operation.

  • Run the below command to collect the packet captures by initiating mount request. 
    pktcap-uw --vmk vmkX --dir 2 -o /vmfs/volumes/<datastoreID>vmk.pcapng

  • Packet captures confirms that NFS mount requests were being retransmitted, especially after reaching port 1063.

Resolution

Verify with the Veeam team if port 1063 is required for NFS communication in the backup server environment. If confirmed:

  1. Coordinate with the internal network team to ensure port 1063 is open between the ESXi host and the NFS server.
  2. Retry the NFS mount operation after confirming network accessibility.

Additional Information

Veeam Backup & Replication Ports

Broadcom is not responsible for the accuracy or reliability of information published in third-party documentation.

 

Powered by Wolken Software