NFS3 datastore fails to remount after host reboot
search cancel

NFS3 datastore fails to remount after host reboot

book

Article ID: 431281

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • NFS3 datastores on ESXi host appear as unmounted and inaccessible after ESXi host reboot.

  • To bring the datastore back to the accessible state, the NFS datastores can be manually unmounted and re-mounted on the ESXi host.

  • The physical network ports to which the ESXi host's vmnic (network interface) is connected to uses Edge Port. (Changed from Spanning Tree.)

  • The issue is intermittent - meaning during some ESXi host reboots the NFS datastores mount and during some they remain unmounted.

Environment

VMware vSphere ESXi 8.x

Cause

  • When the ESXi host invokes the "restore-nfs-volumes" plugin post-reboot, if the network interfaces required for NFS connectivity (vmnic) are not up within 10 seconds, the NFS datastore remount task fails.

  • And since the ESXi host 8.x lacks a retry mechanism to mount the NFS datastores after the vmnics eventually come online, it leaves the NFS datastores inaccessible until manual unmount and mount is performed.

 

  • Below is example of log snippets from an unsuccessful and successful remount of NFS datastore post host reboot:
    Assume vmnic4 & vmnic5 are the network interfaces used to connect to the NFS storage array.

    1. Unsuccessful NFS datastore remount post ESXi host reboot:

      • Reboot performed on 2026-02-06.

      • From the var/run/log/vmkernel.log, it can be seen that the restore-nfs-volumes plugin is called at 04:03:38 UTC and fails to restore the volumes by 04:03:50 UTC:

        2026-02-06T04:03:38.464Z In(182) vmkernel: cpu54:2097693)Activating Jumpstart plugin restore-nfs-volumes.
        2026-02-06T04:03:38.464Z In(182) vmkernel: cpu29:2097702)Applying start on plugin restore-nfs-volumes
        2026-02-06T04:03:50.731Z Al(177) vmkalert: cpu54:2097693)ALERT: Jumpstart plugin restore-nfs-volumes activation failed.

      • Looking at the var/run/log/vobd.log, it can be seen that the vmnic5 and vmnic4 come up at 04:03:49 and 04:03:50 UTC, respectively: 

        2026-02-06T04:03:49.525Z In(14) vobd[2097718]:  [netCorrelator] 47737058us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
        2026-02-06T04:03:50.896Z In(14) vobd[2097718]:  [netCorrelator] 49108273us: [vob.net.vmnic.linkstate.up] vmnic vmnic4 linkstate up

      • And on this day, if the order in which the vmnics came online is checked, the below can be observed, var/run/log/vobd.log:

        2026-02-06T04:03:41.381Z In(14) vobd[2097718]:  [netCorrelator] 39591629us: [vob.net.vmnic.linkstate.up] vmnic vmnic6 linkstate up
        2026-02-06T04:03:41.871Z In(14) vobd[2097718]:  [netCorrelator] 40082625us: [vob.net.vmnic.linkstate.up] vmnic vmnic7 linkstate up
        2026-02-06T04:03:46.525Z In(14) vobd[2097718]:  [netCorrelator] 44736977us: [vob.net.vmnic.linkstate.up] vmnic vmnic8 linkstate up
        2026-02-06T04:03:47.717Z In(14) vobd[2097718]:  [netCorrelator] 45928599us: [vob.net.vmnic.linkstate.up] vmnic vmnic9 linkstate up
        2026-02-06T04:03:49.525Z In(14) vobd[2097718]:  [netCorrelator] 47737058us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
        2026-02-06T04:03:50.896Z In(14) vobd[2097718]:  [netCorrelator] 49108273us: [vob.net.vmnic.linkstate.up] vmnic vmnic4 linkstate up

      • The time taken for all the vmnics to come online, from the first to the last, is ~9 seconds. (from vmnic6 to vmnic4)

      • It can be seen that the vmnic4 & vmnic5 used for the NFS connectivity comes up last.

      • From the above, it appears that the vmnics were not online when the ESXi storage initialization stack had called the "nfs-restore-volumes" plugin and timed out.

      • The time between the restore-nfs-volumes call and fail is ~12 seconds.

      • The time between the restore-nfs-volumes call and the first NFS vmnic up is ~11 seconds.

    2. Successful NFS datastore remount post ESXi host reboot:

      • Reboot performed on 2026-02-09.

      • From the var/run/log/vmkernel.log, it can be seen that the restore-nfs-volumes plugin is called at 04:11:36 and started mounting the first nfs volume at 04:11:47 UTC:

        2026-02-09T04:11:36.443Z In(182) vmkernel: cpu4:2097693)Activating Jumpstart plugin restore-nfs-volumes.
        2026-02-09T04:11:36.444Z In(182) vmkernel: cpu13:2097702)Applying start on plugin restore-nfs-volumes
        2026-02-09T04:11:47.161Z In(182) vmkernel: cpu13:2097702)NFS: 349: Command: (remount) Server: (<nfs_server_fqdn>) IP: (<nfs_server_ip>) Connections: (1) Vmknic: (None)  Path: (/<nfs_share_name>) Label: (<nfs_datastore_name>) Options: (None)
        2026-02-09T04:11:47.352Z In(182) vmkernel: cpu13:2097702)NFS: 349: Command: (remount) Server: (<nfs_server_fqdn>) IP: (<nfs_server_ip>) Connections: (1) Vmknic: (None)  Path: (/<nfs_share_name>) Label: (<nfs_datastore_name>) Options: (None)
        2026-02-09T04:11:47.512Z In(182) vmkernel: cpu4:2097693)Jumpstart plugin restore-nfs-volumes activated.

      • On this reboot, the vmnic5 & vmnic4 came up at 04:11:45 and 04:11:48 respectively, var/run/log/vobd.log:

        2026-02-09T04:11:45.016Z In(14) vobd[2097717]:  [netCorrelator] 45202625us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
        2026-02-09T04:11:48.146Z In(14) vobd[2097717]:  [netCorrelator] 48332613us: [vob.net.vmnic.linkstate.up] vmnic vmnic4 linkstate up

      • And on this day, if the order in which the vmnics came online is checked, the below can be observed, var/run/log/vobd.log:

        2026-02-09T04:11:38.469Z In(14) vobd[2097717]:  [netCorrelator] 38654960us: [vob.net.vmnic.linkstate.up] vmnic vmnic6 linkstate up
        2026-02-09T04:11:38.967Z In(14) vobd[2097717]:  [netCorrelator] 39152966us: [vob.net.vmnic.linkstate.up] vmnic vmnic7 linkstate up
        2026-02-09T04:11:45.016Z In(14) vobd[2097717]:  [netCorrelator] 45202625us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
        2026-02-09T04:11:45.241Z In(14) vobd[2097717]:  [netCorrelator] 45427133us: [vob.net.vmnic.linkstate.up] vmnic vmnic8 linkstate up
        2026-02-09T04:11:47.368Z In(14) vobd[2097717]:  [netCorrelator] 47554871us: [vob.net.vmnic.linkstate.up] vmnic vmnic9 linkstate up
        2026-02-09T04:11:48.146Z In(14) vobd[2097717]:  [netCorrelator] 48332613us: [vob.net.vmnic.linkstate.up] vmnic vmnic4 linkstate up

      • Here too, the time taken for all the vmnics to come online, from the first to the last, is ~9 seconds. (from vmnic6 to vmnic4)

      • But here it can be seen that vmnic5 comes online 3rd, at about ~6 seconds after the first vmnic comes online.

      • Here the time between the restore-nfs-volume call and the first NFS vmnic up is ~8 seconds.

      • Due to the vmnic being available before the 10second timeout, the NFS datastore gets mounted.

      • The time between the first NFS volume remount and vmnic up is ~2 seconds.

  • Based on the above findings:

    • The NFS datastores successfully remount, post reboot, as long as vmnic4 or vmnic5 comes online within 10 seconds after the "restore-nfs-volumes" plugin is invoked.

    • If it takes longer than 10 seconds, the NFS datastores fail to remount.

    • Also, there is no mechanism to retry the remount of NFS datastores during boot up in ESXi 8.x, due to which the NFS datastores remain unmounted even after vmnic4 & vmnic5 come online.

Resolution

  • Currently, ESXi 8.0U3 does not attempt to remount the NFS datastores after unsuccessful mount attempt during reboot of the host.

  • An additional restore NFS mount command can be added to the ESXi host's initialization script to retry the NFS datastore remount.

  • Below are the steps to enable this:

    Note: This script will not run if UEFI secure boot is enabled.

    1. SSH to an ESXi host.

    2. Validate if secure boot is enabled using the command: /usr/lib/vmware/secureboot/bin/secureBoot.py -s

    3. Edit the file using vi: vi /etc/rc.local.d/local.sh

    4. Add the following NFS datastore remount command to the file, above the "exit 0": esxcfg-nas -r
      This command, restores all NFS version 3 mounts from the configuration file.

    5. Save the file by pressing "Esc" and typing in ":wq!".

    6. View the file to validate that the change is updated: cat /etc/rc.local.d/local.sh

    7. Reboot the host to test.

Additional Information

Powered by Wolken Software