ESXi host boot process stuck on "swapobj loaded successfully"

Products

• VMware vSphere ESXi

Issue/Introduction

The ESXi host boot process hangs after displaying the message "swapobj loaded successfully", leading to boot times that take up to 30 minutes or greater for the ESXi host.

Once the boot process completes, the following can be found within the relevant logs. 

• Within /var/log/boot, the plugin "mount-disk-fs" starts after the "swapobj" plugin completes. 

         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)swapobj loaded successfully.
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)Jumpstart plugin tag-boot-bank activated.
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)Jumpstart plugin vsan activated.
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)Activating Jumpstart plugin vmkapps.
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)Activating Jumpstart plugin mount-disk-fs.

• Following the start of the plugin, logs similar to the following, indicating an inability of the ESXi host to connect to the NFS storage.

         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)NFS: ###: NFS mount ###.###.###.###:/ESXi_Logs failed: Unable to connect to NFS server.
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)NFS: ###: Command: (remount) Server: (###.###.###.###) IP: (###.###.###.###) Path: (/Dir1/Dir2/Dir3) Label: (PROD) Options: (None)
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)StorageApdHandler: ###: APD Handle XXXXXXXX-XXXXXXXX Created with lock[StorageApd-0x############]
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 1 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 1 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 2 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 2 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 3 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Synchronous RPC cancel for client 0x############ IP XXX.XXX.XXX.XXX proc 3 xid 0x######## attempt 3 of 3
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Destroying world 0x######
         YYYY-MM-DDTHH:MM:SS.###Z cpu#:#######)SunRPC: ####: Destroying world 0x######

 

• Eventually, the following log entry appears indicating the failure of the plugin to start.

        YYYY-MM-DDTHH:MM:SS.###Z cpu##:#######)Jumpstart plugin restore-nfs-volumes activation failed: restore-nfs-volumes->start() failed: error while executing the cli

• Within /var/log/jumpstart-stdout.log, a simplified view of the information above:

        YYYY-MM-DDTHH:MM:SS.###Z executing start plugin: mount-disk-fs
        YYYY-MM-DDTHH:MM:SS.###Z executing start plugin: restore-nfs-volumes
YYYY-MM-DDTHH:MM:SS.###Z Executor failed executing esxcli command: boot storage restore --nfs-volumes
        YYYY-MM-DDTHH:MM:SS.###Z Method invocation failed: restore-nfs-volumes->start() failed: error while executing the cli

• Within /var/log/jumpstart-esxcli-stdout.log, you may also see 

        YYYY-MM-DDTHH:MM:SS.###Z StorageInfoImpl: getaddrinfo(xssxhostFQDNxxxx) failed: Temporary failure in name resolution
        YYYY-MM-DDTHH:MM:SS.###Z StorageInfoImpl: getaddrinfo(xssxhostFQDNxxxxl) failed: Temporary failure in name resolution
        YYYY-MM-DDTHH:MM:SS.###Z StorageInfoImpl: getaddrinfo(xssxhostFQDNxxxx) failed: Temporary failure in name resolution

Environment

ESXi 6.X, 7.x and 8.x

Cause

This error appears when there is failed connectivity to the backend NFS storage during the boot process. 

Resolution

Investigate the network connectivity , DNS resolution (if the NFS is mounted via FQDN)  between the ESXi host and the NFS server.