Migration or power on operations fail in VMware environments using NFS datastores

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

While conducting a Live Migration or while trying to power on a virtual machine after a migration, you may experience one of the following symptoms:

VMware Infrastructure (VI) Client generates the error:

Failed to open the swap file.
VI Client generates the error:

A general system error occurred. Source detected that destination failed to resume
While using the Migration wizard, you receive the warning:

The virtual machine must be powered off to perform this function

Environment

VMware ESX Server 3.0.x
VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESX 4.1.x

Resolution

Cause

Migration or power on operations may fail due to a mismatch in the datastore UUID. If two ESX hosts represent that datastore using a different UUID in /vmfs/volumes you may experience issues performing power on or migration operations.

Identifying the issue

To verify if you have a mismatch between UUIDs:

Log into the ESX host which is successfully running the virtual machine as root via SSH or at the console.
Run the command:

ls -l /vmfs/volumes

The output appears similar to:

drwxrwxrwt 1 root root 980 May 29 09:25 8687c82b-3e59cbae
lrwxr-xr-x 1 root root 17 Jun 11 12:51 vm_nfs_disk -> 8687c82b-3e59cbae
Make note of the UUID (highlighted in red).
Log into the ESX host which cannot power on or accept migration of the virtual machine as root via SSH or at the console.
Run the command:

ls -l /vmfs/volumes

The output appears similar to:

drwxrwxrwt 1 root root 980 May 29 09:25 fef0f955-dceeecfc
lrwxr-xr-x 1 root root 17 Jun 11 12:51 vm_nfs_disk -> fef0f955-dceeecfc
Make note of the UUID (highlighted in blue).
If the values recorded in step 3 and in step 6 are not the same, then you have a UUID mismatch.

Solution

A UUID mismatch between two datastores occurs because the UUID is based on a hash of the NFS Server and Path, as seen by running esxcfg-nas. If you have specified the NFS server information using different methods on different hosts, then the hash value, and ultimately the UUIDs will be different.

To resolve the mismatch:

Log into the ESX host which is successfully running the virtual machine as root via SSH or at the console.
Run the command:

esxcfg-nas -l

The output appears similar to:

nfs_datastore is /vol/nfs_datastore from nfs-server.private.example.com mounted

Note: The UUID is based on /vol/nfs_datastore and the DNS name nfs-server.private.example.com.
Make note of the method being used to identify the NFS server highlighted in red.
Log into the ESX host which cannot power on or accept migration of the virtual machine as root via SSH or at the console.
Run the command:

esxcfg-nas -l

The output appears similar to:

nfs_datastore is /vol/nfs_datastore from 192.168.1.150 mounted

Note: The UUID is based on /vol/nfs_datastore and the IP address 192.168.1.150.
Make note of the method being used to identify the NFS server highlighted in blue.
Choose one method to identify the NFS server (DNS or IP).
Select the host which is not using the method selected in step 7.
Connect to the host using VI Client with the appropriate permissions.
Power off (or relocate if possible) the virtual machines residing on the NFS datastore which has the mismatch.
Remove the NFS datastore.
Add the same datastore using the method selected in step 7. For more information, see Creating an NFS-Based Datastore documentation for the applicable version of VMware product in the ESX Configuration Guide.
Repeat the operation until all ESX hosts reference the same NFS server in the same method selected in step 7.

Additional Information

Note: If the two hosts with mismatched NFS datastore mounts are managed by vCenter Server, one of the NFS datastore mounts may be renamed with a (1) suffix to differentiate them. To prevent this issue from occurring, ensure that the NFS mount has been unmounted from all ESX/ESXi hosts before mounting using the IP address.