Removing an inaccesible NFS datastore with SIOC enabled fails

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot remove an inactive or inaccessible NFS datastore.
• Storage DRS is enabled on the NFS datastore.
• Cannot unmount a read only NFS datastore that is mounted as read-write on another host with Storage IO control (SIOC) enabled from the vCenter Server.
• Unmounting the NFS datastore in vCenter Server fails.
• You see this error similar to:
  
  'Call "HostDatastoreSystem.RemoveDatastore" form object "datastoreSystem-28" on vCenter Server "xxxx" failed.
  CannotRemove datastore 'datastore_name' because Storage I/O control is enabled on it. Correct it and re-try the operation
   
• Disabling Storage I/O Control on the datastore fails.
  You see this error similar to:
  
  Datastore is not accessible

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.5

Cause

This issue occurs as this is an expected behavior. You cannot remove an inactive NFS datastore with Storage I/O Control (SIOC) enabled. Furthermore, when the back end volume/mount is ungracefully removed, disabling SIOC will fail.

Resolution

To resolve this issue, manually remove the NFS mount/datastore through the command line after restarting the SIOC specific service. 
 

Note: Before attempting to remove the datastore, ensure that there are no virtual machines running or registered against the datastore. Backup programs or third-party tools may also be using the datastore. For more information on requirements for unmounting a datastore, see Unmounting a LUN checklist in How to detach a LUN device from ESXi hosts (2004605).

 

To manually remove the datastore:

1. Run this command to list the mounted datastores:
   
   esxcli storage nfs list 
   
   For example: Where NFS1 is the NFS datastore name.
   
   
    
2. Run this command to stop the SIOC service:
   
   /etc/init.d/storageRM stop
    
3. Initiate a storage rescan on the ESXi host through the vCenter Server GUI by right click on the ESXi host and select Storage > Rescan Storage) or through the command line by executing this command:
   
   vmkfstools -V
    
4. After the rescan completes, run this command to restart the SIOC service:
   
   /etc/init.d/storageRM start
    
5. Run this command to unmount the NFS datastore:
   
   esxcli storage nfs remove -v NFS1 

After running the esxcli storage nfs remove -v NFS1 command and it fails with this error similar to:

Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Sysinfo error: BusySee VMkernel log for details.

In the /var/log/vmkernel.log file, you see entries similar to:

WARNING: NFS: 2104: NFS1 has open files, cannot be unmounted.

To resolve this issue:

1. Isolate the UUID of the NFS mount by running this command:
   
   vmkfstools -Ph -v10 /vmfs/volumes/NFS1
   
   Example: 
   
   vmkfstools -Ph -v10 /vmfs/volumes/NFS1/
   
   NFS-1.00 (Raw Major Version: 0) file system spanning 1 partitions.
   File system label (if any): NFS1
   Mode: public
   Capacity 86.5 GB, 86.5 GB available, file block size 4 KB, max supported file size 16777216 TB
   Disk Block Size: 512/512/0
   UUID: 003e96eb-9f51034a-0000-000000000000
   Logical device: 192.168.10.120 /mnt/Prod/NFS1
   Partitions spanned (on "notDCS"):
           nfs:003e96eb-9f51034a -----> UUID 
   NAS VAAI Supported: NO
   Is Native Snapshot Capable: NO
   OBJLIB-LIB: ObjLib cleanup done.
   WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0
    
2. List out the open files under this UUID by running this command:
   
   lsof | grep 003e96eb-9f51034a
   
   Example:
   
   lsof | grep 003e96eb-9f51034a
   2101956     sh                    cwd                        -1   
    
3. Stop the process by running this command:
   
   kill -9 PID
   
   Example:
   
   kill -9 2101956
    
4. You should now be able to remove the NFS mount successfully.