Error: "An error occurred while communicating to remote host" while attempting to unmount one or more datastores

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x
• VMware vCenter Server
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• While attempting to unmount one or more datastores, you experience an error similar to the following: 

• All VM's are unregistered from the datastore
• The datastore is not used for HA Datastore heartbeating
• System core dump files are not configured to use the datastore
• ESXI System scratch is not configured to use the datastore
• Storage DRS and Storage I/O control are disabled.

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

• There is an an underlying storage acessibility issue on the ESXi host(s). There can be many causes for this, such as Fiber Channel fabric issues, networking connectivity issues, SAN issues, local storage issues, or corruption on the datastores.  NOTE: These are only some examples and are not an exhaustive list. `
• As a result of the storage layer issues, the hostd agent on the ESXi hosts may become unresponsive (see KB: Troubleshooting an ESXi host in a "not responding" state and unable to communicate with vCenter reliably until the underlying issues are fully resolved. 

Resolution

• The underlying problems with the datastore(s) will need to be identified and corrected.
• Please open a case with Broadcom Support for assistance, if needed. 

 

Workaround:

• You can manually unmount the datastore(s) from the hosts, if needed:
  
  
  • Before attempting to unmount or delete any datastore, please verify/complete the following:
    • Remove or migrate all virtual machines from the datastore.
    • Disable Storage DRS for the datastore. For more information, see TechDocs: Activate and Deactivate Storage DRS.
    • Disable Storage I/O Control for the datastore. For more information, see TechDocs: Enable Storage I/O Control.
    • Ensure that the datastore is not used for vSphere HA heartbeating. For more information, see TechDocs: Configure Heartbeat Datastores.
    • Ensure that the datastore to be removed is not use for system scratch. See KB: Creating a Persistent ESXi Scratch Location
    • Ensure that the datastore is not used for system coredump files: See KB: Configuring ESXi coredump to file instead of partition
      
      
  • Once all of that is verified and/or completed, you can try to manually unmount the datastore(s) with the following command:
    
    

localcli storage filesystem unmount -l <Datastore_Name>

NOTE:  this command can take a few seconds to 30 minutes or more to complete depending on the severity of the underlying issues. 

• • Repeat for each host, as required.

Additional Information

• For more information on troubleshooting storage issues on VMware products, see KB:  Troubleshooting storage issues when using VMware products