Unable to remove inaccessible datastore entry from vCenter

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Datastore is Inaccessible in VC UI.

• No Files are available in VC UI for the respective Datastore.

• Datastore ‘{name}’ is not accessible. “No connected and accessible host is attached to the datastore.”

  

• Datastores may also show as accessible but will not have any reference to the esxi hosts or virtual machines.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This issue can be caused due to multiple reasons -

• The underlying storage device or LUN associated with the datastore may have been removed, decommissioned, or failed without proper cleanup.

• vCLS VMs were forcefully deleted from the VMs without following retreat mode. In this case, follow the steps mentioned in this KB to validate that the datastore entry is still present because of vCLS VMs only, and if that's the case try to enable retreat mode to flush the vCLS VMs entry and then disable retreat mode.

• vCenter database (VCDB) retains inaccessible datastore records when they are not properly removed, leaving objects in the environment.

Resolution

Perform a cluster level rescan on all the clusters where the datastore was mounted previously. Even after cluster level rescan, if the inaccessible datastore entry is seen, proceed with the below steps.

Note : Please make sure to take a snapshot or backup of the VCSA before making any changes to the VCDB, if the vCenter is standalone. Incase vCenter is in Enhanced Linked Mode (ELM), then please refer - VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Below are the steps we need to follow for identifying & removing the objects completely from the VCDB which are responsible for holding the datastore entry in the 

• Enable SSH login with root credential.
• Stop the vpxd service using the command : service-control --stop vpxd
• Run the below command connect to the vCenter database : 

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Note: "\d" will list all the tables in VCDB.

• Then we run the below command to search for the inaccessible datastore entry in VCDB :
  
  VCDB=# SELECT id FROM vpx_entity WHERE name = 'Datastore Name';
  
  

• Then we get the Datastore Id and using Datastore Id we can find out what objects are seen by the database inside the Datastore (like template or VMs) :

Example: 

  id
--------
16##67
(1 row)   

• So now that we know the Id of the datastore, we can find out if it is being associated with some objects which were present in the datastore :                                                                                                                                                  

 Example :

VCDB=# SELECT * FROM vpx_ds_assignment WHERE ds_id=16##67;

ds_id   | entity_id | accessible | mount_path | mount_id | mount_mode | mounted
--------+-----------+------------+------------+----------+------------+---------
16##67  |    16yy36 |            |            |          |            |
16##67  |    16yy48 |            |            |          |            |
(2 rows)

• There are two objects in the inventory that are still connected to this datastore. To find out the name of the objects please run the below commands :

Example :

VCDB=# SELECT * FROM vpx_entity WHERE id=16##36;
   id   |       name        | type_id | parent_id
--------+-------------------+---------+-----------
16##36  | <VM_Name>         |       0 |    16##90
(1 row)

VCDB=# SELECT * FROM vpx_entity WHERE id=16##48;
   id   |      name        | type_id | parent_id
--------+------------------+---------+-----------
16##48  | <VM_Name>        |       0 |    16##90
(1 row)

• Once we have all the details regarding VMs and templates with its IDs available inside the database, we can delete it with below commands one by one ;

DELETE FROM vpx_datastore WHERE id=16##67;
DELETE FROM vpx_ds_assignment WHERE ds_id=16##67;
DELETE FROM vpx_vm_ds_space WHERE ds_id=16##67;
DELETE FROM vpx_entity where id=16##67;

Once all the entries from the database are deleted then we have to start the vpxd service for the vCenter using the command: service-control --start vpxd

service-control --start vpxd

Additional Information

Additionally, if the datastore was removed or deleted from storage before removing from the host and is now

 - showing as inaccessible from vCenter

 - is present but errors out when accessing from the host client

 - cannot be removed from vCenter or host client while failing with "Not a known device"

 - does NOT show from the host CLI

The issue may be resolved by rebooting hosts showing the datastore attached.