vCenter is stuck or down
search cancel

vCenter is stuck or down

book

Article ID: 389605

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • vCenter is stuck and cannot be accessed.
  • ESXi server on which the vCenter VM is hosted or residing may be disconnected from vCenter and can be managed through host cli or UI.
  • Virtual machines may not be responding and working as expected.

Validation:

1. While querying the Filesystem on the ESXi host, one or more file system may report size as '0' B. 

Login into ESXi Host and the command "localcli storage filesystem list" to validate the filesystem size.

[root@Hostname:/vmfs/volumes/########-####-####-########/log] localcli storage filesystem list
/vmfs/volumes/########-####-####-########  Testdatastore  ########-####-####-########   false  VMFS-6     0       0

 

2. VMFS extent query on host will fail with messages "Error getting data for filesystem".

Login into ESXi Host and the command "localcli storage vmfs extent list" and this fill query the extents.

[root@Hostname:/vmfs/volumes/########-####-####-########/log] localcli storage vmfs extent list
Errors:
Error getting data for filesystem on '/vmfs/volumes/########-####-####-########': Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Not found, skipping.

 

3. You may observe below messages in /var/run/log/vmkernel.log

2025-03-01T04:03:37.111Z cpu18:2101895)Vol3: 2204: Could not open device 'naa.#####################:1' for volume open: Not found
2025-03-01T04:03:37.111Z cpu18:2101895)Vol3: 4462: Failed to get object 28 type 1 uuid ########-####-####-######## FD 0 gen 0 :Not found

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Host may get into not responding state and disconnected from vCenter as the Storage device attached to ESXi host is forcefully removed from backend storage array without performing proper cleanup on host end. As the vCenter VM resides on ESXi host which is not responding so managing the vCenter is not possible. VCenter VM may not work as expected.

Storage device is reported be in PDL (Permanent device loss) condition. You may observe below messages in /var/run/log/vmkernel.log

2025-03-01T07:00:33.738Z cpu25:2097937)WARNING: NMP: nmp_PathDetermineFailure:3527: Cmd (0x84) PDL error (0x5/0x25/0x0) - path vmhba64:C0:T1:L5 device naa.###################### - triggering path evaluation

 

Host is attempting to recover from a PDL, hostd will become unresponsive. This results in the host’s inability to communicate with vCenter and vCenter displaying the host, associated VMs and resources as inaccessible, disconnected, or inactive.

 

Resolution

Perform filesystem rescan manually on each host using cli as the ESXi host were the vCenter VM is registered may be still down until rescan is completed.

 
Below command can be run to rescan the filesystem on the all the ESXi host were the datastore backing the storage device which was forcefully removed from storage array. By running the following command also rescans for datastores and associated paths. Datastore pretending to be mounted or stale entry will be cleared.
 
localcli storage filesystem rescan
 
Further you may also restart all the host services using the command "services.sh restart". It is always good to reboot your host to remove any PDL residue.


Additional Information

Powered by Wolken Software