vCenter Server is Inaccessible "No Healthy Upstream Error" due to Local Storage Hardware Failure

Products

• VMware vCenter Server

Issue/Introduction

• vCenter is hosted on a local filesystem
• It frequently reports "No healthy upstream"
• Reboot the vCenter may temporarily resolve the issue
• The errors match the KB The Datastore is inaccessible ( FC iSCSI / Local )
• Server BMC reports health warnings against the local drive.
• journalctl command in vCenter may report below errors:

[1209295.941796] systemd-journald[957]: Failed to write entry (22 items, 774 bytes), ignoring: Read-only file system
[1209295.941831] systemd-journald[957]: Failed to write entry (22 items, 793 bytes), ignoring: Read-only file system
[1209295.941865] systemd-journald[957]: Failed to write entry (22 items, 774 bytes), ignoring: Read-only file system

Cause

The hard disk that is backing the local filesystem where the vCenter is hosted experiencing failures.

Resolution

• Engage the hardware vendor for the hard disk replacement
• Depending upon the hardware failure, the files may or may not be copied from the faulty hard disk. 
• If the files could not be copied, restore the vCenter from the backup.

Additional Information

File-Based Backup and Restore of vCenter Server