vSAN on-disk format upgrade to version 20 fails due to inaccessible objects

Products

• VMware vSAN

Issue/Introduction

During a vSAN on-disk format upgrade to version 20, the upgrade pre-check fails. The system reports that objects are inaccessible, preventing the process from proceeding.

• The following error is displayed in the vSAN Health or Disk Management UI: Unknown object type - object inaccessible. unable to proceed with disk management upgrade
• Specific object UUIDs are identified as inaccessible (e.g., ########-####-####-####-############).
• Data availability is lost for these specific objects (APD - All Paths Down).

 

Environment

•  vSAN 8.X

Cause

The vSAN on-disk format upgrade pre-check mandates that all objects be in a healthy state to ensure metadata consistency and prevent potential data loss during the format transition.

In this scenario, the failure is caused by stale vCLS (vSphere Cluster Services) namespaces or orphaned metadata components.

These objects often remain in an ABSENT state due to previous host or disk failures. Because the required components are missing and the metadata (CSN) is inconsistent, the objects are marked as inaccessible, which blocks the cluster-wide upgrade.

Resolution

To resolve this issue, manually delete the stale, inaccessible objects using the objtool utility. This clears the entries from the vSAN object directory (CMMDS) and allows the pre-check to pass.

Prerequisites:

1. Verify the vSAN datastore has no active production workloads that might be impacted.
2. Identify the specific UUIDs of the inaccessible objects from the error logs or via the CLI.
3. It is recommended to take a file-based backup or snapshot of the vCenter Server before performing manual cluster-wide object deletions.

Steps:

1. Access ESXi host in the cluster via SSH as root.
2. Confirm the health and path of the objects. Use the following command to list inaccessible objects: Review this command before running it.

   esxcli vsan debug object list --health=inaccessible

3. Delete the stale objects using the UUIDs identified in the pre-check failure. Replace <UUID> with the actual object ID. This command cannot be undone. Verify every parameter before running.

   /usr/lib/vmware/osfs/bin/objtool delete -u <UUID> -f

Additional Information

• For more information on vSAN object health, see vSAN Health Service - Object Health.