Platform Health Check Error: "The object 'vim.Datastore:' has already been deleted or has not been completely created" when updating Fleet lifecycle component

Products

• VCF Automation

Issue/Introduction

When attempting to upgrade VMware Cloud Foundation (VCF) Services Runtime from 9.1 GA to 9.1.1 (or any subsequent 9.1 patch versions), the Platform Health Check fails and reports the following error:

 

Platform Health Check Error - unable to validate vSphere objects: unable to resolve datastore identifier Datastore :<id>: ServerFaultCode: The object 'vim.Datastore:<datastore-id>' has already been deleted or has not been completely created

This guide provides instructions on how to manually update the stale datastore reference within the VCF Services Runtime environment to resolve the health check failure.

Environment

VCF services runtime 9.1
VCF Automation 9.1

Cause

This issue occurs when the original datastore assigned to the VCF Services Runtime 9.1 cluster is renamed or deleted. This typically happens if the VCF Services Runtime 9.1 Virtual Machines (VMs) were migrated to a new datastore -either manually or automatically via Storage DRS (Distributed Resource Scheduler) - leaving the environment with an outdated datastore reference.

Resolution

1.Migrate VM Templates:

Ensure the VM template is successfully loaded and available on the new datastore. For detailed steps, please refer to KB article: https://knowledge.broadcom.com/external/article?articleNumber=434922

2. Note the new datastore MoID. 

1. Log in to the vSphere Client.
   
   
2. Navigate to the Storage tab and click on the new datastore.
   
   
3. Look at your browser's address bar. The URL will contain a string similar to:
   https://<vcenter-fqdn>/ui/app/datastore;nav=h/obj/datastore-1234/summary
   
   
4. The ID you need is the value after /obj/ (in this example, datastore-1234).

3.Establish SSH Session: 

Open an SSH session to a VCF Services Runtime cluster node. For detailed instructions on accessing cluster nodes, refer to the official Broadcom Technical Documentation: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/fleet-management/configuring-management-components/enable-ssh-access-to-the-vcf-services-runtime-cluster.html.

4.Elevate Privileges:

Switch to the root user by running the following command:

 

sudo su -

5.Stage the Patch Script:

Copy the attached patch-datastore.sh script into the target directory:

 

/home/vmware-system-user/

Ensure the script has executable permissions:

 

chmod +x /home/vmware-system-user/patch-datastore.sh

6.Export Kubeconfig:

Set the Kubernetes configuration environment variable to ensure proper cluster access:

 

export KUBECONFIG=/etc/kubernetes/admin.conf

7.Execute the Patch Script.

Navigate to the directory, review available options (optional), and execute the script using the new valid datastore ID and datastore URL

 

# To view available options: ./patch-datastore.sh --help

 

# To update the datastore reference:

# If you already know both the Datastore ID and its URL, pass both arguments directly: ./patch-datastore.sh --datastore-id Datastore:<new-datastore-moid> --datastore-url "ds://.."

 

# Example: ./patch-datastore.sh --datastore-id "Datastore:datastore-1234" --datastore-url "ds:///vmfs/volumes/5f8a1234-abcd5678-0000-000000000000/"

 

# If --datastore-url is omitted, the script will automatically query vCenter via govc to look up the URL. Ensure GOVC_URL, GOVC_USERNAME and GOVC_PASSWORD are set in your environment prior to running:

# Set your vCenter password (and optionally host/username)

# export GOVC_PASSWORD="your-vcenter-password" # export GOVC_URL="vcenter.example.com"

# Optional: Script will prompt if omitted

# export GOVC_USERNAME="[email protected]"

# Optional: Script will prompt if omitted

# Run the patch script ./patch-datastore.sh --datastore-id "Datastore:datastore-1234"

# Example: ./patch-datastore.sh --datastore-id "Datastore:datastore-1234"