Periodic SDRS task failures seen on the vSphere GUI. "The object 'vim.StoragePod:group-p####' has already been deleted or has not been completely created"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

A failed SDRS task similar to the following is seen in the vSphere UI at regularly occurring intervals:

 

There are one or more scheduled SDRS tasks configured on vCenter and they are failing to run with the same error seen on the vCenter task view:

 

The vpxd log (/var/log/vmware/vpxd/vpxd.log) shows similar entries to the following:

2025-01-21T16:20:01.889Z warning vpxd[07007] [Originator@6876 sub=Vmomi opID=ProcessScheduledTaskFiring-6d3e5b43] VMOMI activation LRO failed; <<521005d1-60db-c66b-ce3b-2f36f798525d, >, StorageResourceManager, vim.StorageResourceManager.configureStorageDrsForPod, (null), (null)>, N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl.fault.ManagedObjectNotFound

<SNIP>

2025-01-21T16:20:01.892Z info vpxd[07007] [Originator@6876 sub=vpxLro opID=ProcessScheduledTaskFiring-6d3e5b43] [VpxLRO] -- FINISH task-2041
2025-01-21T16:20:01.892Z error vpxd[07007] [Originator@6876 sub=Default opID=ProcessScheduledTaskFiring-6d3e5b43] [VpxLRO] -- ERROR task-2041 -- 521005d1-60db-c66b-ce3b-2f36f798525d -- StorageResourceManager -- vim.StorageResourceManager.configureStorageDrsForPod: :vmodl.fault.ManagedObjectNotFound

<SNIP>

vim.StorageResourceManager.configureStorageDrsForPod: :vmodl.fault.ManagedObjectNotFound
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    obj = 'vim.StoragePod:########-####-####-####-########:group-p1004'
-->    msg = ""
--> }
--> Args:
-->
--> Arg pod:
--> 'vim.StoragePod:########-####-####-####-########:group-p1004'

Environment

vCenter Server Appliance 7x

vCenter Server Appliance 8x

Cause

This can happen if an SDRS datastore cluster was deleted after creating an SDRS scheduled task.

Deleting a datastore cluster does not remove the associated scheduled task. This causes the scheduled task to run against a pod ID (datastore cluster ID) that no longer exists.

 

NOTE: This KB describes a specific issue with a stale scheduled task. To verify you are hitting this issue:

* Determine the pod ID (datastore cluster ID), this can be taken from the error on the GUI or from the vpxd.log:

Pod ID from GUI:
The object 'vim.StoragePod:group-p1004' has already been deleted or has not been completely created

Pod ID from vpxd.log:
obj = 'vim.StoragePod:########-####-####-####-########:group-p1004'


In this example the pod ID is 1004

* Determine if the pod ID (datastore cluster ID) exists on vCenter:

SSH to the vCenter and run the following command:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select pod_id from vpx_sdrs_pod_config;"

Example output:


From the example output above it can be seen that there is only one SDRS datastore cluster configured for this vCenter and its pod ID is 1005. There is no pod ID of 1004, this means that the scheduled SDRS task is configured to run against a datastore cluster that no longer exists in vCenter and this KB is applicable. 

Resolution

Delete the stale SDRS scheduled task.