Periodic SDRS task failures seen on the vSphere GUI. "The object 'vim.StoragePod:group-p####' has already been deleted or has not been completely created"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• A failed SDRS task similar to the following is seen in the vSphere UI at regularly occurring intervals:
  

 

• There are one or more scheduled SDRS tasks configured on vCenter and they are failing to run with the same error seen on the vCenter task view:

 

• The vpxd log (/var/log/vmware/vpxd/vpxd.log) shows similar entries to the following:

  <SNIP>
  
  ########-####-#### info vpxd[07007] [Originator@6876 sub=vpxLro opID=ProcessScheduledTaskFiring-6d3e5b43] [VpxLRO] -- FINISH task-2041
  ########-####-#### error vpxd[07007] [Originator@6876 sub=Default opID=ProcessScheduledTaskFiring-6d3e5b43] [VpxLRO] -- ERROR task-2041 -- 52####d1-60db-c66b-ce3b-2f3#####525d -- StorageResourceManager -- vim.StorageResourceManager.configureStorageDrsForPod: :vmodl.fault.ManagedObjectNotFound
  
  <SNIP>
  
  vim.StorageResourceManager.configureStorageDrsForPod: :vmodl.fault.ManagedObjectNotFound
  --> Result:
  --> (vmodl.fault.ManagedObjectNotFound) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    obj = 'vim.StoragePod:########-####-####-####-########:group-p1004'
  -->    msg = ""
  --> }
  --> Args:
  -->
  --> Arg pod:
  --> 'vim.StoragePod:########-####-####-####-########:group-p1004'

Environment

vCenter Server Appliance 7x

vCenter Server Appliance 8x

Cause

• This can happen if an SDRS datastore cluster was deleted after creating an SDRS scheduled task.
• Deleting a datastore cluster does not remove the associated scheduled task. This causes the scheduled task to run against a pod ID (datastore cluster ID) that no longer exists.

Resolution

NOTE: This KB describes a specific issue with a stale scheduled task. To verify you are hitting this issue:

• Determine the pod ID (datastore cluster ID), this can be taken from the error on the GUI or from the vpxd.log:
  
  Pod ID from GUI:
  The object 'vim.StoragePod:group-p1004' has already been deleted or has not been completely created
  
  Pod ID from vpxd.log:
  obj = 'vim.StoragePod:########-####-####-####-########:group-p1004'
  
  In this example the pod ID is 1004
• Determine if the pod ID (datastore cluster ID) exists on vCenter:
  
  SSH to the vCenter and run the following command:
  
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select pod_id from vpx_sdrs_pod_config;"
  • Example output:
    
    
    From the example output above it can be seen that there is only one SDRS datastore cluster configured for this vCenter and its pod ID is 1005. There is no pod ID of 1004, this means that the scheduled SDRS task is configured to run against a datastore cluster that no longer exists in vCenter and this KB is applicable. 
• Delete the stale SDRS scheduled task.

• Remove Stuck Task via MOB (In case if you are not able to remove from vSphere console UI)

  1. Verify the Stale ID (Optional but recommended):

     • Select your current Datastore Cluster in the vSphere Client.

     • Look at the URL in your browser. It will end in something like ...%3Agroup-p5555.

     • Compare this ID (group-p5555) with the ID in your error message. They will likely match names but have different IDs.

  2. Log in to the MOB:

     • Open a web browser to: https://<vcenter-fqdn>/mob

     • Login with [email protected].

  3. Locate the Scheduled Task Manager:

     • Click on Content (Value: ServiceInstance -> ServiceContent).

     • Scroll down to the row scheduledTaskManager and click the link (e.g., scheduledTaskManager).

  4. Find the "Configure Storage DRS" Task:

     • Click the more... link in the value column for the scheduledTask property to expand the list.

     • You will see a list of IDs like schedule-101, schedule-102.

     • Click on each ID to check its info -> name property until you find the one named "Configure Storage DRS" (or the specific task name failing).

  5. Delete the Task:

     • Once you have identified the correct schedule-ID (e.g., schedule-205):

       • Copy the ID or go back to the scheduledTaskManager page.

     • At the bottom of the scheduledTaskManager page, locate the method: RemoveScheduledTask.

     • Click the link RemoveScheduledTask.

     • In the pop-up window, replace the task value MOID with the ID you found (e.g., schedule-205).

     • Click Invoke Method.

     • Result should be "void".

  6. Verification:

     • Refresh the vSphere Client. The stuck task should be gone.

     • You can now create a new Storage DRS schedule if needed against the current valid object.