After SDDC Manager is rebooted, several “Queued” tasks appear in the vCenter Server task bar, indicating “Creating Workload Domain” and/or “Commissioning Host(s) to VMware Cloud Foundation.”

Products

• VMware SDDC Manager / VCF Installer
• VMware vCenter Server

Issue/Introduction

• When SDDC Manager is rebooted, events such as the following may appear in the vCenter Server Task Bar
• SDDC Manager may have been recently updated.

• The tasks will eventually time out and fail.
• Rebooting the vCenter Server causes the tasks to be cancelled.
• In the vCenter Server journalctl -xe logs, events similar to the following may be observed:
  [YYYY-MM-DDTHH:MM:SS.######] [vim.event.TaskEvent] [info] [[email protected]] [] [########] [Task: Creating Workload Domain]
• However, no references to these tasks are found when reviewing the /var/log/vmware/vpxd/vpxd.log on vCenter.
• In SDDC Manager, the /var/log/vmware/vcf/operationsmanager/operationsmanager.log may contain entries similar to the following:
  YYYY-MM-DDTHH:MM:SS.######] WARN  [vcf_om,################,####] [c.v.v.t.services.TaskPublisher,om-scheduler-1] Failed to check for leftover tasks
com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.Task:task-#####' has already been deleted or has not been completely created

Environment

VMware Cloud Foundation 5.2.x (before 5.2.3)
vCenter Server 8.x 

Cause

After an SDDC Manager reboot, the system tries to restart tasks recorded in the database that are no longer current. These processes will either fail or time out because the conditions necessary for them to complete are no longer in place.

Resolution

Addressing this issue requires the manual deletion of specific entries from the SDDC database. 

Ensure backups of SDDC are completed before making any changes 

 

Stale tasks in SDDC database are causing the issue. These tasks appear to be in relation to host/cluster commission and NSX edge cluster commission also. 

The script attached to this case will clear all tasks from the task_metadata table older than the specified number of days used in its execution. 

Use script cleartSDDCtasks.sh to clear out all old tasks from SDDC DB.

1. Download script attached to this KB 
2. WinSCP script to SDDC /home/vcf folder 
3. Make script executable
   • chmod +x cleartSDDCtasks.sh
4. Run the command below to retain tasks from the last 90 days (3 months) and delete all older tasks.
   •  ./cleartSDDCtasks.sh 90
5. Restart SDDC services to ensure changes take affect 
   •  /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh 

If the above options don't resolve the issue, open a case with Broadcom Support for further assistance with these steps.

Additional Information

This issue is resolved with a configurable retention policy (set to a 365 day default) in SDDC Manager 5.2.3.  Utilize the following steps to set the policy to something other than the 365 day default default, only if the environment is currently running SDDC Manager 5.2.3 or higher.

Task retention (default: 365 days)
SDDC Manager keeps tasks from the last 365 days by default. To use a different retention period, follow the steps below:

To change the task retention period

1. Log in to the SDDC Manager appliance (sddc-m) as root with the root account password.
2. Edit /opt/vmware/vcf/commonsvcs/conf/application-prod.properties and set the number of days, for example: tasks.old.number.days=<number_of_days>
   Replace <number_of_days> with the desired retention in days.
3. Apply the change by restarting the service:
   systemctl restart commonsvcs