SDDC Manager's task panel shows multiple tasks (e.g. SDDC-Manager Backup task) as pending "Fetching task info..."

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

On the SDDC Manager UI, the tasks panel shows many tasks (e.g. SDDC-Manager Backup tasks or/and adding ESXi host in cluster, etc ) as "Fetching task info...".

A similar error is observed in the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log file in SDDC appliance:

mm dd Hr:min:sec ERROR [common,#################,####] [c.v.e.s.t.services.TaskCollector,http-nio-127.0.0.1-7100-exec-212] collect() failed
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{"errorCode":"GET_BACKUP_FAILED_INVALID_ID","arguments":[],"message":"Failed to retrieve backup workflow, invalid id","referenceToken":"#####"}"

 

Environment

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Cause

The cause of this issue could be the file /opt/vmware/vcf/sddc-support/status.json is corrupted, deleted, or modified.

Resolution

Note: Before proceeding, ensure to take a backup/ snapshot of the SDDC Manager appliance.

Follow the below mentioned steps to verify that the pending tasks are of SDDC Manager Backup:

1. Click on any pending "Fetching task info..." task.
2. Find the task ID in the address bar. For example, the address bar looks like this "https://{sddc-manager-fqdn}/ui/sddc-manager/dashboard(monitoring-panel:monitoring/tasks/subtasks/{subtask-id})". 
3. SSH into SDDC-Manager and change to root.
4. Connect to platform DB and verify that the pending task is of SDDC Manager Backup operation.

1. Run "psql -U postgres -h localhost"
2. Run "\c platform"
3. Run "select task_url, task_type from task_metadata where id='<above-task-id>';"
4. Verify that the task_type is 'SDDCMANAGER_BACKUP'.
5. Get the task_url from step c. For example, it looks like "http://localhost/operationsmanager/backups/tasks/#######-####-####-####-########"
6. Verify that the backup task is not found in the backup inventory. i.e., the backup status API should fail with the error code "GET_BACKUP_FAILED_INVALID_ID". 

Example: Run "curl http://localhost/operationsmanager/backups/tasks/#######-####-####-####-########". It should fail with the error code "GET_BACKUP_FAILED_INVALID_ID".

 

To workaround the issue, delete all SDDC Manager Backup tasks in the task_metadata, so that the Tasks panel does not pull these tasks details.

Note: As the SDDCMANAGER_BACKUP tasks will be deleted from the task registry, there will be no SDDCMANAGER_BACKUP tasks that are executed till now in the task panel.

1. SSH into SDDC-Manager as a root user.
2. Connect to the platform database and delete the SDDC Manager Backup tasks from the task_metadata table:

1. Run "psql -U postgres -h localhost"
2. Run "\c platform"
3. Run "delete from task_metadata where task_type='SDDCMANAGER_BACKUP';"
4. Run "\q"

3. Restart commonsvcs application. Run 

         systemctl restart commonsvcs