Tasks not visible in the AppsManager and cf cli in Tanzu Application Service

Products

• VMware Tanzu Application Service

Issue/Introduction

Unable to view the Job/Task history for specific applications in the Apps Manager or via the Cloud Foundry CLI.

• Apps Manager: When navigating to the Task tab of an application, the page fails to load history or displays a generic error.
• CF CLI: Running cf tasks <APP_NAME> fails with the following error:

Unexpected Response
Response Code: 500
Code: 10001, Title: UnknownError, Detail: An unknown error occurred.

Cause

This issue is caused by data inconsistency in the Cloud Controller Database (ccdb), specifically regarding Orphaned Task Records.

When the Cloud Controller API attempts to list tasks, it iterates through history records. For each task, it checks if the task was based on a Docker image by inspecting the associated Droplet (compiled code artifact).

The crash occurs because:

1. A Task record exists in the tasks table.
2. The Task record points to a droplet_guid.
3. That specific Droplet no longer exists in the droplets table (likely cleaned up by a retention policy or migration script).
4. The code attempts to load the Droplet, receives nil, and crashes with the following stack trace in cloud_controller_ng.log:

"detail"=>"docker? delegated to droplet, but droplet is nil"

Resolution

To resolve this issue, the Platform Administrator must identify and remove the orphaned task records from the Cloud Controller Database.

Note: This procedure involves direct database manipulation. Ensure a backup of the ccdb is taken before proceeding.

Step 1: Verify the Error in Logs

Search the Cloud Controller logs (/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log) for the specific crash signature:

grep "docker? delegated to droplet, but droplet is nil" cloud_controller_ng.log

Step 2: Identify Orphaned Records (SQL)

Log in to the Cloud Controller Database (ccdb) and run the following query to identify tasks pointing to non-existent droplets:

SELECT t.guid, t.name as task_name, t.app_guid, t.droplet_guid 
FROM tasks t
LEFT JOIN droplets d ON t.droplet_guid = d.guid
WHERE d.guid IS NULL;

Note: If this returns 0 rows, the issue is not caused by orphaned droplets. If it returns rows, proceed to Step 3.

Step 3: Identify Affected Applications (Optional)

To verify which applications own these corrupted records, use the app_guid from Step 2 with the CF CLI:

cf curl "/v3/apps/<APP_GUID>?include=space.organization"

Step 4: Deleting the entry from the CCDB

Delete the corrupted history records. Replace <GUID_FROM_STEP_2> with the specific GUIDs found.

DELETE FROM tasks 
WHERE guid IN ('<GUID_1>', '<GUID_2>');

You can now verify by checking the tasks from cf cli or from the Apps Manager