Understanding scheduled task behaviors when the Cloud_controllers are down

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Users might be interested to understand scheduled task behaviors when the Cloud_controllers are down.

Resolution

We have two scenarios as follows:

1. A task is scheduled to start at 2:00 PM, however, all cloud_controllers are not available (e.g., in down status) at 2:00 PM:
   • Result: The scheduled task will never start or be rescheduled even after the cloud_controllers are back online.
2. A task is scheduled to start at 2:00 PM and is expected to complete at 2:30 PM, however, all cloud_controllers become unavailable (e.g., in down status) during 2:10 PM~3:00 PM: 
   • Result: The scheduled task will start and complete normally. The container created for the task will be destroyed once the task completes and you will observe the following messages in the app logs 

     • [APP/TASK/SOME-GUID-1-|-SOME-GUID-2/0] OUT Exit status 0
       [CELL/0] OUT Cell SOME-SHA stopping instance SOME-INSTANCE-GUID
       [CELL/0] OUT Cell SOME-SHA destroying container for instance SOME-INSTANCE-GUID
       [CELL/0] OUT Cell SOME-SHA successfully destroyed container for instance SOME-INSTANCE-GUID

   • The completed task will show as "Failed" with reason "Unable to determine completion status", but it was actually completed normally.  

     • $ cf curl "/v3/tasks?states=FAILED&per_page=100&order_by=created_at" | jq -r '.resources[] | select(.name | startswith("SOME-GUID-1")) | .guid, .created_at, .result.failure_reason + "\n"'
       SOME-INSTANCE-GUID
       
       Unable to determine completion status

Please contact Tanzu support team is you need any further information or have any additional questions.