How to interact with the scheduler database and CF commands when investigating an abnormal task.

Products

• VMware Tanzu Application Service

Issue/Introduction

The KBA How to track failed tasks provides a very comprehensive guide on how to track failed tasks using an API way.

This KBA aims to provide some additional information on how to interact with the scheduler database and CF commands when there is a need to investigate an abnormal task.

Resolution

1. Open at least two terminals and use one of the terminals to connect to the scheduler database using the following procedure.

1.1. Connecting to the ORG-system & SPACE-scheduler. 

$ cf t -o system -s scheduler

1.2. Confirm the mysql service instance name for the scheduler app. 

$ cf env scheduler
Getting env variables for app scheduler in org system / space scheduler as admin...
System-Provided:
VCAP_SERVICES: {
  "p.mysql": [
......
      "instance_name": "scheduler-mysql",
      "label": "p.mysql",
      "name": "scheduler-mysql",
......

1.3. Connect to the mysql service instance and use the service_instance_db database. 

$ cf service scheduler-mysql --guid
SOME_GUID

$ bosh -d service-instance_SOME_GUID ssh mysql/0

$ sudo -i 

mysql --defaults-file=/var/vcap/jobs/mysql/config/mylogin.cnf

mysql> USE service_instance_db;

 

2. Use another terminal to target the cloud foundry endpoint and execute some CF commands. 

2.1 The following cf job-schedules command outputs tell us there is a schedule for the job job-test and the schedule ID is 456-xyz.

$ cf job-schedules
                                                              
App Name: spring-music
job-test                 pwd
                         456-xyz   0,30 22-9 * * ?   Etc/UTC

2.2 The following API result tells us the GUID for the job-test is 123-abc. 

$ curl -k -X GET -H 'Accept: application/json' -H "authorization: $(cf oauth-token)" "https://scheduler.<DOMAIN>/jobs?space_guid=<SPACE_GUID>" | jq -r '.resources[] | .guid,.name +"\n"'

123-abc
job-test

2.3 A task name is formed then, which is JOB_GUID-|-SCHEDULE_GUID. It can be confirmed by the results of the command cf tasks APP_NAME and the execution record in the app logs.

$ cf tasks spring-music

id    name                state       start time                      command
...
1     123-abc-|-456-xyz   SUCCEEDED   Tue, 08 Jul 2025 03:30:00 UTC   pwd

$ cf logs spring-music --recent 
...
 [CELL/0] OUT Cell DIEDO_CELL_GUID successfully created container for instance APP_INSTANCE_ID
 [APP/TASK/123-abc-|-456-xyz/0] OUT Invoking pre-start scripts.
 [APP/TASK/123-abc-|-456-xyz/0] OUT Invoking start command.
 [APP/TASK/123-abc-|-456-xyz/0] OUT /home/vcap/app
 [APP/TASK/123-abc-|-456-xyz/0] OUT Exit status 0
 [CELL/0] OUT Cell DIEDO_CELL_GUID stopping instance APP_INSTANCE_ID
 [CELL/0] OUT Cell DIEDO_CELL_GUID destroying container for instance APP_INSTANCE_ID
 [CELL/0] OUT Cell DIEDO_CELL_GUID successfully destroyed container for instance APP_INSTANCE_ID
...

 

3. Go back to the scheduler DB terminal, we can validate the execution result from the histories table.

mysql> SELECT * FROM histories WHERE scheduled_time="2025-07-12" AND schedule_guid="456-xyz";
+---------+---------------------+----------------------+---------------------+-----------------------------------------------+-----------+---------------+--------------------------------------+---------------------+-------------+
| guid    | scheduled_time      | execution_start_time | execution_end_time  | message                                       | state     | schedule_guid | task_guid                            | created_at          | counted_job |
----------+---------------------+----------------------+---------------------+-----------------------------------------------+-----------+---------------+--------------------------------------+---------------------+-------------+
| 789-efg | 2025-07-12 00:00:00 | 2025-07-12 00:00:00  | 2025-07-12 00:00:01 | 202 ACCEPTED - Cloud Controller Accepted Task | SUCCEEDED | 456-xyz       | b59bc812-8bd0-45fe-830b-67fba8be58eb | 2025-07-12 00:00:00 |           0 |
+---------+---------------------+----------------------+---------------------+-----------------------------------------------+-----------+---------------+--------------------------------------+---------------------+-------------+
1 row in set (0.00 sec)

 

4. The guid in the above table represents the Execution GUID in the outputs of the command cf job-history job-test. 

$ cf job-history job-test
1 - 50 of 429 Total Results
Execution GUID   Execution State   Scheduled Time                  Execution Start Time            Execution End Time              Exit Message
...
789-efg          SUCCEEDED         Sat, 12 Jul 2025 00:00:00 UTC   Sat, 12 Jul 2025 00:00:00 UTC   Sat, 12 Jul 2025 00:00:01 UTC   202 ACCEPTED - Cloud Controller Accepted Task
...