Jobs do not happen as scheduled with error "422 UNPROCESSABLE_ENTITY - [memory_in_mb exceeds organization memory quota]"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Jobs are not executed as scheduled.

Environment

Product Version: 2.12

Resolution

1- Install Scheduler CLI plugin by referencing Prerequisites

2- Check the outputs for below commands.

• cf jobs
• cf job-schedules
• cf job-history JOB_NAME
• cf tasks APP_NAME

$ cf jobs
Listing jobs for org pikachu / space pikachu as admin
Job Name     App Name       Command
pwd7         spring-music   pwd

$ cf job-schedules
Getting scheduled jobs for org pikachu / space pikachu as admin
App Name: spring-music
pwd7                     pwd
                         466e15b8-7bbb-473a-8959-0316d5757f75   16/18 * * * ?
                         b0d81823-a5cf-4edc-893f-6737d72d2cd4   40 * * * ?

$ cf job-history pwd7
Getting scheduled job history for pwd7 in org pikachu / space pikachu as admin
1 - 50 of 708 Total Results
Execution GUID                         Execution State   Scheduled Time                  Execution Start Time            Execution End Time              Exit Message
1cc7581c-ceef-4718-9294-0119f62d5892   SUCCEEDED         Fri, 07 Oct 2022 10:52:00 UTC   Fri, 07 Oct 2022 10:52:00 UTC   Fri, 07 Oct 2022 10:52:01 UTC   202 ACCEPTED - Cloud Controller Accepted Task
4fac0a8a-522b-4559-a830-94aa1e1aee73   FAILED            Fri, 07 Oct 2022 10:40:00 UTC   Fri, 07 Oct 2022 10:40:00 UTC   Fri, 07 Oct 2022 10:40:02 UTC   422 UNPROCESSABLE_ENTITY - [memory_in_mb exceeds organization memory quota]


$ cf tasks spring-music
Getting tasks for app spring-music in org pikachu / space pikachu as admin...
id     name                                                                          state       start time                      command
7306   12debc94-e85e-4850-a5cb-6bd2bee4bcc1-|-b0d81823-a5cf-4edc-893f-6737d72d2cd4   FAILED      Fri, 07 Oct 2022 10:40:01 UTC   pwd

3- Error message [memory_in_mb exceeds organization memory quota] basically means that the ORG runs out of memory quota when the containers are being created for the scheduled jobs and customer can solve this issue with either increasing the org quota or re-creating the jobs with less memory allocated. 

    3-a Increase the ORG quota:

# Check which quota is being used for the ORG
$ cf org pikachu
Getting info for org pikachu as admin...
 
name:                 pikachu
domains:              apps.internal, cfapps.example.com
quota:                default   
spaces:               pikachu
isolation segments:
 
# Check the allocated resource info for the quota. 
$ cf org-quota default
Getting org quota default as admin...
total memory:         10G
instance memory:      unlimited
routes:               1000
service instances:    100
paid service plans:   allowed
app instances:        unlimited
route ports:          0
 
# Increase the ORG quota
$ cf update-org-quota default -m 80G
Updating org quota default as admin...
OK
 
$ cf org-quotas
Getting org quotas as admin...
name      total memory   instance memory   routes   service instances   paid service plans   app instances   route ports
default   80G      unlimited   1000     100        allowed  unlimited       0
runaway   100G     unlimited   1000     unlimited  allowed  unlimited       0

   
3-b Re-create the job with less memory allocated. 

$ cf create-job spring-music pwd20 pwd --disk 256M --memory 256M

* If the disk/memory size is not set when creating a job, then the container will be created based on the Default app memory and Default disk quota per app fields in the App Developer Controls pane of the VMware Tanzu Application Service for VMs (TAS for VMs) tile. Check Manage Tasks .