cf push staging droplet fails with out of memory error on Ubuntu Jammy Stemcell 1.351

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

After upgrading Bosh Ubuntu Jammy Stemcell to 1.351 or later cf push fails with out of memory error during staging. Not all apps will observe this behavior. 

Go App

[STG/0] ERR Failed to compile droplet: Failed to run finalize script: exit status 12
[STG/0] OUT Exit status 223 (out of memory)
[STG/0] OUT Cell b9375e22-9a0d-4739-bab3-87f921a13740 stopping instance 6efc5f6b-4e99-4102-b8c7-d673b4b4083d
[STG/0] OUT Cell b9375e22-9a0d-4739-bab3-87f921a13740 destroying container for instance 6efc5f6b-4e99-4102-b8c7-d673b4b4083d
[API/1] ERR Failed to stage build: staging failed

Java App

[STG/0] [OUT] Exit status 137 (out of memory)
[STG/0] [OUT] Cell 8d715f56-0299-48b9-9895-af9109ec9dda stopping instance dd6d6b98-8bae-4bbc-b36a-73a0f253f766
[STG/0] [OUT] Cell 8d715f56-0299-48b9-9895-af9109ec9dda destroying container for instance dd6d6b98-8bae-4bbc-b36a-73a0f253f766
[API/1] [ERR] Failed to stage build: staging failed

Resolution

Upgrading to Jammy Stemcell 1.404+ resolves this issue.

To workaround the out of memory errors during staging you can increase the memory limit for the given app during cf push.  

cf push -m 2G

You can then chose to scale down the memory for the given instance after cf push is successful and the staging process created the new droplet.  scaling the app will restart the container without requiring a restage. 

cf scale myapp -m 1G