Droplet creation requires large staging container disk

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This KB article will detail a workaround for applications that require large amounts of disk space to stage its droplet.

There is documentation that covers how to deploy a large app to Tanzu Application Service (TAS), however we will be covering an aspect of the droplet staging process specifically. 

When an application is pushed to TAS, several actions take place within the platform. More information about these actions are detailed in this documentation. We will focus only on the concept that two containers are created during a cf push, one container for staging the droplet and one container for running the droplet.

When a staging container is requested, it will contain a disk size specification. Currently this size specification value is hard coded to 6GB in a TAS environment:

$ bosh -d cf-a9eaa91d4bbbca8a09a4 manifest | grep staging_disk_limit_mb
        staging_disk_limit_mb: 6144

This staging_disk_limit_mb configuration is the minimum limit staging disk size.

Some applications require dependency installations during its droplet staging process. Node apps for example will leverage yarn or npm to install dependencies locally within the staging container to compile the app droplet. Sometimes these dependency installations require a large amount of disk space which in combination with all other actions during the staging process may exhaust that 6GB worth of disk space and fail the droplet creation. An example error message may look like the following:

2023-01-19T13:46:57.35-0600 [STG/0] ERR Failed to compress build artifacts: exit status 2

Most times such an error can be solved by implementing offline mode. In the event that all other measures have been attempted but the disk size still gets exceeded then there is another workaround available.

Environment

Product Version: 2.12

Resolution

For apps that require large staging container disks, there are exposed CAPI endpoints that will allow us to specify the staging container disk size for its build. We will walk through a demo together with pushing a new application and specifying its staging container disk to be 8GB. Please note that this method should be applicable to existing applications as well but may need to be crafted to align with your pipeline deployment needs.

STEP 1
Push the application without starting it:

cf push --no-start

STEP 2
Obtain the app guid:

cf app simpleapp100 --guid

36e436eb-91ac-4d04-8d94-89ce269e3a17

STEP 3
Leverage CAPI to obtain the package guid for your latest push:

cf curl '/v3/apps/36e436eb-91ac-4d04-8d94-89ce269e3a17/packages?order_by=-created_at' | jq -r '.resources[0].guid'

c67fc480-b70e-486a-ac3a-dd5c222385eb

STEP 4
Leverage CAPI to create a build of the package while specifying the staging container disk size (criteria must be met for this step, please see below section "Criteria"):

cf curl /v3/builds -X POST -H "Content-type: application/json" \
-d '{
    "staging_disk_in_mb": 8192,
    "package": {
       "guid": "c67fc480-b70e-486a-ac3a-dd5c222385eb"
    }
}'

This will initiate a build with the specified staging container disk size. When reviewing the garden logs on the staging container's diego_cell we can see a log similar to the following that reflects the staging disk size was accepted: 

garden/garden.stdout.log:{"timestamp":"2023-02-16T21:35:47.425677455Z","level":"info","source":"guardian","message":"guardian.create.volume-creator.image-plugin-create.grootfs.create.groot-creating.making-image.overlayxfs-creating-image.applying-quotas.run-tardis.tardis.ending","data":{"args":["limit","--disk-limit-bytes","8589934592" <rest of log omitted for brevity>

"--disk-limit-bytes","8589934592"

Criteria
The following criteria must be met before step 4 is successful:

• The requested staging disk limit must be greater than the minimum staging disk limit. Earlier we mentioned "This staging_disk_limit_mb is the minimum limit staging disk size." The staging_disk_limit_mb value is hardcoded to 6GB and currently not configurable. There is logic within the staging container disk calculator that will compare the requested staging container disk size to the minimum limit staging disk size and if the requested size below the minimum size then the minimum size will be used. Therefore it is not possible to request a staging container disk size thats less than 6GB. 
• The requested staging_disk_in_mb for the build must be less than the platform configured maximum_app_disk_in_mb. For example, to request an 8GB staging container disk then the max allowed disk size must be greater than 8GB - in this example we have it set to 10GB:

  bosh -d cf-a9eaa91d4bbbca8a09a4 manifest | grep maximum_app_disk_in_mb
          maximum_app_disk_in_mb: 10240

  The maximum_app_disk_in_mb is exposed in the TAS tile:

STEP 5
Obtain the build's droplet link:

 cf curl '/v3/apps/36e436eb-91ac-4d04-8d94-89ce269e3a17/builds?order_by=-created_at' | jq '.resources[0].links.droplet'

{
  "href": "https://api.run-03.slot-34.tanzu-gss-labs.vmware.com/v3/droplets/e023dfc6-1d6f-4282-bc57-fb3ceab5f903"
}

STEP 6
Craft the link to download the droplet:

cf curl '/v3/droplets/e023dfc6-1d6f-4282-bc57-fb3ceab5f903/download' --output ~/simpleapp100/droplet.tgz

OK

STEP 7
Push the droplet:

cf push --droplet ~/simpleapp100/droplet.tgz