SCDF dataflow app crashes due to high disk utilization

Products

• Support Only for Spring

Issue/Introduction

SCDF Dataflow or Skipper app crashes due to failed health check, here is sample error message: 

   2022-01-06T21:59:56.70+0530 [APP/PROC/WEB/0] OUT 2022-01-06 16:29:56.700  WARN 16 --- [nio-8080-exec-5] o.s.b.a.system.DiskSpaceHealthIndicator  : Free disk space below threshold. Available: 24576 bytes (threshold: 10485760B)
   2022-01-06T21:59:56.83+0530 [HEALTH/0] ERR Failed to make HTTP request to '/management/health' on port 8080: received status code 503 in 11ms

 To confirm disk utilization, please execute the following steps:

1. Target the backend app org/space (Admin privilege is required)
     $ cf target -o p-dataflow -s <SERVICE-GUID>

2. ssh to the app container
     $ cf ssh dataflow

3. Inspect disk space 
     $ execute "df -h"
     $ du -h ~/.m2/repository

Environment

Spring Cloud Dataflow for Cloud Foundry: 1.x

Resolution

Option 1 - Increase disk size

The default disk allocation for the dataflow app and skipper app is 2GB. Depending on the number of stream and task apps your deploying/launching, evaluate your disk requirements. Consider increase the disk allocation.

However, note the default is "auto-delete-maven-artifacts": true which means that the jar artifacts are deleted right after a task is launch or a stream deployed. Verify if this configuration is in place by checking the dataflow/skipper environment variables.

To scale the disk execute

   cf update-service <MY-DATAFLOW-SERVICE> -c '{"dataflow": { "disk": "4GiB" } }'

Reference:
Configuring App Setting for Data Flow Server and Skipper


Option 2 - Create the app metadata.jar and configure during app registration

The application metadata can be included inside the uber jar. However, depending on the app size, this can be small or huge. Downloading the entire app jar during app registration could potentially cause:
1) Noticeable delay especially for huge jar
2) Disk space issue

Creating a Dedicated Application Metadata jar is strongly recommended. This being a few kilobytes in size.