Troubleshooting Container Start Failures (No Space Left on Device) in the Tanzu Platform for Cloud Foundry

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service
• VMware Tanzu Platform
• VMware Tanzu Platform Spring
• VMware Tanzu Platform Spring Essentials

Issue/Introduction

Resolving "exit status 2" and "No space left on device" errors during Application Startup or Task Execution in TAS/EAR 

Symptoms include 

• Logs stating: Copying into the container failed: stream-in: nstar: error streaming in: exit status 2.

• Detailed output: tar: ./app/BOOT-INF/lib/...: Cannot open: No space left on device.

• BOSH vitals for the Diego Cell show healthy disk usage, yet the specific App/Task fails to start.

 

 

Environment

Tanzu Application Service
Tanzu Platform for Cloud Foundry
Elastic Application Runtime

Cause

The issue is caused by an insufficient Container Disk Quota.

When an application instance or Task is triggered, Cloud Foundry creates an isolated container and extracts the "Droplet" (which contains the Java Runtime, application JARs, and dependencies) into the container's root filesystem.

 

• Quota Exhaustion: If the combined size of the JRE and the unzipped application exceeds the assigned disk quota (e.g., 128MB), the tar extraction process will fail mid-way.

• JRE Footprint: Java applications typically require a minimum of 300MB–450MB just to accommodate the OpenJDK runtime and the Spring Boot library structure.

• App vs. Cell Limits: While the underlying Diego Cell (VM) may have ample space, the individual container is strictly limited by the quota defined in the application's manifest.

Resolution

To resolve this issue, the application's disk quota must be increased to provide enough "headroom" for the runtime and extracted artifacts.

Scale the application's disk quota using the Cloud Foundry CLI. For Java/Spring Batch applications, a minimum of 512MB is recommended.

You can scale using cf cli

cf scale <APP_NAME> -k 512M

Update the manifest.yml file in your project repository to ensure future deployments use the correct quota.

---
- name: life-service-batch
  memory: 1G
  disk_quota: 512M  # Increase this value

 

Additional Information

After scaling, verify the status of the application:

1. Run cf app <APP_NAME> and check the disk usage line.

2. Monitor logs using cf logs <APP_NAME> --recent to ensure the stream-in process completes without tar errors.

3. If the error persists, ensure the application is not writing large temporary files to the local /tmp directory, as these count against the same disk quota.