Deploying app with large artifact failed with "500 Internal Server Error"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When pushing an application with large artifact to TAS using internal WebDAV as file storage, it failed with "500 Internal Server Error" as shown in image below.

"No space left on device" error was produced to internal_error.log file on nfs_server instance for internal WebDAV file storage. For example,

2022/09/16 04:23:22 [crit] 14#0: *2159812 pwrite() "/var/vcap/data/blobstore/tmp/client_body_temp/0000367893" failed (28: No space left on device), client: 0.0.0.0, server: blobstore.service.cf.internal, request: "PUT /admin/c
c-droplets/f1/85/f18558fa-66fc-4b7a-8bdb-b564954fb36f/bb4110f221b472b9624f43e7bfd2d9a7b30c2941 HTTP/1.1", host: "blobstore.service.cf.internal:4443"

The above error message was complaining that there was no enough space on ephemeral disk (/var/vcap/data). However when checking disk space usage on the node, there was still some free space. As shown in below image, the ephemeral disk was only 55% used when "No space left on device" error" was reported. 

 

Environment

Product Version: 2.11

Resolution

By default internal file storage node (nfs_server) will be configured with a 8GB ephemeral disk. Because BOSH reserves 4GB for internal use, the capacity of file system /var/vcap/data is left 4GB.

nfs_server/22cac737-fdfa-4b6b-88ef-57456c02499e:~# fdisk -l /dev/sdb
Disk /dev/sdb: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 06FAFC28-D946-4EAB-9369-DB3C610332AF

Device       Start      End Sectors  Size Type
/dev/sdb1     2048  8079359 8077312  3.9G Linux filesystem
/dev/sdb2  8079360 16775167 8695808  4.2G Linux filesystem

/var/vcap/data volume on internal file storage node is used as temporary storage for uploading application bits and droplet. In most cases 4GB is enough. But some application might have a very large package which results in a large droplet to be built out. In such cases, uploading the application bits or droplet could fail due to insufficient space on /var/vcap/data. When such thing happens, the solution is to change internal file storage instance (nfs_server) to another VM type with large ephemeral disk while at least keeping persistent disk size same as before (or even larger).