Pushing an app fails with "Failed to perform blobstore operation after three retries." and Cloud Controller logs show "Insufficient Storage"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When either pushing an app or running an errand such as smoke test, it fails with  the error "Failed to perform blobstore operation after three retries."

Job (c85d1ec7-50de-4ea2-ab9e-fe08efdf9dd8) failed: Failed to perform blobstore operation after three retries.

In addition, cloud_controller_ng.log in Cloud Controller VM shows errors that contain "507 Insufficient Storage" and "CF-BlobstoreError". 

"timestamp":1577169316.3136656,"message":"Error with blobstore: Excon::Error::HTTPStatus - Expected(200) <=> Actual(507 Unknown)
excon.error.response
  :body          => \"<Error><Code>Insufficient Storage</Code><Message>The server cannot process the request because there is not enough space on disk</Message><Resource>gtdc-l-f1-packages/3d/76/3d76ef64-fe6f-4166-82ab-f72d5d36e2d0</Resource><RequestId>0a640003:16e7fd4e16f:12864:4e3</RequestId></Error>\"

This error indicates that the Cloud Controller's filesystem configured in Ops Manager > TAS for VMs Tile > File Storage has run out of storage space.

Environment

Product Version: 2.0

Resolution

In order to fix this issue, either increase the storage size or clean it up to make more available space.