Foundation is failing at errands phase for system applications. push-usage-service error "Package failed to process correctly after upload"

Products

• VMware Tanzu Application Service

Issue/Introduction

You are running an Apply Changes on a TAS foundation and its failing at errands phase, specifically for the push-usage-service application. Apply Changes fails with below error:

Errand 'push-usage-service' completed with error (exit code 1)
Exit code 1
Instance clock_global
--------
Pushing app app-usage-scheduler to org system / space system as push_usage_service...
Waiting for API to complete processing files...
Package failed to process correctly after upload. <------------------------------*****************
FAILED

 

Cause

1.) When Cloud Controller uploads the app package to the blobstore but then can't retrieve it, you will see blobstore error in the:

Error: Package failed to process correctly after upload. 

---------------------------

2.) To confirm, check the the VM/instance that threw the error in Apply Change failure, clock_global. This VM (clock_global) schedules and executes the push-usage-service. As shown below, this VM is throwing (Stager) blobstore error.  

# From <clock_global-VM-NAME> /var/vcap/sys/log/push-usage-service/push-usage-service.stdout.log 
------
Staging app and tracing logs...
Unexpected Response
Response Code: 500
Request ID:   <123-EXAMPLE-ABC>
Code: 0, Title: , Detail: {"errors":[{"detail":"Stager error: Failed to get blobstore download url for package <xyz-EXAMPLE-321>","title":"CF-StagerError","code":170011}]}
FAILED

Take notice to this part of the "Stager error: "Failed to get blobstore url".  This suggests that there is some issue with the 'url' (aka 'endpoint') that the TAS components are reaching out to.

Resolution

1.) Validate blobstore config. 

• Confirm what is the 'endpoint:' under the 'packages:' section in blobstore config. See example below:

   packages:
          max_package_size: xx
          max_valid_packages_stored: xx
          app_package_directory_key: xx
          blobstore_type: xx
          fog_connection:
            provider: xx
            aws_access_key_id: xx
            aws_secret_access_key: "((/xx))"
            aws_signature_version: 'xx'
            region: 
            path_style: true
            endpoint: www.<EXAMPLE>.com <--------------------------*****
            enable_signature_v4_streaming: true

* The 'endpoint:' is where TAS components are attempting to reach the blobstore.

 

2.) Investigate and fix any issue with the blobstore's 'endpoint':

• External / self-hosted s-3 (compatible) storage:
  • First check that your TAS VM's can reach (ping) the endpoint defined in the blobstore config.
  • Contact your S-3 provider/vendor to troubleshoot further. 
• For TANZU-provided blobstore:
  • Check if VM's can reach the endpoint/blobstore url. (ping)
  • run command bosh vms --vitals to check (and fix) any unhealthy vm's that could be influencing this behavior (ie, high cpu in nfs, cloud_controller, etc)
  • Engage your platform team and/or create a ticket with TANZU Support.

Additional Information

• For more information, see Broadcom documentation: Configuring file storage for TAS for VMs.