BOSH errands that deploy applications fail when run manually in PAS

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
When running a BOSH errand, such as the deploy-autoscaler or nfsbrokerpush, it fails and you see one of the following error messages: 

• The app is invalid:VCAP::CloudController::BuildCreate::StagingInProgress
• The app package is invalid: bits have not been uploaded

Environment

Cause

The BOSH errand is being run without the '--instance' flag and there are multiple instances of the clock_global job running.

Note: The clock_global job is where BOSH executes colocated errands.

This in turn causes the deploy errand to run on all the clock global instances in parallel, when it should be only running on one Virtual Machine (VM).

This behavior can be observed if you look at the output of bosh run-errand. You can see multiple "Running errand" lines. This is because BOSH is running the errand on all instances of the clock_global job simultaneously.

bosh -d <deployment> run-errand deploy-autoscaler

Task 10287909 | 00:43:26 | Preparing deployment: Preparing deployment (00:00:06)
Task 10287909 | 00:43:32 | Running errand: clock_global/83a96761-65fa-4445-8617-dc7cdcff3f0a (0)
Task 10287909 | 00:43:32 | Running errand: clock_global/09d69bb2-f63b-42ad-a6c3-e8004c6dfcfb (1)
Task 10287909 | 00:43:32 | Running errand: clock_global/0e365929-53cf-44c7-8887-9067fb7b9491 (2)

Resolution

To resolve this issue, use one of the following workarounds: 

• When running your errand, make sure to include the `--instance` flag and specifically tell BOSH to execute the errand on one instance.

bosh -d <deployment> run-errand deploy-autoscaler  --instance clock_global/abcxyz

• Run the following command, where "abcxyz" is the instance guid where you want the errand to run. You can locate your desired instance guid by running 'bosh -d <deployment> instances`.

bosh  -d <deployment> run-errand test-autoscaling  --instance clock_global/abcxyz