Errands in upgrade to PCF or PAS 2.3 are failing as deploying on a stack is not supported by the Buildpack

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
When trying to deploy PAS 2.3.x as part of your PCF 2.3.x upgrade, you notice errands are failing.

Observed the push-apps-manager errand. This could occur in any errand deploying an application. You can see entries in the errand log similar to the following:

2018-10-30T18:48:10.15+0000 [STG/0] OUT Downloading app package...
2018-10-30T18:48:10.23+0000 [STG/0] OUT Downloaded app package (9.3K)
2018-10-30T18:48:10.45+0000 [STG/0] ERR It looks like you're deploying
on a stack (currently set to *cflinuxfs3*) that's not supported by this
buildpack.
2018-10-30T18:48:10.45+0000 [STG/0] ERR That could be because you're
using a recent buildpack release on a deprecated stack.
2018-10-30T18:48:10.45+0000 [STG/0] ERR If you're using the buildpack
installed by your CF admin, please let your admin know you saw this
error message.
2018-10-30T18:48:10.45+0000 [STG/0] ERR If you at one point specified a
buildpack that's at git URL, please make sure you're pointed at a version
that supports this stack.
2018-10-30T18:48:10.45+0000 [STG/0] ERR Failed to compile droplet: Failed
to compile droplet: exit status 44
2018-10-30T18:48:10.48+0000 [STG/0] OUT Exit status 223

When you run the cf buildpacks command using the CF CLI v6.39.0+, you will see a stacks column in the output:

Notice how some of the buildpacks have no association to a stack in the stacks column.

Environment

Cause

Per the PAS 2.3 release notes, a new stack, cflinuxfs3 has been added to PCF. In addition to this, buildpacks must now associate with a particular stack.

At the time of writing, this will be:

cflinuxfs3(based on Ubuntu 18.04 Bionic)
cflinuxfs2 (based on Ubuntu 14.04 Trusty)
windows2016 (based on Windows 2016)
windows2012R2 (based on Windows 2012R2)

In some cases, a buildpack may exist on your PCF foundation but may not be associated with a stack.

The most common cause are custom buildpacks that were added prior to upgrading to PCF 2.3. In this case, the Cloud Controller is unaware the stack to which the buildpack is associated.

Regardless of this, the platform will usually pick the correct buildpack to use, However if there is no stack associated with a buildpack and your application is relying on the buildpack's auto detection feature, there are some edge cases where the wrong buildpack may be picked. In the example above, you can see that a cflinuxfs2 buildpack was picked but a cflinuxfs3 root filesystem was requested by the application.

Resolution

To resolve this issue, you want all existing buildpacks in your cf buildpack listing to have an associated stack in the stack column. This will remove any ambiguity for the platform and allow it to pick the correct buildpack.

You can achieve this three different ways:
1. CF CLI tool version v6.41.0 contains an update for the option to update-buildpack which allows an Operator to assign an unassociated buildpack to a stack, for example:

cf update-buildpack <buildpack-name> --assign-stack cflinuxfs3

2. Associate all of your buildpacks with a stack by using use cf curl on the Assign Buildpack to Stack CAPI endpoint.

Example:

Obtain the guid of your buildpack: cf curl /v2/buildpacks. Look through the JSON output for the name of your buildpack and look for the metadata associated with that buildpack. There will be a guid attribute for that buildpack.
- If you have the excellent jq tool [1] installed, you can run the following command to automatically find your buildpack guid, just insert the name of your buildpack into that command.: cf curl /v2/buildpacks | jq -r '.resources[] | select(.entity.name == "<buildpack name goes here>") | .metadata.guid'.
With the buildpack guid, you can now update the stack for that buildpack. Run: cf curl -X PUT /v2/buildpacks/<buildpack guid goes here> -d '{"stack": "cflinuxfs3"}'

3. Remove any unassociated buildpacks from the foundation. You can run cf delete-buildpack to remove buildpacks. It is not possible at this time to manually set the buildpack stack when running cf create-buildpack. If you delete the buildpack and need to add it back, you'll still need to follow the other instructions to update the stack of the buildpack.