App Metrics 1.6.x pre 1.6.4 are not compatible with VMware Tanzu Application Service for VMs 2.5.20+, 2.6.15+, 2.7.9+ and 2.8.3+

Products

• Operations Manager

Issue/Introduction

App Metrics v1.6.x apps are not compatible with Node.js Buildpack version 1.7.9+. If you have manually upgraded your Node.js buildpack to version 1.7.9+, you will see App Metrics errands fail the next time they are run.

In addition, due to version 1.7.9 of the nodejs-offline-buildpack being shipped in VMware Tanzu Application Service for VMs (TAS for VMs) versions 2.5.20, 2.6.15, 2.7.9, 2.8.3 and later, you will see the same problem with App Metrics after upgrading to one of the versions of PAS.

App Metrics is currently only compatible with version 1.7.8 or lower of the nodejs-offline-buildpack. This note is included with the Known Issues for App Metrics.

The issue is resolved in Metrics 1.6.4 and 2.0

Resolution

There are a few options for working around this issue. The issue is resolved in Metrics 1.6.4 and 2.0:

Option #1 - Temporarily Rollback to an Older Node.js Buildpack

For this workaround, you will upload version 1.7.8 of the Node.js buildpack updating the nodejs_buildpack and at the same time locking the buildpack. This will make it compatible with App Metrics and lock the buildpack so that it's not updated during the Apply Changes. The applications deployed by App Metrics rely on the system Node.js buildpack so with this change they will automatically use the 1.7.8 Node.js buildpack which allows them to run correctly.

At this point, you can Apply Changes and the App Metrics errands should all be able to run successfully. After they finish running and App Metrics is working, you can then go back, and update the buildpack to have the most recent Node.js buildpack version and unlock it. Finally, go into the App Metrics tile and disable the errands (or use Operations Manager's selective deploy feature to make sure that App Metrics is not upgraded).

The are two impacts from this change. First, you cannot upgrade App Metrics and you cannot run the errands. If the errands run, the apps will break again and you will need to reapply the fix in this KB. Second, during the window where the buildpack is locked to the older version, it is possible for applications deployed to TAS for VMs during this window to pick up and use the older buildpack. The older buildpack will contain older versions of Node.js which may contain known bugs or security vulnerabilities. If an application were to be pushed during this period, the application developer would need to cf restage it after the buildpack is returned to the most recent version and unlocked. This re-stage will run the new buildpack and cause it to update to the latest Node.js version.

To implement, follow the below steps:

1. Download the Node.js 1.7.8 buildpack from Pivotal Networks.
2. Run the command cf buildpacks. Make a note of the current version of the Node.js buildpack (this may not be the most recent version of the Node.js depending on the version of TAS for VMs you have deployed).

• Download the version of the Node.js buildpack that you found in step #2 from Pivotal Networks.
• If you have already tried to Apply Changes and it failed with an error during the "push apps" errand for App Metrics, then proceed. If you are planning ahead and have not yet upgraded to an affected version, please perform a selective Apply Changes with App Metrics unselected. This will upgrade your foundation to the latest TAS for VMs version. When this succeeds you will have all tiles upgraded except App Metrics. Now proceed with the rest of the instructions.

Option #2 - Do Not Upgrade or Fix App Metrics

If you have not yet upgraded TAS for VMs, you may do so but you need to either use Ops Manager's selective deploy feature and unselect App Metrics or you need to disable the App Metrics errands. The errands will fail because of the buildpack change. If you prevent the errands from running, the existing App Metrics applications should continue to work with your newer version of TAS for VMs.

If you have already started the upgrade and hit a failure, then App Metric will be in a failing state and you have the option to simply not use App Metrics until it is fixed. While App Metrics can be a helpful tool for developers, it is not a critical part of the platform and will not cause outages to the foundation or customer applications running on the platform.


Option #3 - Hold Back PAS Upgrades

If you have not upgraded to TAS for VMs 2.5.20+, 2.6.15+, 2.7.9+ or 2.8.3+, then simply hold off updating. When a compatible version of App Metrics v1.6 or v2.0 is released, you will be able to upgrade it and then continue with your TAS for VMs upgrades. At this time, Support cannot provide a date for either a App Metrics v1.6 or v2.0 release with a fix.

Please be aware that by taking this option, you will be holding back bug and security fixes for the platform. Before taking this option, Support recommends reviewing the TAS for VMs release notes to see what fixes will be delayed by taking this approach as well as talking with your Platform Architect about the implications of holding back updates.

Recommendations

1. Run the command:

cf update-buildpack nodejs_buildpack -p path/to/nodejs_buildpack-cached-cflinuxfs3-v1.7.8.zip

This will downgrade the version of the buildpack that is installed.

2. Run the command cf update-buildpack nodejs_buildpack --lock. This will lock the buildpack so that it is not updated by the TAS for VMs tile.

3. Apply Changes. If you are in the middle of an upgrade that failed, simply continue the upgrade making sure that the App Metrics errands run. If you planned ahead and previously did a selective deploy with App Metrics unselected, now you can perform a selective deploy with everything unselected except App Metrics. In both cases, this should update the App Metrics deployed applications and put App Metrics into a working state.

4. Run the command: 

cf update-buildpack nodejs_buildpack --unlock

This will unlock the buildpack so that it can be kept up-to-date by the TAS for VMs tile during future updates.

5. Run the command: 

cf update-buildpack nodejs_buildpack -p path/to/nodejs_buildpack-cached-cflinuxfs3-v<version from step 2>.zip

This will restore the original version of the buildpack that is installed.

6. Go into Ops Manager, select the App Metrics tile and go to the Errands screen. Disable all of the errands. Save. No Apply Changes is necessary here. This is to just make sure that the errands do not run on future updates. Once a product fix is available, you will need to enable the errands again. If you have yet to upgrade, Support recommends that you take the first option, and perform a selective deploy or disable the Apps Metrics errands. This is the simplest path forward until there is a product fix for Apps Metrics.

If you have already started the upgrade process and are hitting an issue, Support recommends that you use the second option to temporarily rollback the buildpack and get the applications working again. You can then continue upgrading your environment by using selective deploy bypass the App Metrics tile or by disabling the App Metric tile's errands.

While it is an option to work around this, Support does not recommend taking the third option and. holding back upgrades to TAS for VMs as this can compromise the security of the foundation.