Node.JS app staging failures Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Node.JS app dependency installations can be handled by npm or yarn.

This KB is specific to Node.JS app dependency installations via npm. 

When pushing a Node.JS app on Tanzu Application Service for VMs, the Node.JS buildpack will run a series of npm commands to ensure dependencies are met and in sync while staging the droplet. Due to changes within npm, staging a Node.JS app may fail with newer buildpack releases that previously worked with the older buildpack releases. This KB article aims to discuss some of the observed staging failure scenarios.

Three observed reasons for staging failures:

1. Performance regression.
2. Network calls to registry when vendoring dependencies.
3. Missing dependencies during script execution or during application runtime.

An example of the error observed during cf push if the application is effected from items 1 or 2 above:

2021-12-09T12:17:03.10-0500 [STG/2] OUT        Installing node modules (package.json + package-lock.json)
2021-12-09T12:32:10.24-0500 [API/9] ERR Failed to stage build: staging failed

Examples of errors observed during cf push if the application is effected from item 3 above:

   2022-02-28T09:38:06.68-0500 [STG/0] OUT        Installing any new modules (package.json + package-lock.json)
   2022-02-28T09:38:10.47-0500 [STG/0] OUT > my-nodejs-service@1.0.0 postinstall
   2022-02-28T09:38:10.47-0500 [STG/0] OUT > ./scripts/my-post-install-script.sh
   2022-02-28T09:38:10.48-0500 [STG/0] OUT > my-nodejs-service@1.0.0 prepare
   2022-02-28T09:38:10.48-0500 [STG/0] OUT > husky install
   2022-02-28T09:38:10.48-0500 [STG/0] OUT sh: 1: husky: not found
   2022-02-28T09:38:10.49-0500 [STG/0] OUT npm ERR! code 127
   2022-02-28T09:38:10.49-0500 [STG/0] OUT npm ERR! path /tmp/app
   2022-02-28T09:38:10.49-0500 [STG/0] OUT npm ERR! command failed
   2022-02-28T09:38:10.49-0500 [STG/0] OUT npm ERR! command sh -c husky install
   2022-02-28T09:38:10.55-0500 [STG/0] OUT npm ERR! A complete log of this run can be found in:
   2022-02-28T09:38:10.55-0500 [STG/0] OUT npm ERR!     /home/vcap/.npm/_logs/2022-02-28T14_38_10_495Z-debug.log

 

   2022-02-28T10:51:11.64-0500 [APP/PROC/WEB/0] OUT > MY-APP@1.0.0 start
   2022-02-28T10:51:11.64-0500 [APP/PROC/WEB/0] OUT > node index.js
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR node:internal/modules/cjs/loader:936
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR   throw err;
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR   ^
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR Error: Cannot find module 'nodemon'
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR Require stack:
   2022-02-28T10:51:11.76-0500 [APP/PROC/WEB/0] ERR - /home/vcap/app/index.js

Before elaborating further on these observed staging failure scenarios, lets review key information and gather relevant links:

• The Node.JS buildpack packages several versions of Node, but will use one as a default. 
• Node comes packaged with npm.

See the Node.JS buildpack details table to see which node versions come packaged with each buildpack version. See the Node.JS version table to see which npm version comes with a given node release. See the Node.JS buildpack releases to get insight into default node versions used per buildpack.

We can use the above links to gather information regarding the node and npm versions per buildpack. For example, see the following table representing a few different buildpack versions along with their associated default node and npm versions:



Note that these node and npm versions are the defaults used. Recall that each buildpack ships with several node versions and it is possible to specify a different node version via the engines directive in the package.json file (this is explained further in Workaround 2 later in this KB).

Prior to Node.JS buildpack v1.7.62, npm v6 was the default npm version used during staging. Starting with Node.JS buildpack v1.7.62, npm v6 is no longer the default npm version used.

Performance Regression

Security patches were introduced to npm starting with v6.14.15 and v7.21.0, resulting a performance regression.

For more information about this issue, refer to NPM Restore 15 times slower on versions > 6.14.13 #3939.

The performance regression has been partially patched in npm v7.21.1 and npm v8, however, there are no plans to backport the patch to npm v6. 

Node.JS Buildpack v1.7.61 provides npm v6.14.15 as the default npm version. Due to this, Node.JS applications may fail to stage when using this buildpack. The failure will be observed while installing dependencies referenced in package.json if either of the following is true:

•  Dependencies are not packaged with the application.

•  A working node and/or npm version combination isn't specified by the application's package.json.

Network Calls To Registry When Vendoring Dependencies

This failure scenario is specific to apps that have dependencies vendored, which is a common use case for air-gapped environments or environments that have networking rules in place to block internet bound traffic egressing from Diego Cells. Vendoring dependencies with the app means that the app is pushed with a package-lock.json file and a node_modules directory containing all dependencies needed to stage and run the application.

By vendoring the dependencies with the app, there should not be a need for the staging container to reach out to the npm registry. 

It has been observed that network calls to the npm registry can happen due to an incompatibility of the package-lock.json files between different versions of npm.

The following events may result in this staging failure scenario:
Locally installing dependencies for the application with npm v6 and pushing the application with a newer buildpack that contains npm v7+. 

Npm debug log output would show something similar to:

npm WARN old lockfile 
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile 
npm WARN old lockfile This is a one-time fix-up, please be patient...

In this scenario, staging would fail if the npm registry can not be reached from the staging container. 
 

Missing Dependencies When Scrips Run or During Application Runtime

Starting with Node.JS buildpack v1.5.35 the environment variable NODE_ENV is set to "production".

Starting with npm v7, the npm config property "omit" is set to "dev" when the environment variable NODE_ENV is set to "production" as seen in the documentation:

omit
Default: 'dev' if the NODE_ENV environment variable is set to 'production', otherwise empty.

This "omit" property is not in the config for npm v6. 

If an application has scripts or runtime dependencies that rely on the dependencies under the devDependencies section in the package.json file then those dependencies will not be available when called if the following is true:

• environment variable NODE_ENV is set to "production"
• npm v7+ is used

Please see the "Notes" section in the Resolution below for how to direct newer npm versions to include devDependencies.

Environment

Product Version: 2.11

Resolution

The following workarounds are recommended to choose from, in order of preference:
 

Workaround 1 

Vendor dependencies if you can.

Due to the incompatibility of the package-lock.json files between different versions of npm it is recommended to run npm install locally against your app using the target version of npm that the buildpack uses. For example if you are using buildpack 1.7.63, then use npm v8.0.0 to run npm install locally.

Workaround 2 

If dependencies cannot be vendored, pin to a known working version of node and/or npm within package.json. See the Node.JS buildpack docs for details on how to do this. It’s not necessary to specify an npm version if the specified node version comes packaged with a working npm version already.

For example:

"engines": {
  "node": "12.x.x"
}

 

For each of the above recommendations, it’s advisable to use the most recent version of the buildpack available to you. If neither of the above recommendations work, however, you may pin the buildpack itself to a known working version. See Managing Custom Buildpacks.

Notes

• If an application has scripts or runtime dependencies that rely on devDependencies, then provide a .npmrc file with the project with the following entry to include devDependencies:

  include=dev

• If using an internal registry, providing that registry information in the .npmrc file may be necessary.
• Workaround 2 requires internet access to download, if the specified version is not already provided by the buildpack. Secured environments may have to ensure that their firewall settings are configured correctly, or they may see an error message similar to:

  **ERROR** We're unable to download the version of npm you've provided.

• If choosing not to vendor dependencies, it’s advisable to provide a package-lock.json to ensure build reproducibility.

• It is possible to lock a buildpack before a TAS upgrade if you wish for a buildpack to not be upgraded during the deploy.
  
  For example, if you wish to lock Node.JS Buildpack v1.7.57 before a TAS upgrade so apps can be verified to work with newer buildpacks first then running the following before the upgrade will help:

  cf update-buildpack nodejs_buildpack --lock

  
  This should keep the nodejs_buildpack locked from updating during the deploy. Then when ready to deploy the new buildpack to the environment, you can download it from Tanzunet and upload it to the environment. For more information on how to do this, refer to Managing Custom Buildpacks. 

Next steps

If all the steps mentioned before do not resolve the issue, enabling Verbose logging and changing the log level can help gather more detailed information about the application staging process.

To do this, enable the environment variables NODE_VERBOSE and NPM_CONFIG_LOGLEVEL in the application manifest. For an example see the following.

---
applications:
- name: my-app-name
  buildpack: nodejs_buildpack
  env:
   NODE_VERBOSE: true
   NPM_CONFIG_LOGLEVEL: silly

This link provides detailed information about the different log levels that can be applied.

If this does not bring visibility about the on going issue, please collect all these logs and contact Tanzu Support.