Performance issues when using the New Relic Nozzle for VMware Tanzu

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

New Relic Nozzle is a Partner tile.

If you are seeing performance issues in

VMware Tanzu Application Service (TAS) for VMs, as in Cloud Controllers getting overloaded, API calls that are failing inconsistently, or various CAPI failure, it is likely due to the New Relic Nozzle polling expensive endpoints (/v2/apps) with deprecated parameter inline-relations-depth. 

Please check the Router logs and if you see that there are frequent occurrences of  inline-relations-depth=2 calls, most likely this is what is responsible for the performance issues. The GoRouter Access logs can be found here /var/vcap/sys/log/gorouter/access.log

Resolution

This issue was first introduced in 1.x and has been introduced again in 2.5.x (somewhere between 2.5.9 and 2.5.12).To fix this issue, follow the steps listed below:

1. If you are in 1.x, please upgrade the New Relic Nozzle version to at least 1.1.22.
   If you are in 2.5.x, please upgrade the New Relic Nozzle version to at least 2.5.15.

2. Please increase the App Detail Collection Interval for fetching application metadata from Cloud Controller. The default value for this interval is 1 minute. Please increase this to a minimum of 30 minutes, but no later than 60 minutes. This reduces the amount of load the New Relic Nozzle causes the Cloud Controllers.

3. The Nozzle version 1.1.22 and onwards offers Redis as a Database option for Storage. If you turn on this option, there will be fewer calls made to Cloud Controller. Let's say there are 6 New Relic Nozzle instances, instead of 6 calls, there will be just 1 call to Cloud Controller. Redis will cache this data among other instances. Using Redis as an option minimizes the impact that the Noozzle calls have on Cloud Controllers.
   
   Note: To use this feature, you must have Redis installed in the same PCF Foundation as the New Relic Nozzle.