Credhub causes high CPU on MySQL due to Spring Cloud config-server clients healthcheck

Products

VMware Tanzu Application Service for VMs

Environment

Product Version: 2.10

Resolution

Checklist:

This issue occurs when the following conditions are met:

[Enable credhub as a backend for config server] is enabled in Spring Cloud Services configuration.
Large number of config-servers and corresponding Java client apps in the cluster.
HTTP healthcheck is enabled with /actuator/health endpoint of the client apps. The check is also possibly not directly enabled by developers but could be enabled by some application plugins.
Spring Cloud dependencies version is lower than 2020.0, for example Hoxton release.

HTTP healthcheck against /actuator/health of (config-server)client apps triggers the cascading check with the backend config-servers, then each config-server reaches out to CredHub at 5 minutes interval when "Enable credhub as a backend for config server". CredHub also queries the backend database. In the case of large number of apps on the platform, the large volume of SQL queries finally result in extremely high CPU load on MySQL.

Please refer to system diagram below:

Screenshot 2021-09-28 at 15.28.50.png

To confirm the issue:

`cf logs` with config-server app, check if there is incoming request from any single client side app every 30 seconds.
on Credhub side, check if there are large volume of request `/c/p.spring-cloud-services-scs-service-broker/config-server-****` in security audit logs.

As a temporary workaround, the check to backend config-sever can be disabled by setting health.config.enabled=false, the setting is being used by config-server client library, and it's default true.

With true configuration, config-server client library issues heathcheck to backend config-server which is unnecessary. After changing the settings, all Java client apps must be restarted or restaged to take effect. Be aware to not override existing runtime-environment variables.

As shown in the example below, key1/key2 are existing variables, keep them and append health.config.enabled as false.

1. Fetch current running environment variable group, for example

cf running-environment-variable-group
Retrieving the contents of the running environment variable group as admin...
OK
Variable Name   Assigned Value
key1            value1
key2            value2

2. Append health.config.enabled to existing settings

$ cf set-running-environment-variable-group '{"key1": "value1", "key2": "value2", "health.config.enabled": "false"}'
Setting the contents of the running environment variable group as admin...
OK

3. Confirm settings

$ cf running-environment-variable-group
Retrieving the contents of the running environment variable group as admin...
OK
Variable Name           Assigned Value
health.config.enabled   false
key1                    value1
key2                    value2

4. Restart Java apps which are binding to config-servers, then healthcheck to config-server will be disabled.

This issue has been fixed in Spring Cloud Dependencies release 2020.0 and above. For example, with Spring Boot apps built with maven, please update spring-cloud-dependencies to fixed releases for permanent resolution.

      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>2020.0.4</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>