Spring Cloud Services config-server may report Vault backend unhealthy at HA mode

Products

• Support Only for Spring

Issue/Introduction

Vault supports a multi-server mode for high availability. This mode protects against outages by running multiple Vault servers. Please refer to the details in the following links:

• https://www.vaultproject.io/docs/concepts/ha
• https://www.vaultproject.io/api/system/health.html

Spring Cloud Services config-server runs health checks against vault backend at path /sys/health. In the case of HA mode without a load balancer in front of the Vault cluster, config-server could hit the standby or performance standby nodes, the response codes are 429 and 473, though the Vault cluster is healthy. 

Error message with Vault status code 429:

 2018-11-08T13:54:16.48-0500 [APP/PROC/WEB/0] OUT 2018-11-08 18:54:16.489  INFO 18 --- [nio-8080-exec-2] i.p.s.c.h.ConfigServerHealthEndpoint     : Health status: DOWN. Health details: {vault=DOWN {vault-1=DOWN {server={scheme=https, port=****, host=*****}, initialized=true, sealed=false, standby=true, replication_performance_mode=disabled, replication_dr_mode=primary, server_time_utc=1541703256, version=0.10.3+prem, cluster_name=****, cluster_id=*****, error=Vault is in standby mode.}}, git=UP {git-1=UP {repository={uri=https://****/***.git}}}, diskSpace=UP {total=1073741824, free=905228288, threshold=10485760}}

Error message with Vault status code 473:

   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT [http-nio-8080-exec-5] DEBUG o.s.web.servlet.DispatcherServlet.logResult - Failed to complete request: java.lang.IllegalArgumentException: No matching constant for [473]
   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT [http-nio-8080-exec-5] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet].log - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalArgumentException: No matching constant for [473]] with root cause
   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT java.lang.IllegalArgumentException: No matching constant for [473]
   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT     at org.springframework.http.HttpStatus.valueOf(HttpStatus.java:538)
   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT     at org.springframework.http.client.AbstractClientHttpResponse.getStatusCode(AbstractClientHttpResponse.java:33)
   2020-09-08T14:34:42.02-0400 [APP/PROC/WEB/0] OUT     at 

In order to avoid the problem, it's recommended to place a load balancer in front of the Vault cluster. In this way, a health check from the config-server can be forwarded to the active Vault node correctly.
 

Resolution

There are extra flags added in /sys/health endpoints for the vault to make sure the errors from the standby vault are handled correctly.

This is permanently fixed in Spring Cloud Services 3.1.16