Config server with vault backend returns error "Vault is in standby mode"

Products

• Support Only for Spring

Issue/Introduction

Config-server with a vault cluster backend returns the error "Vault is in standby mode" randomly. Details for the error message in config server logs is shown below:

 2018-11-08T13:54:16.48-0500 [APP/PROC/WEB/0] OUT 2018-11-08 18:54:16.489  INFO 18 --- [nio-8080-exec-2] i.p.s.c.h.ConfigServerHealthEndpoint     : Health status: DOWN. Health details: {vault=DOWN {vault-1=DOWN {server={scheme=https, port=xxxx host=vaulg.xxx.xxx}, initialized=true, sealed=false, standby=true, replication_performance_mode=disabled, replication_dr_mode=primary, server_time_utc=1541703256, version=0.10.3+prem, cluster_name=vault-cluster, cluster_id=xxxxxxxxxx, error=Vault is in standby mode.}}, git=UP {...}}

Environment

Product Version: Other

Resolution

According to Vault High Availability, the vault end endpoint in the cluster could be at either standby state or active state.

While the config-server vault healthcheck reaches out to the standby endpoint, config-server doesn’t retry the redirected url but simply reports that the vault server is down. 

A github issue was logged at open source spring-cloud-config project, developers recommend placing a load balancer in front of vault cluster. This way, the requests from config-server can be proxied to an active vault endpoint.