How to troubleshoot Spring Cloud Services (SCS) for VMware Tanzu - Config Server with Vault Backend
search cancel

How to troubleshoot Spring Cloud Services (SCS) for VMware Tanzu - Config Server with Vault Backend

book

Article ID: 298012

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Please see the below sections for information on troubleshooting issues with Config Server service instances in Spring Cloud Services (SCS) for VMware Tanzu 3.x.
  • Known Issues
  • Enabling Logs in Vault
  • Client App Dependencies
  • Docs - Quick References


Environment

Product Version: 2.7

Resolution

Known Issues / Limitations

  • p-config-server instances gets unauthorized response from CC API
  • Renewing Vault Token results in 400 Bad Request in Client App and error 500 in Config Server
  • Enabling Logs in Vault

p-config-server instances gets unauthorized response from CC API

When using SCS Config Server for VMware Tanzu with Vault as the backend, you will see errors such as following, although this will not cause the properties to malfunction.

2020-04-15T11:59:53.775+02:00 [RTR/2] [OUT] config-4b5ef318-####-####-####-b246a074d4cd.apps.sandbox.#########.io - [2020-04-15T09:59:53.748706975Z] "POST /vault/v1/auth/token/renew-self HTTP/1.1" 500 17 158 "-" "Java/1.8.0_232" "10.##.###.128:38044" "10.##.###.74:61054" x_forwarded_for:"10.##.###.128" x_forwarded_proto:"https" vcap_request_id:"512e1a5d-####-####-####-e3d45c8d321c" response_time:0.026177 gorouter_time:0.000153 app_id:"3a08d1f5-####-####-####-b3abacadd226" app_index:"0" x_b3_traceid:"79e9######64d5" x_b3_spanid:"79e9b######e064d5" x_b3_parentspanid:"-" b3:"79e9b######64d5-79e######064d5"
22020-04-15T12:00:00.675+02:00 [APP/PROC/WEB/0] [OUT] 2020-04-15 10:00:00.674 ERROR 20 --- [nio-8080-exec-5] i.p.s.cf.CloudControllerServiceImpl : GET https://api.system.sandbox.#########.io/v2/service_instances/4b5ef318-####-####-####-b246a074d4cd/permissions resulted in response code 401


These errors are not happening with Config Service instances when using Git as the backend.

After doing some investigation, we see that the app bound to Config Server tries to refresh the Vault token periodically, resulting in the above errors. The 500 code the you are getting is because you are using the root token, which is not renewable. However, if you use a different token, you still get similar errors. The errors above are from my test, where we used a non-root token.

The errors below are produced because Config Service checks some permissions in the process of refreshing the Vault token periodically.

2020-04-21T17:29:17.89+0200 [APP/PROC/WEB/0] OUT 2020-04-21 15:29:17.892 ERROR 27 --- [nio-8080-exec-2] i.p.s.cf.CloudControllerServiceImpl      : GET https://api.run-04.####-##.###.#########.io/v2/service_instances/d248b95c-####-####-####-52fb95976556/permissions resulted in response code 401
   2020-04-21T17:29:17.90+0200 [RTR/0] OUT config-d248b95c-####-####-####-52fb95976556.cfapps-04.####-##.###.######.io - [2020-04-21T15:29:17.866231740Z] "POST /vault/v1/auth/token/renew-self HTTP/1.1" 200 17 393 "-" "Java/1.8.0_242" "10.###.##.###:42806" "10.###.##.##:61054" x_forwarded_for:"10.###.##.##, 10.###.##.250" x_forwarded_proto:"https" vcap_request_id:"1207ab4f-ada9-4824-6269-fe6345a12d28" response_time:0.034804 gorouter_time:0.000228 app_id:"85775bd7-####-####-####-6e9429fa9488" app_index:"0" x_b3_traceid:"0107c######9f117" x_b3_spanid:"0107c########17" x_b3_parentspanid:"-" b3:"010#########17-0107c########117"


Renewing Vault Token results in 400 Bad Request in Client App and error 500 in Config Server

When using Vault root token in your Client App you see errors containing the following stack trace in your Client App:

2020-04-15T11:59:06.580+02:00 [APP/PROC/WEB/0] [OUT] at io.pivotal.springcloud.config.vault.VaultRelay.renewSelf(VaultRelay.java:27) ~[classes/:na]
2020-04-15T11:59:06.580+02:00 [APP/PROC/WEB/0] [OUT] at io.pivotal.springcloud.config.vault.VaultRelayController.relayRenewSelf(VaultRelayController.java:26) ~[

You also see the following 500 response in your Config Server App:

2020-04-15T11:59:53.775+02:00 [RTR/2] [OUT] config-d248b95c-c262-####-#####-52fb95976556.cfapps-04.####-##.###.######.io - [2020-04-15T09:59:53.748706975Z] "POST /vault/v1/auth/token/renew-self HTTP/1.1" 500 17 158 "-" "Java/1.8.0_232" "10.##.###.###:38044" "10.##.###.##:61054" x_forwarded_for:"10.##.###.###" x_forwarded_proto:"https" vcap_request_id:"512e1a5d-####-####-####-e3d45c8d321c" response_time:0.026177 gorouter_time:0.000153 app_id:"3a08d1f5-####-####-####-b3abacadd226" app_index:"0" x_b3_traceid:"79e9x######4d5" x_b3_spanid:"79e9######64d5" x_b3_parentspanid:"-" b3:"79e9b#####e064d5-79e9b#######4d5"

This is because Vault root token is note renewable and for this reason we recommend against using it. When changing to a non-root token, these errors will disappear.


Enabling Logs in Vault

Audit devices are the components in Vault that keep a detailed log of all requests and response to Vault. Since every operation with Vault is an API request or response, the audit log contains every authenticated interaction with Vault, including errors.

Multiple audit devices can be enabled and Vault will send the audit logs to both. This allows you to not only have a redundant copy, but also a second copy in case the first is tampered with.

When a Vault server is first initialized, no auditing is enabled. Audit devices must be enabled by a root user using vault audit enable.

When enabling an audit device, options can be passed to it to configure it. For example, the command below enables the file audit device:

$ vault audit enable file file_path=/var/log/vault_audit.log



Powered by Wolken Software