Spring Cloud Services config-server fails to start due to CredHubException: 401 Unauthorized

Products

• Support Only for Spring

Environment

Product Version: 3.0

Resolution

Checklist:

During config-server starts, it depends on Java buildpack key manager to load

• /etc/cf-instance-credentials/instance.cert
• /etc/cf-instance-credentials/instance.key

then interpolate credhub-ref of mirror service in VCAP_SERVICES, for example: 

{
 "VCAP_SERVICES": {
  "p.mirror-service": [
   {
    "binding_guid": "f2c64258-0bc0-40b7-b563-76e6c365b3c2",
    "binding_name": null,
    "credentials": {
     "credhub-ref": "/c/p.spring-cloud-services-scs-mirror-service/5b1f7f18-8cf0-4163-84ef-784ef80afcbc/credentials"
    },
    "instance_guid": "5b1f7f18-8cf0-4163-84ef-784ef80afcbc",
    "instance_name": "mirror-svc",
    "label": "p.mirror-service",
    ...
   }
  ]
 }
}
If the key manager is disabled by environment variable or running-environment-variable-group, interpolation will fail as credhub refuses the interpolation request. 

   2023-03-12T08:17:34.11+0400 [APP/PROC/WEB/0] OUT org.springframework.credhub.core.CredHubException: 401 Unauthorized
   2023-03-12T08:17:34.11+0400 [APP/PROC/WEB/0] OUT 	at org.springframework.credhub.core.CredHubTemplate.doWithRest(CredHubTemplate.java:189)
   2023-03-12T08:17:34.11+0400 [APP/PROC/WEB/0] OUT 	at org.springframework.credhub.core.credential.CredHubCredentialTemplate.getByName(CredHubCredentialTemplate.java:159)
   2023-03-12T08:17:34.11+0400 [APP/PROC/WEB/0] OUT 	at io.pivotal.spring.cloud.configserver.env.CredHubPropertiesPostProcessor.postProcessEnvironment(CredHubPropertiesPostProcessor.java:60)

If the error is observed in config-server logs, please check environment variables of config-server with `cf env config-server` under p-spring-cloud-service org / <GUID> space. If JBP_CONFIG_CONTAINER_SECURITY_PROVIDER is configured as  '{key_manager_enabled: false}', key_manager_enabled should be changed to true. 

Usually it is not necessary to disable key_manager, but as workaround for a bug which has been identified within the Container Security Provider Framework, some customers disabled key_manager. This bug has been fixed since Java buildpack v4.57 release, once upgrade Java buildpack to v4.57 or above, the workaround can be removed.