Gorouter fails to authenticate after changing the System Domain

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
During an attempt to Apply Changes from the Ops Manager, the task fails, with the Gorouter is emitting the following error:

`{"log_level":6,"timestamp":xxx.xxx,"message":"routing-api-connection-failed","source":"vcap.gorouter","data":{"error":"unauthorized"}}`  

The following output can be found in the Routing APIlogs:

`{"timestamp":"xxx.xxx","source":"routing-api","message":"routing-api.api-server.uaa-client.decode-token-failed","log_level":2,"data":{"error":"invalid issuer","session":"1.13797"}}`.

Environment

Cause

When changing the system domain, the Routing API may cache the UAA issuer, resulting in one of the following outcomes:

• The Routing API does not restart after the UAA client restarts.
• The Routing API happened to talk to a UAA client that had not yet updated with the new issuer.

Both of these possibilities cause the Routing API to not update the cached issuer, and subsequently fail to decode tokens sent by the Gorouter. Thus, it is impossible for the Routing API's cached issuer to match the new (correct) issuer in the token used by the Gorouter.

Resolution

Restart the Routing API that is emitting the "invalid-issuer" errors. This will allow the cache to be purged and then populated with the correct issuer. Below is the procedure for restarting the Routing API:

1. SSH into the Ops Manager VM.
2. Run 'bosh vms' to list the VMs.
3. Run 'bosh ssh' to SSH into one of the control VMs listed in the previous step.
4. Run 'sudo monit restart all' to restart all processes.
5. Run 'sudo monit summary' to check the status of all processes on the VM.
6. Once all processes on the control VM have returned to a working state, repeat steps 3-5 for any remaining control VMs.

Apply Changes again and the errors will cease.