Hit an Error "Credhub is currently in read only mode" when accessing the credhub service instance

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When the app tries accessing the credhub service instance, it might hit the following error:

202#-0#-0#T##:##:##.##-0000 [APP/PROC/WEB/0] ERR Unable to interpolate credhub refs: 
Unable to interpolate credhub references: Service Unavailable. 
Credhub is currently in read-only mode.

When apply changes to TAS or HW tile in OpsMan UI, user might hit the following errors:

Error inserting the .properties.metrics_agent_metrics_tls credential for cf-4cdd######## into CredHub: CredhubClient::FailedRequest - Credhub request returned 503 with error Service Unavailable. Credhub is currently in read-only mode.

Error inserting the .grafana.generated_ssl_certificate credential for p-healthwatch2-8df4######## into CredHub: CredhubClient::FailedRequest - Credhub request returned 503 with error Service Unavailable. Credhub is currently in read-only mode.

Error inserting the .grafana.mysql_client_certificate credential for p-healthwatch2-8df4######## into CredHub: CredhubClient::FailedRequest - Credhub request returned 503 with error Service Unavailable. Credhub is currently in read-only mode.

Error inserting the .properties.metric_registrar_orchestrator_tls credential for cf-4cdd######## into CredHub: CredhubClient::FailedRequest - Credhub request returned 503 with error Service Unavailable. Credhub is currently in read-only mode.

Resolution

There are several factors causing this issue. One common reason is that a BBR backup job has been triggered and then fails, which leaves various jobs still locked.

BBR writes back up logs to the current directory in a file named bbr-TIMESTAMP.err.log when any error occurs. We can filter out the log file with the key words, "error", "ERROR" or "failed". Here are several examples:

1 error occurred:
failed to find instances for deployment cf-xxxxx
ERROR: Job failed: exit code 1

If this is the case in your side, please run "BBR backup-cleanup" command to unlock the jobs. For detailed instructions on "BBR backup-cleanup" command, please refer to the manual at Canceling a backup.