Redis On Demand Broker in a crashed state after Ops Manager Upgrade

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
During the upgrade BOSH will try to find an instance of CredHub. It will fail with the following error: 
 

[on-demand-service-broker] 2018/11/02 08:10:19.507861 error connecting to runtime credhub: lookup credhub.service.cf.internal on 192.XXX.X.X:XX: no such host

Environment

Cause

The cause of this is that during the upgrade BOSH tries to find an instance of CredHub however it does not exist. This can be confirmed by running nslookup. 

You will receive the following error: 
 

credhub.service.cf.internal
Server:         169.XXX.X.X
Address:        169.XX.X.X#XX

*** Can't find credhub.service.cf.internal: No answer

Resolution

There are two resolutions to this issue, depending on whether or not you want to use CredHub with Redis. 

1. If you want to use CredHub with Redis then you will need to create a CredHub VM. This can be done by navigating to the resource config tab in the PAS tile and selecting a minimum of 1. You can then apply changes and this will create a CredHub instance. 
2. The alternate option, if you do not want to use CredHub is to disable this in the PAS tile. Please be aware this option will disable CredHub integration for all services, not just Redis. In order to do this you will need to go to the CredHub tab on the PAS tile and untick the following and then apply changes again: 
   • Secure Service Instance Credentials