Redis On Demand Services do not rotate /services/tls_ca leaf certs resulting in app downtime

Products

• Redis for VMware Tanzu

Issue/Introduction

Following the standard /services/tls_ca rotation procedure may result in application downtime for apps that bind to Redis On Demand Services. Applications might return some form of a certificate validation error similar to the one below:

OpenSSL::SSL::SSLError - SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

There is a bug in the service instance manifest generation process in Redis for VMware Tanzu Tile 2.2.x and 2.3.x that places the update_mode key in the wrong section of the manifest. The update_mode key is placed in the features block when it should be in the Redis leaf certificate variable block.

Since the update_mode key is misplaced, Redis Service Instances does not rotate their leaf certificates. You can check which instances have not been rotated by comparing the expiration date of the signing CA using the following command:

credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq '.certificates[] | .signs[]' | egrep redis | while read line ;do echo $line ; credhub get -n $line -k ca | openssl x509 -noout -enddate ; done

Environment

Product Version: Other
OS: 2.2 & 2.3

Resolution

This issue is fixed in VMware Tanzu for Redis 2.4. This bug will also be back ported to VMware Tanzu for Redis 2.2 and 2.3. 

Workaround

Insert this workaround into the existing /services/tls_ca procedure. Stop following the standard procedure when you reach the point in the /services/tls_ca procedure that removes the old cert and then asks you to do the final Apply Changes. Instead apply this workaround before proceeding to remove the old certificate.

Note: This procedure needs to be applied to all Reids On Demand Service instances with TLS enabled.

Download the service instance manifest

Replace <GUID> with the actual GUID of your service instance.

bosh -d service-instance_<GUID> manifest > /tmp/redis-<GUID>.yml

Modify the manifest

Edit the manifest by moving the update_mode key into the correct variable block.

Bad Example

- name: redis_certificate
  type: certificate
  options:
    ca: /services/tls_ca
    common_name: Redis Certificate
  consumes:
    alternative_name:
      from: custom-redis-service-address
      properties:
        wildcard: "true"
    common_name:
      from: custom-redis-service-address
      properties:
        wildcard: "true"
features:
  randomize_az_placement: true
  update_mode: converge <<<<<<<<<<<<<<< BAD!!!!!!

Correct Example

- name: redis_certificate
  type: certificate
  update_mode: converge <<<<<<<<<<<<<<< FIXED!!!
  options:
    ca: /services/tls_ca
    common_name: Redis Certificate
  consumes:
    alternative_name:
      from: custom-redis-service-address
      properties:
        wildcard: "true"
    common_name:
      from: custom-redis-service-address
      properties:
        wildcard: "true"
features:
  randomize_az_placement: true

Deploy the Redis On Demand Service Instance

Using the modified manifest, replace <GUID> with the actual GUID of your service instance and start the deployment to trigger a leaf certificate rotation.

bosh -d service-instance_<GUID> deploy /tmp/redis-<GUID>.yml