VMware Tanzu GemFire root CA certificate expiration issue

Products

• VMware Tanzu Gemfire

Issue/Introduction

This issue is applicable to all versions of VMware Tanzu GemFire.

The VMware Tanzu GemFire Pulse access fails with the following error: 

502 Bad Gateway: Registered endpoint failed to handle the request. 

Resolution

Check if the gemfire-ssl/root CA certificates are expired using the following command:

credhub get -n /services/tls_ca -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"

Currently the VMware Tanzu GemFire product does not support rotating a CA certificate, if it is already expired. The only way to recover the system is shutting each cluster down entirely and bringing it back up with a new CA certificate.

As of now, there is no way that an expired cert can be rotated while the VMware Tanzu GemFire services instances are up. You will have to manually bosh stop each service instance and provide a new certificate manually and then do bosh start. This is a scenario where non-persistent data will be lost, make sure to add the following command:

$ bosh stop --skip-drain

To prevent this issue from happening again in the future, periodically validate the certificate expiry as specified in the Reference link below. These steps can be scripted into your CI job that is checking expiry dates.

References:

Check Expiration Dates

• https://docs.pivotal.io/p-cloud-cache/1-9/rotating-ca.html#check

Managing Certificates

• https://docs.pivotal.io/p-cloud-cache/1-9/rotating-ca.html

Rotating Certificates

• https://docs.pivotal.io/platform/2-7/security/pcf-infrastructure/api-cert-rotation.html