PCF Services do not Start due to 'x509' certificate has expired or is no longer valid

Products

• Operations Manager

Issue/Introduction

Symptoms:

Error Message:

consul_agent.stderror.log (any failed service could potentially hit this error if certs are expired)

consul_server/55e8ad09-5a22-4987-8ad6-93a97e43a2e2:
/var/vcap/sys/log/consul_agent$ cat consul_agent.stderr.log
error during start: timeout exceeded: 
"Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: 
"Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"

Environment

Cause

Certificates in Cloud Foundry have a 2 year expiration period. Certificates should be regenerated within 2 years of installation. In this instance, it is the non-configurable internal certificates that require regeneration.

 

Resolution

The internal non-configurable certificates on Cloud Foundry require regeneration. 

Follow the steps to resolve this issue:

1. IMPORTANT Prior to performing this procedure, it is required to scale Consul to 1 instance temporarily in order to avoid certificate mismatches. Navigate to Resource Config in Elastic Runtime under Operations Manager and set Consul to 1 VM. 

2. From your local machine, target your Operations Manager UAA server:

   $ uaac target https://OPS-MAN-FQDN/uaa
   

3. Retrieve your token to authenticate. When prompted for a passcode, retrieve it from https://OPS-MAN-FQDN/uaa/passcode.

   $ uaac token owner get
   Client ID: opsman
   Client secret: [Leave Blank]
   User name: OPS-MAN-USERNAME (by default 'admin')
   Password: OPS-MAN-PASSWORD
   

   Replace OPS-MAN-USERNAME and OPS-MAN-PASSWORD with the credentials that you use to log in to the Operations Manager web interface.

4. List your tokens:

   $ uaac contexts
   

   Locate the entry for your Operations Manager FQDN. Under client_id: opsman record the value for access_token.

5. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing Operations Manager Director:

   $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \ 
   -X POST \ 
   -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \ 
   -H "Content-Type: application/json" \ 
   -d '{}'
   

6. Click Apply Changes in Operations Manager UI.
    
7. If Consul was scaled down (in step 1), then you may now scale it back up to original value once apply changes has successfully completed.

Thus the certificates will be regenerated and re-applied to the system. The certificate has expired errors should now be fixed.