Cloud Controller fails to create service instance when there is a expired cert installed on the system

Products

• Operations Manager

Issue/Introduction

Error message returned by cf cli when creating the service instance:

Unable to create service instance - SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)

All versions of the Xenial Stemcell release using Openssl version 1.1.0 could fail if there are two certificates for the same CA where one is expired and one is valid.  Depending on which CA cert is processed first will determine the success or failure of the ssl handshake.

Here is a excerpt from the Openssl 1.1.1 man page:
 

"If several CA certificates matching the name, key identifier, and serial number condition are available, only the first one will be examined. This may lead to unexpected results if the same CA certificate is available with different expiration dates. If a "certificate expired" verification error occurs, no other certificate will be searched. Make sure to not have expired certificates mixed with valid ones."

The above passage describes the current behavior of Openssl version 1.1.0 used in the Xenial stemcells. Openssl 1.1.1 is believed to have changed this behavior as per bug-1840767 and we currently do not have any plans to upgrade Openssl in the Xenial stemcell release.

 

BOSH stemcell version 445.112 or earlier include an old expired root CA "AddTrust_External_Root".  and you might experience this issue when upgrading to this stemcell and your load balancer terminates SSL with a cert signed by this root authority.

 

Environment

Product Version: 2.9
OS: Xenial

Resolution

In this specific case, Cloud Controller uses this HTTP client which imports the systems default trust store (/etc/ssl/certs/ca-certificate.crt) to send requests to the service broker. 

Resolution when there is an old private root CA installed on the system

To resolve simply review the certificates in Operations Manager -> Bosh Tile -> Security -> Trusted certificates field. Identify and remove any expired certificates from that field and apply the changes to all tiles installed on the system.

Resolution for stemcell public expired CA

Upgrading to stemcell version 445.113 removes the expired "AddTrust_External_Root" root CA from this stemcell line.