"Certificate verify failed (certificate has expired)" error when binding or unbinding a service instance and using Load Balancer or Gorouter certificates from Let's Encrypt

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When trying to perform service instance operations using cf CLI, you receive the following error when binding or unbinding a service instance and using Load Balancer (LB) or Gorouter certificates from Let's Encrypt:

Error: Certificate verify failed (certificate has expired)

Note: This can also occur when running the Tanzu Application Service for VMs (TAS for VMs) smoke-test errand, since there will be service instance operations being conducted during the life-cycle of the test.

This error occurs because the well-known Certificate Authority (CA), DST Root CA X3 (DST_Root_CA_X3.crt), in Xenial Stemcells expired September 30th, 2021. 

There is a newer Root CA available for Let's Encrypt in the Stemcell, however, based on the order the certificates are read, it's possible the expired certificate is consumed first and subsequently the verification fails immediately.

For more information about the expiration of DST Root CA X3 (DST_Root_CA_X3.crt), refer to OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates.

Environment

Product Version: 2.7

Resolution

To resolve this issue, ensure you are using a Stemcell version equal to or greater than the following versions: 

• Ubuntu Xenial 456.194+
• Ubuntu Xenial 621.160+

Using a newer version of the Ubuntu Xenial Stemcell remove the expired certificate in question.