Orgs and Spaces got deleted after certificate update.

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Application Service
• Operations Manager

Issue/Introduction

A user does some cert update to their TAS foundation and Apply Changes fails with below error: 

Running errand: clock_global/96######-8###-4###-a###-012######### (1) (TIME) Running errand 'smoke_tests': Expected task '123456' to succeed but state is 'error' L Error: Response exceeded maximum allowed length 

Cause

As per the vm error mentioned in the above Introduction field, clock_global vm error logs will show:

Error: Get "https://SMOKES-APP-1######8-1###.apps.pcf.abc.abcd.com/json": tls: failed to verify certificate: x509: certificate is valid for *.apps.pcf.abc.abcd.com, *.apps.pcfprod.abc.abcd.com, not SMOKES-APP-1######8-1###.apps.pcf.abc.abcd.com
 

This means trust is broken between TAS and Bosh Director.

Resolution

There are two common fixes for this issue, both should be considered:

1. Using this doc (near bottom), provide all the required wildcard domains in the gorouter certs. These wildcards are indicated in the error from clock_global logs (in above 'Cause' section). Save, and Apply Changes to TAS tile only (with all errand on).
2. Trusted Certificates on the Bosh Director tile need to be "cleaned up".