Unable to regenerate certificate /services/tls_ca using due to maestro commands timeout

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Maestro command used to regenerate certificate /services/tls_ca exit with error caused by a timeout.

Cause

In bigger foundations that handle large amount of service instances the regeneration of certificate /services/tls_ca can cause a lot of overhead causing maestro to show credhub timeouts.

Resolution

After the command stops showing an error due to timeout it is common that credhub continues working even when the command has outputted a timeout error. The following verification steps can be executed to in order to confirm the transaction was completed and it is safe to continue the rotation process:

• Download a support bundle for your foundation.
• Once downloaded, extract all files and look for the logs located at
  • bosh_director_logs (the file inside this folder needs to be extracted too) 
    • create-env-vm-logs-xxxxxxxx-xxxxxx-xxxxxxxxx
      • credhub
        • credhub_security_events.log
• In this log look for an entry similar to the following:

  • 2025-XX-XXTXX:XX:XX.XXXZ: CEF:0|cloud_foundry|credhub|2.12.107|POST /api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/regenerate|POST /api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/regenerate|0|rt=1755186784144 suser=ops_manager suid=uaa-client:ops_manager cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/regenerate requestMethod=POST cs3Label=versionUuid cs3=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx cs4Label=httpStatusCode cs4=200 src=xx.xxx.xx.xx dst=xx.xxx.xx.xx cs2Label=resourceName cs2=/services/tls_ca cs5Label=resourceUuid cs5=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx deviceAction=REGENERATE_CERTIFICATE cs6Label=requestDetails cs6={"transitional":true}

• Once you find the entry verify that the value for cs4 is 200 as seen in the example above: cs4=200. This indicates a successful regeneration of the CA.

 

• Next, in the support bundle go to certificates -> maestro_topology.yml.
  • Search for /services/tls_ca and verify that you can see a new version of the CA created, this one should be the first version in the list under /services/tls_ca and it should be marked as transitional. To confirm it is marked as transitional, make sure it shows the value transitional: true in the new version.

If you can confirm both of the conditions mentioned above, this indicates that the cli threw a timeout error, but credhub continued working on the backgroud and completed the process. You can continue with the certificate rotation safely.

If you can't see a cs4=200 response on credhub_security_events.log or there is no new version marked as transitional, please contact the Support team for investigation.