maestro command fails with error "context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

While running maestro commands the execution fails with timeout errors similar to the following:

maestro update-transitional signing --name "/services/tls_ca" updated:  - name: /services/intermediate_tls_ca   certificate_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx update_failed:  - name: /services/tls_ca   certificate_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   version_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   exclusion_reason: 'credhub error: Put "https://xx.xxx.xxx.xx:8844/api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/update_transitional_version": context deadline exceeded (Client.Timeout exceeded while awaiting headers)'error: there was an error updating some certificates

 

ubuntu@opsman:~$ time ./maestro regenerate ca --name "/services/tls_ca" --force
regenerate_failed:
    - name: /services/tls_ca
      certificate_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      exclusion_reason: 'credhub error: Post "https://xx.xx.xx.xx:8844/api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/regenerate": context deadline exceeded (Client.Timeout exceeded while awaiting headers)'
error: could not regenerate some certificates

 

This can also be observed when using calls directly from the Credhub API:

credhub curl -p /api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/update_transitional_version -d '{"version": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" }' -X PUT
Put "https://xx.xxx.xxx.xx:8844/api/v1/certificates/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/update_transitional_version": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Cause

These kind of error messages are related to maestro-CLI default timeout. The default value is 45 seconds. On large environments that have big amounts of certificates information in Credhub the execution of maestro commands can take a long time to respond which can eventually exceed this timeout setting.

Resolution

One option is to find the search on the Credhub API documentation for the call corresponding to the maestro command required and execute from there.

If an immediate workaround is needed, the value for the timeout can be overwritten using the command --http-timeout=. You can find more information about the command on the Credhub commands documentation:

Credhub CLI commands: https://github.com/cloudfoundry/credhub-cli/blob/17db04838e07702b567dbc4fe685dbe4ffc64234/commands/commands.go#L29

Lastly, you can find the following information on the credhub cli help:

--http-timeout= Http timeout for http-client. Needs to have unit passed in (i.e. 30s, 1m) [$CREDHUB_HTTP_TIMEOUT]

Use the command to set a new value for the timeout and try the required maestro command again.