Blobstore verifier in Operations Manager fails and causes an Apply Changes to halt

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Sometimes the Blobstore verifier fails with the ambiguous error message "An unknown error occurred: 'read timeout reached'. Please contact support.", which causes Apply Changes to halt in Ops Manager.

This error message is unclear and it is necessary to debug the logs from Excon.

Environment

Resolution

To successfully troubleshoot this issue, you need to set `EXCON_DEBUG` to 'true' to enable debug mode in Ops Manager.

Follow the instructions below to see the debug logs:

1. ssh into the Ops Manager VM.

2. Become the root user.

3. For Ops Manager 2.2, edit /etc/init/tempest-web.conf and add environment variable EXCON_DEBUG="true" to the list of environment variables.

For Ops Manager 2.3 and above, edit /etc/tempest.d/tempest-web and add the environment variable, EXCON_DEBUG="true" to the list of environment variables.

4. Restart the tempest-web process.

5. Tail /var/log/opsmanager/tempest-web.log.

6. Visit the Ops Manager GUI again, log in, and save the Director Config to trigger the BOSH Director Blobstore verifier to see the debug information. For the PAS Blobstores, save the appropriate form on the PAS tile.