BOSH verifier throws nondescript error in Ops Manager during Apply Changes

Products

• Operations Manager
• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Platform Core

Issue/Introduction

When hitting Apply Changes to remove the old CA during a full CA and leaf cert rotation, we encountered the following error:

error alert message, An unexpected error occurred. Contact VMware Technical Support to report the problem. Details: cannot call errors_as_hash on a step with no errors

 

Resolution

The first step is to log into the opsman VM and check /var/log/opsmanager/production.log for the stack trace. This file can also be viewed in the Support Bundle as production.log in the ops_manager_logs folder. Check for the stack trace by looking at the timestamp of the Apply Changes (this can be found in ops_manager_apply_changes_logs/deployment_XXX/install_log.txt) and/or searching for the error text as displayed on the GUI.

In this case, we searched for "errors_as_hash" and found the following text in the log file:

E, [2022-09-23T19:14:40.502241 #798] ERROR -- :         Exception caught via rescue_from (code 500):
        (See app/controllers/application_controller.rb)
        cannot call errors_as_hash on a step with no errors
    /home/tempest-web/tempest/web/app/models/tempest/verifiers/internal/step.rb:14:in `errors_as_hash'
/home/tempest-web/tempest/web/app/models/tempest/verifiers/internal/bosh_product_verifiers.rb:19:in `block in problem_steps'
/home/tempest-web/tempest/web/app/models/tempest/verifiers/internal/bosh_product_verifiers.rb:17:in `each'
/home/tempest-web/tempest/web/app/models/tempest/verifiers/internal/bosh_product_verifiers.rb:17:in `problem_steps'
/home/tempest-web/tempest/web/app/presenters/api/v0/staged/product_verifiers_presenter.rb:16:in `errors_as_hash'

The bosh_product_verifiers did not state specifically what condition is causing the verifier test to fail. If they had, this could point to a correctable condition. In this case, we theorized that a transient condition, such as a network blip, may have caused the failure, and we advised the customer to run Apply Changes again. The second time, Apply Changes complete successfully. If the error does not clearly point to a specific cause for the failure, open a case with Tanzu Support.