Failed to download support bundle on Ops Manager with error "undefined method `utc' for nil:NilClass"

Products

• Operations Manager

Issue/Introduction

When trying to fetch the support resource bundle on the Ops Manger web console, it fails with the  following error:

The following log messages could be seen in the Ops Manger log file (production.log):

I, [2020-07-20T05:45:46.381809 #779]  INFO -- : Completed 500 Internal Server Error in 35774ms (Views: 0.2ms | ActiveRecord: 15.6ms)
I, [2020-07-20T05:45:51.748518 #779]  INFO -- : BoshExecutor stop  cmd=/usr/local/bin/bosh --no-color --non-interactive --tty --environment=10.117.69.10 vms --vitals; pwd=/home/tempest-web/tempest/web exit_code=0
E, [2020-07-20T05:45:51.859525 #779] ERROR -- :         Exception caught via rescue_from (code 500):
        (See app/controllers/application_controller.rb)
        undefined method `utc' for nil:NilClass
    /home/tempest-web/tempest/web/app/controllers/api/v0/support_bundle_controller.rb:99:in `bundle_product_manifest_and_configs'
/home/tempest-web/tempest/web/app/controllers/api/v0/support_bundle_controller.rb:73:in `block in bundle_deployed_manifest_and_configs'
/usr/local/lib/ruby/2.6.0/delegate.rb:83:in `each'
/usr/local/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
/home/tempest-web/tempest/web/app/controllers/api/v0/support_bundle_controller.rb:72:in `bundle_deployed_manifest_and_configs'
/home/tempest-web/tempest/web/app/controllers/api/v0/support_bundle_controller.rb:17:in `show'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:194:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/home/tempest-web/tempest/web/app/controllers/application_controller.rb:172:in `emit_audit_log_event'

Environment

Product Version: 2.8

Resolution

The issue could arise if an older version of Ops Manager was upgraded to Ops Manager 2.7+, from which the support bundle feature was introduced. Some old installation records contain NULL value for one column called "started_at". When downloading support bundle. the started_at column with a NULL value might be referenced and caused the error.

Workaround

To workaround this issue, follow the steps below:

1. ssh into Ops Manger VM and connect to database:

$ sudo -u tempest-web psql tempest_production

2. Create a backup table first.

tempest_production=# create table installation_changes_backup as select * from installation_changes;

3. Check if there are any records with a NULL value of in the started_at column.

tempest_production=# select count(*) from installation_changes where started_at is null;

4. Start a transaction and update the NULL value with the created_at value from another table with same installation ID:

tempest_production=# begin;
tempest_production=# UPDATE installation_changes
   SET started_at = (SELECT created_at
                        FROM installs
                        WHERE installs.id = installation_changes.install_id)
   WHERE started_at IS NULL
     AND EXISTS(SELECT id
                        FROM installs
                        WHERE installs.id = installation_changes.install_id);
tempest_production=# end;

5. Verify that the support bundle can be downloaded successfully.