Operations Manager fails to change the decryption passphrase
search cancel

Operations Manager fails to change the decryption passphrase

book

Article ID: 293594

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

Symptoms:
The following error is observed after changing the decryption passphrase in Ops Manager.  
500 An error occurred.
Contact Pivotal Technical Support to report the problem.
Back to dashboard

ActiveRecord::RecordInvalid
Validation failed: Deployment status is not included in the list
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/validations.rb:78:in `raise_validation_error'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/validations.rb:50:in `save!'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/attribute_methods/dirty.rb:43:in `save!'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:313:in `block in save!'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:233:in `transaction'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:210:in `transaction'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:313:in `save!'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/suppressor.rb:46:in `save!'
/home/tempest-web/tempest/web/app/models/installation_change.rb:104:in `block in reencrypt!'
/home/tempest-web/tempest/web/app/models/installation_change.rb:102:in `each'
/home/tempest-web/tempest/web/app/models/installation_change.rb:102:in `reencrypt!'
/home/tempest-web/tempest/web/app/models/use_cases/change_passphrase.rb:19:in `block (2 levels) in save'
/home/tempest-web/tempest/web/app/models/use_cases/change_passphrase.rb:16:in `tap'
/home/tempest-web/tempest/web/app/models/use_cases/change_passphrase.rb:16:in `block in save'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
/usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/transactions.rb:210:in `transaction'
/home/tempest-web/tempest/web/app/models/use_cases/change_passphrase.rb:14:in `save'
/home/tempest-web/tempest/web/app/controllers/encryption_passphrases_controller.rb:16:in `update'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/abstract_controller/base.rb:186:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:131:in `run_callbacks'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/rescue.rb:20:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications.rb:166:in `block in instrument'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/notifications.rb:166:in `instrument'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.6.1/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/abstract_controller/base.rb:124:in `process'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionview-5.1.6.1/lib/action_view/rendering.rb:30:in `process'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal.rb:189:in `dispatch'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_controller/metal.rb:253:in `dispatch'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/routing/route_set.rb:31:in `serve'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/journey/router.rb:33:in `each'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/journey/router.rb:33:in `serve'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/routing/route_set.rb:844:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:189:in `call!'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/strategy.rb:167:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/omniauth-1.6.1/lib/omniauth/builder.rb:63:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.6.1/lib/active_support/callbacks.rb:97:in `run_callbacks'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/railties-5.1.6.1/lib/rails/rack/logger.rb:36:in `call_app'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/railties-5.1.6.1/lib/rails/rack/logger.rb:26:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/request_id.rb:25:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/static.rb:125:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.6.1/lib/action_dispatch/middleware/ssl.rb:68:in `call'
/home/tempest-web/tempest/web/lib/rack/streaming.rb:63:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/railties-5.1.6.1/lib/rails/engine.rb:522:in `call'
/home/tempest-web/tempest/web/lib/stack_prof_middleware.rb:78:in `call'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'
/home/tempest-web/tempest/web/vendor/bundle/ruby/2.4.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:1077:in `block in spawn_threadpool'

Environment


Cause

This is a known issue that is fixed in Ops Manager version 2.3.6, and 2.2.11. 

The problem is triggered when there is a null field in the installation_changes table for field deployment_status. Certain tile updates cause the deployment_status field to be null. 

When this error is observed the stored value of the decryption passphrase in the Ops Manager database is updated with the new passphrase. However, the local files, installation.yml and actual-installation.yml are still encrypted with the original passphrase.

Resolution

There are two procedures to workaround this problem. 

Procedure 1 will override the Ops Manager decryption passphrase manually and allow Operators to recover the environment.

Procedure 2 should be run after Procedure 1 has been successfully implemented.  Procedure 2 prevents Operations Manager from failing to change the decryption passphrase.

IMPORTANT NOTE: Please ensure the following statements are true before proceeding with this procedure:

Procedure 1 - Recover Operations Manager

In this procedure, you will restore the original passphrase before a change was attempted.

1. First generate a bcrypt hash using your original passphrase that you tested decrypting the installation.yml and actual-installation.yml with.

a. https://bcrypt-generator.com/ 
b. Change the maximum number of rounds from 4 to 10. In this example, the password is set to 123456789 with the maximum number of rounds set to 10.
c.  In this example, the bcrypt generator inputs and output are as follows:
 
2. Insert the password hash result into the Ops Manager database.
a. Connect to the database using command sudo -u tempest-web psql tempest_production.
b. Create a backup of the application_unlock_infos table using the following command:
CREATE TABLE application_unlock_infos_backup AS select * from application_unlock_infos ;
c. Then update the application_unlock_infos tables with the password digest generated by the bcrypt calculator. 
UPDATE application_unlock_infos SET password_digest = (decode(encode('< Password Hash >', 'hex'), 'hex'));

3. Verify the password is inserted into the Ops Manager database.

tempest_production=# select convert_from(password_digest, 'UTF8') from application_unlock_infos;
                         convert_from
--------------------------------------------------------------
< Password Hash >
(1 row)

4. Restart tempest web using the command, sudo service tempest-web restart5. When it starts back up, unlock the database using the original decryption passphrase.


5. If the procedure is successful, reconnect to the Ops Manager database and remove the backup table. application_unlock_infos_backup.
DROP TABLE application_unlock_infos_backup;

Now that Ops Manager decryption passphrase is restored, you can now remove the null deployment_status column in the installation_changes table by updating all the rows that have this problem.

Procedure 2 - Prevent Operations Manager from failing again in the future

Follow the instructions from step 2 of Procedure 1 above to connect to the Ops Manager database.

1. Execute the following query to update all the rows that have a null deployment status.

update installation_changes set deployment_status='failed' where deployment_status is null;

2. Proceed to change the operations decryption passphrase.

Note: Pivotal Support recommends upgrading Ops Manager to the latest version to prevent this issue from happening in the future. 



Powered by Wolken Software