Enabling Ops Manager Rescue Mode 500 An error occurred. ActiveRecord::Encryption::Errors::Decryption
search cancel

Enabling Ops Manager Rescue Mode 500 An error occurred. ActiveRecord::Encryption::Errors::Decryption

book

Article ID: 367917

calendar_today

Updated On:

Products

VMware Tanzu Kubernetes Grid Integrated (TKGi) VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Kubernetes Grid Integrated Edition (Core) VMware Tanzu Kubernetes Grid Integrated Edition 1.x VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)

Issue/Introduction

When entering Ops Manager rescue mode after typing the decryption passphrase, the following error occurs:

500 An error occurred.

ActiveRecord::Encryption::Errors::Decryption
ActiveRecord::Encryption::Errors::Decryption

 

Symptoms

All conditions under symptoms must be met for the issue to be present:

  1. The error message is displayed after typing the decryption passphrase in rescue mode.
  2. The decryption of the installation YAMLs files succeeds.
  3. The bcrypt.checkpw function returns false for the hashed application_unlock_infos password_digest from the Ops Manager database.

 

Steps to confirm:

  1. When entering rescue mode in Ops Manager after typing the decryption passphrase, you see this error: 
    500 An error occurred.

    ActiveRecord::Encryption::Errors::Decryption
    ActiveRecord::Encryption::Errors::Decryption

    /home/tempest-web/tempest/web/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/encryption/encryptor.rb:581:in `rescue in decrypt'


  2. Decrypting of the installationYAMLs succeeds from Ops Manager with these two commands (run via SSH into Ops Manager):
    $ cd /tmp
    $ sudo -u tempest-web SECRET_KEY_BASE="key" RAILS_ENV=production /home/tempest-web/tempest/web/scripts/decrypt /var/tempest/workspaces/default/actual-installation.yml /tmp/decrypted-actual-installation.yml

    Expected Output:

    Decryption successful. File saved to /tmp/decrypted-actual-installation.yml

    $ sudo -u tempest-web SECRET_KEY_BASE="key" RAILS_ENV=production /home/tempest-web/tempest/web/scripts/decrypt /var/tempest/workspaces/default/installation.yml /tmp/decrypted-installation.yml

    Expected Output:

    Decryption successful. File saved to /tmp/decrypted-installation.yml


  3. Install pip and bcrypt (Note: The pip and Python versions may vary). If Ops Manager is restricted, use an alternate machine that has access to install these:
    sudo apt install python3-pip -y
    pip install bcrypt

    Using the same decryption passphrase from Step 2, bcrypt.checkpw returns false for the hashed application_unlock_infos password_digest from the Ops Manager database:

    IMPORTANT: Make a backup of the select statement output below. If the bcrypt.checkpw output below is False, we will be updating the database password_digest under the "Resolution" section.
    $ echo "select * from application_unlock_infos" | sudo -u tempest-web psql tempest_production

    id | password_digest | eula_accepted----+----------------------------------------------------------------------------------------------------------------------------+---------------1 | \x243262243132243735554e4b69674d573572614e6b2e3776764939712e664d6755687a46436464445a532e5a2e6e414f52594831746357427a6f7a53 | true(1 row)

    $ echo '243262243132243735554e4b69674d573572614e6b2e3776764939712e664d6755687a46436464445a532e5a2e6e414f52594831746357427a6f7a53' | xxd -p -r

    $2b$12$75UNKigMW5raNk.7vvI9q.fMgUhzFCddDZS.Z.nAORYH1tcWBzozS

    $ python3.10 -c 'import bcrypt; password = b"same decryption passphrase"; hashed = b"$2b$12$75UNKigMW5raNk.7vvI9q.fMgUhzFCddDZS.Z.nAORYH1tcWBzozS"; print(bcrypt.checkpw(password, hashed))'

    False

Environment

Ops Manager versions 2.x, 3.x

Cause

The error occurs when the passphrase for the Ops Manager installation YAMLs and database password_digest do not match.

Resolution

Ensure you are logged into the same Ops Manager VM or machine where pip and bcrypt were installed.


Generate a bcrypt hash of the passphrase that matches the successful decryption of YAMLs:

$ python3.10 -c 'import bcrypt; pw = "password that matches successful decrypt of yamls"; print(bcrypt.hashpw(pw.encode("utf-8"), bcrypt.gensalt(rounds=12)))'

$ echo -n '$2b$12$3ktko62x3pnFE4nc2oFOH.Yu4FwBd4foUyX2rc/adbGpgsZBFPyxi' | xxd -p | tr -d '\n'


Update the passphrase in the Ops Manager database (must be run via SSH into Ops Manager):

$ sudo -u tempest-web psql tempest_production

tempest_production=# update application_unlock_infos set password_digest = '\x243262243132243735554e4b69674d573572614e6b2e3776764939712e664d6755687a46436464445a532e5a2e6e414f52594831746357427a6f7a53' where id = 1;

tempest_production=# quit

$ sudo service tempest-web restart

 

Close all browsers and then login to the Ops manager UI using the decryption passphrase that was updated into the database.

Additional Information

Powered by Wolken Software