lockb.ekey is referred by Device server despite choosing Advanced Security during installation.

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

lockb.ekey is referred by Device server despite choosing Advanced Security 
Error "Lockbox Access Error: The primary internal decryption routine failed" is seen on console when lockbox is unlocked or when job is scheduled.

• Unable to run any user scheduled job like pull/push/compliance audit/Test credential.
• However system created job like healthcheck would work as it does not look for lockbox key 

Environment

All NCM versions below 24.x versions

Cause

Lockbox has corrupted entries due to which Device Server unable to recognize security selected during installation hence looks for default lockb.ekey.

Resolution

NOTE: Take backup of $VOYENCE_HOME/data/lockb.clb to non-NCM path ex: /tmp before proceeding with below steps.

Steps to remove any corrupted/inconsistencies which might have been created during installation/migration:

1. 1. Set environmental variable as 

      source /etc/voyence.conf

   2. Create a copy of the original lockbox from AS (lockb.clb) and change the name to csp.clb under $VOYENCE_HOME/lockbox/xml/ directory as: 

      cp $VOYENCE_HOME/data/lockb.clb $VOYENCE_HOME/lockbox/xml/csp.clb

   3. Navigate to $VOYENCE_HOME/bin, and run ./cstadmin list-hosts, provide the lockbox password here when prompted.
      This command will list details of all hosts which includes IP, hostname and FQDN that has been attached to lockbox ($VOYENCE_HOME/lockbox/xml/csp.clb)
   4. Find the entries that are relevant to the Device Server where issue (lockb.ekey is expected when advanced security is chosen) is seen, there will be multiple entries, all need to be removed one-by-one.
   5. To remove entries from the lockbox, run ./cstadmin remove-host <hostname>, provide the lockbox password here when prompted.
      Example: 

      ./cstadmin remove-host <IP_ADDRESS>
      OR
      ./cstadmin remove-host <HOSTNAME>

   6. Output will be similar to: 

      cstadmin: Host #.#.#.# removed from authorized host list for the Lockbox $VOYENCE_HOME/lockbox/xml

   7. Once all the relevant entries have been removed, run ./cstadmin list-hosts to double check and see if all hosts have been removed properly.

Step to add the hostname and IP again to the lockbox:

1. 1. Run ./cstadmin add-host <hostname>, provide the lockbox password here when prompted.
      Example: 

      ./cstadmin add-host <IP_ADDRESS>
      OR
      ./cstadmin add-host <HOSTNAME>

   2. Output will be similar to : 

      cstadmin: Host #.#.#.# added to authorized host list for the Lockbox $VOYENCE_HOME/lockbox/xml.

   3. Once the host has been added, again run ./cstadmin list-hosts to double check and see if the entry is present.

Step to copy lockbox to DS:

1. 1. Copy lockbox ($VOYENCE_HOME/lockbox/xml/csp.clb) to the Device Server whose detail (IP/hostname/FQDN) is just added and copy it under $VOYENCE_HOME/data/lockb.clb, make sure the copy is correctly done as 

      cp $VOYENCE_HOME/lockbox/xml/csp.clb $VOYENCE_HOME/data/lockb.clb

   2. Use the cksum command to verify and then change the permissions to root:cst
   3. Once copy has been done, and permissions set properly, unlock the lockbox using the command:
      1. Navigate to $VOYENCE_HOME/bin

      2. ./cstdriver -lockbox $VOYENCE_HOME/data/lockb.clb -passphrase <passphrase>

   4. Restart vcmaster services as service vcmaster restart, and schedule a job in problematic Device server.