The lookup_password utility is fails to retrieve passwords in the SDDC manager VM.

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

• An error may occur when using the lookup_password utility, preventing login to the SDDC manager.
• Errors similar to the following may also be found in /var/log/vmware/vcf/sddc--manager-ui-app/sddcManagerServer.log:
  
  

  YYYY-MM-DDTHH:MM:SS.395+0000 ERROR [664de4da36c443e1, 8189a31f6e084dd8] [services/pscUtils. js, init-pscs, attemptPSCInit :62] Caught error from await primaryPsc Init
  YYYY-MM-DDTHH:MM:SS.396+0000 WARN [664de4da36c443e1, 8189a31f6e084dd8] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry : ###] ###. ###: VError : PSC Initilization attempt "5" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: 500 - "\"Error executing remote command via SSH: WARNING: Your password has expired. nPassword change required but no TTY available. \"" at Object. initializationPscError (/opt/VMware/vcf/sddc-manager-ui-app/server/src/errors/VCFError. js : ###:#) at attemptPSCInitWithRetry (/opt/VMware/vcf/sddc-manager-ui-app/server/src/services/pscUtils. js: 99:46)at process. processTicksAndRejections (node: internal/process/task_queues: 95:5 Error Info: {"retryCount":5, "status":403, "errorModule":100, "errorCode": 109}
  caused by : ##.###: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: 500 - "\"Error executing remote command via SSH: WARNING: Your password has expired. nPassword change required but no TTY available. \""

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

• The lookup_password utility is failing due to an issue with token creation.
• This issue may arise due to one or more of the following factors.
  
  • The root password on the vCenter has expired.
  • The certificates on the vCenter have expired.
  • The vCenter SSO password has been changed.

Resolution

The following steps are only applicable in cases where the local account has not yet been set up and the user cannot generate a token using the regular SSO credentials. 

1. SSH to SDDC Manager using vcf user credentials and then switch to the root account, and run the following commands.
   

   mkdir -p /etc/security/local
chown root:vcf_services /etc/security/local
chmod 650 /etc/security/local
echo -n "" > /etc/security/local/.localuserpasswd
chown root:vcf_services /etc/security/local/.localuserpasswd
chmod 660 /etc/security/local/.localuserpasswd

   # Set the new password in place of ########
echo -n "########" | openssl dgst -sha512 -binary | openssl enc -base64 | tr -d '\n' > /etc/security/local/.localuserpasswd


2. After setting up the local account, run the lookup_password utility using the local account, and use the same password configured in Step #1.

   lookup_passwords
   
   Output:

   Password lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Administration Guide for setting up ADMIN user.

   Supported entity types: ESXI VCENTER PSC NSX_MANAGER NSX_CONTROLLER NSXT_MANAGER NSX_ALB NSXT_EDGE VRSLCM VRLI VROPS VRA WSA BACKUP VXRAIL_MANAGER AD
Enter an entity type from above list: PSC
Enter page number (optional):
Enter page size (optional, default=50):
Enter Username: admin@local
Enter Password: ########
        PSC
        identifiers: ###.###.###.###,###.####.###
        workload: ###-###-###
                username: [email protected]
                password: ########
                type: SSO
                account type: SYSTEM

     Page : 1/1, displaying 1 of total 1 entities in a page.
   
   

Note: If the local account password needs to be changed after the environment is recovered (e.g., after retrieving credentials), it can be done by following the steps outlined in Update SDDC Manager Local Account Password