Unable to remediate passwords for the NSX accounts in VCF Operation UI due to error "Remediate for appliance <FQDN>has failed. Failed to perform specified operation on SDDC manager".

Products

• VMware SDDC Manager / VCF Installer
• VCF Operations

Issue/Introduction

• On remediating NSX user credentials within VCF operation UI, the below error is observed:
  
  "Remediate for appliance <FQDN> has failed. Failed to perform specified operation on SDDC manager"

• Following log evidences are observed on SDDC Manager - less /var/log/vmware/vcf/operationsmanager/operationsmanager.log:
  
  YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_om,0000000000000000,0000] [c.v.v.p.r.DiagnosticMessageProcessor,Thread-134] Pre validation of password REMEDIATE has failed, error : Resources : [FQDN]are not in ACTIVE state
  YYYY-MM-DDTHH:MM:SSZ ERROR [vcf_om,0000000000000000,0000] [c.v.v.p.r.NewPasswordTransactionHandler,Thread-134] REMEDIATE operation with id : 1XX failed !
   
   
   

Environment

VCF 9.x

Cause

This issue occurs because the status of the NSX-T Manager is incorrectly recorded as an ERROR state in the SDDC Manager's platform database.

Resolution

• To remediate this issue, follow the below steps:
  • SSH into the SDDC Manager appliance using the vcf user and switch to the root user.
  • Confirm the status of the affected NSX node from the SDDC Manager by executing the below command:
    psql -h localhost -U postgres -d platform -c "select id, status, cluster_fqdn from nsxt where status!='ACTIVE';"
  • Note the id of the affected resource from the command output and Update the resource status to ACTIVE by executing the applicable command (replace <id> with the exact ID obtained from previous step):
    psql -h localhost -U postgres -d platform -c "update nsxt set status='ACTIVE' where id='<id>';"
  • Refresh the VCF Ops UI and retry the password remediation task.

Note: Ensure to take snapshot of the SDDC Manager VM before proceeding with database modifications.

Contact Broadcom Technical Support for further assistance or queries using Contact Broadcom Support