Upgrade to 3.2.X fails with "Management service is not available. The credentials were incorrect or the account specified has been locked."
search cancel

Upgrade to 3.2.X fails with "Management service is not available. The credentials were incorrect or the account specified has been locked."

book

Article ID: 322509

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • Upgrade fails during the preparing for upgrade phase of an upgrade of NSXT to 3.2.X.
  • Can happen when upgrading to 3.2.0, 3.2.0.1, and 3.2.1.
  • In the NSX-T manager log /var/log/upgrade-coordinator/upgrade-coordinator.log we see the following entries:

2022-06-02T22:03:58.524Z INFO http-nio-127.0.0.1-7442-exec-5 UpgradeCoordinatorFacadeImpl 1307 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Converted Uc Upgrade status UcUpgradeStatus{percentage=0, status=FAILED, progressMessages=[], errorMessages=[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]} to UcUpgradeStatus{state='FAILED', progressPercentage='0', status='', progressMessages='[]', errors='[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]'}

2022-06-02T22:03:58.524Z INFO http-nio-127.0.0.1-7442-exec-6 UpgradeCoordinatorFacadeImpl 1307 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Converted Uc Upgrade status UcUpgradeStatus{percentage=0, status=FAILED, progressMessages=[], errorMessages=[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]} to UcUpgradeStatus{state='FAILED', progressPercentage='0', status='', progressMessages='[]', errors='[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]'}

  • In the NSX-T manager log /var/log/syslog we see the following entries:

2022-06-02T22:03:58.524Z nsx-mngr-01.corp.local NSX 1307 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Converted Uc Upgrade status UcUpgradeStatus{percentage=0, status=FAILED, progressMessages=[], errorMessages=[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]} to UcUpgradeStatus{state='FAILED', progressPercentage='0', status='', progressMessages='[]', errors='[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]'}

2022-06-02T22:03:58.524Z nsx-mngr-01.corp.local NSX 1307 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] message repeated 2 times: [Converted Uc Upgrade status UcUpgradeStatus{percentage=0, status=FAILED, progressMessages=[], errorMessages=[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]} to UcUpgradeStatus{state='FAILED', progressPercentage='0', status='', progressMessages='[]', errors='[[192.168.10.3] Management service is not available. The credentials were incorrect or the account specified has been locked.]'}]

  • In the NSX-T manager log /var/log/proxy/reverse-proxy.log we see the following entries:

2022-04-03T13:45:49.037Z ERROR https-jsse-nio-192.168.120.1-443-exec-13 NsxRestAuthenticationEntryPoint - - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] The credentials were incorrect or the account specified has been locked.

2022-04-03T15:59:10.901Z ERROR https-jsse-nio-192.168.120.1-443-exec-13 NsxRestAuthenticationEntryPoint - - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] The credentials were incorrect or the account specified has been locked.


Environment

VMware NSX-T Data Center

Cause

  • There is multiple RegistrationTokenDto.class files contained in jar files for each service, but only one will be used.
  • Some of the token files are not the correct ones.
  • If the wrong token file is randomly selected the issue will be present.
  • These incorrect token files are contained in all versions of NSXT, however it will only impact upgrades to 3.2.X because of the rolling upgrade feature introduced in this version.

Resolution

This issue is resolved in NSX-T 3.2.2 available at VMware downloads.

Workaround:

Note:
If you encounter this issue, that is that the Upgrade coordinator upgrade has failed, DO NOT change the Orchestrator node by running the following command set repository-ip, as it can lead to other issues, instead follow the workaround steps below.

The workaround for Local NSX-T managers

Note: The work around should only be followed on the impacted unified appliances. (Appliances getting the 403 errors with "The credentials were incorrect" error.

  • Follow the below procedure to remove the bad token files.
  • As root user on the NSX-T manager, run the below 2 commands to remove the incorrect token files.:
    • zip -d /opt/vmware/proxy-tomcat/webapps/ROOT/WEB-INF/lib/libreverse-proxy-compile.jar com/vmware/nsxapi/registrationtokenendpoint/dto/RegistrationTokenDto.class
    • zip -d /opt/vmware/proton-tomcat/webapps/nsxapi/WEB-INF/lib/libvmc-aws-core.jar com/vmware/nsxapi/registrationtokenendpoint/dto/RegistrationTokenDto.class
  • Then, restart these two services:
    • su admin -c restart service http
    • su admin -c restart service manager
  • After that, go back to the NSX UI and continue with the upgrade.


The workaround for Global NSX-T managers

Note: The work around should only be followed on the impacted unified appliances. (Appliances getting the 403 errors with "The credentials were incorrect" error.

Note: The workaround for Global NSX-T managers is slightly different, as the libvmc-aws-core.jar jar file is not used on global managers, only the libreverse-proxy-compile.jar file needs to be fixed if this issue occurs:

  • Follow the below procedure to remove the bad token files.
  • As root user, run the below command to remove the incorrect token files.:
    • zip -d /opt/vmware/proxy-tomcat/webapps/ROOT/WEB-INF/lib/libreverse-proxy-compile.jar com/vmware/nsxapi/registrationtokenendpoint/dto/RegistrationTokenDto.class
  • Then, restart these two services:
    • su admin -c restart service http
    • su admin -c restart service manager
  • After that, go back to the NSX UI and continue with the upgrade.


Note:
This may affect one or more managers. 
Please start the workaround with the NSX Manager showing in the error message.


Powered by Wolken Software