NSX Host Upgrade Fails immediately with error "Unexpected error while upgrading upgrade unit: can't parse argument number: <EOL>"
Article ID: 390285
Updated On:
Products
Issue/Introduction
- NSX Host Upgrade Fails, and the upgrade status is Not Started.
- Below error are found in /var/log/upgrade-coordinator/upgrade-coordinator.log
XXXX-XX-XXTXX:XX:XX.XXXZ INFO ### UpgradeQueryServiceImpl 2766473 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For component type HOST componentUpgradeStatus is FAILED 0%
XXXX-XX-XXTXX:XX:XX.XXXZ INFO ### UpgradeQueryServiceImpl 2766473 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Returning upgrade status summary for HOST details as null.
XXXX-XX-XXTXX:XX:XX.XXXZ INFO ### UpgradeQueryServiceImpl 2766473 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [From DB] For component type MP componentUpgradeStatus is NOT_STARTED 0% XXXX-XX-XXTXX:XX:XX.XXXZ INFO ### FacadeInterceptorHelperImpl 1075133 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Caught error in facade interceptor java.lang.IllegalArgumentException: can't parse argument number: <EOL> "details" : "Some error has occurred."
Environment
VMware NSX-T Data Center 3.2.x
VMware vCenter Server 7.x
Cause
The NSX service user account fails to acquire a token and to log into vCenter because the vCenter VMDIR Service is in Read-Only Mode.
## vCenter
* /var/log/vmware/vmdird/vmdird-syslog.log
XXXX-XX-XXTXX:XX:XX err vmdird ###: InternalModifyEntry: VdirExecutePostModifyCommitPlugins - code(9114)
XXXX-XX-XXTXX:XX:XX err vmdird ###: VmDirSendLdapResult: Request (Modify), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in read-only mode), (0) socket (127.0.0.1)
XXXX-XX-XXTXX:XX:XX info vmdird ###: MOD 1,rep,certificateRevocationList: (-----BEGIN X509 CRL-----
* /var/log/vmware/vapi/endpoint/endpoint.log
XXXX-XX-XXTXX:XX:XXZ | WARN | sso8 | StsFacade | Call to STS failed
XXXX-XX-XXTXX:XX:XXZ | ERROR | sso8 | UserPassTokensRequest | Unable to acquire token for user nsxt_<Service User UUID>@vsphere.local
XXXX-XX-XXTXX:XX:XXZ | INFO | sso8 | BaseSessionImpl | Cannot authenticate user login request com.vmware.vapi.security.UserPassSecurityContext@1ebcd4de with error com.vmware.vapi.endpoint.sso.StsException: Call to STS failed.
XXXX-XX-XXTXX:XX:XXZ | WARN | jetty-default-32 | RequestRateLimitedProvider | User name cannot be obtained.
* /var/log/vmware/sso/svcaccountmgmt.log
XXXX-XX-XXTXX:XX:XXZ ERROR svcaccountmgmt[52:tomcat-http--19] [CorId=### OpId=] [com.vmware.vcenter.svcaccountmgmt.impl.PasswordMgmt] resetPassword: Got Directory Exception
XXXX-XX-XXTXX:XX:XXZ ERROR svcaccountmgmt[52:tomcat-http--19] [CorId=### OpId=] [com.vmware.vcenter.svcaccountmgmt.impl.DirectoryError] Internal Server Error, VMware directory error[9100]
XXXX-XX-XXTXX:XX:XXZ INFO svcaccountmgmt[36:tomcat-http--3] [CorId=### OpId=] [com.vmware.vcenter.svcaccountmgmt.impl.PasswordMgmt] Changing Account Password for account: nsxt_<Service User UUID>
Resolution
Changing the VMDIR Read-Only mode to Normal mode will fix the issue.
1. Access the vCenter CLI
2. Determine whether the vmdir service is in Read-Only mode:
-
/usr/lib/vmware-vmafd/bin/dir-cli state get
3. Change the state to Normal mode:
-
/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL
4. Confirm that the configuration has been changed:
-
/usr/lib/vmware-vmafd/bin/dir-cli state get
5. Retry the NSX Host Upgrade.
Feedback
