Password remediation failed for the NSX node root, admin and audit

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Error snippet:

Cause

This /etc/passwd error is expected when the ROOT credentials are not in sync with SDDC Manager. The SDDC Manager needs to login as ROOT to download the /etc/passwd file to change the API/AUDIT passwords. 

Which means the ROOT password needs to be remediated first.

However if the ROOT password is 123 on SDDC and the password was modified on NSX to 456 using passwd. The remediate will still fail for ROOT  in SDDC Manager with the  new (456) updated password - "failed to authenticate with the Guest OS"

This is because the root password has not fully changed in NSX despite using passwd command. 

Logs:

operationamanager.log

YYYY:MM:DDTHH:MM:SS.345+0000 ERROR [vcf_om,69956701ef9cf0460fa63147050b85c0,be2a] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-7] Failed to validate (vim.vm.guest.NamePasswordAuthentication) {
   dynamicType = null,
   dynamicProperty = null,
   interactiveSession = false,
   username = root,
   password = (not shown)
} credentials on <hostname>, error: Failed to authenticate with the guest operating system using the supplied credentials.
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed to validate (vim.vm.guest.NamePasswordAuthentication) {
   dynamicType = null,
   dynamicProperty = null,
   interactiveSession = false,
   username = root,
   password = (not shown)
} credentials on <hostname>, error: Failed to authenticate with the guest operating system using the supplied credentials.
.

.

.

YYYY-MM-DDTHH:MM:SS.489+0000 ERROR [vcf_om,691ca62d38a15d375f0c8b864f826ad9,f1d1] [c.v.v.p.v.u.CredentialsValidationTaskExecutor,om-exec-2] Error occurred in
 validation tasks: com.vmware.vcf.passwordmanager.exception.Fault: Failed executing command in VM.
java.util.concurrent.ExecutionException: com.vmware.vcf.passwordmanager.exception.Fault: Failed executing command in VM.
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at com.vmware.vcf.passwordmanager.validation.utils.CredentialsValidationTaskExecutor$1.call(CredentialsValidationTaskExecutor.java:133)
        at com.vmware.vcf.passwordmanager.validation.utils.CredentialsValidationTaskExecutor$1.call(CredentialsValidationTaskExecutor.java:84)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.vmware.vcf.passwordmanager.exception.Fault: Failed download /etc/passwd in VM with FQDN <hostname>, error: Failed to authenticate with the guest operating system using the supplied credentials.
        at com.vmware.vcf.passwordmanager.helper.GuestProgramService.download(GuestProgramService.java:296)
        at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.validatePassword(NsxtEdgeChanger.java:145)
        at com.vmware.vcf.passwordmanager.update.changers.NsxtEdgeChanger.doTest(NsxtEdgeChanger.java:111)
        at com.vmware.vcf.passwordmanager.service.PasswordValidationService.testPasswordValidity(PasswordValidationService.java:644)
        at com.vmware.vcf.passwordmanager.service.PasswordValidationService.validatePasswordForEntity(PasswordValidationService.java:453)
        at jdk.internal.reflect.GeneratedMethodAccessor1187.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:702)
        at com.vmware.vcf.passwordmanager.service.PasswordValidationService$$SpringCGLIB$$0.validatePasswordForEntity(<generated>)
        at com.vmware.vcf.passwordmanager.validation.utils.CredentialsValidationTaskExecutor$2.call(CredentialsValidationTaskExecutor.java:295)
        at com.vmware.vcf.passwordmanager.validation.utils.CredentialsValidationTaskExecutor$2.call(CredentialsValidationTaskExecutor.java:290)
        ... 5 common frames omitted
Caused by: com.vmware.vim.binding.vim.fault.InvalidGuestLogin: Failed to authenticate with the guest operating system using the supplied credentials.

Resolution

• From SDDC manager fetch the password from the lookup_passwords for NSX edge node.
  (For example if the password on the edge node in SDDC shows VMware123!)
• Set the same ROOT password(VMware123!) on the EDGE node using document : https://knowledge.broadcom.com/external/article/316043/nsx-edge-nodes-disconnected-in-password.html  
• From SDDC Manager - run UPDATE on the passwords page and provide a new password for ROOT
• Once ROOT password is in sync/the UPDATE operation is successful.
• Follow the same steps for admin and audit user as well if needed.