"Password rotation fails with error" | "Unknown error occurred, check log further"
search cancel

"Password rotation fails with error" | "Unknown error occurred, check log further"

book

Article ID: 431578

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

"Password rotation fails with error" | "Unknown error occurred, check log further"

Environment

SDDC Manager 5.x

Cause

Password rotation for service account fails with error.

SDDC Manager logs : 

operationsmanager.log on SDDC Manager located at /var/log/vmware/vcf/operationsmanager/ shows entries similar to:

YYYY-MM-DDT00:00:00.342+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Update operation started asynchronously
YYYY-MM-DDT00:00:00.343+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Current stage UNKNOWN
YYYY-MM-DDT00:00:00.343+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Successfully obtained old credentials of VCFQDN
YYYY-MM-DDT00:00:00.343+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Is service account : VCENTER
YYYY-MM-DDT00:00:00.343+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Setting old and service credentials for account with entityId: 3eacc1bf-4654-4e85-a623-9eb468877014, entityName: VCFQDN, credentialType: SSO, username: VCSERVICEACCOUNTVMware vCenter Server@SSODOMAINNAME
YYYY-MM-DDT00:00:00.343+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.helper.CredentialHelper,om-exec-18] Fetching credentials for entityId==3eacc1bf-4654-4e85-a623-9eb468877014;credentialType==SSO;username==VCSERVICEACCOUNTVMware vCenter Server@SSODOMAINNAME;entityType==VCENTER
YYYY-MM-DDT00:00:00.445+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.helper.CredentialHelper,om-exec-18] Unable to find credentials for entity with Id 3eacc1bf-4654-4e85-a623-9eb468877014, status 500, exception com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:
        at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:788)
        at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.execute(ApiClient.java:708)

        at java.base/java.lang.Thread.run(Thread.java:840)
YYYY-MM-DDT00:00:00.449+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Failed to populate old and service credentials for account with entityId: 3eacc1bf-4654-4e85-a623-9eb468877014, entityName: VCFQDN, credentialType: SSO, username: VCSERVICEACCOUNTVMware vCenter Server@SSODOMAINNAME
com.vmware.vcf.passwordmanager.exception.LookupException:
        at com.vmware.vcf.passwordmanager.helper.CredentialHelper.queryCredentialsWithEntityIdCredentialTypeUsernameEntityType(CredentialHelper.java:759)
Caused by: com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException:

        ... 22 common frames omitted
YYYY-MM-DDT00:00:00.449+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18]
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException:
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:636)
        ... 8 common frames omitted
YYYY-MM-DDT00:00:00.449+0000 DEBUG [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Error Message : , Error Token : P77QC2, Error Cause : {}
YYYY-MM-DDT00:00:00.450+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Exception in Async Cannot invoke "com.vmware.cloud.foundation.rest.commonsvcs.model.Credential.getSecret()" because the return value of "com.vmware.vcf.passwordmanager.model.Entity.getOldCredential()" is null
java.lang.NullPointerException: Cannot invoke "com.vmware.cloud.foundation.rest.commonsvcs.model.Credential.getSecret()" because the return value of "com.vmware.vcf.passwordmanager.model.Entity.getOldCredential()" is null
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:649)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:201)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
YYYY-MM-DDT00:00:00.451+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-18] Unknown Error occurred : Cause Cannot invoke "com.vmware.cloud.foundation.rest.commonsvcs.model.Credential.getSecret()" because the return value of "com.vmware.vcf.passwordmanager.model.Entity.getOldCredential()" is null
YYYY-MM-DDT00:00:00.515+0000 ERROR [vcf_om,699b98806a7f7ee845a5211f01be5e3f,3b3b] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-18] Exception while performing Async Job for password operation: Unknown error occurred, check log further. Error : {0}.
com.vmware.vcf.passwordmanager.exception.PasswordManagerException: Unknown error occurred, check log further. Error : {0}.
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:250)


vcf-commonsvcs.log on SDDC Manager located under /var/log/vmware/vcf/commonsvcs/shows entries similar to:

YYYY-MM-DDT00:00:00.337+0000 INFO  [common,699b9880a2443b5bbab99a2db8702acd,d581] [c.v.e.s.i.s.EntityInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-715] Update Entity type - VCENTER Entity id - 3eacc1bf-4654-4e85-a623-9eb468877014 Status - ACTIVATING
YYYY-MM-DDT00:00:00.340+0000 INFO  [common,699b9880a2443b5bbab99a2db8702acd,d581] [c.v.e.s.i.r.a.c.ResourceInventoryController,http-nio-127.0.0.1-7100-exec-715] patchEntity(): id = 3eacc1bf-4654-4e85-a623-9eb468877014, type = VCENTER, status = ACTIVATING
YYYY-MM-DDT00:00:00.344+0000 INFO  [common,699b988013b8b4d3c381a88c1693c647,c986] [c.v.e.s.c.s.CredentialServiceImpl,http-nio-127.0.0.1-7100-exec-742] Query Credentials search entityId==3eacc1bf-4654-4e85-a623-9eb468877014;credentialType==SSO;username==VCSERVICEACCOUNTVMware vCenter Server@SSODOMAINNAME;entityType==VCENTER
YYYY-MM-DDT00:00:00.380+0000 ERROR [common,0000000000000000,0000] [o.a.c.c.C.[.[.[.[dispatcherServlet],http-nio-127.0.0.1-7100-exec-742] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: cz.jirutka.rsql.parser.RSQLParserException: cz.jirutka.rsql.parser.ParseException: Encountered " <UNRESERVED_STR> "vCenter "" at line 1, column 107.
Was expecting one of:
    <EOF>
    <AND> ...
    <OR> ...
    ] with root cause
cz.jirutka.rsql.parser.ParseException: Encountered " <UNRESERVED_STR> "vCenter "" at line 1, column 107.
YYYY-MM-DDT00:00:00.435+0000 ERROR [common,699b9880ba8d9a218c2062c1981313ad,3ede] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-742] [FAERGB] VCF_RUNTIME_ERROR Unhandled error caught. To get details about the error, search for [VONGO8] in the logs
com.vmware.evo.sddc.common.core.error.LocalizableRuntimeException: Unhandled error caught. To get details about the error, search for [VONGO8] in the logs
        at com.vmware.evo.sddc.exception.handler.VcfFallbackErrorController.defaultFallbackErrorHandler(VcfFallbackErrorController.java:69)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


Standard vCenter service account naming format : svc-<sddc-name>-<vcenter-name>
In this scenario, the service account name is having space : VCSERVICEACCOUNTVMware vCenter Server@SSODOMAINNAME

Due to the space, the RSQL query parsing failed, leading to:

  • Credential lookup failure
  • NullPointerException
  • Password rotation task failure
  • Database locks



 

Resolution

If you encounter the symptoms outlined above, contact Broadcom Technical Support for further assistance, as the resolution involves modifying database instance, which should be performed with caution.

Powered by Wolken Software