Cannot place host into maintenance mode even though WCP service is in started state
search cancel

Cannot place host into maintenance mode even though WCP service is in started state

book

Article ID: 319384

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Cannot put host into maintenance mode from vCenter web client.
  • However, from host client it works.
    vpxd.log     
    [YYYY-MM-DDTHH:MM:SS] info vpxd[23194] [Originator@6876 sub=vpxLro opID=######-777-auto-lm-h5:70000263-36] [VpxLRO] -- BEGIN task-140043 -- host-64099 -- vim.HostSystem.enterMaintenanceMode -- ######-####-####-####-#########(######-####-####-####-#########)
[YYYY-MM-DDTHH:MM:SS] warning vpxd[23194] [Originator@6876 sub=MoHost opID=######-777-auto-lm-h5:70000263-36] [Invoke] Host 'localhost' Failed to acquire Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
[context]zKq7AVECAAAAANjkCgEXdnB4ZAAAEcAubGlidm1hY29yZS5zbwAATdwZAGX5GABHZBYBY/pTdnB4ZAABB0phAQV/xAHgWlkB4V5ZAbBgWQHgB1wBvBRcApoU7WxpYnZpbS10eXBlcy5zbwABbcx2AerIdQGZ2nUBUWV1AdI+dgB2viIA1BQjAKytKgOHfwBsaWJwdGhyZWFkLnNvLjAABL81D2xpYmMuc28uNgA=[/context]
[YYYY-MM-DDTHH:MM:SS] info vpxd[31190] [Originator@6876 sub=vpxLro opID=vb-161:auto-name-23:01-2e] [VpxLRO] -- BEGIN lro-16320 -- ResourceModel -- vim.dp.ResourceModel.queryBatch -- ######-####-####-####-#########(######-####-####-####-#########)
[YYYY-MM-DDTHH:MM:SS] info vpxd[31190] [Originator@6876 sub=vpxLro opID=vb-161:auto-name-23:01-2e] [VpxLRO] -- FINISH lro-16320
[YYYY-MM-DDTHH:MM:SS] info vpxd[23194] [Originator@6876 sub=MoHost opID=######-777-auto-lm-h5:70000263-36] WCP enterMaintenanceMode vAPI returns error: Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->    vapi.security.authentication.invalid<Unable to authenticate user>
-->
[YYYY-MM-DDTHH:MM:SS] error vpxd[23194] [Originator@6876 sub=MoHost opID=######-777-auto-lm-h5:70000263-36] [Delete] Failed to delete vAPI session. Error:
--> Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->    vapi.security.authentication.invalid<Unable to authenticate user>
 
  • wcpsvc.log
[YYYY-MM-DDTHH:MM:SS] debug wcp [opID=hok-auth-handler] Getting HOK signer; store: wcp, alias: wcp
[YYYY-MM-DDTHH:MM:SS] error wcp [opID=hok-auth-handler] STS Issue HOK request failed; err: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found :: More than one solution user found
:AttributeValue xsi:type="xsd:string">false</saml2:AttributeValue></saml2:Attribute></saml2:AttributeStatement></saml2:Assertion>', audience: vmware-tes:vapi, err: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found :: More than one solution user found
[YYYY-MM-DDTHH:MM:SS] error wcp [opID=vapi] Authentication failed.
[YYYY-MM-DDTHH:MM:SS] error wcp [opID=vapi] Unable to authenticate user
[YYYY-MM-DDTHH:MM:SS] debug wcp [opID=vapi] Sending response with output {"error":{"ERROR":{"com.vmware.vapi.std.errors.unauthenticated":{"data":{"OPTIONAL":null},"error_type":{"OPTIONAL":"UNAUTHENTICATED"},"messages":[{"STRUCTURE":{"com.vmware.vapi.std.localizable_message":{"args":[],"default_message":"Unable to authenticate user","id":"vapi.security.authentication.invalid","localized":{"OPTIONAL":null},"params":{"OPTIONAL":null}}}}]}}}}
 
  •  wcpsvc.log (may also show)
[YYYY-MM-DDTHH:MM:SS] debug wcp [ssolib/sts.go:87] [opID=hok-auth-handler] Getting HOK signer; store: wcp, alias: wcp
[YYYY-MM-DDTHH:MM:SS] error wcp [ssolib/sts.go:122] [opID=hok-auth-handler] STS Issue HOK request failed; err: ns0:FailedAuthentication: Invalid credentials
 
  • vmware-identity-sts.log
[YYYY-MM-DDTHH:MM:SS] ERROR sts[53:tomcat-http--21] [CorId=######-####-####-####-#########] [com.vmware.identity.idm.server.IdentityManager] Failed to find solution user by subject DN [OU=VMware Engineering,O=VMware,L=Palo Alto,ST=California,C=US,CN=CA] in tenant [vsphere.local]
[YYYY-MM-DDTHH:MM:SS] ERROR sts[53:tomcat-http--21] [CorId=######-####-####-####-#########] [com.vmware.identity.idm.server.ServerUtils] Exception 'java.lang.IllegalStateException: More than one solution user found'
java.lang.IllegalStateException: More than one solution user found
at com.vmware.identity.idm.server.provider.vmwdirectory.VMware
[YYYY-MM-DDTHH:MM:SS] ERROR sts[53:tomcat-http--21] [CorId=######-####-####-####-#########] [com.vmware.identity.idm.server.ServerUtils] Caught an unexpected exception java.lang.IllegalStateException: More than one solution user found


Environment

VMware vCenter Server 7.0.x

Cause

This issue can be caused due any one of the following reasons :

  1. Duplicate solution user certificate for the WCP service in vCenter linked mode setup. Replacing the solution user certificate via the certificate-manager utility would cause this issue and VMware vCenter Server 7.0 Update 2 and above versions has the fix for this issue.
  2. Expired WCP solution user certificate. By default all the solution user certificates has 10 years validity except for 'wcp' solution user. Only for 'wcp' solution user, the certificate validity was set to 2 years which caused this certificate getting expired when compared to other solution users.
  3. This issue is fixed in vCenter Server 7.0 U3 and above versions by changing the default validity of 'wcp' solution user to 10 years.

Resolution

Note: Take snapshot before proceeding.

If only WCP solution user certificate is impacted, follow the manual steps from KB "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" errors for the WCP service causing multiple workflow failures to regenerate only the WCP solution user certificate.

To regenerate all the solution user certificates, you can choose from options 5, 6 or 8 depending on customer environment.
For more information about how to generate the certificates refer below KBs:

How to use vSphere Certificate Manager to Replace SSL Certificates


Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA

Additional Information

To regenerate all the solution user certificates, you can choose from options 5, 6 or 8 depending on customer environment.
For more information about how to generate the certificates see the below KBs:


How to use vSphere Certificate Manager to Replace SSL Certificates


Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA

Powered by Wolken Software