vCenter Tags are not available after removing vCenter from ELM
Article ID: 400273
Updated On:
Products
Issue/Introduction
Issue:
vCenter was recently unlinked from a group of two vCenters in ELM mode
The vCenter that was removed from linked mode still sees the tags it had prior to the removal, but none of the tags can be added to a VM.
Additionally:
- If a new tag is created in an existing category the tag is not seen in the list of tags that can be added to a VM
- A new tag category cannot be created directly (but can be created inside the "New tag" section)
Environment
vCenter 7.0 U3
Cause
STS users needed to be recreated after vCenter was removed from ELM mode
In the vmware-identity-sts and vpxd-svcs log files you see errors related to solution users
/var/log/vmware/sso/vmware-identity-sts.log
INFO sts[39:tomcat-http--5] [CorId=#####-####-####-####-#########][com.vmware.identity.sts.InvalidCredentialsException] Censored exception
com.vmware.identity.sts.InvalidCredentialsException: Solution user's certificate does not match the one in BST!
at com.vmware.identity.sts.auth.impl.BSTAuthenticator.checkMatchingCertificate(BSTAuthenticator.java:230) ~[sts-7.0.0.jar:?]
at com.vmware.identity.sts.auth.impl.BSTAuthenticator.doAuthenticate(BSTAuthenticator.java:120) ~[sts-7.0.0.jar:?]
...
/var/log/vmware/vpxd-svcs/vpxd-svcs.log
[tomcat-exec-292 [] INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=404de701-63d5-4958-b17b-e4b72258f818] Provided credentials are not valid.
2025-06-04T15:16:02.221-04:00 [tomcat-exec-292 [] ERROR com.vmware.cis.server.authentication.impl.TokenLoginContext opId=404de701-63d5-4958-b17b-e4b72258f818] Failed to get a renewable act-as HoK token
com.vmware.cis.server.authentication.exception.TokenProviderException: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:60) ~[inventory-server.jar:?]
at com.vmware.cis.server.ssoauthentication.impl.AbstractTokenProvider.refreshSamlToken(AbstractTokenProvider.java:49) ~[inventory-server.jar:?]
at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.getValidSamlToken(SolutionTokenProvider.java:39) ~[inventory-server.jar:?]
at com.vmware.cis.server.authentication.impl.TokenLoginContext.getRenewableActAsToken(TokenLoginContext.java:131) [inventory-server.jar:?]
at com.vmware.cis.server.authentication.impl.TokenLoginContext.getVapiEndpointSessionId(TokenLoginContext.java:101) [inventory-server.jar:?]
at com.vmware.cis.authorization.impl.SessionAuthDataImpl.getVapiEndpointSessionId(SessionAuthDataImpl.java:61) [inventory-client.jar:?]
at com.vmware.cis.server.util.VapiConnectionManager.getVpxdVapiEndpointSession(VapiConnectionManager.java:532) [inventory-server.jar:?]
at com.vmware.cis.core.tagging.vapi.TagAssociationsProviderImpl.list(TagAssociationsProviderImpl.java:56) [inventory-server.jar:?]
...
Resolution
1. Create a powered off snapshot of the vCenter server
2. Review the VDT log and resolve any errors
- Using the VCF Diagnostic Tool for vSphere (VDT)
https://knowledge.broadcom.com/external/article/344917
3. Use lsdoctor to recreate the STS users
- Using the 'lsdoctor' Tool
https://knowledge.broadcom.com/external/article/320837/using-the-lsdoctor-tool.html - Start with --lscheck then --solutionusers
- --lscheck, this option checks for common issues in the lookup service. Does not make any changes to the environment. This will show issues found on any node in the SSO domain. See output for findings and path to JSON report.
- --solutionusers, This option is used to recreate solution users for a node. There are many reasons a solution user may be missing or inconsistent, but this script will delete any existing entry and recreate them from scratch.
- --solutionusers will show the following, confirm that you have taken the offline snapshot(s) to proceed
4. Restart the vCenter services
- service-control --stop
- service-control --start
5. Log into vCenter and the tag permissions will be fixed
6. Remove the snapshot(s)
Feedback
