Multiple vCenter operations fails with error: A general system error occurred: PBM error occurred during "task": Fault cause: pbm.fault.InvalidLogin

Products

• VMware vCenter Server

Issue/Introduction

 

• PBM error while adding and expanding a vmdk or while trying to create a VM:

A general system error occurred: PBM error occurred during PreCreateCheckCallback: Fault cause: pbm.fault.InvalidLogin

• Unable to vMotion:

A general system error occurred: PBM error occurred during PreMigrateCheckCallback: Fault cause: pbm.fault.InvalidLogin
 

• The following entries are observed in the /var/log/vmware/vmware-sps/sps.log file:

yyyy-mm-ddThh:mm:ss.006-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.pbm.auth.impl.SessionManagerImpl - Logging in to PBM...
yyyy-mm-ddThh:mm:ss.017-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.identity.token.impl.SamlTokenImpl - SAML token for SubjectNameId [[email protected] format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
yyyy-mm-ddThh:mm:ss.022-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.identity.token.impl.SamlTokenImpl - Error parsing SAML token.
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
        at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:755)
        at com.vmware.identity.token.impl.SamlTokenImpl.<init>(SamlTokenImpl.java:277)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:69)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:77)
        at com.vmware.vim.storage.common.security.CommonSessionManager.parseSamlToken(CommonSessionManager.java:258)
        at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:158)
        at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
        at sun.reflect.GeneratedMethodAccessor1182.invoke(Unknown Source)
...
yyyy-mm-ddThh:mm:ss.022-06:00 [pool-3-thread-4] ERROR opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.vim.storage.common.VmodlErrorStrings - Login to PBM failed
com.vmware.vim.storage.common.fault.LoginException: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
        at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:176)
        at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
        at sun.reflect.GeneratedMethodAccessor1182.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:78)
        at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)
        at com.vmware.vim.storage.common.task.opctx.RunnableOpCtxDecorator.run(RunnableOpCtxDecorator.java:38)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
...
yyyy-mm-ddThh:mm:ss.022-06:00 [pool-3-thread-4] ERROR opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.vim.storage.common.VmodlErrorStrings - Login to SMS failed

Environment

VMware vCenter Server 8.0.x

VMware vCenter Server 7.0.x

Cause

Incorrect Service Registrations cause the PBM login to fail.

Resolution

Identify The Issue

1. List the service registrations on vCenter:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk --no-check-cert > /tmp/listservices.txt

2. Determine the environment's current and correct SSO domain

/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

Example:
vsphere.local

3. Identify the SSO domain in the listservices.txt file

cat /tmp/listservices.txt |grep -i "owner id"
 

If output does not match the SSO domain in Step 2, we are running into this issue.

Examples of incorrect SSO domain in listservices.txt:
VSPHERE.local
VSPHERE.LOCAL

Note: if SSO domain is custom from as seen in the get-domain-name command and SSO domain within listservices.txt shows vsphere.local, the issue is present here as well. 

Fix

1. Download lsdoctor from KB: Using the 'lsdoctor' Tool
2. Run lsdoctor -r > option 2 to replace all services
3. Restart all vCenter services: 

service-control --stop --all && service-control --start --all

Note: lsdoctor -r may not fix custom SSO domains. Manual export, edit, register of spec file of corresponding Service Registration will be required.