In vSphere 7.0, A general system error occurred: PBM error occurred during <task>: Fault cause: pbm.fault.InvalidLogin

Products

• VMware vCenter Server

Issue/Introduction

 

• PBM error while adding and expanding vmdk and while provisioning VMs

A general system error occurred: PBM error occurred during PreCreateCheckCallback: Fault cause: pbm.fault.InvalidLogin

• Unable to vMotion

A general system error occurred: PBM error occurred during PreMigrateCheckCallback: Fault cause: pbm.fault.InvalidLogin
 

• /var/log/vmware/vmware-sps/sps.log

yyyy-mm-ddT09:15:01.006-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.pbm.auth.impl.SessionManagerImpl - Logging in to PBM...
yyyy-mm-ddT09:15:01.017-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.identity.token.impl.SamlTokenImpl - SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
yyyy-mm-ddT09:15:01.022-06:00 [pool-3-thread-4] INFO opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.identity.token.impl.SamlTokenImpl - Error parsing SAML token.
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
        at com.vmware.identity.token.impl.SamlTokenImpl.parseTokenXmlToDom(SamlTokenImpl.java:755)
        at com.vmware.identity.token.impl.SamlTokenImpl.<init>(SamlTokenImpl.java:277)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:69)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:77)
        at com.vmware.vim.storage.common.security.CommonSessionManager.parseSamlToken(CommonSessionManager.java:258)
        at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:158)
        at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
        at sun.reflect.GeneratedMethodAccessor1182.invoke(Unknown Source)
...
yyyy-mm-ddT09:15:01.022-06:00 [pool-3-thread-4] ERROR opId=kitbjgme-70397-auto-1ibi-h5:70021757-29-01 com.vmware.vim.storage.common.VmodlErrorStrings - Login to PBM failed
com.vmware.vim.storage.common.fault.LoginException: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.
        at com.vmware.vim.storage.common.security.CommonSessionManager.loginByToken(CommonSessionManager.java:176)
        at com.vmware.pbm.auth.impl.SessionManagerImpl.loginByToken(SessionManagerImpl.java:44)
        at sun.reflect.GeneratedMethodAccessor1182.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:78)
        at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)
        at com.vmware.vim.storage.common.task.opctx.RunnableOpCtxDecorator.run(RunnableOpCtxDecorator.java:38)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.sso.client.exception.MalformedTokenException: Error parsing SAML token.

Environment

VMware vCenter Server 7.0.x

Cause

Incorrect Service Registrations cause the PBM login to fail.

Resolution

Identify The Issue

1. List the service registrations on vCenter:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk --no-check-cert > /tmp/listservices.txt

2. Determine the environment's current and correct SSO domain

/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost

Example:
vsphere.local

3. Identify the SSO domain in the listservices.txt file

cat /tmp/listservices.txt |grep -i "owner id"
 

If output does not match the SSO domain in Step 2, we are running into this issue.

Examples of incorrect SSO domain in listservices.txt:
VSPHERE.local
VSPHERE.LOCAL

Note: if SSO domain is custom from as seen in the get-domain-name command and SSO domain within listservices.txt shows vsphere.local, the issue is present here as well. 

Fix

1. Download lsdoctor from KB * Using the 'lsdoctor' Tool : https://knowledge.broadcom.com/external/article?articleNumber=320837
2. Run lsdoctor -r > option 2 to replace all services
3. Restart all vCenter services: 

service-control --stop --all && service-control --start --all

Note: lsdoctor -r may not fix custom SSO domains. Manual export, edit, register of spec file of corresponding Service Registration will be required.