"A general system error occurred: Invalid fault" while deploying VM's on SDRS enabled cluster

Products

• VMware vCenter Server

Issue/Introduction

• Unable to deploy VM's from template or create VM's on SDRS enabled cluster.
• You will see similar entries in vpxd.log file location: /var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS.659Z info vpxd[01206] [Originator@6876 sub=vpxLro opID=k23k082r-######-auto-3r3m-h5:#######-2] [VpxLRO] -- BEGIN lro-2384xxxx -- StorageResourceManager -- vim.StorageResourceManager.recommendDatastores -- 52xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx(52xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
YYYY-MM-DDTHH:MM:SS.703Z info vpxd[01206] [Originator@6876 sub=vpxCrypt opID=k23k082r-######-auto-3r3m-h5:#######-2] Failed to read X509 cert; err: 151441516
YYYY-MM-DDTHH:MM:SS.749Z error vpxd[13815] [Originator@6876 sub=vmomi.soapStub[61447]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
YYYY-MM-DDTHH:MM:SS.749Z warning vpxd[13815] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
YYYY-MM-DDTHH:MM:SS.758Z info vpxd[01206] [Originator@6876 sub=vpxLro opID=k23k082r-######-auto-3r3m-h5:#######-2] [VpxLRO] -- FINISH lro-2384xxxx
YYYY-MM-DDTHH:MM:SS.758Z info vpxd[01206] [Originator@6876 sub=Default opID=k23k082r-######-auto-3r3m-h5:#######-2] [VpxLRO] -- ERROR lro-2384xxxx -- StorageResourceManager -- vim.StorageResourceManager.recommendDatastores: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Invalid fault"
-->    msg = "Received SOAP response fault from [<cs p:00007f86241235f0, TCP:localhost:10080>]: queryMapOfAttachedTags
--> com.vmware.identity.interop.ldap.ProtocolErrorLdapException"
--> }

• You will see similar entries in vpxd-svcs.log file location: /var/log/vmware/vpxd-svcs/vpxd-svcs.log

YYYY-MM-DDTHH:MM:SS.750Z [pool-8-thread-8  ERROR com.vmware.identity.interop.ldap.OpenLdapClientLibrary  opId=k23k082r-######-auto-3r3m-h5:#######-2] Exception when calling ldap_search_s: base=cn=urn:vmomi:InventoryServiceScope:default-scope:GLOBAL,cn=Tagging,cn=services,dc=vsphere,dc=local, scope=1, filter=(&(|)(objectClass=vmwTaggingCategoryModel)), attrs=[Ljava.lang.String;@691e9a41, attrsonly=0
com.vmware.identity.interop.ldap.ProtocolErrorLdapException: Protocol error
LDAP error [code: 2]
        at com.vmware.identity.interop.ldap.LdapErrorChecker$2.RaiseLdapError(LdapErrorChecker.java:55)

 

• From the above log snippet  (vpxd-svcs.log) exception raised when calling ldap_search on Tagging container.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

The issue is caused due to data inconsistency between tag associations on cis_kv_keyvalue table from VCDB and actual tags available in vmdird. This condition is usually seen when vCenter server is re-deployed or re-installed using existing DB.

Resolution

To resolve this issue, it is necessary to identify and clean up stale tag associations within the vCenter Server database.

We recommend contacting the Broadcom Technical Support team by raising a support case.