VCSA patching fails at 80% with an error "Invalid Type, expected String, instead got NoneType"

Products

• VMware vCenter Server

Issue/Introduction

• Patching of vCenter in ELM fails at 80% with an error message
  "Invalid Type, expected String, instead got NoneType"
  
  
• Below error messages are observed in following logs:
  • var/log/vmware/applmgmt/PatchRunner.log 
YYYY-MM-DD HH:MM:SSZ trustmanagement:Patch INFO patch_02 Updating global permissions trust management
YYYY-MM-DD HH:MM:SSZ  trustmanagement:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'trustmanagement:Patch' failed.
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Failed to serialize response',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   reason = 'SystemError required property reason not set'
}
YYYY-MM-DD HH:MM:SSZ  ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.

• • var/log/vmware/vpxd-svcs/vpxd-svcs.log
YYYY-MM-DD HH:MM:SSZ  [dataservice-4 [] ERROR com.vmware.cis.core.authz.accesscontrol.impl.AuthorizationServiceInternalImpl opId=########-####-####-####-############] Store Exceptioncom.vmware.cis.core.authz.exception.StoreException: java.lang.Exception: Failed to add entry to lotus cn=Trust.Manage,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
        Caused by: java.lang.Exception: Failed to add entry to lotus cn=Trust.Manage,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
        Caused by: com.vmware.identity.interop.ldap.UnwillingToPerformLdapException: Server is unwilling to perform
YYYY-MM-DD HH:MM:SSZ  [completion-step-4 [] ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask opId=########-####-####-####-############] Failed to serialize response com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
        Caused by: com.vmware.v im.vmomi.core.exception.MarshallException: SystemError required property reason not set

• • var/log/vmware/vmdird/vmdird-syslog.log
YYYY-MM-DD HH:MM:SS err vmdird  t@###############: VmDirSendLdapResult: Request (Modify), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in read-only mode), (0) socket (127.0.0.1 )


• Verified that the partner status of the vCenters in linked mode is not working as expected with the following command:

  • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers -h localhost -u administrator
  • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator
  • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator


Environment

VMware vCenter Server 7.0.x

Cause

• VMDir status is in read-only state. It reverts to read-only mode after setting it to normal using the command : /usr/lib/vmware-vmdir/bin/vdcadmintool

Resolution

NOTE: Before applying the steps below, please take a snapshot (in powered-off state) of the vCenter Server Appliance. If the vCenter is part of a Linked Mode replication setup, please take offline snapshot all replicating nodes. 
Please refer VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Execute the below steps to fix the issue:

• Download the fix-psc-master script from the KB :VMDir State is Read Only with vmdir inconsistencies 
• Rebuild vmdir with the help of Healthy vCenter node. Run the following command on the broken unhealthy node pointing to the good healthy node as denoted by --healthy-psc-fqdn.

./fixpsc rebuild --healthy-psc-fqdn <Healthy vCenter FQDN>