"Roles Cache not initialised correctly.. java.lang.NumberFormatException" resulting in custom roles not replicating across vCenter Servers in ELM

Products

VMware vCenter Server

Issue/Introduction

Custom role created on one node in a enhanced linked mode (ELM) vCenter Server (VC) not listed on the other VC nodes
Validating the replication using vdcrepadmin does not report VMDir replication inconsistency. Refer to Determining replication agreements and status with the Platform Services Controller (PSC)

/var/log/vmware/vpxd-svcs/vpxd-svcs.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS [refresh-lotus-locator-task [] INFO com.vmware.cis.lotus.LotusLocator opId=] Lotus hostname URL : <VCSA FQDN>
YYYY-MM-DDTHH:MM:SS [refresh-lotus-locator-task [] INFO com.vmware.cis.lotus.LotusLocator opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
YYYY-MM-DDTHH:MM:SS [refresh-lotus-locator-task [] INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
YYYY-MM-DDTHH:MM:SS [pool-5-thread-1 [] FATAL com.vmware.cis.core.authz.accesscontrol.impl.RolesLotusCache opId=] Get Roles from Lotus Store failed - Roles Cache not initialised correctly.. java.lang.NumberFormatException: null

Cause

In the vmdird (Lotus) schema, every role object is expected to carry an AuthzRoleVersion attribute which acts as a schema-enforcement mechanism. If a role exists but lacks this parameter, the LDAP Search Filter fails to parse the object, resulting in the "Get Roles from Lotus Store failed" exception.

Resolution

To resolve the issue, Role with the vmwAuthzRoleVersion parameter missing needs to be removed.

Note: Ensure a backup or an offline-snapshot (in powered-off state) of the vCenter Server is present. In case if the VC is in ELM, powered-off snapshot of all the nodes are required.

Log in to VC using ssh
Execute the below commands to list the Role ID with the RoleVersion parameter missing

read -sp "LDAP Password: " LDAP_PASS; DOMAIN_DN=$(cat /etc/vmware/install-defaults/vmdir.domain-name | sed 's/\./,dc=/g;s/^/dc=/'); ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=services,$DOMAIN_DN" -D "cn=Administrator,cn=Users,$DOMAIN_DN" -w "$LDAP_PASS" -LLL -o ldif-wrap=no "(&(objectClass=vmwAuthzRole)(!(vmwAuthzRoleVersion=*)))" dn | grep '^dn:' | awk '{print $2}'

Note: The command would prompt for SSO administrator password (e.g [email protected])

Sample Output:

root@vcenter [ / ]# read -sp "LDAP Password: " LDAP_PASS; DOMAIN_DN=$(cat /etc/vmware/install-defaults/vmdir.domain-name | sed 's/\./,dc=/g;s/^/dc=/'); ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=services,$DOMAIN_DN" -D "cn=Administrator,cn=Users,$DOMAIN_DN" -w "$LDAP_PASS" -LLL -o ldif-wrap=no "(&(objectClass=vmwAuthzRole)(!(vmwAuthzRoleVersion=*)))" dn | grep '^dn:' | awk '{print $2}'
LDAP Password:
cn=-########,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local

Delete the ID listed in the output of Step 2

read -sp "LDAP Password: " LDAP_PASS; DOMAIN_DN=$(cat /etc/vmware/install-defaults/vmdir.domain-name | sed 's/\./,dc=/g;s/^/dc=/'); /opt/likewise/bin/ldapdelete -D "cn=administrator,cn=users,$DOMAIN_DN" -w "$LDAP_PASS" "cn=########,cn=RoleModel,cn=VmwAuthz,cn=Services,$DOMAIN_DN"

Note: In the above command, cn=######## is the value from Step 2

Perform the same command on all the vCenter Server in ELM
Restart the services on all the vCenter server post the affected Roles are cleared

service-control --stop --all && service-control --start --all

Additional Information

In order to manually compare the entries, use the below commands to count the entries for Roles with vmwAuthzRoleVersion being present

Count of Role entries under Authz

ldapsearch -H ldaps://localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -b "cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local" "(objectclass=vmwAuthzRole)" -W | grep "dn: c" | wc -l Count of Role entries under Authz with Role Version parameter

ldapsearch -H ldaps://localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -b "cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local" "(objectclass=vmwAuthzRole)" -W | grep vmwAuthzRoleVersion | wc -l