SDDC UI unreachable, after removing ELM, when upgrading to vSphere 9 - UI error "Internal Server Error"

Products

• VMware SDDC Manager

Issue/Introduction

Is some scenarios, in vSphere 9.X, when breaking the ELM as per https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/what-is/points-to-consider-while-setting-up-vmware-cloud-foundation-sso/deactivate-enhanced-link-mode--elm--for-upgraded-vmware-cloud-foundation-vcenters.html we run into the following error.

SDDC UI unreachable, login redirects to the Management vCenter, however the SDDC UI hangs on "Internal Server Error".

Under /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log, we see the following entry  

<DATE && TIME> ERROR [common,69931aad2da44f53d78a0e08c9fd9521,855e] [c.v.e.s.c.c.v.vsphere.VsphereClient,cs-exec-19] Failed to connect to https://<VCSA_FQDN>.com:443/sdk as svc-<USER>@<domain.tbd>
java.util.concurrent.ExecutionException: (vim.fault.NoPermission) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = e870c323-b05b-4709-a0f2-408ca63cf6c7,
   privilegeId = System.View,
   missingPrivileges = (vim.fault.EntityPrivileges) [
      (vim.fault.EntityPrivileges) {
         dynamicType = null,
         dynamicProperty = null,
         entity = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = e870c323-b05b-4709-a0f2-408ca63cf6c7,
         privilegeIds = (STRING) [
            System.View
         ]
      }
   ]
}
        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:122)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:548)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:508)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:521)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:475)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:549)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:62)
        at com.vmware.vcf.inventory.sync.utils.VcSyncManagerUtil.startMonitoringvCenter(VcSyncManagerUtil.java:55)
        at com.vmware.vcf.inventory.sync.services.impl.ClusterSyncAdapterImpl.lambda$startSyncService$0(ClusterSyncAdapterImpl.java:74)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.vim.binding.vim.fault.NoPermission: Permission to perform this operation was denied.
        at jdk.internal.reflect.GeneratedConstructorAccessor325.newInstance(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)

We can find the Service account user referenced in the vcf-commonsvcs.log in teh LDIF export 
See KB https://knowledge.broadcom.com/external/article/326305/how-to-export-vmdir-information-from-vce.html 

/opt/likewise/bin/ldapsearch -LLL -h localhost -p 389 -x -b "dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W  + '*'> $(hostname)_"`date +"%d-%m-%Y"`".ldif

 

Environment

SDDC 5.X upgrading to vSphere 9

Cause

As the error states the issue is permission related.
 

Resolution

Take appropriate [offline] snapshots as per https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html 
This should be a prerequisite when starting to remove/ split the ELM construct as per https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/what-is/points-to-consider-while-setting-up-vmware-cloud-foundation-sso/deactivate-enhanced-link-mode--elm--for-upgraded-vmware-cloud-foundation-vcenters.html 

Add the SVC account to the Administrators group via CLI.

# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add svc-<account crated by SDDC>

Restart SDDC services 

# /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh