/var/log/log/vcf/domainmanager/domainmanager.log reveals a 404 Not Found error during a REST API call to the Workspace ONE broker.
These logs indicate that the VCF Domain Manager is failing a configuration drift check because it cannot find the SDDC Manager registered as an OAuth2 client within the Workspace ONE Broker.
While the security certificates are valid, the 404 Not Found response confirms that the required Client ID is missing from the authorization server.
YYYY-MM-DDTHH:MM:SS.528+0000 DEBUG [vcf_dm,########,4222] [c.v.vcf.vapi.vsphere.WS1BRestClient,dm-exec-3] Executing REST request: Type GET, URL https://########/acs/t/########/broker/oauth2-clients/########YYYY-MM-DDTHH:MM:SS.528+0000 DEBUG [vcf_dm,########,4222] [c.v.v.s.h.VcfRestTemplateHttpClient,dm-exec-3] Http request connect timeout milliseconds 180000YYYY-MM-DDTHH:MM:SS.528+0000 DEBUG [vcf_dm,########,4222] [c.v.v.s.h.VcfRestTemplateHttpClient,dm-exec-3] Http request read timeout milliseconds 180000YYYY-MM-DDTHH:MM:SS.528+0000 DEBUG [vcf_dm,########,0000] [c.v.vcf.vapi.vsphere.VapiRestClient,ForkJoinPool.commonPool-worker-3] Successfully executed REST request with body: , and received response with body: {"access_token":"########","token_type":"Bearer","expires_in":3599}YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_dm,########,0000] [c.v.vcf.vapi.vsphere.WS1BRestClient,ForkJoinPool.commonPool-worker-3] Successfully retrieved admin client token for ########YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_dm,########,0000] [c.v.vcf.vapi.vsphere.WS1BRestClient,ForkJoinPool.commonPool-worker-3] Executing REST request: Type GET, URL https://########/acs/t/########/broker/oauth2-clients/########YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_dm,########,0000] [c.v.v.s.h.VcfRestTemplateHttpClient,ForkJoinPool.commonPool-worker-3] Http request connect timeout milliseconds 180000YYYY-MM-DDTHH:MM:SS.529+0000 DEBUG [vcf_dm,########,0000] [c.v.v.s.h.VcfRestTemplateHttpClient,ForkJoinPool.commonPool-worker-3] Http request read timeout milliseconds 180000YYYY-MM-DDTHH:MM:SS.530+0000 DEBUG [vcf_dm,########,4222] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-3] Security config retrieved {"fipsMode":false}YYYY-MM-DDTHH:MM:SS.531+0000 DEBUG [vcf_dm,########,0000] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,ForkJoinPool.commonPool-worker-3] Security config retrieved {"fipsMode":false}YYYY-MM-DDTHH:MM:SS.550+0000 DEBUG [vcf_dm,########,0000] [c.v.v.s.t.DynamicTrustManager,ForkJoinPool.commonPool-worker-3] Checking validity of certificate chain C=########, CN=########YYYY-MM-DDTHH:MM:SS.551+0000 DEBUG [vcf_dm,########,0000] [c.v.v.s.t.DynamicTrustManager,ForkJoinPool.commonPool-worker-3] Certificate chain C=########, CN=######## is validYYYY-MM-DDTHH:MM:SS.553+0000 DEBUG [vcf_dm,########,4222] [c.v.v.s.t.DynamicTrustManager,dm-exec-3] Checking validity of certificate chain C=########, CN=########YYYY-MM-DDTHH:MM:SS.557+0000 DEBUG [vcf_dm,########,4222] [c.v.v.s.t.DynamicTrustManager,dm-exec-3] Certificate chain C=########, CN=######## is validYYYY-MM-DDTHH:MM:SS.647+0000 ERROR [vcf_dm,########,0000] [c.v.v.v.v.SddcManagerRelyingPartyServiceImpl,ForkJoinPool.commonPool-worker-3] Caught exception while getting registered SDDC Manager as Relying Party 404 Not Found: "{"errors":[{"code":"oauth2.client.with.client.id.not.found","message":"OAuth2 Client with client id ######## does not exist","parameters":{"oauth2_client_id":"########"}}]}"org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"errors":[{"code":"oauth2.client.with.client.id.not.found","message":"OAuth2 Client with client id ######## does not exist","parameters":{"oauth2_client_id":"########"}}]}"at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:168)at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825)at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783)at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717)at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:608)at com.vmware.vcf.vapi.vsphere.WS1BRestClient.exchange(WS1BRestClient.java:110)at com.vmware.vcf.vapi.vsphere.WS1BRestClient.exchange(WS1BRestClient.java:50)at com.vmware.vcf.vapi.vsphere.SddcManagerRelyingPartyServiceImpl.isSddcManagerRegisteredAsRelyingParty(SddcManagerRelyingPartyServiceImpl.java:166)at com.vmware.vcf.configreconciler.config.checks.WorkspaceOneBrokerConfigDriftCheck.isSddcManagerRelyingParty(WorkspaceOneBrokerConfigDriftCheck.java:53)at com.vmware.vcf.configreconciler.config.checks.WorkspaceOneBrokerConfigDriftCheck.isConfigurationRealized(WorkspaceOneBrokerConfigDriftCheck.java:109)at com.vmware.vcf.configreconciler.service.ResourceCacheService.performIsConfigRealizedCheck(ResourceCacheService.java:1002)at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshFailedConfigRealizedCheck(ResourceCacheService.java:1043)at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshConfigApplicability(ResourceCacheService.java:922)at com.vmware.vcf.configreconciler.service.ResourceCacheService.lambda$refreshLocalInventoryWithConfigApplicability$0(ResourceCacheService.java:263)at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)YYYY-MM-DDTHH:MM:SS:39.647+0000 ERROR [vcf_dm,########,4222] [c.v.v.v.v.SddcManagerRelyingPartyServiceImpl,dm-exec-3] Caught exception while getting registered SDDC Manager as Relying Party 404 Not Found: "{"errors":[{"code":"oauth2.client.with.client.id.not.found","message":"OAuth2 Client with client id ######## does not exist","parameters":{"oauth2_client_id":"########"}}]}"org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: "{"errors":[{"code":"oauth2.client.with.client.id.not.found","message":"OAuth2 Client with client id ######## does not exist","parameters":{"oauth2_client_id":"########"}}]}"
VCF 5.x
The WorkspaceOneBrokerConfigDriftCheck failed because the vcf-sddc-manager-client-id could not be located on the identity broker.
Reconfigure the SDDC Manager Depot settings to re-establish the connection and recreate the missing client registration.
This populates the missing vcf-sddc-manager-client-id on the Workspace ONE broker, satisfying the realization check and allowing the drift check (WorkspaceOneBrokerConfigDrift) to pass.