Symptoms:
- Adding a host or a cluster to an existing VMware Cloud Foundation 3.7 workload domain fails
- A password rotation against all components in the affected workload domain was recently completed.
- No new metrics are being received by one or more vCenter Server adapters in vRealize Log Insight.
- You see messages similar to the following in the /var/log/vmware/domainmanager/domainmanager.log file on the SDDC Manager VM:
2019-02-28T09:52:49.878+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Making request: POST https://load-balancer.vrack.vsphere.local:9543/api/v1/sessions
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response: for POST request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/sessions, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"userId":"76fb1c40-2b33-####-####-####-########707","sessionId":"fllIpiI9O4ZNSk5S1FIwQAG1UjEt3PBzk6BEzLuuNzbeQrcBga8OyrcCBw6YdG/Le8bY8qp1x086Y2RA1lg5dmuMMBHQl5oX605M/DTR5tqVg0kTXlKk33GTAJ65gksbYnmRNIlO/jQPi5kLAOkeGkrxiyu6Kap9fEJNEQRsV+t9K2KNyUunh4K9xmnx/E63jsiSZ2F4DsgFqP3VqmuUc5SOoP6K9yK2uWQnonHb/dN9DSTpuU79m61I8WtuvOCrGJgn20nmim1k4eMgvaSD3w==","ttl":1800}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Starting PUT request from host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Making request: PUT https://load-balancer.vrack.vsphere.local:9543/api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts
2019-02-28T09:53:08.758+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response with status: 401 reason: PUT ... for Unauthorized request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:53:08.759+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"errorMessage":"Invalid username or password.","errorCode":"*****","errorDetails":{"errorCode":"com.vmware.loginsight.api.errors.vsphere.unauthorized"}}
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.s.l.i.LogInsightApiConfigurator,pool-2-thread-1] POST request to vRLI load-balancer.vrack.vsphere.local failed. Response code 401
2019-02-28T09:53:08.761+0000 WARN [601be10b47ffecbe,87ac] [c.vmware.vcf.domainmanager.AppConfig,pool-2-thread-1] Attempt #5: Retryable method public void com.vmware.evo.sddc.common.services.loginsight.impl.LogInsightApiConfigurator.retryPutRequest(java.lang.Object,java.lang.String,int) threw exception Failed post request in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.f.l.i.ConfigureVCenterInLogInsightWithSyslog,pool-2-thread-1] Failed to configure vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.764+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-1] [6AV1OO] VRLI_VSPHERE_HOSTS_CONFIGURATION_FAILURE Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.BaseVCenterInLogInsightConfiguration.execute(BaseVCenterInLogInsightConfiguration.java:84)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:36)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:19)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:164)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:151)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:324)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:417)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:96)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
at reactor.bus.EventBus$3.accept(EventBus.java:317)
at reactor.bus.EventBus$3.accept(EventBus.java:310)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.