Excessive apigw.log logging warning sent to Syslog server

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Excessive apigw.log log events are being sent to the syslog server continuously. 
• In vCenter var/log/vmware/vsphere-ui/logs/apigw.log file, similar log entries are available.
  [YYYY-MM-DDTHH:MM:SS.258-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.505-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.505-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.
[YYYY-MM-DDTHH:MM:SS.506-04:00] [WARN ] data-service-pool-784 70028635 101174 200061 ApiGwServicePrincipal [] The token with id '_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' for domain vsphere.local(yyyyyyyy_yyyy_yyyy_yyyy_yyyyyyyyyyyy) is unusable (EXPIRED). Will acquire a fresh one.

Environment

VMware vCenter Server 8.0.x

Cause

Possible issue with STS causing an endless loop in vsphere-ui service, which is causing the apigw.log filled up with expired token events.

Resolution

VMware is aware of this issue and working to fix it in a future release.

Workaround:

1. Take Snapshot of the vCenter server (Offline Snapshots if the vCenter servers are in Enhanced Linked Mode).
2. Backup the original file using below command.
   cp vmware-services-vsphere-ui.conf vmware-services-vsphere-ui.conf.backup
3. Edit vmware-services-vsphere-ui.conf file using below command.
   vi /etc/vmware-syslog/vmware-services-vsphere-ui.conf
4. Type i for insert.
5. Change the Severity for apigw.log from "info" to "Error"
6. To save and exit the file, press Esc :wq!
7. Restart the vsphere-ui & vmware-stsd services using below command.
   service-control --restart vsphere-ui
service-control --restart vmware-stsd