apigw.log is flooded with the warning messages "com.vmware.vapi.std.errors.Unauthenticated"

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

• In certain conditions, the syslog server may flag invalid authentication warnings from the vCenter Server in its dashboards.
• The events will be reflected under logs, location , /var/log/vmware/vsphere-ui/logs/apigw.log
  
  

  [YYYY-MM-DDTHH:MM:SS] [ERROR] ns-poll-executor r0023072 ###### 205602 ApiGwApiProvider$AsyncHandleSandbox [sub:xxxx-xxxx-xxxx-xxxx-xxxx] Failed to forward com.vmware.cdc.change_log.get_changes to http://localhost:1080/wcp : com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
  [YYYY-MM-DDTHH:MM:SS] [ERROR] ns-poll-executor r0023097 ###### 205608 ApiGwApiProvider$AsyncHandleSandbox [sub:xxxx-xxxx-xxxx-xxxx-xxxx] Failed to forward com.vmware.cdc.change_log.get_changes to http://localhost:1080/wcp : com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
  [YYYY-MM-DDTHH:MM:SS] [ERROR] ns-poll-executor r0023097 ###### 205608 ApiGwApiProvider$AsyncHandleSandbox [sub:xxxx-xxxx-xxxx-xxxx-xxxx] Failed to forward com.vmware.cdc.change_log.get_changes to http://localhost:1080/wcp : com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
  [YYYY-MM-DDTHH:MM:SS] [ERROR] ns-poll-executor r0023097 ###### 205608 ApiGwApiProvider$AsyncHandleSandbox [sub:xxxx-xxxx-xxxx-xxxx-xxxx] Failed to forward com.vmware.cdc.change_log.get_changes to http://localhost:1080/wcp : com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {

Environment

VMware vCenter Server 7.0

Cause

The errors are caused by a rare case in which, after destroying the API Gateway session, the corresponding polling tasks of the NotificationService that retrieve changes from the WCP service continue to run. This means the subscriptions have failed to be removed for some reason. After three unsuccessful retries for each subscription, it automatically gets removed. This could be occurring due to a race condition.

Resolution

This warning can be safely ignored and you may suppress the event in the monitoring tool, as these events have no impact nor cause any issues.
Filter the event in your monitoring tool with the parameter "Failed to forward com.vmware.cdc.change_log.get_changes"

To prevent these errors from occurring, explicitly unsubscribing all the subscriptions in NotificationService for a given API GW session and not relying upon the consumer to unsubscribe manually.