"Access is denied" error and Healthwatch v2.x causes a large amount of uaa.log output in Operations Manager

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The Ops Manager UAA is logging every unauthenticated call to Ops Manager when Healthwatch v2.x does the Ops Manager canary healthcheck.

It returns the following error and causes a large amount of of uaa.log output in Operations Manager:

[2021-04-20 00:00:15.324] uaa - 5124 [http-nio-127.0.0.1-8080-exec-1] .... DEBUG --- ExceptionTranslationFilter: Access is denied (user is anonymous); redirecting to authentication entry point
org.springframework.security.access.AccessDeniedException: Access is denied

Environment

Product Version: 2.10

Resolution

This is a known issue in Operations Manager v2.7.30 and v2.10.8, where uaa.log is full and not getting rotated. 

R&D determined an issue in Healthwatch v2.x that occurs when the blackbox exporter checks the Ops Manager URL.

R&D have slowed down the logging rate by not following redirects for http 2xx requests on the Healthwatch side in Healthwatch v2.1.1. They are targeting to have this bug fixed in the upcoming Heatlhwatch v2.2 release.

To work around this issue before the Healthwatch official fix comes out, R&D confirmed and recommended updating the Ops Manager FQDN on the Canary URLS page to be: <opsmanager-fqdn>/api/v0/info.

Note: This does significantly reduce the log load on Ops Manager that is generated by the Healthwatch Canary.