"Unexpected VMware Cloud Director error. [ ... ] This operation is denied." error in the Events and Notifications view of Cloud Director Availibility 4.1

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• In the Events and Notifications view of the Cloud Director Availability Provider Portal, Settings and Events sections are missing and you see an error similar to:

Unexpected VMware Cloud Director error. [ UI-240237dd-####-####-####-########218-mH-5dbbbc15-####-####-####-########6ed ] This operation is denied.

• In the /opt/vmware/h4/cloud/log/cloud.log file on the Cloud Replication Management Appliance, you see entries similar to:

2021-01-27 08:50:32.012 DEBUG - [19cc1d95-####-####-####-########091] [health-check-5] h.c.s.VcloudSessionHealthCheckingService : Initiating vCD sessions health check.

2021-01-27 08:50:38.139 ERROR - [UI-240237dd-####-####-####-########218-mH] [https-jsse-nio-8443-exec-9] c.v.h.c.c.error.ExceptionAdvisorBase     : A GET request from  user[IP Address] to /config/notifications/provider failed.

com.vmware.vcloud.client.exception.VcloudException: (Major code = 403, minor code = ACCESS_TO_RESOURCE_IS_FORBIDDEN) - [ UI-240237dd-####-####-####-########218-mH-5dbbbc15-####-####-####-########6ed ] This operation is denied.
        at com.vmware.vcloud.client.VcloudClient.lambda$defaultErrorDeserializer$0(VcloudClient.java:247)
        at com.vmware.rest.client.AbstractRestClient.convert(AbstractRestClient.java:225)
        at com.vmware.vcloud.client.XmlRestClient.exchange(XmlRestClient.java:92)
        ...

• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file on the Cloud Director cell, you see entries similar to:

2021-01-27 10:50:38,122 | DEBUG    | pool-jetty-289322         | CSecurityManager               | Building security context for user [loginUserId = 156ec36f-####-####-####-########4bb, operatingOrgId = a93c9db9-####-####-####-########5f9, isSystemAdministrator = true, canSwitchOrgContext= true] | requestId=UI-240237dd-####-####-####-########218-mH-5dbbbc15-####-####-####-########6ed,request=GET https://vcd_address/api/admin/user/ff079561-####-####-####-########6b1/grantedRights,requestTime=1611737438099,remoteAddress=10.204.98.47:48638,userAgent=Apache-HttpClient/4.5.13 (Java/11.0.9),accept=application/*+xml;version 31.0
2021-01-27 10:50:38,130 | DEBUG    | pool-jetty-289322         | CSecurityManager               | Access attempted for disabled userid = 156ec36f-####-####-####-########4bb. | requestId=UI-240237dd-####-####-####-########218-mH-5dbbbc15-####-####-####-########6ed,request=GET https://vcd_address/api/admin/user/ff079561-####-####-####-########6b1/grantedRights,requestTime=1611737438099,remoteAddress=<VCDA_IP>:48638,userAgent=Apache-HttpClient/4.5.13 (Java/11.0.9),accept=application/*+xml;version 31.0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue can occur when you have configured Cloud Director email notifications in Cloud Director Availability and there are disabled System Administrator users in Cloud Director.

When Cloud Director email notifications are enabled, Cloud Director Availability attempts to retrieve the email addresses for all Cloud Director System Administrator users. When there are disabled System Administrator users present, Cloud Director Availability is unable to retrieve their email addresses resulting in this behavior.

Resolution

This issue is resolved in Cloud Director Availability 4.2, available at VMware Downloads.

Workaround:
To work around this issue if you do not want to upgrade, enable all System Administrator users in Cloud Director.