After disabling and enabling Organization Deployments are failing due to Extensibility error

Products

• VCF Automation

Issue/Introduction

After you disable the Organization name and enable it you notice Deployments may fail with extensibility error

Extensibility triggered task failed. Event ID: ##################. Failure: Extensibility error received for topic compute.allocation.pre, eventId = ' ##################': [] SubscriberID: service-account-vro-gateway-serviceaccount, RunnableID:  ################## and SubscriptionID: sub_ ################## failed with the following error: com.vmware.automation.vro.gateway.common.RemoteVroGatewayException: VCF Operations Orchestrator Gateway received error 500 INTERNAL_SERVER_ERROR for request: /sessions/exchange error:

{"status":"500 INTERNAL_SERVER_ERROR","serverMessage":"Internal server error!","serverErrorId":" ##################"}

Environment

VMware Cloud Foundation Automation 9.0.x

Cause

Session service in tenant portal uses OIDC refresh tokens issued by Tenant Manager to allow user vRO workflows to make API calls with user tokens. Disabling a tenant clears existing OIDC refresh tokens for this tenant and in versions before 9.1 session service is not aware when the tenant is disabled. This causes errors to be thrown by Tenant Manager when a workflow / deployment attempts to acquire a user token.

Resolution

This issue is fixed in VMware Cloud Foundation Automation 9.1 release where the session service in Tenant Portal is aware of when a tenant is modified and makes it so refresh tokens are re-issued when users login after modification of the tenant.

Workaround for VMware Cloud Foundation Automation 9.0.x : Clear the sessions table in project_db. This causes sessions to be re-created with new refresh tokens. This needs to be performed every time a tenant is disabled. If there are deployments running at the time the table is cleared, they will fail.

To clear the table:

1. Log in to VCFA over ssh using the vmware-system-user account. Once logged in, a root shell can be opened by running sudo -i

2. Set the kubeconfig path:
export KUBECONFIG=/etc/kubernetes/admin.conf

3. run:

kubectl exec -n prelude $(kubectl get pods -n prelude -l spilo-role=master,application=spilo -o jsonpath='{.items[0].metadata.name}') -c postgres -- psql -U postgres -d project_db -c "delete from session;"

4. Users should re-login