Bad Gateway when trying to access Aria Automation portal after decomissioning domain controller /Updating DNS server

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• Bad Gateway message is seen when trying to access Aria Automation portal after decommissioning domain controller

• The etc/resolv.conf file only contains the entries for old decommissioned domain controllers, newly added domain controllers are not present in the file
  
  
• From the identity-service-app.log, we can see the following error:
  
  " ERROR identity-service [host='identity-service-app-xxxxxxxxxx-xxxxx' thread='reactor-http-epoll- 4' user='' org='' trace='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'] c.v.i.common.util.HealthUtilComponent.onErrorMarkDown:70 - nested exception is java.nio.channels.UnresolvedAddressException"

•  Inside the Identity service pods container, VIDM URL is not able to resolve the hostname: 
  
  connect identity service pod: kubectl exec -- bash -it <identity-service-app-pod-name> 

curl -k https://<vIDM-url>/SAAS/API/1.0/REST/system/health/heartbeat

curl: (6) Could not resolve host: <IDM-URL>

• The correct procedure has been followed to Update the DNS assignment for Aria Automation, despite this the issue persists.

Environment

VMware Aria Automation 8.x

Cause

Existing domain controllers have been decommissioned with new domain controller servers not yet added to the etc/resolv.conf files.

In addition to performing the procedure to Update the DNS assignment for Aria Automation, it is sometimes necessary to restart the coredns pods to have the etc/resolv.conf files updated to reflect the changes.

This issue occurs when the identity-service-app pod fails to resolve or establish connectivity with the vIDM. This typically indicates a DNS resolution failure or network connectivity issue between the pod and the vIDM endpoint.

 

Resolution

1. Take a snapshot without memory of your vRealize Automation / Aria Automation appliances before making any changes.
2. Log into the Automation appliance with the root user.
3. Run the following command to list the pods:

kubectl get pods --all-namespaces 

     Document the name of each coredns pod from the above command (if you have 3 nodes you will have 3 pods) 

     4.Restart the coredns pods using the following command:

kubectl -n kube-system delete pod coredns-xxxxx

Note: Replace coredns-xxxxxx with the pod names documented step 3

     5. Power off any existing decommissioned domain controller servers & confirm that you can log into the vRealize Automation / Aria Automation portal with the only the new      domain controllers active.