Bad Gateway when trying to access Aria Automation portal after decomissioning domain controller

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Bad Gateway message is seen when trying to access Aria Automation portal after decommissioning domain controller

• The etc/resolv.conf file only contains the entries for old decommissioned domain controllers, newly added domain controllers are not present in the file.
• The logfiles contain Unresolved Address Exceptions similar to:

org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is java.nio.channels.UnresolvedAddressException 

• The correct procedure has been followed to Update the DNS assignment for Aria Automation, despite this the issue persists.

Environment

VMware vRealize Automation 8.x

Cause

Existing domain controllers have been decommissioned with new domain controller servers not yet added to the etc/resolv.conf files.

In addition to performing the procedure to Update the DNS assignment for Aria Automation, it is sometimes necessary to restart the coredns pods to have the etc/resolv.conf files updated to reflect the changes.

 

Resolution

1. Take a snapshot without memory of your vRealize Automation / Aria Automation appliances before making any changes.
2. Log into the Automation appliance with the root user.
3. Run the following command to list the pods

kubectl get pods --all-namespaces 
 

Document the name of each coredns pod from the above command (if you have 3 nodes you will have 3 pods) 

     4. Restart the coredns pods using the following command

kubectl -n kube-system delete pod coredns-xxxxx

         Note: Replace coredns-xxxxxx with the pod names documented step 3

     5. Power off any existing decommissioned domain controller servers & confirm that you can log into the vRealize Automation / Aria Automation portal with the only the new domain controllers active.

 

Workaround: