Newly deployed VMware Aria Automation Orchestrator appliances are experiencing pod startup failures.

Products

VMware Aria Suite

Issue/Introduction

Kubernetes Prelude namespace pods fail to start post new deployment of Aria Orchestrator appliance when setting NTP on initial configuration.
After deploying a new Aria Automation Orchestrator appliance, when running command kubectl get pods -n prelude in CLI on appliance, the following response can be seen:
root@ [ /var/log/vmware/prelude ]# kubectl get pods -n prelude
Config not found: /etc/kubernetes/admin.conf
The connection to the server localhost:8080 was refused - did you specify the right host or port?
When searching through the vracli-service-status.log the following error's can be seen:
root@ [ /var/log/vmware/prelude ]# tail -f vracli-service-status.log

2024-08-14 14:29:02,357 - service - INFO - Service kube-dns. There is no information about nodes.
2024-08-14 14:29:02,359 - service - INFO - Service etcd-service. There is no information about nodes.
2024-08-14 14:29:02,361 - service - INFO - Service kube-apiserver. There is no information about nodes.
2024-08-14 14:29:02,362 - service - INFO - Service kube-flannel-ds. There is no information about nodes.
2024-08-14 14:29:02,363 - service - INFO - Service health-reporting-service. There is no information about nodes.
2024-08-14 14:29:02,364 - service - INFO - Service kube-controller-manager. There is no information about nodes.
2024-08-14 14:29:02,365 - service - INFO - Service kube-proxy. There is no information about nodes.
2024-08-14 14:29:02,366 - service - INFO - Service kube-scheduler. There is no information about nodes.
2024-08-14 14:29:02,367 - service - INFO - Service predictable-pod-scheduler. There is no information about nodes.
2024-08-14 14:29:02,368 - service - INFO - Service kubelet-rubber-stamp. There is no information about nodes.
2024-08-14 14:30:02,139 - service.kube - ERROR - Error get status for all kubernetes items: Config not found: /etc/kubernetes/admin.conf\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?\n'
2024-08-14 14:30:02,186 - service.kube - ERROR - Error get status for all pods in all namespaces: Config not found: /etc/kubernetes/admin.conf\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?\n'

When attempting to access the controlcenter, it fails to load with the following image:

Environment

Aria Automation Orchestrator 8.x

Cause

Scenario 1:

This is a known issue present in the following OVF versions:

O11N_VA-8.12.0.30728-21620161_OVF10.ova
O11N_VA-8.16.2.34719-23466433_OVF10.ova
O11N_VA-8.17.0.35210-23787547_OVF10.ova
O11N_VA-8.18.0.35770-24024334_OVF10.ova

When setting an NTP server prior to the deployment the pod services never come up, and the control center is inaccessible.

Scenario 2:

Aria Automation Orchestrator IP address resolves to multiple FDQN's.

Resolution

Scenario 1:

NOTE: Do not assign an NTP server if deploying 8.10.x + as it causes the kube pods to not create successfully on firstboot. There is a known issue where setting NTP before firstboot causes the docker pods to not create. Add the NTP servers in the OVA properties after the initial docker containers are created, or run this command from the CLI after the initial firstboot after the docker pods are created (also detailed further down in doc during appliance configuration process):

'vracli ntp systemd --set <IP_Addr_1>,<IP_Addr_2>' after the firstboot completes

You can tail this log file after log file of initial power on to see if you are running into this -- /var/log/bootstrap/firstboot.log

Scenario 2:

The Aria Automation Orchestrator IP address must resolve to a unique FQDN.

To verify the DNS lookup, execute the following command:

nslookup <Aria Automation Orchestrator IP address>
nslookup <Aria Automation Orchestrator  FQDN>

Refer the Network Requirements for Automation Orchestrator on the below document to get more detials.

Automation Orchestrator system requirements