New deployment fails with glance pods in a CrashLoopBackOff state

Products

• VMware Integrated OpenStack

Issue/Introduction

• Glance pods went in a CrashLoopBackOff state during new deployments

osget po | grep glance
glance-api-xxxxx             1/2 CrashLoopBackOff 13 47m
glance-vmw-replicator-xxxxx  0/1 CrashLoopBackOff 13 47m

• glance-api.log
  May 21 11:21:11 controller-xxxxx  glance-api[968]: ERROR: HTTPSConnectionPool(host='vcenter.example.com', port=443): Max retries exceeded with url: /sdk/vimService.wsdl (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f40900ee890>: Failed to establish a new connection: [Errno -2] Name or service not known'))
• glance-vmw-replicator.log
  May 21 11:16:17 controller-xxxxxxx glance-vmw-replicator[968]: 2024-05-21 11:16:17.378 1 ERROR glance dns.resolver.NoNameservers: All nameservers failed to answer the query vcenter.example.com.
• Health check shows glance-api pod alarm.

Name: vio health check
+-------+----------+----------------------------+
| NAME | RESULT | ALARM |
+-------+----------+----------------------------+
| basic | Alarms:1 | Pod glance-api unreachable |
| | Passed:3 | |
+-------+----------+----------------------------+

Cause

This is an external DNS server configuration issue.

Resolution

Check your networking and DNS server configuration.

• The controllers are using the VIO API network as the default gateway.
  Please make sure VIO API network gateway can access your DNS server and that your DNS server is configured correctly. 
• Verify reply from your server with dig command.
  dig +short  @<dns server ip> vcenter.example.com
• Check if DNS port 53 is open with a netcat tool.

Example
root@controller-mw4qdzc74l [ ~ ]# nc -zv <your DNS server IP> 53 
ad.example.com [192.168.x.x] 53 (domain) open

• Capture and analyze DNS traffic from controller VM to DNS server.

Additional Information

Check our KB 'Using the pktcap-uw tool in ESXi 5.5 and later'