Nova compute Failed to establish a new connection: [Errno 113] No route to host

Products

• VMware Integrated OpenStack

Issue/Introduction

Symptoms:

• Nova compute pod(s) in CrashLoopBackOff

root@photon-machine [ ~ ]# osctl get pods | grep compute-0
compute-526e3ee4-c20803-compute-0 1/2 CrashLoopBackOff 84 2d12h

• Compute pod(s) showing error similar to:

++ jq -r --arg host compute-526e3ee4-c20856 -c ‘(.[] | select(.Binary == “nova-compute”) | select(.Host==$host)).State’
Unable to establish connection to https://172.17.241.100:8774/v2.1/ca050e82e9a84b319926d970006a02dc/os-services: HTTPSConnectionPool(host=‘172.17.241.100’, port=8774): Max retries exceeded with url: /v2.1/ca050e82e9a84b319926d970006a02dc/os-services (Caused by NewConnectionError(‘<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd5055b0f90>: Failed to establish a new connection: [Errno 113] No route to host’,))

Environment

6.x
7.x

Cause

nova-compute pod(s) are in crashing loop because nova-compute service takes a long time to start (setting its status to up), longer than the livenessProbe set on nova-compute pod. So nova-compute pod is restarted when livenessProbe limit reached before nova-compute service starts properly.

Resolution

This is fixed in VMware Integrated Openstack 7.0.

Workaround:

1. ssh into the manager node
2. Find the statefulset for compute.

root@oms [ ~ ]# osctl get statefulset |grep compute
compute-f154d6c2-c28-compute   1/1     64

3. Edit the statefulset. 

osctl edit statefulset compute-f154d6c2-c28-compute

4. change failureThreshold of nova-compute's livenessProbe to 30

        image: docker-registry.default.svc.cluster.local:5000/vmware/nova-compute:6.0.0.15329962
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /tmp/nova-compute-liveness.sh
          failureThreshold: 30 <<<<<<
          initialDelaySeconds: 50
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nova-compute