Error: "No valid host was found. There are not enough hosts available."

Products

• VMware Integrated OpenStack

Issue/Introduction

• An instance is launched, and it will promptly fail.
• The error returned in the Horizon UI of VMware Integrated OpenStack is: "No valid host was found. There are not enough hosts available."

Environment

7.x

Cause

The instance ID or request ID must be tracked through the logs to find the true cause of the errors. For services nova-scheduler will have the most details.

The number of available hosts that OpenStack can deploy the instance to is zero. Here's a list of the example of potential causes:  

Note:  This is not a complete list of potential causes.

• If nova-compute is down as a service we will start with zero hosts to deploy to.
• Resource quotas could be met. 
• A custom value in image or flavor metadata could be invalidating available compute or datastores.

Resolution

Note:  The message returned is generic and further troubleshooting must be done for the root cause.

• Collect the list of pods with: osctl get pods
• Find the request ID for the instance in the nova compute pod with: osctl log <compute pod UUID>
• Then search the nova-scheduler pod's logs the request, and find the result that invalidated available hosts: osctl log <nova-scheduler pod UUID>
• There is a possibility all hosts were filtered in which we need to track the request ID or the instance ID in the /var/log/nova/nova-scheduler.log on the controllers.
• Check the services and components of VMware Integrated OpenStack for errors.
• Are there any ESXi host or storage failures?
• Is communication working with the NSX Manager or vCenter?
• Is the failure only happening with a specific message or metadata requirement?
• Check Openstack to see what resources it has marked as used using the CLI:
  
  #openstack allocation candidate list --os-placement-api-version 1.10 --resource VCPU=1 --resource MEMORY_MB=2048 --resource DISK_GB=20
+---+----------------------------------+--------------------------------------+----------------------------------------------+
| # | allocation | resource provider | inventory used/capacity |
+---+----------------------------------+--------------------------------------+----------------------------------------------+
| 1 | VCPU=1,MEMORY_MB=2048,DISK_GB=20 | ########-####-####-####-############ | VCPU=3/40,MEMORY_MB=1536/36862,DISK_GB=3/699 |
+---+----------------------------------+--------------------------------------+----------------------------------------------+

[root@vioadmin1-vioshim-6dbdb59f56-mq7lb /]# openstack resource provider list
+--------------------------------------+-------------------------------------------------+------------+
| uuid | name | generation |
+--------------------------------------+-------------------------------------------------+------------+
|  ########-####-####-####-############| domain-c20.########-####-####-####-############ | 6 |
+--------------------------------------+-------------------------------------------------+------------+

[root@vioadmin1-vioshim-6dbdb59f56-mq7lb /]# openstack resource provider inventory list  ########-####-####-####-############

Password:
+----------------+------------------+----------+----------+-----------+----------+-------+
| resource_class | allocation_ratio | max_unit | reserved | step_size | min_unit | total |
+----------------+------------------+----------+----------+-----------+----------+-------+
| VCPU           |             10.0 |        8 |        0 |         1 |        1 |    16 |
| MEMORY_MB      |              1.5 |    20479 |        0 |         1 |        1 | 40958 |
| DISK_GB        |              1.0 |      670 |        0 |         1 |        1 |  1199 |
+----------------+------------------+----------+----------+-----------+----------+-------+
• If the allocation ratios need to be adjusted, see KB Changing allocation ratios