VIO How to troubleshoot a "No valid host was found. There are not enough hosts available." error.

Products

• VMware
• VMware Integrated OpenStack

Issue/Introduction

Symptoms:

• An instance is launched, and it will promptly fail.
• The error returned in the Horizon UI of VMware Integrated OpenStack is: "No valid host was found. There are not enough hosts available."

Environment

VMware Integrated OpenStack 6.x
VMware Integrated OpenStack 4.x
VMware Integrated OpenStack 5.x
VMware Integrated Openstack 7.x

Cause

The instance ID or request ID must be tracked through the logs to find the true cause of the errors. For services nova-scheduler will have the most details.

The number of available hosts that OpenStack can deploy the instance to is zero. Here's a list of the example of causes:

• If nova-compute is down as a service we will start with zero hosts to deploy to.
• Resource quotas could be met. 
• A custom value in image or flavor metadata could be invalidating available compute or datastores.

Resolution

Note:  The message returned is generic and further troubleshooting must be done for the root cause.
There are several areas to check in 5.1:

• On the compute node where the instance was attempting to launch, review the /var/log/nova/nova-compute.log for errors related to the request id.
• There is a possibility all hosts were filtered in which we need to track the request ID or the instance ID in the /var/log/nova/nova-scheduler.log on the controllers.
• Check the services and components of VMware Integrated OpenStack for errors.
• Are there any ESXi host or storage failures?
• Is communication working with the NSX Manager or vCenter?
• Is the failure only happening with a specific message or metadata requirement?

In 6.0 or 7.x for VMware Integrated OpenStack:

• Collect the list of pods with: osctl get pods
• Find the request ID for the instance in the nova compute pod with: osctl log<compute pod UUID>
• Then search the nova-scheduler pod's logs the request, and find the result that invalidated available hosts: osctl log <nova-scheduler pod UUID>