Error: "Resource CREATE failed: ResourcelnError resources(1].resources #######: Went to status ERROR due to "Message: Exceeded maximum number of retries" during instance creation failure.

Products

• VMware Integrated OpenStack
• VMware NSX
• VMware Telco Cloud Platform

Issue/Introduction

• Instance creation during scale out operations do not complete successfully. 
• The instances are part of heat stack template in VIO.
• There were DHCP configuration was disabled directly from the the NSX UI and not from openstack horizon dashboard.
• Nova conductor logs point to an exception at neutron service as below
  
  nova.exception.RescheduledException: Build of instance ########-####-####-####-############ was re-scheduled: Request Failed: internal server error while processing your request.\nNeutron server returns request_ids: ['req-########-####-####-####-############']\n"]
  
  
• Neutron logs show that there was a configuration change directly made at the NSX layer. 
  
  WARNING vmware_nsxlib.v3.cluster [req-########-####-####-####-############ - - - - -] Request failed due to: The object was modified by somebody else. Please retry. details: Operation failed because of conflicting transaction. Transaction ID:########-####-####-####-############ Address: 4846656856: vmware_nsxlib.v3.exceptions.StaleRevision: The object was modified by somebody else. Please retry. details: Operation failed because of conflicting transaction. Transaction ID: ########-####-####-####-############ Address: 4846656856

Environment

VIO 7.3
NSX 4.1.2.2
TCP 3.0 

Cause

• Modifying DHCP configurationchanges directly from NSX UI in is not supported when the infrastructure is managed by VIO.
• VIO  acts as the master controller for NSX. When a network is created via OpenStack, the OpenStack Neutron database stores the respective configuration.
• When any configuration is changed from NSX manuallythe neutron's database goes out of sync leading to deployment failures with 500 neutronclient.common.exceptions.InternalServerError Error.
  
  INFO neutron.wsgi [req-########-####-####-####-############ - ########-####-####-####-############] ###.##.###.###,127.0.0.1 "PUT /v2.0/ports/########-####-####-####-############ HTTP/1.1" status: 500  len: 363 time: 0.8419664

 

Resolution

Option 1: 

Disable the DHCP configuration directly from the openstack UI. 

1. Log in to the OpenStack Horizon Dashboard as an administrator or project member.
2. Navigate to Project > Network > Networks.
3. Click on the name of the network containing your target subnet.
4. Locate the subnet in the list and click Edit Subnet on the right side.
5. Uncheck the Enable DHCP checkbox in the subnet detail configuration.
6. Click Save.

Option 2:

Force a Network Update via OpenStack CLI.

1. Perform a dummy update to the network or subnet via the OpenStack CLI. This forces the OpenStack NSX plugin to re-fetch the latest object from NSX-T.
   
   openstack network set --description "Syncing with NSX" <network_name>
   
   
2. Disable DHCP using openstack CLI
   
   openstack subnet set --no-dhcp <subnet_name>