Enterprise PKS cluster creation fails to apply NSX properties to VM 'vm-xxxxxx' with error: The object was modified by somebody else

Products

• VMware

Issue/Introduction

Symptoms:

• Enterprise PKS cluster creation fails intermittently
• The version of Opsman in use is below 2.9.2
• You see an error similar to the following:

Error: Unknown CPI error 'Unknown' with message 'The object was modified by somebody else. Please retry.' in 'create_vm' CPI method 

 

• You see an error similar to the following when running bosh task <task_number> --cpi  log:

I, [2020-07-17T11:04:19.054478 #1252]  INFO -- [req_id cpi-794570]: LogicalPorts found for vm 'vm-########-####-####-####-########9b9a': ["########-####-####-####-########83b1"]'
I, [2020-07-17T11:04:19.054598 #1252]  INFO -- [req_id cpi-794570]: Adding LogicalPorts: ["########-####-####-####-########83b1"] to NSGroup '########-####-####-####-########a762'
I, [2020-07-17T11:04:19.115245 #1252]  INFO -- [req_id cpi-794570]: Failed to apply NSX properties to VM 'vm-########-####-####-####-########9b9a' with error: The object was modified by somebody else. Please retry.
I, [2020-07-17T11:04:19.115359 #1252]  INFO -- [req_id cpi-794570]: Deleting VM 'vm-########-####-####-####-########9b9a'...
I, [2020-07-17T11:04:19.115500 #1252]  INFO -- [req_id cpi-794570]: Deleting vm: vm-########-####-####-####-########9b9a

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware PKS 1.x

Cause

Cluster Creation is failing when CPI tries adding VMs to nsgroups(security groups) in NSX-T. This may happen if multiple CPI processes try to update a security group simultaneously. 
 

Some processes might succeed and add VMs to a security group and others might still have a stale copy of the security group since `retrieve nsgroups' is called only once before updating the nsgroups.

 

Resolution

The issue is fixed in Opsman version 2.9.2

Additional Information

https://github.com/cloudfoundry/bosh-vsphere-cpi-release/issues/256