ERROR | Backend-activity-pool-10385 | FutureUtil | Failed to send create affinity rule requests to VC
java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
...
ERROR | task-service-activity-pool-11 | FutureUtil | Task future is not complete.
com.vmware.ssdc.util.LMException: Internal Server Error
...
Caused by: java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
... 7 more
(vim.cluster.AntiAffinityRuleSpec) {
dynamicType = null,
dynamicProperty = null,
key = 1xxxx,
status = null,
enabled = true,
name = anti-affinity-rule,
mandatory = false,
userCreated = true,
inCompliance = null,
ruleUuid = DRS_Rule_UUID,
vm = (vim.VirtualMachine) [
ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null,
ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null
]
...
[vcId=VC_UUID, moref=domain-cxxxx], xxxxx-xxxx-xxxxx-xxxx-xxxxxx) |
java.lang.NullPointerException
at com.vmware.ssdc.backend.util.VmAffinityRuleUtil.isShellVm(VmAffinityRuleUtil.java:198)
at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.getRuleResourcePoolMoref(ClusterComputeResourceListener.java:408)
at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.createDrsVmVmAffinityRule(ClusterComputeResourceListener.java:369)
at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.handleVmVmDrsRuleUpdates(ClusterComputeResourceListener.java:283)
at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.onClusterDrsRulesChanged(ClusterComputeResourceListener.java:137)
at com.vmware.vcloud.inventory.events.impl.ClusterComputeResourceDrsRulesChangedEvent.dispatchTo(ClusterComputeResourceDrsRulesChangedEvent.java:31)
...
This issue can occur if one or more VMs, which are part of an affinity/anti-affinity rule, no longer exist in vCenter. Cloud Director reads the whole rule spec, and if one or more VMs in the spec are missing, a NullPointerException (NPE) is thrown.
In vCenter the invalid VMs will show with the name "null" in the affinity/anti-affinity rule.
To resolve the issue invalid VMs showing as "null" in the affinity/anti-affinity rule in vCenter must be removed from the rule.
If there are a large number of affinity/anti-affinity rules in vCenter find the list of VMs in the rulespec, and then find which of those VMs are missing from the Cloud Director vm_inv table:
To count the total number of VMs in the spec:
$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u | wc -l
74
Then output the list of VMs into a file:
$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u > /tmp/drsvmlist.txt
On the machine where the VCD database is, using the same file, count how many of these VMs exist in the vm_inv table - change the vc_id and database name accordingly:
# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line' and vc_id = 'VC_UUID';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep -v "(0 rows)" | wc -l
73
In this example, there are 74 VMs in the rulespec, but only 73 of these exist in the vm_inv table. To find which ones are missing from vm_inv:
# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep "(0 rows)"
vm-52147 (0 rows)
Once you have identified the missing VM(s), find the VM in the DRS rule in vCenter and if it shows "null", remove it from the rule.