"Failed to send create affinity rule requests to VC" error when creating DRS rules in VCD

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Affinity rule creation in Cloud Director fails with Internal Server Error
• The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file shows entries similar to:

ERROR  | Backend-activity-pool-10385 | FutureUtil           | Failed to send create affinity rule requests to VC
java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
    at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
    ...

ERROR  | task-service-activity-pool-11 | FutureUtil           | Task future is not complete.
com.vmware.ssdc.util.LMException: Internal Server Error
    ...
Caused by: java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
    at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
    ... 7 more

• The /opt/vmware/vcloud-director/logs/vcloud-container-info.log file shows entries similar to:

(vim.cluster.AntiAffinityRuleSpec) {
dynamicType = null,
dynamicProperty = null,
key = 1xxxx,
status = null,
enabled = true,
name = anti-affinity-rule,
mandatory = false,
userCreated = true,
inCompliance = null,
ruleUuid = DRS_Rule_UUID,
vm = (vim.VirtualMachine) [
 ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null,
 ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null
]
...
[vcId=VC_UUID, moref=domain-cxxxx], xxxxx-xxxx-xxxxx-xxxx-xxxxxx) |
java.lang.NullPointerException
    at com.vmware.ssdc.backend.util.VmAffinityRuleUtil.isShellVm(VmAffinityRuleUtil.java:198)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.getRuleResourcePoolMoref(ClusterComputeResourceListener.java:408)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.createDrsVmVmAffinityRule(ClusterComputeResourceListener.java:369)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.handleVmVmDrsRuleUpdates(ClusterComputeResourceListener.java:283)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.onClusterDrsRulesChanged(ClusterComputeResourceListener.java:137)
    at com.vmware.vcloud.inventory.events.impl.ClusterComputeResourceDrsRulesChangedEvent.dispatchTo(ClusterComputeResourceDrsRulesChangedEvent.java:31)
    ...

Environment

VMware Cloud Director for Service Provider 9.x

Cause

This issue can occur if one or more VMs, which are part of an affinity/anti-affinity rule, no longer exist in vCenter. Cloud Director reads the whole rule spec, and if one or more VMs in the spec are missing, a NullPointerException (NPE) is thrown.
In vCenter the invalid VMs will show with the name "null" in the affinity/anti-affinity rule.

Resolution

To resolve the issue invalid VMs showing as "null" in the affinity/anti-affinity rule in vCenter must be removed from the rule.

If there are a large number of affinity/anti-affinity rules in vCenter find the list of VMs in the rulespec, and then find which of those VMs are missing from the Cloud Director vm_inv table:

 

To count the total number of VMs in the spec:

$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u | wc -l

74

 

Then output the list of VMs into a file:

$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u  > /tmp/drsvmlist.txt

 

On the machine where the VCD database is, using the same file, count how many of these VMs exist in the vm_inv table - change the vc_id and database name accordingly:

# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line' and vc_id = 'VC_UUID';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep -v "(0 rows)" | wc -l

73

 

In this example, there are 74 VMs in the rulespec, but only 73 of these exist in the vm_inv table. To find which ones are missing from vm_inv:

# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep "(0 rows)"

vm-52147 (0 rows)

 

Once you have identified the missing VM(s), find the VM in the DRS rule in vCenter and if it shows "null", remove it from the rule.