"Failed to send create affinity rule requests to VC" error when creating DRS rules in VCD
search cancel

"Failed to send create affinity rule requests to VC" error when creating DRS rules in VCD

book

Article ID: 322126

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

Symptoms:
  • Affinity rule creation in Cloud Director fails with Internal Server Error
  • The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file shows entries similar to:

ERROR  | Backend-activity-pool-10385 | FutureUtil           | Failed to send create affinity rule requests to VC
java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
    at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
    ...

ERROR  | task-service-activity-pool-11 | FutureUtil           | Task future is not complete.
com.vmware.ssdc.util.LMException: Internal Server Error
    ...
Caused by: java.lang.RuntimeException: Failed to configure drs Rule Id DRS_Rule_UUID in cluster ManagedObjectReference: type = ClusterComputeResource, value = domain-cxxxx, serverGuid = null
    at com.vmware.vcloud.val.internal.impl.ReconfigureClusterDrsRuleActivity$FinalPhase.invoke(ReconfigureClusterDrsRuleActivity.java:304)
    ... 7 more

  • The /opt/vmware/vcloud-director/logs/vcloud-container-info.log file shows entries similar to:

(vim.cluster.AntiAffinityRuleSpec) {
dynamicType = null,
dynamicProperty = null,
key = 1xxxx,
status = null,
enabled = true,
name = anti-affinity-rule,
mandatory = false,
userCreated = true,
inCompliance = null,
ruleUuid = DRS_Rule_UUID,
vm = (vim.VirtualMachine) [
 ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null,
 ManagedObjectReference: type = VirtualMachine, value = vm-morefID, serverGuid = null
]
...
[vcId=VC_UUID, moref=domain-cxxxx], xxxxx-xxxx-xxxxx-xxxx-xxxxxx) |
java.lang.NullPointerException
    at com.vmware.ssdc.backend.util.VmAffinityRuleUtil.isShellVm(VmAffinityRuleUtil.java:198)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.getRuleResourcePoolMoref(ClusterComputeResourceListener.java:408)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.createDrsVmVmAffinityRule(ClusterComputeResourceListener.java:369)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.handleVmVmDrsRuleUpdates(ClusterComputeResourceListener.java:283)
    at com.vmware.ssdc.backend.vimeventlistener.ClusterComputeResourceListener.onClusterDrsRulesChanged(ClusterComputeResourceListener.java:137)
    at com.vmware.vcloud.inventory.events.impl.ClusterComputeResourceDrsRulesChangedEvent.dispatchTo(ClusterComputeResourceDrsRulesChangedEvent.java:31)
    ...


Environment

VMware Cloud Director for Service Provider 9.x

Cause

This issue can occur if one or more VMs, which are part of an affinity/anti-affinity rule, no longer exist in vCenter. Cloud Director reads the whole rule spec, and if one or more VMs in the spec are missing, a NullPointerException (NPE) is thrown.
In vCenter the invalid VMs will show with the name "null" in the affinity/anti-affinity rule.

Resolution

To resolve the issue invalid VMs showing as "null" in the affinity/anti-affinity rule in vCenter must be removed from the rule.

If there are a large number of affinity/anti-affinity rules in vCenter find the list of VMs in the rulespec, and then find which of those VMs are missing from the Cloud Director vm_inv table:

 

To count the total number of VMs in the spec:

$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u | wc -l

74

 

Then output the list of VMs into a file:

$ sed -n '/AntiAffinityRuleSpec/,/NullPointerException/p' vcloud-container-info.log | grep "ManagedObjectReference: type = VirtualMachine, value = vm-[0-9]*, serverGuid = nul[l$\|l,$]" | grep -v 'requestId\|INFO' | awk -F "value = " '{print $2}' | cut -d \, -f1 | sort -u  > /tmp/drsvmlist.txt

 

On the machine where the VCD database is, using the same file, count how many of these VMs exist in the vm_inv table - change the vc_id and database name accordingly:

# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line' and vc_id = 'VC_UUID';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep -v "(0 rows)" | wc -l

73

 

In this example, there are 74 VMs in the rulespec, but only 73 of these exist in the vm_inv table. To find which ones are missing from vm_inv:

# cat /tmp/drsvmlist.txt | while read line ; do echo $line && echo "select * from vm_inv where moref = '$line';" | sudo -i -u postgres psql vcloud ; done | grep 'vm-\|row' | grep -v vmx | sed 'N;s/\n/ /' | grep "(0 rows)"

vm-52147 (0 rows)

 

Once you have identified the missing VM(s), find the VM in the DRS rule in vCenter and if it shows "null", remove it from the rule.