Configuration drift update fails during a HA reconfigure task in vCenter

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• The following is logged in /var/log/vmware/vcf/lcm/lcm.log

2021-04-21T11:31:09.382+0000 ERROR [vcf_lcm,ccd694bcc6f64191,df0d,upgradeId=9c0d8cb8-1ceb-4e6e-bad8-1ffa445e24e2,resourceType=THIRD_PARTY,resourceId=8ac9070c-b64b-49bf-a300-346007bdb5b0,bundleElementId=615d80d0-4b88-4f85-ad3d-280ddb970e28] [c.v.e.s.l.p.e.i.LcmThirdPartySWUpgradeService,ThreadPoolTaskExecut
or-10] received COMPLETED_WITH_FAILURE while polling /var/log/vmware/vcf/lcm/thirdparty/upgrades/9c0d8cb8-1ceb-4e6e-bad8-1ffa445e24e2/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade.status upgrade status with error code 0
 

• You see messages similar to the following in the /var/log/vmware/vcf/lcm/thirdparty/upgrades/9c0d8cb8-1ceb-4e6e-bad8-1ffa445e24e2/sddcmanager-migration-app/sddcmanager_migration_app_upgrade.log file on SDDC Manager:

2021-04-21 11:30:55.809  INFO [,,,] 15900 --- [ool-2-thread-12] c.v.e.s.c.c.v.vsphere.VcManagerBase      : Waiting for enabling HA feature on vCenter cluster VCF_Cluster to complete.
2021-04-21 11:30:57.812  WARN [,,,] 15900 --- [ool-2-thread-12] c.v.v.v.c.h.i.HttpProtocolBindingBase    : Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
2021-04-21 11:30:57.827 ERROR [,,,] 15900 --- [ool-2-thread-12] c.v.e.s.c.c.v.vsphere.VcManagerBase      : Task: (MOR:task-86858) (Name:reconfigureEx) Entity: (MOR:domain-c9) (Name:VCF_Cluster) is failed

2021-04-21 11:30:57.827 ERROR [,,,] 15900 --- [ool-2-thread-12] c.v.e.s.c.c.v.vsphere.VcManagerBase      : Task: (MOR:task-86858) (Name:reconfigureEx) Entity: (MOR:domain-c9) (Name:VCF_Cluster) is failed
com.vmware.vim.binding.vim.fault.InvalidDasConfigArgument: The setting of vmConfig is invalid for cluster VCF_Cluster.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0-internal]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0-internal]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0-internal]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0-internal]
        at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0-internal]
        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.StackContextFactory.newContext(StackContextFactory.java:113) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.FaultStackContext.getNestedContext(FaultStackContext.java:86) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallContext.beginElement(UnmarshallerImpl.java:355) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallContext.parse(UnmarshallerImpl.java:265) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallContext.parse(UnmarshallerImpl.java:218) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallContext.unmarshall(UnmarshallerImpl.java:201) ~[vlsi-core-gobuild.6.0.0-4652991.RELEASE.jar:na]
 

• In the vSphere Client, a number of Reconfigure Cluster tasks fail with the following error message:

The setting of is invalid for cluster VCF_Cluster
 

• Messages similar to the following are seen in /var/log/vmware/vpxd/vpxd.log on vCenter:

2021-04-21T11:30:55.808Z info vpxd[56394] [Originator@6876 sub=Default opID=4c3b254d] [VpxLRO] -- ERROR task-86858 -- domain-c9 -- vim.ComputeResource.reconfigureEx: vim.fault.InvalidDasConfigArgument:
--> Result:
--> (vim.fault.InvalidDasConfigArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = "vmConfigSpec",
-->    entry = "vmConfig",
-->    clusterName = "VCF_Cluster"
-->    msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.cluster.ConfigSpecEx) {
-->    vmSwapPlacement = <unset>,
-->    spbmEnabled = <unset>,
-->    defaultHardwareVersionKey = <unset>,
-->    desiredSoftwareSpec = (vim.DesiredSoftwareSpec) null,
-->    maximumHardwareVersionKey = <unset>,
-->    dasConfig = (vim.cluster.DasConfigInfo) null,
-->    dasVmConfigSpec = (vim.cluster.DasVmConfigSpec) [
-->       (vim.cluster.DasVmConfigSpec) {
-->          operation = "edit",
-->          removeKey = <unset>,
-->          info = (vim.cluster.DasVmConfigInfo) {
-->             key = 'vim.VirtualMachine:8de4ddd8-1163-4492-8c79-e4adcaf841e3:vm-7386',
-->             restartPriority = <unset>,
-->             powerOffOnIsolation = <unset>,
-->             dasSettings = (vim.cluster.DasVmSettings) {
-->                restartPriority = <unset>,
-->                restartPriorityTimeout = <unset>,
-->                isolationResponse = "clusterIsolationResponse",
-->                vmToolsMonitoringSettings = (vim.cluster.VmToolsMonitoringSettings) null,
-->                vmComponentProtectionSettings = (vim.cluster.VmComponentProtectionSettings) null
-->             }
-->          }
-->       },
-->       (vim.cluster.DasVmConfigSpec) {
-->          operation = "edit",
-->          removeKey = <unset>,
-->          info = (vim.cluster.DasVmConfigInfo) {
-->             key = 'vim.VirtualMachine:8de4ddd8-1163-4492-8c79-e4adcaf841e3:vm-7384',
-->             restartPriority = <unset>,
-->             powerOffOnIsolation = <unset>,
-->             dasSettings = (vim.cluster.DasVmSettings) {
-->                restartPriority = <unset>,
-->                restartPriorityTimeout = <unset>,
-->                isolationResponse = "clusterIsolationResponse",
-->                vmToolsMonitoringSettings = (vim.cluster.VmToolsMonitoringSettings) null,
-->                vmComponentProtectionSettings = (vim.cluster.VmComponentProtectionSettings) null
-->             }
-->          }
-->       }
-->    ],
-->    drsConfig = (vim.cluster.DrsConfigInfo) null,
-->    drsVmConfigSpec = <unset>,
-->    rulesSpec = <unset>,
-->    orchestration = (vim.cluster.OrchestrationInfo) null,
-->    vmOrchestrationSpec = <unset>,
-->    dpmConfig = (vim.cluster.DpmConfigInfo) null,
-->    dpmHostConfigSpec = <unset>,
-->    vsanConfig = (vim.vsan.cluster.ConfigInfo) null,
-->    vsanHostConfigSpec = <unset>,
-->    groupSpec = <unset>,
-->    infraUpdateHaConfig = (vim.cluster.InfraUpdateHaConfigInfo) null,
-->    proactiveDrsConfig = (vim.cluster.ProactiveDrsConfigInfo) null,
-->    inHciWorkflow = <unset>,
-->    cryptoConfig = (vim.cluster.CryptoConfigInfo) null
--> }
--> Arg modify:
--> true

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 4.x

Cause

The virtual machines referenced in the logging do not have the relevant overrides configured in HA.

Resolution

This is a known issue affecting VMware Cloud Foundation. Currently, there is no resolution.

Workaround:
To work around this, add a VM override setting to the affected virtual machines:

1. Connect to the vSphere Client.
2. Navigate to the Cluster object.
3. Under the Configure tab for the cluster, select the VM Overrides sub-tab which is under Configuration.
4. Click "Add..."
5. Select the relevant VMs from the list.
6. Set the VM restart priority to high and keep the HA isolation response unchecked.

The impacted virtual machines are not named in the logging. Their Managed Object IDs (MOIDs) are referenced instead (e.g: vm-7386). Check the vCenter MOB to determine which VMs these MOIDs belong to.

1. Connect to the vCenter MOB at the following address: https://vCenter_FQDN/mob
2. Under properties, click 'content'.
3. Select from the row 'rootFolder': 'group-dx (Datacenters)'
4. Select childEntity: 'datacenter-x (vcf-ds)'
5. Select vmFolder: 'group-vx (vm)'
6. Select the relevant group from the childEntity row: 'group-vxx'
7. The childEntity row will contain both the MOID and the hostname of the impacted VM(s) in the value column.