Error: "A problem has occurred on the server. Please retry or contact the service provider and provide the reference token." when running clusterexpansion API

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

The cluster expansion API POST /v1/clusters/########-####-####-####-############/validations fails at the stage:

Validate Uplink Teaming Policy for Management Port Groups for L3 Hosts
From /var/log/vmware/vcf/domainmanager/domainmanager.log logfile, we see the following stacktrace:
ERROR [vcf_dm,###############################,####] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-15] method invocation target not found
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.getParams(ProcessingTaskSubscriber.java:312)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:354)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:540)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
at jdk.internal.reflect.GeneratedMethodAccessor786.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:209)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at com.vmware.vcf.hostmanager.adapters.clients.HostManagerParamBuilderClient.lambda$getDvsToMgmtPortGroupMoidToNames$7(HostManagerParamBuilderClient.java:375)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at com.vmware.vcf.hostmanager.adapters.clients.HostManagerParamBuilderClient.getDvsToMgmtPortGroupMoidToNames(HostManagerParamBuilderClient.java:366)
at com.vmware.vcf.hostmanager.adapters.clients.HostManagerParamBuilderClient.getManagementL3PortgroupUplinkTeamingValidationParam(HostManagerParamBuilderClient.java:395)
at com.vmware.vcf.hostmanager.adapters.ManagementL3PortgroupUplinkTeamingValidationParamBuilder.toPlugin(ManagementL3PortgroupUplinkTeamingValidationParamBuilder.java:20)
... 17 common frames omitted
ERROR [vcf_dm,###############################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-15] [123A4B] VCF_ERROR_INTERNAL_SERVER_ERROR Param Building for prefix 'preValidate' part of task ManagementL3PortgroupUplinkTeamingValidationAction in plugin HostNetworkingPlugin failed with exception.
com.vmware.evo.sddc.common.core.error.InternalServerErrorException: Param Building for prefix 'preValidate' part of task ManagementL3PortgroupUplinkTeamingValidationAction in plugin HostNetworkingPlugin failed with exception.
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:362)

Environment

VMware Cloud Foundation 9.0.2

Cause

This issue happens when SDDC manager cannot find any details of the management portgroups to carry out the translation from IDs to names.

Resolution

This is a known issue and is resolved in VMware Cloud Foundation 9.1 found at Broadcom Downloads.

To workaround the issue,carry out the steps below:

Execute the cluster expansion once to get the latest execution ID we require in step 7 below before proceeding.
Backup/snapshot SDDC manager
SSH to the SDDC manager as vcf user and elevate to root with su.
Run the below command to enable HTTP access used by Domain Manager:
echo vcf.vault.http-access=true >> /etc/vmware/vcf/domainmanager/application-prod.properties && systemctl restart domainmanager
Wait for domain manager to come up,the following request stops returning 502 gateway error:

watch "curl localhost/domainmanager/about"
From /var/log/vmware/vcf/domainmanager/domainmanager.log, identify the Execution ID of the failed task using the keyword 'executionId'.

<timestamp> INFO [vcf_dm,################################,####] [c.v.v.h.s.i.HostManagerSpecClientImpl,dm-exec-15] Getting AddHostUberSpec for executionId: 123#####-####-####-####-############

-- This can change every time.
Run the below command to copy the workflow of the failed task in workflow.json file:

curl -s http://localhost:7200/domainmanager/internal/vault/<execution ID identified from domainmanager.log> -X GET > spec.json

Example : curl -s http://localhost:7200/domainmanager/internal/vault/123#####-####-####-####-############ -X GET > spec.json
Open the spec in an editor and find "pgIdToNameMap",it should only contain two keys(UUID#1 and UUID#2).Add the two missing keys (UUID#3 and UUID#4 in the map below) for the L3 port groups, as follows:

"pgIdToNameMap": {
"UUID#1": "PG_NAME_VLAN_01",
"UUID#2": "PG_NAME_VLAN_02", "UUID#3": "PG_NAME_VLAN_03", "UUID#4": "PG_NAME_VLAN_04"
},

Note: UUID#3 and UUID#4 can be retrieved/identified from the spec.json file retrieved in step 7.
Save the spec
Update the vault with the newly updated spec:

curl -s http://localhost:7200/domainmanager/internal/vault/<execution ID identified from domainmanager.log> -X PUT -H "Content-type: text/plain" -d @spec.json

Example:
curl -s http://localhost:7200/domainmanager/internal/vault/123#####-####-####-####-############ -X PUT -H "Content-type: text/plain" -d @spec.json
Restart the workflow in the UI
After the workflow completes successfully, please delete the property vcf.vault.http-access=true from /etc/vmware/vcf/domainmanager/application-prod.properties
Restart domain manager with the command "systemctl restart domainmanager"

Error: "A problem has occurred on the server. Please retry or contact the service provider and provide the reference token." when running clusterexpansion API

Article ID: 447865

Updated On:

Products

Issue/Introduction

Environment

Cause

Resolution

Feedback