Getting license error during cluster deployement in SDDC manager as "Obtain Licenses for Cluster Addition" and failing validation
search cancel

Getting license error during cluster deployement in SDDC manager as "Obtain Licenses for Cluster Addition" and failing validation

book

Article ID: 392467

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

Cluster deployment in SDDC Manager is failing with error:  "Obtain Licenses for Cluster Addition" failed on validation, even though the licenses are correct have been applied on the clusters. 

Task details-
DescriptionﾠObtain Licenses for Cluster Addition
Progress MessagesﾠFailed to fetch Add Cluster Licenses
Error
Message: Failed to fetch Add Cluster Licenses
Remediation Message:
Reference Token: 69A70A
cause: Cannot skip 20 ESXi Host(s) ([8ba73b1b-####-####-####-2354990#####, e29aa8b2-####-####-####-54ac236#####, 3b6b2b52-####-####-####-34039a9#####, 423b2479-####-####-####-1b453191####, dae143db-####-####-####-aed0100b####, 48e1bd52-####-####-####-97f070d4####, e605d5bf-####-####-####-c5717716####, 7ae7a91d-####-####-####-c7bd0409####, 3a63125f-####-####-####-f945787d####, 300de4da-####-####-####-6daf7a08####, d878xx5f-####-####-####-4d3xx1a####, 3864c603-####-####-####-e823d354####, 6f2e0254-####-####-####-66aac6b3####, e0265c04-####-####-####-0a6e1c2b####, 0eb397ac-####-####-####-606aa79c####, 816e9c81-####-####-####-5aebdb08####]) as only 0 ESXi host(s) would remain and the minimum is 5

 

Log snippets:

  • Task/op id-
    ┃ Name                                                         ┃ Workflow ID                          ┃ Status    ┃ Creation Time                 ┃
    │ Adding cluster XX-######## to domain XX-########         │ bd8f82c9-####-####-####-abc5817##### │ Failed    │ Wed, DD MM YYYY HH:MM:SS UTC │

 

  • Common Services-
    path - /var/log/vmware/vcf/commonsvcs/

commonsvcs.all:YYYY-MM-DDTHH:MM:SS ERROR [common,67db1470ba29b7a002e5e2460333####,b1d7] [c.v.e.s.t.services.TaskServiceImpl,vac-scheduler-1] updateCachedMutables(): failed to collect data for workflow: bd8f82c9-####-####-####-abc5817c####
....
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-7100-exec-11351] Creating admin client using psc host xx-########.xxx.xx.####.xxx
YYYY-MM-DDTHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDtHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] Possible remote API mismatch detected. Operation will continue, but errors are likely.
YYYY-MM-DDTHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AdminClientImpl,http-nio-127.0.0.1-7100-exec-11351] Client was created successfully
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] Client was disposed successfully
YYYY-MM-DDTHH:MM:SS INFO  [common,67db147026add01eaa8b30e########,4287] [c.v.e.s.a.u.a.r.CertificateController,http-nio-127.0.0.1-7100-exec-11360] Inside getCertificateConfiguration
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.sso.client.impl.SamlTokenImpl,http-nio-127.0.0.1-7100-exec-11351] SAML token for subject {Name: Administrator, Domain: VSPHERE.LOCAL} successfully parsed from Element
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.c.i.SecurityTokenServiceImpl,http-nio-127.0.0.1-7100-exec-11351] Successfully acquired token for user: [email protected]
YYYY-MM-DDTHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDTHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] Possible remote API mismatch detected. Operation will continue, but errors are likely.
YYYY-MM-DDTHH:MM:SS WARN  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] ******* WARNING ****** WARNING ****** WARNING *******
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.e.s.i.s.services.PscServiceImpl,http-nio-127.0.0.1-7100-exec-11351] Created admin client successfully for psc host xx-########.xxx.xx.####.xxx
YYYY-MM-DDTHH:MM:SS INFO  [common,67db14709e9f5639f041dec########,7f35] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7100-exec-11351] Client was disposed successfully
YYYY-MM-DDTHH:MM:SS ERROR [common,67####71fd6####654a####479d1c643,da06] [c.v.e.s.t.services.TaskCollector,http-nio-127.0.0.1-7100-exec-11344] collect() failed
org.springframework.web.client.HttpClientErrorException$NotFound: 404 : [no body]
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:112)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
YYYY-MM-DDTHH:MM:SS ERROR [common,67db1470ba29b7a002e5e2460333####,b1d7] [c.v.e.s.t.services.TaskServiceImpl,vac-scheduler-1] updateCachedMutables(): failed to collect data for workflow: bd8f82c9-####-####-####-abc5817c####

 

  • Domain Manager:

path - /var/log/vmware/vcf/domainmanager/

domainmanager.all:YYYY-MM-DDTHH:MM:SS. DEBUG [vcf_dm,67####89f####0bf18e####ecf####dc,a32f] [c.v.v.c.s.i.ClusterManagerSpecClientImpl,dm-exec-8]  Updating cluster associated with execution bd8f82c9-####-####-####-abc5817c#### with status ERROR. Skipping status update for hosts []
...

 

YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.c.s.a.l.LicenseValidationUtil,dm-exec-19]  Provided ESXI license version: 7.0
YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.c.s.a.l.LicenseValidationUtil,dm-exec-19]  Major version comparison - provided version: 7, expected version: 8
YYYY-MM-DDTHH:MM:SS WARN  [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.c.s.a.l.LicenseValidationUtil,dm-exec-19]  Invalid license for the ESXi version provided for host with id 3b6b2b52-####-####-####-34039a923####, skipping the host
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.c.s.a.l.LicenseServiceAdapterImpl,dm-exec-19]  Retrieving license service adapter with timeout 12 and base path http://127.0.0.1:80.
YYYY-MM-DDTHH:MM:SS WARN  [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.c.s.a.l.LicenseValidationUtil,dm-exec-19]  Invalid license for the ESXi version provided for host with id 423b2479-####-####-####-1b453191####, skipping the host
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.v.c.f.a.FetchAddClusterLicensesAction,dm-exec-19]  Domain is in perpetual license mode, retrieving license keys from spec
YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.v.h.a.h.FetchAddHostLicensesAction,dm-exec-19]  Fetching ESXI perpetual licenses from spec

 

YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.v.clustermanager.ClusterSpecUtil,dm-exec-19]  Cannot skip 20 ESXi host(s) ([8ba73b1b-####-####-####-23549906####, e29aa8b2-####-####-####-54ac236e####, 3b6b2b52-####-####-####-34039a92####, 423b2479-####-####-####-1b453191####, dae143db-####-####-####-aed0100b####, 48e1bd52-####-####-####-97f070d4####, e605d5bf-####-####-####x-c5717716####, 7ae7a91d-####-####-####-c7bd0409####, 3a63125f-####-####-####-###########, 300de4da-####-####-####-6daf7a08####, d8786d5f-####-####-####-4d3eb1a3####, ec07416d-####-####-####-8d089cd3####
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-19]  [BONJ5I] FAILED_TO_FETCH_ADD_CLUSTER_LICENSES Failed to fetch Add Cluster Licenses
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to fetch Add Cluster Licenses
        at com.vmware.vcf.clustermanager.fsm.actions.FetchAddClusterLicensesAction.execute(FetchAddClusterLicensesAction.java:114)
        at com.vmware.vcf.clustermanager.fsm.actions.FetchAddClusterLicensesAction.execute(FetchAddClusterLicensesAction.java:42)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.evo.sddc.common.core.error.InvalidStateException: Cannot skip 20 ESXi Host(s) ([8ba73b1b-####-####-####-23549906####, e29aa8b2-####-####-####-54ac236e####,6f2e0254-####-####-####-66aac6b3####, e0265c04-####-####-####-0a6e1c2b####, 0eb397ac-####-####-####-606aa79c####, 816e9c81-####-####-####-5aebdb08####]) as only 0 ESXi host(s) would remain and the minimum is 5
....
....
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,67db1488d1b83595d46787########,c3cb] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-19]  Collected the following errors for task with name FetchAddClusterLicensesAction and ID 7f01-####-1a53-####-afc7da3c####: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,67db148873960b91a3bd2c97306d####,b9bd] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-19]  Invoking task FetchAddClusterLicensesAction.UNDO Description: Fetch the ESXi and/or vSAN licenses, Plugin: ClusterManagerPlugin, ParamBuilder null, Input map: {skipFailedHostsEnabled=addCluster____9__AddClusterHeader____1__AddClusterTrailingHeader____12___3_skipFailedHostsEnabled, hostSpec=addCluster____9__AddClusterHeader____1__AddClusterTrailingHeader____12__hostSpec, domainId=addCluster____9__AddClusterHeader____1__AddClusterTrailingHeader____12__domainId, datastoreSpec=addCluster____9__AddClusterHeader____1__AddClusterTrailingHeader____12__datastoreSpec}, Id: 7f01-####-####-8195-afc7da3c#### ...
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,67db148873960b91a3bd2c97306d####,b9bd] [c.v.e.s.o.c.c.ContractParamBuilder,dm-exec-19]  Contract task Fetch the ESXi and/or vSAN licenses input: {"

 

  • Operation Manager:
    path - /var/log/vmware/vcf/operationsmanager/

 

operationsmanager.YYYY-MM-DD.3.log:YYYY-MM-DDTHH:MM:SS WARN  [vcf_om,32bd4a23f6b7####,3fe4] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7300-exec-196] Possible remote API mismatch detected. Operation will continue, but errors are likely.
operationsmanager.YYYY-MM-DD.3.log:YYYY-MM-DDTHH:MM:SS WARN  [vcf_om,32bd4a23f6b7####,3fe4] [c.v.v.s.a.c.v.impl.AbstractClient,http-nio-127.0.0.1-7300-exec-196] ******* WARNING ****** WARNING *****
* WARNING *******
....

 

YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.r.a.c.v.LicenseManagerController,http-nio-127.0.0.1-7300-exec-181] Fetching license keys with input filters productTypes: [ESXI], l
icenseKeyStatuses: null, productVersion: 8.0.3-24194597
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] Validating a license key...
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] Checking license for product VMware ESX Server having license version 7.0
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] The License key ####-####X-####X-####X-#### is decoded successfully with this file /opt/vmware/vcf/operationsmanager/dlfs/esxi/license-esx-XX-XXX-XX-####X
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] Successfully set Total 11200
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] Successfully set enabled features ["vsmp:0","<host_name>","esxHost","vimap
i","contentlib","vstorage","vmotion","xswitchvmotion","das","dr","endpoint","replication","vshield","hotplug","svmotion","smartcard","ft:8","vvolumes","storageawarenessapi","spbm","vaai","drs","serialuri:2"
,"mpio","bigdataex","rem","dvs","hostprofile","autodeploy","sriov","sioshares","dpvmotion","storagedrs","metrovmotion","viewaccel","appha","xvcvmotion","vgpu","proactiveha","vmcrypt","encryptedvmotion","pre
dictivedrs","pmem","trustedplatform","wcp","bitfusion"].
YYYY-MM-DDTHH:MM:SS INFO  [vcf_om,0ff5ccd8c7f6####,5896] [c.v.v.l.s.v.LicenseKeyValidator,http-nio-127.0.0.1-7300-exec-181] Successfully set license edition esx.enterprisePlus.cpuPackageCoreLimited

 

 

Environment

VMware SDDC Manager 4.x 
VMware SDDC Manager 5.x

Cause

Customer software SuperCell may be holding on to the old ESXi 7.0 licenses and presenting them in the spec sheet instead of the 8.0 licenses. This is causing the commission to fail.

 

Resolution

  1. Delete licenses - Do it from vSphere Client and SDDC Manager the usual UI way 

  2. Delete cluster - For this refer steps from How to manually remove a stale cluster from the SDDC Manager database after an unsuccessful cluster deletion attempt.

  3. Delete hosts  - For this refer steps from article Unable to remove ESXi host from VCF SDDC manager due to database mismatch | Steps to remove ESXi Hosts from the SDDC Manager DB

  4. Update spec in SuperCell - Develop section inside SDDC Manager and edit and run spec file from there 

  5. Recommission hosts - This can be done from the SDDC Manager 

  6. And last recreate the cluster  - This you can do either from SDDC Manager or vSphere Client based on the type of deployment (greenfield/brownfield) 

Powered by Wolken Software