License Assignment to ESX 9.1 fails with "License not available to perform the operation" due to intermittent License Server network failure
search cancel

License Assignment to ESX 9.1 fails with "License not available to perform the operation" due to intermittent License Server network failure

book

Article ID: 453977

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Following an ESXi host upgrade to 9.1, the host experiences licensing failures that prevent standard operations, such as powering on virtual machines.

  • Attempts to disconnect and reconnect the host on the vCenter yields the following error:

    License not available to perform the operation. The Evaluation license assigned to "Host" <ESX_IP> has expired. Recommend updating the license




  • The /storage/log/vcops/log/vcf-licensing-plugin.log  on VCF Operations shows License Server related error for the concerned vCenter:

    YYYY-MM-DDThh:mm:ss NOTICE vef-licensing-plugin #### [ops@#### threadId="####" threadName="pool-####-thread-####" operationId=""] [com.vmware.vrops. licensing. vcf.
    task.TaskHistory. storeCompletedSyncUsageTaskDetailsToDb] - Storing completed sync usage task details to DB: [AssignEntitlementStatus{allocationId='####', vcServerGuid=<vc_guid>, vcHostname='<vc_fqdn>', vcAdapterName='<vc_adapter>', vcfAdapterName='null',
    status=LICENSE_SERVER_FAILURE) ]

    NOTE: Command to retrieve the vCenter GUID from concerned vCenter CLI:
    grep instanceUuid /etc/vmware-vpx/instance.cfg

  • The vCenter's /var/log/vmware/cis-license/cis-license.log indicates a failure to communicate with the License Server, resulting in a fallback to an expired evaluation license:

    YYYY-MM-DDThh:mm:ss ERROR CISLicense #### [vc@#### threadName="invocation-vmomi-executor-2" logger="server.core.oldmanagement.impl.LicenseAssignmentManagerImpl" opId="<op_id>"] Failed license entity [<ESX_FQDN/IP>] operation in License Server. Entity will remain in evaluation if evaluation has not expired.
    com.vmware.cis.license.vpx.server.core.oldmanagement.dao.licstorage.LicenseStorageException: com.vmware.cis.license.embedded.server.core.management.licenseserver.LicenseServerManagementListener$Errors$OnUpdateAssetsLicenseException$LicenseServerConnectException: com.vmware.cis.license.licenseserver.client.exception.LicenseServerException$InvalidApiKey: License Server returned unexpected status code: 403 and response Forbidden
    ...
    YYYY-MM-DDThh:mm:ss WARNING CISLicense ####[vc@#### threadName="invocation-vmomi-executor-2" logger="server.core.oldmanagement.impl.LicenseAssignmentManagerImpl" opId="<op_id>"] Not a valid failover license key: ####-####-####-####-#### for entityId: host-<host_id>
    YYYY-MM-DDThh:mm:ss INFO CISLicense #### [vc@#### threadName="invocation-vmomi-executor-3" logger="server.core.oldmanagement.vmomi.LicenseAssignmentManagerMoImpl" opId="<op_id>"] updateEntity result:
    LicenseAssignment host-<host_id>
    scope: <scope_id>
    entityDisplayName: <ESX_FQDN/IP>
    ...
    LicenseInfo
    name: Unlicensed
    editionKey: Unlicensed
    evaluationKey=####-####-####-####-####,
    evaluation={hours: 0 minutes: 0 date: TTTT-MM-DDThh:mm:ss},

  • The License Server continuously toggles to a Connectivity Disconnected(Unknown_Network_Failure) state, which persists across reboots.

  • Additionally reviewing the ESX host license status from the Host CLI, shows the expired evaluation state:

    root [<esx>]# vim-cmd vimsvc/license -- show

    [200] Sending request for installed licenses ... [200] Complete, result is:
    serial: 00000-00000-00000-00000-00000
    vmodl key: esx.eval.entitlement.cpuCoreMin
    nanm: VMware ESX Server
    total: 1
    used: 1
    unit: host
    Properties:
    [system_time] = YYYY-MM-DDThh:mm:ss
    [startDate] = <startDate>
    [expirationDate] = <endDate>
    [expirationHours] = 0
    [expirationMinutes] = 0
    [diagnostic] = Evaluation period has expired, please install license.

Environment

vCenter 9.1

Cause

The issue is triggered by underlying network connectivity disruptions between vCenter, VCF Operations, and the License Server infrastructure. Because vCenter cannot reach the License Server to retrieve a valid 9.1 license, it fails over to the host's previous evaluation license. Since this evaluation period has already expired, vCenter fails to add the host

Resolution

  1. Ensure reliable network communication between vCenter, the License Server, and VCF Operations. To isolate and rule out physical network or routing issues, one of the methods can be temporarily migrating the vCenter, License Server, and Ops VMs to the same ESXi host.

     

  2. Once the License Server stops flapping and network stability is confirmed, restart the hostd and vpxa services. This forces the agents to re-initialize and rebuild their communication channels with vCenter, clearing the stalled state.

    /etc/init.d/hostd restart
    /etc/init.d/vpxa restart

  3. Right-click the disconnected ESX host in the vCenter inventory and select Connect to add it back into the cluster.

Additional Information

Error: "Licenses could not be assigned to vCenters", seen during license assignment on VCF 9.1

Powered by Wolken Software