Licenses could not be assigned to vCenters, license assignment failure on VCF 9.1 due to vCenter UUID case mismatch
search cancel

Licenses could not be assigned to vCenters, license assignment failure on VCF 9.1 due to vCenter UUID case mismatch

book

Article ID: 440982

calendar_today

Updated On:

Products

VCF Operations VMware Cloud Foundation VMware vSphere Foundation

Issue/Introduction

  • Assigning a license from VCF Ops shows the following error message:

    This license could not be assigned to the selected vCenter System. Check the network connection from the selected vCenter system to the license server and try again. Please refer to KB article 424533 for more information.

  • The license server FQDN "<license_server_hostname>.example.com" is reachable from local network, vCenter and from VCF Ops.
  • There are no port connectivity issues between vCenter, VCF Ops, and the License Server.
  • License plugin logs on VCF Operations show the LICENSE_SERVER_FAILURE error message.

    /storage/log/vcops/log/vcf-licensing-plugin-<uid>.log

    YYYY-MM-DDTHH:MM:SS.###Z NOTICE vcf-licensing-plugin ID [ops@#### threadId="#####" threadName="pool-##-thread-#" operationId=""] [com.vmware.vrops.licensing.vcf.task.AssignEntitlementTaskExecutor.executeAssignEntitlementsOnAdapterInstance] - Executed assign entitlement task on adapter instance completed with result [AssignEntitlementStatus{allocationId='#######-####-####-####-##############', vcServerGuid=#######-####-####-####-###########,vcHostname='<vcenter_hostname>.example.com', vcAdapterName='example_adapter', vcfAdapterName='<sddc_hostname>.example.com', status=LICENSE_SERVER_FAILURE}]

  • License service logs on vCenter show error 403.

    /var/log/vmware/cis-license/license.log

    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-#" logger="license.licenseserver.client.http.LicenseServerHttpRequestInterceptor"] HTTP Request: GET http://localhost:1080/external-vecs/http1/<license_server_hostname>.example.com/443/lsc/entitlements?allocation_ids=<id>
    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-7" logger="license.licenseserver.client.http.LicenseServerHttpResponseInterceptor"] HTTP Response: 403 Forbidden
    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-7" logger="license.licenseserver.client.http.LicenseServerHttpResponseInterceptor"] HTTP Response Body: <html>
    <head><title>403 Forbidden</title></head>

    YYYY-MM-DDTHH:MM:SS.###Z ERROR CISLicense ## [vc@4413 threadName="http-nio-0.0.0.0-#####-exec-7" logger="managment.server.vapi.impl.LicenseServerEntitlementsDelegate"] Caught exception: com.vmware.cis.license.entitlement.management.exception.LicenseServerManagementException: Failed to get Entitlements [<id>] from License Server [<server id>] with URL [<licenseserver_hostname>.example.com] with response [License Server returned unexpected status code: 403 and response Forbidden]
            at com.vmware.cis.license.entitlement.management.LicenseReader.getEntitlement(LicenseReader.java:167)

Environment

  • VVF 9.1.
  • VCF 9.1.

Cause

  • vCenter Instance UUID case mismatch between vCenter and VCF Operations causes this issue. Reviewing the VCF Operations and vCenter logs confirms this.
    • In VCF Operations, ManagementAdapter logs show that it uses the lowercase vCenter UUID:

      /storage/log/vcops/log/adapters/ManagementAdapter/ManagementAdapter_##.log

      YYYY-MM-DDTHH:MM:SS.###Z ERROR ManagementAdapter id [ops@### threadId="#####" threadName="vcf-licensing-assign-######" instanceId="##"] [(##) com.vmware.adapter.management.components.licensing.vcf.H.A] - Error while assigning entitlement to VC: <lowercase_uuid> at <vcenter_hostname>.example.com

    • However, the HTTP header from vCenter to the License server uses an uppercase vCenter UUID. This mismatch causes the License server to throw a 403 error message.

      /var/log/vmware/cis-license/license.log on vCenter:

      YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@### threadName="http-nio-0.0.0.0-####-exec-#" logger="license.licenseserver.client.http.LicenseServerHttpRequestInterceptor"] HTTP Request Header: [User-Agent: cis-license/<UPPECASE_UUID>]

Resolution

Modify the vCenter UUID to lowercase by following the steps below:

  1. Take a snapshot of the vCenter VM.
  2. Validate if VMware Live Recovery solutions are configured to connect to this vCenter server. If yes, export the configuration data. Changing the vCenter GUID impacts VLR/SRM functionality. Hence, it is important to collect a VLR/SRM configuration data backup.
  3. Open an SSH connection to vCenter as root.
  4. Update "instance.cfg" and "vpxd-service-spec.prop" to use lowercase letters for the vCenter GUID. 

    1. Edit "/etc/vmware-vpx/instance.cfg" and update the instanceUuid value from uppercase letters to lowercase letters.

      # vi /etc/vmware-vpx/instance.cfg
      Change line "instanceUuid=########-####-####-####-############" to use lowercase letters

    2. Edit "/etc/vmware-vpx/firstboot/vpxd-service-spec.prop", and update the cmreg.serviceid value from uppercase letters to lowercase letters.

      # vi /etc/vmware-vpx/firstboot/vpxd-service-spec.prop
      Change line "cmreg.serviceid=########-####-####-####-############" to lowercase letters

  5. Update the vpxd service registration with "lsdoctor" to reflect the changes made.

    1. Download the lsdoctor tool from the KB 'Using the 'lsdoctor' Tool' and upload it into vCenter.
    2. Run "python lsdoctor.py -r"
    3. Select option 3 when prompted (3. Replace individual service)
    4. Select the endpoint named "vcenterserver" 

      Note: When selecting Option 3, most templates are provided in the templates directory, and you would only need to select a template if it doesn’t exist.  If the template for the existing build of vCenter does not exist, select one that most closely matches the vCenter build.

  6. Restart the vCenter services using the command below:

    service-control --stop --all && service-control --start --all

  7. Once all the services start, check that the required services are running.

    service-control --status --all

    Note: Certain services may require few minutes to become operational. If vSphere HA reports errors after a service restart, disable HA on each cluster and wait for the removal to complete. Re-enabling HA then resolves the issue.

  8. Fix the license tied to the vCenter UUID

    1. Obtain the current vCenter LDU ID with the command:

      # /usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhost
      Output:   ########-####-####-####-############

    2. Remove the license associated with the uppercase UUID using the command below:

      # /opt/likewise/bin/ldapdelete -r "cn=AssetEntity_<instanceUuid>-<LDU_ID>,cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W

      1. For example, the AssetEntity is a combination of the vCenter's instanceUUID and LDU ID. It should look something like this: AssetEntity_########-####-####-####-############-########-####-####-####-############
      2. Adjust dc=vsphere,dc=local to reflect the environment's SSO domain.
      3. <instanceUuid> is from step 4.a in upper case letters.
      4. <LDU_ID> is from Step 8.a

  9. Restart all the services.

    service-control --stop --all && service-control --start --all

  10. Assign the License from VCF Ops

Additional Information

If SRM is used in the environment, export the configuration data. When the vCenter GUID changes, SRM will stop functioning. Hence, needs to unregister both of them then re-register and import the xml.

Powered by Wolken Software