Licenses could not be assigned to vCenters, license assignment failure on VCF 9.1 due to vCenter UUID case mismatch
search cancel

Licenses could not be assigned to vCenters, license assignment failure on VCF 9.1 due to vCenter UUID case mismatch

book

Article ID: 440982

calendar_today

Updated On:

Products

VCF Operations VMware Cloud Foundation VMware vSphere Foundation

Issue/Introduction

  • Assigning License from VCF Ops shows below error message:

    This license could not be assigned to the selected vCenter System. Check the network connection from the selected vCenter system to the license server and try again. Please refer to KB article 424533 for more information.

  • License server FQDN is reachable "<license_server_hostname>.example.com" from local network, vCenter and from VCF Ops.
  • No port connectivity issues between vCenter, VCF Ops, and License Server.
  • License plugin logs on VCF Operations shows LICENSE_SERVER_FAILURE error message.

    /storage/log/vcops/log/vcf-licensing-plugin-<uid>.log

    YYYY-MM-DDTHH:MM:SS.###Z NOTICE vcf-licensing-plugin ID [ops@#### threadId="#####" threadName="pool-##-thread-#" operationId=""] [com.vmware.vrops.licensing.vcf.task.AssignEntitlementTaskExecutor.executeAssignEntitlementsOnAdapterInstance] - Executed assign entitlement task on adapter instance completed with result [AssignEntitlementStatus{allocationId='#######-####-####-####-##############', vcServerGuid=#######-####-####-####-###########,vcHostname='<vcenter_hostname>.example.com', vcAdapterName='example_adapter', vcfAdapterName='<sddc_hostname>.example.com', status=LICENSE_SERVER_FAILURE}]

  • License service logs on vCenter shows error 403.

    /var/log/vmware/cis-license/license.log

    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-#" logger="license.licenseserver.client.http.LicenseServerHttpRequestInterceptor"] HTTP Request: GET http://localhost:1080/external-vecs/http1/<license_server_hostname>.example.com/443/lsc/entitlements?allocation_ids=<id>
    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-7" logger="license.licenseserver.client.http.LicenseServerHttpResponseInterceptor"] HTTP Response: 403 Forbidden
    YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@#### threadName="http-nio-0.0.0.0-#####-exec-7" logger="license.licenseserver.client.http.LicenseServerHttpResponseInterceptor"] HTTP Response Body: <html>
    <head><title>403 Forbidden</title></head>
    <body>
    <center><h1>403 Forbidden</h1></center>
    </body>
    </html>

    YYYY-MM-DDTHH:MM:SS.###Z ERROR CISLicense ## [vc@4413 threadName="http-nio-0.0.0.0-#####-exec-7" logger="managment.server.vapi.impl.LicenseServerEntitlementsDelegate"] Caught exception: com.vmware.cis.license.entitlement.management.exception.LicenseServerManagementException: Failed to get Entitlements [<id>] from License Server [<server id>] with URL [<licenseserver_hostname>.example.com] with response [License Server returned unexpected status code: 403 and response Forbidden]
            at com.vmware.cis.license.entitlement.management.LicenseReader.getEntitlement(LicenseReader.java:167)
            at com.vmware.cis.license.entitlement.management.LicenseServerEntitlementManager.set(LicenseServerEntitlementManager.java:137)

Environment

  • VVF 9.1.
  • VCF 9.1.

Cause

  • This issue is caused due to vCenter Instance UUID case mismatch between vCenter and VCF Operations and the same can be confirmed by reviewing the VCF Operations and vCenter logs.
    • In VCF Operations, ManagementAdapter logs shows that, it is using the lowercase vCenter UUID as mentioned below:

      /storage/log/vcops/log/adapters/ManagementAdapter/ManagementAdapter_##.log

      YYYY-MM-DDTHH:MM:SS.###Z ERROR ManagementAdapter id [ops@### threadId="#####" threadName="vcf-licensing-assign-######" instanceId="##"] [(##) com.vmware.adapter.management.components.licensing.vcf.H.A] - Error while assigning entitlement to VC: <lowercase_uuid> at <vcenter_hostname>.example.com

    • However, the the HTTP header from vCenter to License server uses the vCenter UUID with uppercase UUID. This mismatch caused the License server to throw 403 error message.

      /var/log/vmware/cis-license.log on vCenter:

      YYYY-MM-DDTHH:MM:SS.###Z INFO CISLicense ## [vc@### threadName="http-nio-0.0.0.0-####-exec-#" logger="license.licenseserver.client.http.LicenseServerHttpRequestInterceptor"] HTTP Request Header: [User-Agent: cis-license/<UPPECASE_UUID>]

Resolution

Modify the vCenter UUID to lowercase by following the steps below:

  1. Take snapshot of the vCenter VM.
  2. Validate if VMware Live Recovery solutions are configured to connect to this vCenter server. If yes, export the configuration data. When the vCenter GUID changes, VLR/SRM will break. Hence it is important to collect VLR/SRM configuration data backup.
  3. Open SSH connection to vCenter as root.
  4. Update "instance.cfg" and "vpxd-service-spec.prop" to use lowercase letters for the vCenter GUID. 
    1. Edit "/etc/vmware-vpx/instance.cfg" and update the instanceUuid value from uppercase letters to lowercase letters.

      # vi /etc/vmware-vpx/instance.cfg
      Change line "instanceUuid=########-####-####-####-############" to use lowercase letters

    2. Edit "/etc/vmware-vpx/firstboot/vpxd-service-spec.prop", and update the cmreg.serviceid value from uppercase letters to lowercase letters.

      # vi /etc/vmware-vpx/firstboot/vpxd-service-spec.prop
      Change line "cmreg.serviceid=########-####-####-####-############" to lowercase letters

  5. Update the vpxd service registration with "lsdoctor" to reflect the changes made.
    1. Download lsdoctor tool from KB Using the 'lsdoctor' Tool and upload it into vCenter.
    2. Run "python lsdoctor.py -r"
    3. Select option 3 when prompted (3. Replace individual service)
    4. Select the endpoint named "vcenterserver" 

      Note: When selecting Option 3, Most templates are provided in the templates directory, and you would only need to select a template if it doesn’t exist.  If the template for the existing build of vCenter does not exist, select one that most closely matches with the vCenter build.

  6. Restart the vCenter services using the command below:

    service-control --stop --all && service-control --start --all

  7. Once all the services have started, check that required services are running.

    service-control --status --all

    Note: Certain services may require up to 20 minutes to become operational.  If HA reports errors after a service restart, disable HA on each cluster and wait for the removal to complete—which can take up to 20 minutes. Re-enabling HA should then resolve the issue.

  8. Fix the license tied to the vCenter UUID
    1. Obtain the current vCenter LDU ID with the command:

      # /usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhost
      Output:   ########-####-####-####-############

    2. Remove the license associated with uppercase UUID using below command:

      # /opt/likewise/bin/ldapdelete -r "cn=AssetEntity_<instanceUuid>-<LDU_ID>,cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W

      1. example AssetEntity is a combination of the vCenter's instanceUUID and LDU ID. It should look something like this: AssetEntity_########-####-####-####-############-########-####-####-####-############
      2. Adjust dc=vsphere,dc=local to reflect the environment's SSO domain.
      3. <instanceUuid> is from step 4.a in lower case letters.
      4. <LDU_ID> is from Step 8.a

        Note: Remove the vCenter license using Jxplorer KB - How to remove a License Manually from VCenter using JXplorer if the above step 8.b fails to remove the license with an error similar to below:

        ldap_delete: No such object (32) additional info: (9703)((MDB_NOTFOUND: No matching key/data pair found

  9. Restart all the services.

    service-control --stop --all && service-control --start --all

  10. Assign the License from VCF Ops

Additional Information

  • If SRM is used in the environment, export the configuration data. When the vCenter GUID changes, SRM will break so we will need to unregister both of them then re-register and import the xml.
Powered by Wolken Software