Error: "Licenses could not be assigned to vCenters", seen during license assignment on VCF 9.1
search cancel

Error: "Licenses could not be assigned to vCenters", seen during license assignment on VCF 9.1

book

Article ID: 424533

calendar_today

Updated On:

Products

VCF Operations VMware vCenter Server VMware Cloud Foundation

Issue/Introduction

  • License assignment to a selected vCenter in VCF Operations 9.1 fails with an error message "failed assignments" and "Licenses could not be assigned to the selected vCenter".



  • Selecting the View Details tab for the error message shows the message below:

    This license could not be assigned to the selected vCenter system. Check the network connection between the license server and the selected vCenter system and try again.



  • An error message may show on the licenses page saying: “Licenses could not be synchronized with vCenters.” or  “Add-on licenses could not be synchronized with vCenters.”



  • /storage/log/vcops/log/collector.log on VCF Operations may show the following error:

    com.vmware.adapter3.vmwareinfrahealth.helpers.vcenter.collector.VCenterAvailabilityMetricsCollector.addNTPServerAndDriftMetric] [NTP DRIFT] NTP drift is not performed cause this setup vCenter-<vCenterFQDN> as its vROPs is not configured with NTP server.

Environment

  • VMware Cloud Foundation (VCF) 9.1
  • VMware vSphere Foundation (VVF) 9.1
  • vCenter Server 9.x
  • VCF Operations 9.1

Cause

This issue is a result of the VCF Operations being unable to communicate with the vCenter, or the vCenter being unable to communicate with the license server, or the license server being unable to communicate with the VCF Operations instance.

  1. This issue can most often occur due to network connectivity problems between the VCF Operations instance, vCenter, and/or license server. This includes time drift between these components.
  2. In some cases, it can be a result of an uppercase vcGuid.  This behavior is typically observed in vCenter Server instances that have been upgraded from version 5.x. During such upgrades, the files /etc/vmware-vpx/instance.cfg and /etc/vmware-vpx/firstboot/vpxd-service-spec.prop retain vcGuid in uppercase.

To confirm which of these issues apply to you: 

  1. Checking the connectivity from vCenter to license server
    1. If a license has already been assigned to the vCenter and the vCenter becomes disconnected from the license server, vCenter shows a banner indicating that the connection has been lost.
    2. If a license has not yet been assigned to the vCenter, SSH into the vCenter appliance and run:

      curl -v -k https://<license-server-fqdn>

    3. If the response of the CURL is any HTTP code - the vCenter likely has the connectivity. For more information, see Resolution section.

  2. Checking the connectivity from VCF Operations to the vCenter

    From VCF Operations, go to Operate > Inventory > select the vCenter to assign a license to and then select the “Alerts” tab. If there is a connectivity issue from VCF Operations to the vCenter, it shows the alert “Connectivity to the vCenter is affected”.




  3. Checking the connectivity from license server to VCF Operations

    From VCF Operations, go to Manage > Licensing > Licenses & Registration and under Registration and License Server Status select the Manage dropdown > License Servers. Check the column “Connectivity to VCF Operations”. Above mentioned error message is shown if any of these are “Not Connected” or does not appear at all after enrollment.



    Note:  Connectivity to the license server can show as connected in this UI if the Licensing Server vAPP was deployed with a valid IP but has invalid DNS records.

  4. Checking if the vCenter instanceUuid is uppercase

    1. From the vCenter Appliance shell, run:

      grep instanceUuid /etc/vmware-vpx/instance.cfg

      Example output:
      instanceUuid=DBE#####-1##B-##BC-CA##-##CDC#####

    2. If the instanceUuid is in uppercase, requests from the vCenter to license server will fail.

    3. For resolution to this issue, see the resolution section in the KB Registering VASA provider fails with error "A problem was encountered while registering the provider" to change Instance UUID to lowercase.

Resolution

The license server must be powered on.

  1. Check: From vCenter, go to Inventory -> select the cluster where the license server VM is running, and select the license server VM. Verify the “Power Status” is “Powered On”
  2. Resolution: If the powered status is not “Powered On”, select “Actions” -> “Power On”.

The time of the license server, vCenter, and VCF Operations must be in sync.

  1. Time of the license server should match the time on the vCenter and VCF Operations time. The license server time will match the time of the ESX host that the license server is running on.
    1. Check: The timestamps of vCenter and license server should be within a few seconds of each other. If the drift exceeds 30 seconds, NTP is likely misconfigured and must be fixed. Run the following commands in short succession to confirm the date.
      1. From the ESX host where the license server is running, run command line from SSH session:

        date -u +"%Y-%m-%dT%H:%M:%SZ"

      2. From the vCenter SSH Session, run the following command:

        date -u +"%Y-%m-%dT%H:%M:%SZ"

      3. From the VCF Operations Appliance SSH session, run the command

        date -u +"%Y-%m-%dT%H:%M:%SZ"

    2. Resolution: It is recommended to have same NTP Server configured for the vCenter, VCF Operations, and the host running the license server.
      1. Configure Network Time Protocol (NTP) on the Vcenter 
      2. On the host running the license server VM, Configure Network Time Protocol (NTP) on the ESXi
      3. Reconfigure NTP on the VCF Operations Cluster Nodes 

vCenter must be able to connect to the license server. 

  1. Check if any firewall rule blocking access from the vCenter system to the license server on TCP port 443.
    1. Check: SSH into the vCenter appliance and run the following command:

      curl -v -k https://<license-server-IP>

      1. Alternatively:

        openssl s_client -connect <license-server-IP>:443

    2. Resolution: If the connection is blocked or does not respond with any HTTP status code:
      1. Add an explicit allow rule to your firewall permitting TCP 443 from the vCenter IP to the license server IP. 
      2. Verify that the license server IP address and gateway is unique and correctly configured. In vSphere Client, while the license server VM is powered off, select the license server VM > Configure > vApp Options. After a change is made - power on the license server VM.

  2. Check DNS resolution of license server
    1. Check: Use nslookup to determine if the license server hostname resolves

      Forward lookup resolution:
      nslookup <license-server-hostname> <vCenter-DNS-IP>

      Reverse lookup resolution:
      nslookup <license-server-IPAddress> <vCenter-DNS-IP>

      1. Find the vCenter DNS server:
        1. Go to https://<vcenter_IP_or_FQDN>:5480  and log-in > Click Networking > and under Networking Settings click “Edit” > Next > Expand “Hostname and DNS”. The configured DNS servers will be listed here.
      2. Find the license server hostname and IP Address:
        1. Launch the remote console of the license server VM (via vCenter Inventory) to view the hostname listed under “Hostname” and IP under "IP Address"
    2. Resolution: Add an A record for the license server hostname pointing to the license server IP in the DNS server that vCenter uses. If using Active Directory DNS, add the record via DNS Manager on the AD server.
      1. As a temporary workaround, add the entry to vCenter's hosts file. To do this - SSH into the vCenter appliance and run:

        echo "<license-server-IP> <license-server-hostname>" >> /etc/hosts

        Note: - This temporary fix should be removed once the DNS records have been updated.

License server must be able to connect to the VCF Operations.

From VCF Operations, go to Licensing > Licenses & Registration > Manage > License Servers and open the License Server details page to view health and connectivity status. If the license server appears as disconnected, or fails to appear after enrollment

  1. Check if any firewall rule blocking access from the license server to VCF Operations on TCP port 443.
    1. Check: Verify network connectivity from a host on the same management subnet as the license server:
      1. curl -v -k https://<VCF-Operations-IP>
      2. Alternatively openssl s_client -connect <VCF-Operations-IP>:443
    2. Resolution: If the connection is blocked or does not respond with any HTTP status code:
      1. Add an explicit allow rule to your firewall permitting TCP 443 from the license server to the VCF Operations IP.
      2. Verify that the license server IP address and gateway is unique and correctly configured. In vSphere Client, while the license server VM is powered off, select the license server VM > Configure > vApp Options. After a change is made - power on the license server VM.
  2. Check license server DNS configuration
    1. Check: Use nslookup to determine if the VCF Operations hostname resolves from the license server DNS nslookup <VCF-Operations-hostname> <license-server-DNS-IP>. The returned IP must match the VCF Operations IP. NXDOMAIN or a wrong IP confirms DNS is the issue.
      1. Find the license server DNS server: from the vSphere Client, select the license server VM > Configure > vApp Options > Domain Name Servers.
    2. Resolution: Add an A record for the VCF Operations FQDN in the DNS server configured on the license server. If the DNS server itself cannot be updated, the license server must be updated. In vSphere Client, while the license server VM is powered off, select the license server VM > Configure > vApp Options > Domain Name Servers. After a change is made - power on the license server VM.

VCF Operations must be able to connect to the vCenter

  1. Check if any firewall rule blocking access from VCF Operations to vCenter on TCP port 443
    1. Check: SSH into the VCF Operations appliance and run:
      1. curl -v -k https://<vCenter-IP>
      2. Alternatively: openssl s_client -connect <vCenter-IP>:443
    2. Resolution: If the connection is blocked or does not respond with any HTTP status code: Add an explicit allow rule to your firewall permitting TCP 443 from the VCF Operations IP to the vCenter IP. 
  2. Check VCF Operations DNS configuration.
    1. Check: Use nslookup to determine if the vCenter hostname resolves from the VCF Operations DNS nslookup <vCenter-hostname> <VCF-Operations-DNS-IP>. 
      1. Find the VCF Operations DNS server: SSH into the VCF Operations appliance and run nmctl show_dns.
    2. Resolution: Add an A record for the vCenter FQDN in the DNS server configured on the VCF Operations instance. 

Check for any alerts for the license server, or any resource issue on license server such as high Memory, CPU, or Disk space

  1. Check: From VCF Operations, navigate to Operate > Dashboards > All > License Server Health > Select the license server and review the CPU, memory and disk space.
  2. Resolution: 
    1. If CPU or memory utilization exceeds 90%, reboot the license server. From vCenter, go to Inventory > select the cluster where the license server VM is running, and select the license server VM. Select “Actions” > “Power” > “Reset” > “Yes.
    2. If the disk is full you must first identify which disk is full. From VCF Operations, navigate to Operate > Dashboards > All > License Server Health and check which partition is exhausted (root or data disk). Then expand the disk from vSphere Client:
      1. Power off the license server VM. From vCenter, go to Inventory > select the license server VM > Actions > Power > Shut Down Guest OS. 
      2. Right-click the license server VM > Edit Settings. Expand the Hard Disk that is full and increase its size > Click OK.
      3. Power the VM back on: Actions > Power > Power On.
      4. The appliance will automatically detect and use the additional space on the next boot.

Additional Information

Connectivity diagram between License Server and other components:

Powered by Wolken Software