License synchronization fails with error on VCF Operations as "License could not be synchronized with vCenters" preventing new or reinstalled ESXi hosts from being added back to the affected vCenter

Products

• VMware vCenter Server
• VMware Cloud Foundation

Issue/Introduction

In a VMware Cloud Foundation (VCF) 9.1 environment, license synchronization between VCF Operations and a Workload Domain vCenter fails. This prevents new or reinstalled ESXi hosts from being added back to a cluster because license entitlements cannot be applied.

• Attempt to add an ESXi host to a cluster fail with licensing errors.

• License synchronization tasks in VCF Operations fail to complete.

• Under VCF Operations UI > Manage > License & Registration under 'vCenter Systems' shows error "License could not be synchronized with vCenters",
  

  

• Rebooting License server does not fix the issue

• The /var/log/vmware/cis-license/license.log file on the affected vCenter contains the following error,

  YYYY-MM-DDTHH:MM:SS ERROR CISLicense 129 [vc@4413 threadName="refresh-entitlements-task-executor-1" logger="cis.license.entitlement.management.LicenseServerPeriodicOperations"] Failed to refresh entitlements on license server id ########-####-####-####-############ for allocation ids: [########-####-####-####-############, ########-####-####-####-############, ########-####-####-####-############] java.lang.IllegalStateException: Connection pool shut down

YYYY-MM-DDTHH:MM:SS ERROR CISLicense 130 [vc@4413 threadName="report-addon-usage-task-executor-1" logger="cis.license.entitlement.management.LicenseServerPeriodicOperations"] Failed to report addon usage to license server with id ########-####-####-####-############ for license id urn:vri:com.vmware.license.license:########-####-####-####-############ and allocation id ########-####-####-####-############.

Environment

• VMware Cloud Foundation 9.1
• vCenter Server 9.1
• VCF Operations 9.1

Cause

The vCenter license service (cis-license) connection pool has shut down, preventing communication between the VCF Operations manager and the vCenter license service for synchronizing license keys.

Resolution

To resolve the license synchronization failure, restart the license service on the affected vCenter:

1. Log in to the vCenter Server Appliance via SSH as root.
2. Restart the cis-license service by running the following command,
   
   service-control --restart cis-license
3. Verify that license synchronization resumes in VCF Operations by reloading the VCF Operation UI