License and Authentication Failures in SDDC Manager Updates Tab

Products

VMware vCenter Server VMware SDDC Manager / VCF Installer

Issue/Introduction

In VMware Cloud Foundation (VCF), the Workload Domain summary in SDDC Manager displays the following alert: License check: There are no assigned licenses found for this product.
- You may also see under the domain's Updates tab: "License check: Failed to fetch assigned licenses".
Alert "License check: There are no assigned licenses found for this product" is seen in SDDC Manager and upgrading the system does not resolve the issue.

Attempting to upgrade the environment does not resolve the issue. The /var/log/vmware/vcf/operationsmanager/operationsmanager.log shows errors similar to:

YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,3f929f###5984d9e,1##f] [c.v.v.l.s.u.ResourceLicensingPopulator,om-exec-3] License assignment not found for resource esxi01.example.local
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,3f929f###5984d9e,1##f] [c.v.v.l.s.u.ResourceLicensingPopulator,om-exec-3] License assignment not found for resource esxi02.example.local
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,3f929f###5984d9e,1##f] [c.v.v.l.s.u.ResourceLicensingPopulator,om-exec-3] License assignment not found for resource esxi03.example.local
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,3f929f###5984d9e,1##f] [c.v.v.l.s.u.ResourceLicensingPopulator,om-exec-3] License assignment not found for resource esxi04.example.local

YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,69a82fb9c7496######de7c809f21d3,5c12] [c.v.v.l.s.u.ResourceLicensingPopulator,pool-3-thread-18] License assignment not found for resource VCENTER02.#####.com
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,69a82fd5e#########e6f10fcfd1f,a0bf] [c.v.v.l.s.u.ResourceLicensingPopulator,om-exec-8] License assignment not found for resource VCENTER02.#####.com

Checking SDDC Web page, Inventory -> Hosts -> ALL HOSTS, the hosts are in lowercase:

esxi01.example.local
esxi02.example.local
esxi03.example.local
esxi04.example.local
However in vCenter web client, click the Main menu -> Administration -> Licensing -> Licenses -> Assets -> Hosts, the hosts are in uppercase:

ESXi01.example.local
ESXi02.example.local
ESXi03.example.local
ESXi04.example.local
Or Checking the SDDC Web page, Inventory->Workload Domains->Certificates, you notice upper case vCenter FQDN
If the license check error is thrown, in additon to an error under the domain's Precheck Status that says: "Cannot complete login due to an incorrect user name or password", see kb 418206.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

These errors typically occur due to one of two conditions, or both concurrently:

Hostname Case Mismatch: A case-sensitivity discrepancy (uppercase vs. lowercase) exists between the SDDC Manager database and the vCenter Server inventory.
Credential Desynchronization: The vCenter service account credentials stored in SDDC Manager do not match the credentials in vCenter SSO.

Resolution

Note: The DB edits can be destructive if done incorrectly. Take snapshot of the SDDC Manager. Ref KB: Taking and restoring snapshots of the SDDC Manager appliance.

Resolve this issue by modifying the SDDC Manager PostgreSQL database to match the hostname casing.

Prerequisites:

Ensure that the hostname displayed in vCenter matches the FQDN of the ESXi hosts by verifying it directly on the ESXi servers. If the names match, reach out to Broadcom Support
The rename operation is a SDDC Database edit. Ensure a valid file-based backup or offline snapshot of the SDDC Manager appliance is taken..
- Verify which hosts or the vCenter are impacted by running the below command:
  
  grep -i "License assignment not found for resource" /var/log/vmware/vcf/operationsmanager/operationsmanager.log | awk '{print $11}' | sort | uniq -c | less -iN
```
1       6 esxi01.example.local
2       3 esxi02.example.local
3       5 esxi03.example.local
4       5 esxi04.example.local
5       6 VCENTER02.#####.com
```

Follow the steps below:

SSH to the SDDC Manager appliance and access the BASH shell

Access the postgres platform database:

psql -U postgres -h localhost -d platform

Retrieve the ID for the affected ESXi host or vCenter Server:

In case of ESXi host, Run the following command to list the host with ID's from the host table:

SELECT id, hostname from host;

Example Output

platform=# SELECT id, hostname from host;
                  id                  |       hostname
--------------------------------------+-----------------------
 dbc4a6fd-ee75-##cb-####-e3735c167f5d | ESX01.example.local

In case of vCenter, Run the following command to list the vCenter with ID's from the vm_hostname table:

select id, vm_hostname, type, status from vcenter;

Example Output

platform=# SELECT id, vm_hostname, type, status from vcenter;
                  id                  |     vm_hostname     |    type    | status
--------------------------------------+---------------------+------------+--------
 37840a76-e2d5-####-####-1ac777f3e33b | vcenter01.#####.com | VI         | ACTIVE
 a4b473b2-b28d-####-####-981e568c1d62 | VCENTER02.#####.com | MANAGEMENT | ACTIVE
(2 rows)

Update the database record with the correct lowercase Fully Qualified Domain Name (FQDN) corresponding to the ID identified in Step 3.

For the ESXi host

update host set hostname='esxi01.example.local' where id='dbc4a6fd-ee75-##cb-####-e3735c167f5d';

For the vCenter

update vcenter set vm_hostname='vcenter02.#####.com' where id='a4b473b2-b28d-####-####-981e568c1d62';

Type \q to exit the PostgreSQL prompt.
Refresh the SDDC Manager browser session to verify the alert has cleared.

Additional Information

For the holistic health of the environment, lowercase is the recommended case for the environment. If you find that you have mixed case hostname / FQDN in your SDDC environment it is recommended that you take the following additional steps:

Reconfigure the ESXi hostname to lowercase directly on the host.
Ensure Forward and Reverse DNS records are updated to reflect lowercase formatting.