Unable to activate Private AI Foundation on the workload domain vCenter because of the license entitlement anomaly.

Products

• VCF Private AI Services

Issue/Introduction

• Unable to activate Private AI Foundation on the workload domain vCenter.
  
  
• When running the dcli command inside the vCenter Appliance shell to confirm the Supervisor Cluster bindings, it complains of not having a valid VMware Private AI Foundation license applied. 
  
  Use the below command to obtain the supervisor ID
  
  dcli> com vmware vcenter namespacemanagement supervisors summary list
Username: [email protected]
Password: ####
Do you want to save credentials in the credstore? (y or n) [y]:
items:
   - supervisor: <supervisor-ID>
     info:
        stats:
           cpu_used: ####
           storage_capacity: ####
           memory_used: ####
           cpu_capacity: ####
           memory_capacity: ####
           storage_used: ####
        kubernetes_status: READY
        name: <cluster-name>
        messages: 
        config_status: RUNNING


   

  Use the below command to confirm the supervisor binding with the license. Below is an example output of how it might look like.
  
  dcli> com vmware vcenter namespacemanagement supervisors zones bindings list --supervisor <supervisor-ID>
zones:
   - zone: <cluster-ID>
     marked_for_removal: False
     resource_allocation:
        vm_reservations:
         - reserved_vm_class: <vm-class>
             count: 1

           - reserved_vm_class: <vm-class>
             count: 1

       messages:
      - severity: INFO
          details:
         error reconciling Zone reservation; the failed operation will be retried: The vCenter does not have the VMware Private AI Foundation with NVIDIA, and SQL Support license required by VM Resource Profiles.

          type: MANAGEMENT
        namespaces:
         - <namespace>
         - <namespace>
        status: ERROR

• However the dcli command to check the license assignment confirms that the valid license is present on the vCenter. The same can also be validated from the vSphere Client and VCF Operations UI. The below command output confirms how the expected output looks like.
  
  dcli> com vmware cis license subscription entitlement assetassignments list
  items:
  

     - server_name: <vCenter FQDN>
     asset_name: <vCenter FQDN>
     asset_license_product: VMware Cloud Foundation (cores)
     component_product_family: VMware VirtualCenter Server
     server_guid: <ID>
     component_product_family_display_name: VMware vCenter
     license_state: LICENSED_NO_EVALUATION
     asset: urn:vri:com.vmware.license.asset:<server-guid>-<scope-id>
     expiration_status:
        expiration_date: YYYY-MM-DDT<time>:<timezone>
        license_status: VALID
     asset_usage: 
     asset_product_version: 9.0
     asset_license_name: <Name of the license>

• In /var/log/vmware/vpxd.log of the workload domain vCenter, the DRS can be seen complaining of not being able to obtain the instance ID of the PAIF asset.
  
  error vpxd[] [Originator@6876 sub=drmLogger opId=<ID>] Failed to get instance ID for the PAIF asset

Environment

VMware Cloud Foundation 9.0.x

Cause

DRS does a two step check in order to determine if a deployment can access PAIF features like GPU reservations:

1. Check for the PAIF addon license.
2. Check if the PAIF license is registered with the System Management Service.

In some situations, the deployment does have the PAIF addon license however it does not get registered with the System Management Service. As a result, it is unable to validate the license details correctly.

Resolution

This is a known issue in VCF 9 and engineering is still working on a permanent fix.

Reference this kb to contact Broadcom Support to implement the workaround.