Registering VASA provider fails with error "A problem was encountered while registering the provider"

Products

VMware vCenter Server 8.0

Issue/Introduction

When trying to register VASA provider on vCenter, the following error appears: "A problem was encountered while registering the provider.”

vCenter has been upgraded to 8.x from earlier versions eg.,5.x
The following error snippets are observed in the /var/log/vmware/vmware-sps/sps.log Log file:

virtualHostInfo = (VirtualHostInfo) { SNI = null
multiVcSupported = true
port = null
serviceHost = null
vcGuid = ########-####-####-####-############ <----- This entry of the vcGuid will be in uppercase } } YYYY-MM-DDTHH:MM:SS [pool-38-thread-4] INFO opId=mamo8ouk-4614-auto-3ko-h5:70000458 com.vmware.vim.storage.common.security.CommonActivationValidator - [getRolesFromRoleCacheObject] Validating session for user <user-name> for method QuerySmsTaskInfo having correlator 3647 YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.sms.client.VasaClientImpl - [removeVirtualHost] Remote exception org.apache.axis2.AxisFault: Retrieved vasa version is null. Hostname verification could not be done. YYYY-MM-DDTHH:MM:SS [pool-3-thread-8] INFO opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.cis.localkeyvalue.client.impl.KVClientImpl - KV Client login by SamlToken successful YYYY-MM-DDTHH:MM:SS [pool-3-thread-8] INFO opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.storage.common.serviceclient.SamlTokenLoginHelper - Login successful for KV client. YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.sms.kvstore.KvStorePersistenceManager - VasaProviderInfo could not be found for the provider: ########-####-####-####-############ YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.sms.provider.vasa.VasaProviderInfoPersistenceManager - Exception occured when removing the provider information in KV store:########-####-####-####-############ com.vmware.vim.sms.fault.KvNotFoundException: VasaProviderInfo could not be found for the provider: ########-####-####-####-############ YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] WARN opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.sms.provider.vasa.VasaProviderInfoPersistenceManager - [cleanProvider] Failed to remove provider from KV store! com.vmware.vim.sms.fault.DBPersistenceException: Exception occured when removing the provider information in KV store########-####-####-####-############ YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.storage.common.VmodlErrorStrings - [init] Provider creation failed java.lang.NullPointerException: null YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] INFO opId=mamo8ouk-4614-auto-3k7-h5:70000458 com.vmware.vim.sms.provider.ProviderCache - [removeSNIEntry] Removing SNI entry for URL https://<VASA_provider_URL>:<port>/version.xml

When we execute wget for the .XML file highlighted in the sps logs above, we notice that the same vcGuid, highlighted in the above snippets, is in lower case.

Command:
wget <URL>.xml --no-check-certificate

Output:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><vasa-provider><supported-versions><version id="4" serviceLocation="/vasa"/><version id="5" serviceLocation="/vasa"/><version id="6" serviceLocation="/vasa"/><version id="7" serviceLocation="/vasa"/></supported-versions><serviceVirtualHost><virtualHost vcguid="########-####-####-####-############" virtualHostName="vasa-vhost-########-####-####-####-############" serviceHost="##.##.##.###"/><virtualHost vcguid="########-####-####-####-############" virtualHostName="vasa-vhost-########-####-####-####-############.########.com" serviceHost="##.##.##.###"/></serviceVirtualHost></vasa-provider>

Cause

This behavior is typically observed in vCenter Server instances that have been upgraded from version 5.x.
During such upgrades, the files /etc/vmware-vpx/instance.cfg and /etc/vmware-vpx/firstboot/vpxd-service-spec.prop retained vcGuid in uppercase, as seen in the /var/log/vmware/vmware-sps/sps.log.
However, the version.xml file contains the vcGuid in lowercase. This case-sensitivity discrepancy is the root cause of the registration failure.

Resolution

Update "instance.cfg" and "vpxd-service-spec.prop" to use lowercase letters for the vCenter GUID.

a. Take a snapshot of vCenter
b. Open SSH connection to vCenter as root
c. Edit "/etc/vmware-vpx/instance.cfg" and update the instanceUuid value from uppercase letters to lowercase letters.
# vi /etc/vmware-vpx/instance.cfg
Change line "instanceUuid=########-####-####-####-############" to use lowercase letters

d. Edit "/etc/vmware-vpx/firstboot/vpxd-service-spec.prop", and update the cmreg.serviceid value from uppercase letters to lowercase letters.

# vi /etc/vmware-vpx/firstboot/vpxd-service-spec.prop
Change line "cmreg.serviceid=########-####-####-####-############" to lowercase letters
Update the vpxd service registration with "lsdoctor" to reflect the changes made.

a. Download lsdoctor tool from KB320837 and upload it into vCenter
b. Run "python lsdoctor.py -r"

Select option 3 when prompted (3. Replace individual service)
Select the endpoint named "vcenterserver" -

**Please note that when selecting Option 3, Most templates are provided in the templates directory, and you would only need to select a template if it doesn’t exist. If the template for your build of vCenter does not exist, you will be prompted to select one that most closely matches your system.
Restart the vCenter services using the command below, and reregister the VASA provider
service-control --stop --all && service-control --start --all

Once all the services have started, check that required services are running
service-control --status --all
Fix the license tied to the vCenter

a. Record the current vCenter license key.

b. Obtain the current vCenter LDU ID with the command:
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhostOutput: ########-####-####-####-############
c. Remove the license associated with the command:
# /opt/likewise/bin/ldapdelete -r "cn=AssetEntity_<instanceUuid>-<LDU ID>,cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W
- example AssetEntity is a combination of the vCenter's instanceUUID and LCD ID. It should look something like this: AssetEntity_########-####-####-####-############-########-####-####-####-############

Notes:

Adjust dc=vsphere,dc=local to reflect the environment's SSO domain
instanceUuid is from step 1c in lower case letters
LDC ID is from Step 4.c

d. If the above step (c) fails to remove the license with an error similar to:

ldap_delete: No such object (32) additional info: (9703)((MDB_NOTFOUND: No matching key/data pair found

Remove the vCenter license using Jxplorer KB - How to remove a License Manually from VCenter using JXplorer

e. The "AssetEntity", can be retrieve using Jxplorer.

f. Restart the vCenter services and re-assign the license.

Additional Information

If SRM is used in the environment, please export the configuration data. When the vCenter GUID changes, SRM will break so we will need to unregister both of them then re-register and import the xml.

After implementing KB, if you receive the error: "Provider method implementation threw unexpected exception: vc server [VC UUID] is not available on the server." When trying to make changes to the content library, please open a ticket with support.