Registering VASA provider fails with error "A problem was encountered while registering the provider"
search cancel

Registering VASA provider fails with error "A problem was encountered while registering the provider"

book

Article ID: 368313

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • When trying to register VASA provider on vCenter, the following error appears: "A problem was encountered while registering the provider."
  • vCenter has been upgraded to 8.x from earlier versions (e.g., 5.x).
  • The following error snippets are observed in the /var/log/vmware/vmware-sps/sps.log file:
virtualHostInfo = (VirtualHostInfo) {
SNI = null
multiVcSupported = true
port = null
serviceHost = null
vcGuid = ########-####-####-####-############       <----- This entry of the vcGuid will be in uppercase
}
}

YYYY-MM-DDTHH:MM:SS [pool-38-thread-4] INFO  opId=ma######-####-####-###-##:######458 com.vmware.vim.storage.common.security.CommonActivationValidator - [getRolesFromRoleCacheObject] Validating session for user <user-name> for method QuerySmsTaskInfo having correlator 3647
YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=ma######-####-####-###-##:######458 com.vmware.vim.sms.client.VasaClientImpl - [removeVirtualHost] Remote exception
org.apache.axis2.AxisFault: Retrieved vasa version is null. Hostname verification could not be done.

YYYY-MM-DDTHH:MM:SS [pool-3-thread-8] INFO  opId=ma######-####-####-###-##:######458 com.vmware.cis.localkeyvalue.client.impl.KVClientImpl - KV Client login by SamlToken successful
YYYY-MM-DDTHH:MM:SS [pool-3-thread-8] INFO  opId=ma######-####-####-###-##:######458 com.vmware.vim.storage.common.serviceclient.SamlTokenLoginHelper - Login successful for KV client.
YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=ma######-####-####-###-##:######458 com.vmware.vim.sms.kvstore.KvStorePersistenceManager - VasaProviderInfo could not be found for the provider: ########-####-####-####-############
YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=ma######-####-####-###-##:######458 com.vmware.vim.sms.provider.vasa.VasaProviderInfoPersistenceManager - Exception occured when removing the provider information in KV store:########-####-####-####-############
com.vmware.vim.sms.fault.KvNotFoundException: VasaProviderInfo could not be found for the provider: ########-####-####-####-############
YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] WARN  opId=ma######-####-####-###-##:######458 com.vmware.vim.sms.provider.vasa.VasaProviderInfoPersistenceManager - [cleanProvider] Failed to remove provider from KV store!
com.vmware.vim.sms.fault.DBPersistenceException: Exception occured when removing the provider information in KV store########-####-####-####-############

YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] ERROR opId=ma######-####-####-###-##:######458 com.vmware.vim.storage.common.VmodlErrorStrings - [init] Provider creation failed
java.lang.NullPointerException: null
YYYY-MM-DDTHH:MM:SS [pool-29-thread-3] INFO  opId=ma######-####-####-###-##:######458 com.vmware.vim.sms.provider.ProviderCache - [removeSNIEntry] Removing SNI entry for URL https://<VASA_provider_URL>:<port>/version.xml
  • When you execute wget for the XML file highlighted in the sps logs above, notice that the same vcGuid is in lowercase.

Command:

wget <URL>.xml --no-check-certificate

Output:

root [/var/log/vmware/vmware-sps] # wget https://<VASA_provider_URL>:<port>/version.xml --no-check-certificate
https://<VASA_provider_URL>:<port>/version.xml
Resolving <VASA_provider_URL>... <VASA_provider_IP>
Connecting to <VASA_provider_URL>|<VASA_provider_IP>|:<port>... connected.
Proxy request sent, awaiting response... 200 OK
Length: 188 [text/xml]
Saving to: 'version.xml'

version.xml         100%[===================>]     188  --.-KB/s    in 0s

Cat into version.xml: (cat version.xml)

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><vasa-provider><supported-versions><version id="4" serviceLocation="/vasa"/><version id="5" serviceLocation="/vasa"/><version id="6" serviceLocation="/vasa"/><version id="7" serviceLocation="/vasa"/></supported-versions><serviceVirtualHost><virtualHost vcguid="########-####-####-####-############" virtualHostName="vasa-vhost-########-####-####-####-############" serviceHost="##.##.##.###"/><virtualHost vcguid="########-####-####-####-############" virtualHostName="vasa-vhost-########-####-####-####-############.########.com" serviceHost="##.##.##.###"/></serviceVirtualHost></vasa-provider>

Environment

VMware vCenter Server 8.x

Cause

  • This behavior is typically observed in vCenter Server instances that have been upgraded from version 5.x.
  • During such upgrades, the files /etc/vmware-vpx/instance.cfg and /etc/vmware-vpx/firstboot/vpxd-service-spec.prop retained vcGuid in uppercase, as seen in the /var/log/vmware/vmware-sps/sps.log.
  • However, the version.xml file contains the vcGuid in lowercase. This case-sensitivity discrepancy is the root cause of the registration failure.

Resolution

  1. Validate if VMware Live Recovery solutions are configured to connect to this vCenter server. If yes, please export the configuration data. When the vCenter GUID changes, VLR/SRM will break, so you will need to unregister both sites, then re-register and import the xml.
  2. Update instance.cfg and vpxd-service-spec.prop to use lowercase letters for the vCenter GUID.
    1. Take a snapshot of vCenter.
    2. Open an SSH connection to vCenter as root.
    3. Edit /etc/vmware-vpx/instance.cfg and update the instanceUuid value from uppercase letters to lowercase letters.
      # vi /etc/vmware-vpx/instance.cfg
Change line "instanceUuid=########-####-####-####-############" to use lowercase letters
    4. Edit /etc/vmware-vpx/firstboot/vpxd-service-spec.prop, and update the cmreg.serviceid value from uppercase letters to lowercase letters.
      # vi /etc/vmware-vpx/firstboot/vpxd-service-spec.prop
Change line "cmreg.serviceid=########-####-####-####-############" to lowercase letters
  3. Update the vpxd service registration with lsdoctor to reflect the changes made.
    1. Download the lsdoctor tool from Using the 'lsdoctor' Tool and upload it into vCenter.
    2. Run python lsdoctor.py -r.

      Select option 3 when prompted (3. Replace individual service).
      Select the endpoint named "vcenterserver".
    Note: When selecting Option 3, most templates are provided in the templates directory, and you would only need to select a template if it doesn't exist. If the template for your build of vCenter does not exist, you will be prompted to select one that most closely matches your system.
  4. Restart the vCenter services using the command below, and reregister the VASA provider:
    service-control --stop --all && service-control --start --all
    Once all the services have started, check that the required services are running:
    service-control --status --all
    Note: Certain services may require up to 20 minutes to become operational. If HA reports errors after a service restart, disable HA on each cluster and wait for the removal to complete—which can take up to 20 minutes. Re-enabling HA should then resolve the issue.
  5. Fix the license tied to the vCenter.
    1. Record the current vCenter license key.
    2. Obtain the current vCenter LDU ID with the command:
      # /usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhost
Output:   ########-####-####-####-############
    3. Remove the license associated with the command:
      # /opt/likewise/bin/ldapdelete -r "cn=AssetEntity_<instanceUuid>-<LDU ID>,cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W
      • Example AssetEntity is a combination of the vCenter's instanceUUID and LDU ID. It should look something like this: AssetEntity_########-####-####-####-############-########-####-####-####-############
      • Adjust dc=vsphere,dc=local to reflect the environment's SSO domain.
      • instanceUuid is from step 2.c in lowercase letters.
      • LDU ID is from Step 5.b.
    4. If the above step (c) fails to remove the license with an error similar to:
      ldap_delete: No such object (32) additional info: (9703)((MDB_NOTFOUND: No matching key/data pair    found
      Remove the vCenter license using Jxplorer KB - How to remove a License Manually from VCenter using JXplorer.
    5. The "AssetEntity" can be retrieved using Jxplorer.
    6. Restart the vCenter services and re-assign the license.

Additional Information

If SRM is used in the environment, please export the configuration data. When the vCenter GUID changes, SRM will break, so you will need to unregister both of them, then re-register and import the XML.

After implementing the KB, if you receive the following error when trying to make changes to the content library, engage Broadcom Support:

Provider method implementation threw unexpected exception: vc server [VC UUID] is not available on the server.
Powered by Wolken Software