Unable to remove the storage provider or is greyed out in vCenter
search cancel

Unable to remove the storage provider or is greyed out in vCenter

book

Article ID: 323125

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

To successfully remove the stale entry and register the VASA provider.

Symptoms:
  • Unable to remove the storage provider or is greyed out in vCenter.
  • When reconnecting the SPs "Active"  SP is  stuck in "Rescan Error" Status.
  • Recently there was a  array firmware upgrade or you removed the backend storage provider without removing the VASA provider (VP).
  • During provider un-registration you get  "java.lang.NullPointerException" error.
  • In VC  the Storage Providers (SPs) are in a "Disconnected" stated as well the vVOL Datastore is in an "Inactive" State and is unable to failover to the standby SP.
  • esxcli storage vvol vasaprovider list shows the below

    VP Name: storage01-ct1
    URL: https://<IP of array>:8084/version.xml
    Status: syncError
    Arrays:
          Array Id: com.xxxstorage:a6697aab-02dd-4d78-af24-e434c85c3d33
          Is Active: true
          Priority: 200
 
  • In sps logs we see
        <Timestamp> [Thread-9] ERROR opId=sps-Main-44515-482 com.vmware.vim.sms.provider.vasa.VasaProviderImpl -
      SetContext failed!^M
      com.vmware.vim.sms.fault.VasaServiceException: org.apache.axis2.AxisFault: Connection has been shutdown:
       javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed:
       java.security.cert.CertPathValidatorException: timestamp check failed^M
 
  • In sms logs
2018-08-22T12:21:57.752-07:00 [pool-24-thread-10] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...
2018-08-22T12:21:57.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.sms.StorageManagerImpl - Received API Call: unregisterProvider
2018-08-22T12:21:57.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STARTED: Task for unregisterProviderInt
2018-08-22T12:21:57.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.sms.task.TaskImpl - Created a task - ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = 340b905f-b663-4a6f-a66a-e0074ecfe596
2018-08-22T12:21:57.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.StorageManagerImpl - unregisterProvider() called for provider - 60c61aaa-79ba-4341-aff5-7a7a12a42d0d
2018-08-22T12:21:57.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskImpl - Updated progress to 10% for task ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = 340b905f-b663-4a6f-a66a-e0074ecfe596
2018-08-22T12:21:57.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderFactory - [removeProvider] Removing provider - 60c61aaa-79ba-4341-aff5-7a7a12a42d0d
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderCache - [removeEntry] Remove the cache entry for the provider - 60c61aaa-79ba-4341-aff5-7a7a12a42d0d
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderCache - [removeFdMappingForProvider] No FaultDomainMapping objects found for given fualtDomainId list []
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverHandler - [removeProvider] Removing provider: 60c61aaa-79ba-4341-aff5-7a7a12a42d0d from priority cache
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [FAILOVER] Provider failover started: [provider: 60c61aaa-79ba-4341-aff5-7a7a12a42d0d; array: com.<xxxstorage>:a6697aab-02dd-4d78-af24-e434c85c3d33]
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [getBackupProvider] Looking for the new backup provider...
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Check backup provider 60c61aaa-79ba-4341-aff5-7a7a12a42d0d ...
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Skip the current provider...
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Check backup provider 09fee91b-c729-4110-b71d-41a5639ac805 ...
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - NeedsExplicitActivation is not set. Skipping activating provider...
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [getBackupProvider] Found new backup provider: 09fee91b-c729-4110-b71d-41a5639ac805
2018-08-22T12:21:57.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [updateActiveProvider] Flush event queue for the provider.
2018-08-22T12:21:57.841-07:00 [pool-14-thread-2] ERROR opId= com.vmware.vim.sms.provider.ProviderCache - Error while removing an entry for the provider in the providerCache -
java.lang.NullPointerException
at com.vmware.vim.sms.provider.vasa.VasaProviderImpl.flushEventQueue(VasaProviderImpl.java:1562)
2018-08-22T12:21:57.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskManagerImpl - [markTaskCompleted] Marked SmsTask-52 as completed
2018-08-22T12:21:57.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STOPPED: Task for unregisterProviderInt, TIME TAKEN: 3 ms.
2018-08-22T12:21:57.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskImpl - Task execution failed! ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = 340b905f-b663-4a6f-a66a-e0074ecfe596
  • vasa logs on storage array shows
2018-07-25 12:16:29,079:INFO:[Timer-1] spawnValidateCertTask(): Task awaken validate cert.
2018-07-25 12:16:29,425:ERROR:[Timer-1] spawnValidateCertTask(): Found an exception {}

For location of vCenter log files see KB : https://kb.vmware.com/s/article/2110014

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

Storage Monitoring Service (SMS)  continuously polls  VASA Provider (VP) in cases where in operation state of VP is either "online" or "offline" to ensure both event and alarm managers are properly initialized. When we get this  exception  sms is the VP operation state to "disconnected" and hence event and alarm managers were not initialized. This results in "java.lang.NullPointerException" during provider unregistration.

Resolution

This is fixed in vCenter 6.7 update 1 and 6.5 patch 3.

Workaround:
To workaround this issue

1. Stop SPS service.
2. Make sure Storage VP against which un-registration failed is not reachable from VC.
3. Start SPS service.
4. Trigger unregistration of provider through SMS MOB as below,
   a. Log in to SMS MOB : https://<VC-IP>/sms/mob
      NOTE: <VC_IP> need to be replaced with actual VC IP address.
   b. click on QueryStorageManager link
   c. click on Invoke method, then on the "storageManager" link in the Method Invocation Result
   d. Click on UnregisterProvider_Task link
   e. Fill in the value - 60c61aaa-79ba-4341-aff5-7a7a12a42d0d for "providerId" and click on the invoke method


If the above steps do not resolve please contact VMware Technical Support.

Additional Information

Impact/Risks:
Requires vCenter service restart and also editing of vCenter database.

Powered by Wolken Software