Unable to remove the storage provider or is greyed out in vCenter
Article ID: 323125
Updated On:
Products
VMware vCenter Server
VMware vCenter Server 6.0
VMware vCenter Server 7.0
VMware vCenter Server 8.0
Issue/Introduction
Impact/Risks:
Requires vCenter service restart and also editing of vCenter database.
To successfully remove the stale entry and register the VASA provider.
SetContext failed!^M
com.vmware.vim.sms.fault.VasaServiceException: org.apache.axis2.AxisFault: Connection has been shutdown:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed:
java.security.cert.CertPathValidatorException: timestamp check failed^M
For location of vCenter log files see KB : https://knowledge.broadcom.com/external/article?legacyId=2110014
- Unable to remove the storage provider or is greyed out in vCenter.
- When reconnecting the SPs "Active" SP is stuck in "Rescan Error" Status.
- Recently there was a array firmware upgrade or you removed the backend storage provider without removing the VASA provider (VP).
- During provider un-registration you get "java.lang.NullPointerException" error.
- In VC the Storage Providers (SPs) are in a "Disconnected" stated as well the vVOL Datastore is in an "Inactive" State and is unable to failover to the standby SP.
- esxcli storage vvol vasaprovider list shows the below
VP Name: storage01-ct1
URL: https://<IP of array>:8084/version.xml
Status: syncError
Arrays:
Array Id: com.xxxstorage:########-####-####-####-########3d33
Is Active: true
Priority: 200
- In sps logs we see
SetContext failed!^M
com.vmware.vim.sms.fault.VasaServiceException: org.apache.axis2.AxisFault: Connection has been shutdown:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed:
java.security.cert.CertPathValidatorException: timestamp check failed^M
- In sms logs
YYYY-MM-DDThh-mm-ss.752-07:00 [pool-24-thread-10] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.storage.common.security.CommonActivationValidator - Validating session using session cookie...YYYY-MM-DDThh-mm-ss.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.sms.StorageManagerImpl - Received API Call: unregisterProviderYYYY-MM-DDThh-mm-ss.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STARTED: Task for unregisterProviderIntYYYY-MM-DDThh-mm-ss.838-07:00 [pool-22-thread-9] DEBUG opId=StorageProvidersViewMediator-remove-58424-ngc:70004041 com.vmware.vim.sms.task.TaskImpl - Created a task - ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = ########-####-####-####-########e596YYYY-MM-DDThh-mm-ss.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.StorageManagerImpl - unregisterProvider() called for provider - ########-####-####-####-########2d0dYYYY-MM-DDThh-mm-ss.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskImpl - Updated progress to 10% for task ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = ########-####-####-####-########e596YYYY-MM-DDThh-mm-ss.838-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderFactory - [removeProvider] Removing provider - ########-####-####-####-########2d0dYYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderCache - [removeEntry] Remove the cache entry for the provider - ########-####-####-####-########2d0dYYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.ProviderCache - [removeFdMappingForProvider] No FaultDomainMapping objects found for given fualtDomainId list []YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverHandler - [removeProvider] Removing provider: ########-####-####-####-########2d0d from priority cacheYYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [FAILOVER] Provider failover started: [provider: ########-####-####-####-########2d0d; array: com.<xxxstorage>:########-####-####-####-########3d33]YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [getBackupProvider] Looking for the new backup provider...YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Check backup provider ########-####-####-####-########2d0d ...YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Skip the current provider...YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - Check backup provider ########-####-####-####-########c805 ...YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - NeedsExplicitActivation is not set. Skipping activating provider...YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [getBackupProvider] Found new backup provider: ########-####-####-####-########c805YYYY-MM-DDThh-mm-ss.840-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.provider.failover.ProviderFailoverEngine - [updateActiveProvider] Flush event queue for the provider.YYYY-MM-DDThh-mm-ss.841-07:00 [pool-14-thread-2] ERROR opId= com.vmware.vim.sms.provider.ProviderCache - Error while removing an entry for the provider in the providerCache -java.lang.NullPointerExceptionat com.vmware.vim.sms.provider.vasa.VasaProviderImpl.flushEventQueue(VasaProviderImpl.java:1562)YYYY-MM-DDThh-mm-ss.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskManagerImpl - [markTaskCompleted] Marked SmsTask-52 as completedYYYY-MM-DDThh-mm-ss.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.storage.common.util.SimpleTimeCounter - TIMER STOPPED: Task for unregisterProviderInt, TIME TAKEN: 3 ms.YYYY-MM-DDThh-mm-ss.841-07:00 [pool-14-thread-2] DEBUG opId= com.vmware.vim.sms.task.TaskImpl - Task execution failed! ManagedObjectReference: type = SmsTask, value = SmsTask-52, serverGuid = ########-####-####-####-########e596- vasa logs on storage array shows
YYYY-MM-DDThh-mm-ss,079:INFO:[Timer-1] spawnValidateCertTask(): Task awaken validate cert.YYYY-MM-DDThh-mm-ss,425:ERROR:[Timer-1] spawnValidateCertTask(): Found an exception {}For location of vCenter log files see KB : https://knowledge.broadcom.com/external/article?legacyId=2110014
Environment
- VMware vCenter Server Appliance 8.x
- VMware vCenter Server Appliance 7.x
- VMware vCenter Server Appliance 6.7.x
- VMware vCenter Server 6.5.x
- VMware vCenter Server 6.7.x
- VMware vCenter Server Appliance 6.5.x
Cause
Storage Monitoring Service (SMS) continuously polls VASA Provider (VP) in cases where in operation state of VP is either "online" or "offline" to ensure both event and alarm managers are properly initialized. When we get this exception sms is the VP operation state to "disconnected" and hence event and alarm managers were not initialized. This results in "java.lang.NullPointerException" during provider unregistration.
Resolution
This is fixed in vCenter 6.7 update 1 and 6.5 patch 3.
Workaround:
To workaround this issue
1. Stop SPS service.
2. Make sure Storage VP against which un-registration failed is not reachable from VC.
3. Start SPS service.
4. Trigger unregistration of provider through SMS MOB as below,
a. Log in to SMS MOB : https://<VC-IP>/sms/mob
NOTE: <VC_IP> need to be replaced with actual VC IP address.
b. click on QueryStorageManager link
c. click on Invoke method, then on the "storageManager" link in the Method Invocation Result
d. Click on UnregisterProvider_Task link
e. Fill in the value - ########-####-####-####-########2d0d for "providerId" and click on the invoke method
If the above steps do not resolve please contact VMware Technical Support.
Workaround:
To workaround this issue
1. Stop SPS service.
2. Make sure Storage VP against which un-registration failed is not reachable from VC.
3. Start SPS service.
4. Trigger unregistration of provider through SMS MOB as below,
a. Log in to SMS MOB : https://<VC-IP>/sms/mob
NOTE: <VC_IP> need to be replaced with actual VC IP address.
b. click on QueryStorageManager link
c. click on Invoke method, then on the "storageManager" link in the Method Invocation Result
d. Click on UnregisterProvider_Task link
e. Fill in the value - ########-####-####-####-########2d0d for "providerId" and click on the invoke method
If the above steps do not resolve please contact VMware Technical Support.
Additional Information
Feedback
Yes
No
Powered by
