"VasaServiceException: org.apache.axis2.AxisFault: certificate has expired", SMS Certificate Expiry Alarm after upgrading vCenter Server from 5.x to 6.x

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

After upgrading from vCenter Server 5.x to 6.x, you experience these symptoms:

• You see a critical alarm in the vSphere Client or vSphere Web Client for a certificate (SSL) expiry
• VMware vSphere Profile-Driven Storage Service Health Alarm will show as Warning
• Restarting VMware VirtualCenter Server after acknowledging the alarm cause the alarm to reappear.

In a VMware vSAN environment, you experience these symptoms:

• Cannot see or manually add VMware vSAN Storage Providers in the VMware vSphere Web Client
• Manually adding Storage Provider for vSAN in the vSphere Web client fails
• In the VMware vSphere Web Client, VMware vSAN Storage Providers that were previously online report an offline or disconnected status
• You see the error:
  
  The Register new storage provider operation failed for the entity with the following error message.
  A Problem was encountered while registering the provider
   
• In the sps.log file on vCenter Server, you see entries similar to:
  
  ...] ERROR opId=########-####-####-####-########724a com.vmware.vim.sms.provider.vasa.VasaProviderImpl - SetContext failed!
  com.vmware.vim.sms.fault.VasaServiceException: org.apache.axis2.AxisFault: certificate has expired

Windows vCenter Server log location:

%ProgramData%/VMware/vCenterServer/vmware-sps/sps.log

vCenter Server Appliance log location:

/var/log/vmware/vmware-sps/sps.log

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Cause

SSL alarm will display when the Storage Monitoring Service (SMS) 5.x certificate is still in the VECS (VMware Endpoint Certificate Store) and has expired. This certificate is no longer used in version vCenter Server 6.x.

Resolution

To resolve this issue, remove the expired certificate from the VECS:

To remove the expired certificate from the VECS in Windows-based vCenter Server:

1. Log in to vCenter Server as an administrative user.
2. Open a command prompt and navigate to C:\Program Files\VMware\vCenter Server\vmafdd.
3. List all the stores present in VECS with this command:
   
   vecs-cli store list
    
4. List all the entries in SMS store with this command and check the validity:
   
   vecs-cli entry list --store sms --text | more
    
5. Delete the sms_self_signed certificate:
   
   vecs-cli entry delete --store sms --alias sms_self_signed
    
6. Restart the VMware vSphere Profile-Driven Storage Service. For more information, see Stopping, starting, or restarting VMware vCenter Server 6.0 services (2109881).
   
   Note: To verify that the SMS store certificate is recreated, wait for few minutes and then run the vecs-cli entry list --store sms command.

To remove the expired certificate from the VECS in vCenter Server Appliance:

1. Log in as root using an SSH or console session on vCenter Server Appliance.
2. Run this command to enable the shell:
   
   shell.set --enabled true
    
3. Run this command to launch the shell:
   
   shell
    
4. Navigate to /usr/lib/vmware-vmafd/bin.
5. Run this command to list all the stores present in VECS:
   
   ./vecs-cli store list
    
6. Run this command to list all the entries in SMS store:
   
   ./vecs-cli entry list --store sms --text | more
    
7. Delete the sms_self_signed certificate:
   
   ./vecs-cli entry delete --store sms --alias sms_self_signed
    
8. Restart SPS service with these commands:
   
   service-control --stop vmware-sps
   
   service-control --start vmware-sps
   
   Note: To verify that the SMS store certificate is recreated, wait for few minutes and then run the ./vecs-cli entry list --store sms command.

Additional Information

CertificateStatusAlarm - There are certificate that expired or about to expire / Certificate Status Change Alarm Triggered on VMware vCenter Server
How to stop, start, or restart vCenter Server 6.x services
Manually reviewing certificates in VMware Endpoint Certificate Store for vSphere 6.0