Resolving IOFilter disconnected/offline post upgrade to 7.0u2

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
IOFilters go offline after a vCenter upgrade to 7.0 Update 2

Environment

VMware vCenter Server 7.0.2
VMware vSphere ESXi 7.0.2

Cause

vCenter uses a self-signed certificate stored in SMS trust store to communicate with IOFilter providers running on ESXi hosts.

Resolution

The certificate(sms_self_signed) needs to be cleared and a new certificate is to be generated for the communication between SMS to IOFilter. The newly generated SMS certificate will be signed with sha256WithRSAEncryption.
 

Run the below command on the VC where the issue is seen. 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text

 

This command displays the SMS certificate. Check fields Signature Algorithm. If the Signature Algorithm is sha1WithRSAEncryption, then proceed with the steps below.

Note: Take a snapshot of the vCenter Server in a powered-off state if the VC's are in ELM before following the steps below

 1. Download unreg_vasa.py attached to this KB to the VC.

2. Enhancing logging level to analyze in case issue is not resolved

Edit file /usr/lib/vmware-vpx/sps/conf/log4j.properties in vCenter and change logging to DEBUG for the below entries,

log4.properties

log4j.logger.com.vmware.vim.sms=DEBUG log4j.appender.file.Threshold=DEBUG

3. Run the command : 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_Before.out

 This command stores the cert file signed with sha1WithRSAEncryption.

4. Run the script:

python unreg_vasa.py -s <VC_IP> -d

 

Replace the <VC_IP> with the IP of the VC. The script is used to automatically unregisters all the IOFilter providers which are offline from the VC.
Provide the credentials in the prompt given by the script.
The script identifies the bad providers. This should show the total number of IOFilters that are in a disconnected state.
Hit 'Y' when the prompt asks if you want to unregister the bad providers(IOFilters).

 

All the IOFilter providers will be unregistered one by one.

 

5. Run the command : 

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed

 This command deletes the cert from SMS trust store.
 

6. Run the command : 

vmon-cli -r sps

 Restarts SPS service
 

7. Run the command : 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_After.out

 To verify the newly generated cert is signed with sha256WithRSAEncryption
 

8. You can check the status of SPS service using: 

vmon-cli -s sps

9. IOFilter provider registration starts automatically when SPS restarts. Wait for some time until SPS has finished initialisation and check vCenter→ Configure → Storage Providers to verify that the IOFilters are now online
 

If the above steps have not resolved the issue, please do collect and provide the SupportBundle along with the output generated by above steps.

Additional Information

Impact/Risks:
If the SMS self-signed certificate is signed using sha1WithRSAEncryption signing algorithm, then this will cause a problem once the vCenter is upgraded to 7.0U2 as it is FIPS complaint.