Increased syslog loggings/Resolving IOFilter disconnected/offline

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

IOFilters showing as offline in the vCenter/configure/storage providers
From Aria Operations for Logs there is visible increase of events to the vCenter
There were certificate renewals recently for the vCenter such as Machine SSL, STS

Looking at the /var/log/vmware/dnsmask visible queries from the vCenter to ESXi hosts

dnsmasq[2278]: query[A] vm-#-######.#####-###.com from 127.0.0.1
dnsmasq[2278]:cached vm-#-######.#####-###.com is ##.###.##.##
dnsmasq[2278]: query[A] vca-#-######.#####-###.com from 127.0.0.1
dnsmasq[2278]: cached vca-#-######.#####-###.com is ##.###.##.##1

Environment

VMware vCenter Server 8.0.X

VMware vCenter Server 7.0.X

Cause

vCenter uses a self-signed certificate stored in SMS trust store to communicate with IOFilter providers running on ESXi hosts

Resolution

The certificate(sms_self_signed) needs to be cleared and a new certificate is to be generated for the communication between SMS to IOFilter. The newly generated SMS certificate will be signed with sha256WithRSAEncryption.
Run the below command on the VC where the issue is seen. This command displays the SMS certificate. Check fields Signature Algorithm. If the Signature Algorithm is sha1WithRSAEncryption, then proceed with the steps below.
- ```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text
```
Note: Take a snapshot of the vCenter Server in a powered-off state if the VC's are in ELM before following the steps below

Download unreg_vasa.py attached to this KB to the VC.
Enhancing logging level to analyze in case issue is not resolved
Edit file /usr/lib/vmware-vpx/sps/conf/log4j.properties in vCenter and change logging to DEBUG for the below entries, log4.properties
```
log4j.logger.com.vmware.vim.sms=DEBUG
log4j.appender.file.Threshold=DEBUG
```
Run the command below. This command stores the cert file signed with sha1WithRSAEncryption.
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_Before.out
```
Run the script
```
python unreg_vasa.py -s <VC_IP> -d
```

Replace the <VC_IP> with the IP of the VC. The script is used to automatically unregisters all the IOFilter providers which are offline from the VC.

Provide the credentials in the prompt given by the script.
The script identifies the bad providers. This should show the total number of IOFilters that are in a disconnected state.
Hit 'Y' when the prompt asks to unregister the bad providers(IOFilters).

All the IOFilter providers will be unregistered one by one.

Run the command. This command deletes the cert from SMS trust store.

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed

Expected output

Warning: This operation will delete entry [sms_self_signed] from store [sms]
Do you wish to continue? Y/N [N]
Y
Deleted entry with alias [sms_self_signed] in store [sms] successfully

Run the command below to restarts SPS service
```
vmon-cli -r sps
```
Run the command to verify the newly generated cert is signed with sha256WithRSAEncryption
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_After.out
```
The status of the SPS service can be checked by using the command
```
vmon-cli -s sps
```

Expected output

Name: sps
Starttype: AUTOMATIC
RunState: STARTED
RunAsUser: sps
CurrentRunStateDuration(ms): ####
HealthState: HEALTHY
FailStop: N/A
MainProcessId: ######

IOFilter provider registration starts automatically when SPS restarts. Wait for some time until SPS has finished initialization and check vCenter→ Configure → Storage Providers to verify that the IOFilters are now online

If the above steps have not resolved the issue, please do collect and provide the SupportBundle along with the output generated by above steps.

Additional Information

Impact/Risks:
If the SMS self-signed certificate is signed using sha1WithRSAEncryption signing algorithm, then this will cause a problem once the vCenter is upgraded to 7.0U2 as it is FIPS complaint.

Attachments

unreg_vasa get_app