"Generate a Certificate Signing Request" Task Automatically Triggered by SPS After Upgrading vCenter and ESXi from 7.0 to 8.0

Products

• VMware vSphere ESXi 8.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

1. After upgrading the vCenter Server and ESXi from 7.0 to version 8.0, SPS is generating a certificate signing request every 5 mins.
2. Only the upgraded 8.0 hosts are affected.

YYYY-MM-DD info vmcad YYYY-MM-DD [vmcad][INFO] [RPC] Exiting RpcVMCAGetSignedCertificate, Status = 0
YYYY-MM-DD info vmcad YYYY-MM-DD [vmcad][INFO] [RPC] Exiting RpcVMCAGetSignedCertificate, Status = 0
YYYY-MM-DD info vmcad YYYY-MM-DD [vmcad][INFO] [RPC] Exiting RpcVMCAGetSignedCertificate, Status = 0
YYYY-MM-DD info vmcad YYYY-MM-DD [vmcad][INFO] [OPID :RPC] Entering RpcVMCAGetSignedCertificate
YYYY-MM-DD info vmcad YYYY-MM-DD [vmcad][INFO] [RPC] Exiting RpcVMCAGetSignedCertificate, Status = 0

Environment

VMware vCenter Server 8.0.x

Cause

1. The SPS service keeps generating "Generate a certificate signing request" task every 5 minutes on the ESXi.
2. The IO filters are missing from the storage providers.
3. The SPS is not healthy and always in an initialized state.

Resolution

Before proceeding with the steps below, take both a backup and a snapshot of the vCenter Server Appliance. If the vCenter is part of a Enhanced Linked Mode (ELM) replication setup, also take a backup or offline (powered off) snapshot of all replicating vCenter ELM nodes.

We need to clean up the VASA before re-starting the vmcad service.

Recommendation: To resolve this issue, contact VMware by Broadcom Support. To contact VMware by Broadcom support, see Creating and managing Broadcom support cases

Use the below workaround steps to remove all the VASA Providers from MOB and VCDB.

1. Ensure you have Admin access to the vcenter MOB.
2. Ensure you have permissions to run the script and specified commands in the VCenter VM
3. Stop sps by running "vmon-cli -k sps". Ensure sps service is stopped.
4. Edit the file /usr/lib/vmware-vpx/sps/conf/sms.properties and change the line mentioning sms.threadpool.queueSize to 2000, i.e, sms.threadpool.queueSize=2000
5. Login to VCDB: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB and run following sql queries:
   1. delete from cis_kv_keyvalue where kv_provider='storageArrayKvProvider';
   2. delete from cis_kv_keyvalue where kv_provider='storageContainerKvProvider';
   3. delete from cis_kv_keyvalue where kv_provider='providerInfoKvProvider';
   4. delete from cis_kv_keyvalue where kv_provider='spbmVpCapabilityObjectMetadataInfoProvider';
   5. delete from cis_kv_keyvalue where kv_provider='spbmVpIdToCapabilityProfileIdsStorageIdsMapProvider';
   6. delete from cis_kv_keyvalue where kv_provider='spbmCapabilityProfileProvider';
   7. delete from cis_kv_keyvalue where kv_provider='spbmStorageIdToCapabilityProfileIdsMapProvider';
   8. delete from cis_kv_keyvalue where kv_provider='spbmIoFilterCapabilityInfoProvider';
6. Verify that the provider data is deleted correctly. To do that, follow the steps:
   1. Go to https://<vc-ip>/invsvc/mob/localkv?moid=local-kv-mgr&method=FindKeys
   2. In provider → Enter "providerInfoKvProvider" without quotes
   3. sc → Enter "<sc><keyPrefix></keyPrefix></sc>" without quotes and invoke method. Make sure no provider UUID is returned.
7. Start sps service (vmon-cli -i sps).
8. Restart vsan-health service (vmon-cli -r vsan-health)
9. Wait till sps is restarted successfully and health of sms turns green.

NOTE: In case if customer has VVOL VP's in his environment. Then register of the all the VVOL providers needs to triggered one by one manually. 

       10. Re-start the vmcad service using service-control --restart vmcad
       11. Monitor the environment for next 15-20 mins it will stop generating the CSR request.