Enabling encryption for Data-at-rest encryption in vSAN services on cluster fails with Error: RuntimeFault.Summary

Products

• VMware vCenter Server

Issue/Introduction

While enabling the encryption in vSAN services (Data-at-rest encryption)  fails with error RuntimeFault.Summary

From /var/log/vmware/vsan-health/vsanvcmgmtd.log we see that below entries.

info vsanvcmgmtd[3783592] [vSAN@6876 sub=vmomi.soapStub[10] opID=agw-0001334-bd64] SOAP request returned HTTP failure; <<io_obj p:0x00007fe74c03a1c8, h:56, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /
hgw/host-27/vsan>, method: generateClusterKeys; code: 500(Internal Server Error); fault: (vmodl.RuntimeFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fe74c03a1c8, h:56, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-27/vsan>]: generateClusterKeys
--> N7HostCtl3Lib4Vsan16VsanCtlExceptionE: Failed to create KEK from KMS cluster: KMS-server_Name"
--> }
warning vsanvcmgmtd[3783592] [vSAN@6876 sub=Py2CppStub opID=agw-0001334-bd64] |- EExit host-XX::vim.host.VsanSystemEx.generateClusterKeys (347 ms)

On ESXi /var/run/log/vsansystem.log

In(166) vsansystem[2101330]: [vSAN@6876 sub=Libs opId=agw-0001334-bd64-5b5f] VsanUtil: Create client context for server 10.XX.XX.XX:5696
Er(163) vsansystem[2101330]: [vSAN@6876 sub=Default opId=agw-0001334-bd64-5b5f] {2101330} :0 check_user_trust() - Verified certificate
In(166) vsansystem[2101330]: [vSAN@6876 sub=Libs opId=agw-0001334-bd64-5b5f] VsanUtil: Connect to key server done
Er(163) vsansystem[2101330]: [vSAN@6876 sub=Default opId=agw-0001334-bd64-5b5f] {2101330} :0 qlc_eh_err() - The batch count is empty
Er(163) vsansystem[2101330]: [vSAN@6876 sub=Libs opId=agw-0001334-bd64-5b5f] VsanUtil: Create, Key, Attributes={Algorithm=AES, Usage=Encrypt:Decrypt, Length=256} failed, QLC_ERR_NO_BATCH_COUNT
Wa(164) vsansystem[2101330]: [vSAN@6876 sub=Libs opId=agw-0001334-bd64-5b5f] VsanInfoEncrypt: Failed to create KEK from KMS 'KMS-server_Name' with reason: QLC_ERR_NO_BATCH_COUNT

Cause

When enabling KMS in vCenter using username/password, the setup fails if the user_name field in the KMS configuration is already populated. vCenter expects this field to be empty for a new configuration. If it's not empty, vCenter assumes an inconsistent state or a previous attempt, preventing new settings from being applied and KMS from being enabled.

Resolution

1. Connect to vCenter via SSH:
   Use an SSH client (like PuTTY or Terminal) to connect to your vCenter Server Appliance using the root user.
   
   
2. Stop vCenter Services:
   Stop the vpxd and content-library services to ensure safe database access and prevent conflicts.
   
   service-control --stop vmware-vpxd
service-control --stop vmware-content-library
   
   
3. Connect to the vCenter Database:
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
   
   
4. Check Current Value:
   You can inspect the current data in the vpx_crypto_server table. You'll likely see the user_name field filled with [email protected] or a similar value that needs to be cleared.
   
   select * from vpx_crypto_server;


5. Update user_name to Empty:
   
   update vpx_crypto_server set user_name = '';


6. [Optional] Verify Update: Run the select query again to confirm that the user_name field is now empty.
   
   select * from vpx_crypto_server;


7. Exit the Database: Type \q and press Enter to exit the PostgreSQL command-line interface.
   \q


8. Start vCenter Services:
   Restart the services that were stopped in step 2.
   
   service-control --start vmware-vpxd
service-control --start vmware-content-library


9. Login to vCenter UI and Configure KMS: