Storage provider shows 'Disconnected' or 'Offline: SyncError' post ESXI patch upgrade

Products

• VMware vCenter Server
• VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:

• After patching, ESXi hosts show Storage provider status as 'Disconnected' or 'Offline: SyncError'.
• Storage policies appear as "Incompatible" with vSAN datastores or "Disconnected".

• Unable to deploy new Virtual Machines.
• The following command output on an affected ESXi host shows a Sync Error: esxcli storage vvol vasaprovider list Status: Offline: SyncError
• In vvold.log, the following error is observed: HandleRequest: unexpected exception handling function UpdateVasaProvider: Wrong X.509 Certificate format
• Following command output on one of affected ESXi host shows Sync Error:
  
  # esxcli storage vvol vasaprovider list

VP Name: <VASA-Provider_name>
URL: https://<VP-FQDN>:9997/vasa
Status: Offline: SyncError []
VASA Version: 4
Max Concurrent Operations: 4
Arrays:
      Arrays:
         ArrayId: <Array_ID>
         Is Active: true
         Priority: 255

  VirtualHost:
   VirtualHost:
      IP:
      Port: 0
      SNI:
API Batch Size:
      APIBatchSize:
      BindVirtualVolume: 10
      Default: 1024
      GetTaskUpdateEx: 16
      QueryVirtualVolumeInfo: 1024
      SnapshotVirtualVolume: 10
      SpaceStatsVirtualVolume: 1024
      UnbindVirtualVolume: 16
      UpdateVirtualVolumeMetdataEx: 1024

• Due to this disconnection the ESXi hosts loses access to the VVOL datastores.

Environment

VMware ESXi 8.0 / 8.0 U3

VMware vCenter 8.0

VMware vSAN Witness VM (including Build ####25205845####)

Cause

This issue occurs due to the missing -----BEGIN CERTIFICATE----- header and -----END CERTIFICATE----- footer for the VASA Provider certificate within the ESXi Config Store. This typically happens during host remediation or witness deployment, resulting in a certificate string that is not in the required X.509 format.

A sample output from one of affected ESXI host shows the certificate rather shows up the string without PEM certificate header or footer.

# configstorecli config current get -c esx -g storage_vasa -k vvol -outfile vvol-vpconfig.json

# cat vvol-vpconfig.json | grep -i self_signed_certificate

\nMIIDBBjANBgkqhkiG......... \n

As observed above, the header ---> "-----BEGIN CERTIFICATE-----" and footer "-----END CERTIFICATE-----" are missing in the stored certificate).

This results in host exiting maintenance mode after patch upgrade failing to connect to VASA Provider as the self-signed certificate is not in X.509 format.

Following wrong format certificate error is observed in vvold logs while observing the issue:

YYYY-MM-DDTHH:MM:SS.177Z Er(163) VVold[#######]: [Originator@6876 sub=IpcSvc IpcCtx=2441865-41] HandleRequest: unexpected exception handling function UpdateVasaProvider: Wrong X.509 Certificate format
YYYY-MM-DDTHH:MM:SS.177Z In(166) VVold[#######]: [Originator@6876 sub=Libs] 2441868:VVOLLIB : VVolLib_IpcStorageFaultToVVolLibError:765: Storage Fault TRANSPORT_FAULT (1): Wrong X.509 Certificate format
YYYY-MM-DDTHH:MM:SS.177Z Er(163) VVold[#######]: [Originator@6876 sub=Libs] 2441868:VVOLLIB : VVolLib_UpdateVendorProvider:8041: Failed with error (10) The VASA transport encountered an error

Resolution

This issue is stipulated to be resolved in an upcoming vSphere release -- Broadcom Support Download portal

Workaround : 

 

1. SSH to the affected ESXi host.
2. Extract the saved certificate details to a JSON file:
   
   configstorecli config current get -c esx -g storage_vasa -k vvol -outfile vvol-vpconfig.json
3. Edit the JSON file to include the missing header and footer:
   
   vi vvol-vpconfig.json

Update the self_signed_certificate field to:
   
   "-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----"
4. Import the changes back to the config store:
   
   configstorecli config current set -c esx -g storage_vasa -k vvol -infile vvol-vpconfig.json
5. Restart the vvold service:
   
   /etc/init.d/vvold restart
6. In the vSphere Client, go to Storage Providers and select Resync.