Velero fails to make PV backups using Velero vSphere Plugin method

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Velero Backup has been completed successfully but PVs from a given namespace were not actually backed up
• Data manager log shows following errors:
  • cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
  • Failed at copying to remote repository
  • x509: certificate is valid for <*.mydomain.com>, <mydomain.com>, not <mydomain.local>
  • CopyToRepo Error Received: Failed to delete peinfo from bucket

Environment

vSphere with Tanzu

Velero Plugin for vSphere

 

Cause

The customer used a CA-signed certificate for the S3 backup storage, but the server was referenced with a different internal hostname (FQDN) not included in the certificate's signing. This issue arose because not all Velero-related Pods were restarted, causing some to continue using the old S3 hostname (FQDN).

Resolution

• Remove Existing Velero Supervisor Service
  Uninstall the current Velero supervisor service from the Supervisor cluster and perform a clean reinstall of the service.

• Ensure CA-Signed Certificate for S3 Backend
  Verify that the SSL certificate for the backend S3 storage server is signed by a trusted Certificate Authority (CA). Note that self-signed certificates or certificates signed by a private CA are not supported with the Velero for vSphere plugin.

• Match S3 Server Hostname in Velero Configuration
  Confirm that the backend S3 storage server name matches the hostname specified in the Velero configuration exactly.

• Restart Velero-Related Pods
  After updating the S3 storage server configuration, restart all Velero-related Pods in both the Supervisor and Guest clusters to apply the changes.

Additional Information

The recommended method for backing up and restoring workloads running on TKG clusters is CSI snapshot.

Velero vSphere Plugin is only recommended for Supervisor backup.

For more information on each Velero backup method and comparison, please refer this link - https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/using-tkg-service-with-vsphere-supervisor/backup-and-restore-workloads-using-the-velero-plugin-for-vsphere.html