TMC Data Protection backup failed with error messages of "The backup location is not yet available for backup (failed precondition)"

Products

• Tanzu Mission Control
• Tanzu Kubernetes Grid

Issue/Introduction

TMC Data Protection "Create backup" operation failed with the following error message.

API Error: The backup location - '{LOCATION_NAME}' under Org: '{ORG_ID}' and Cluster: 'c:{CLUSTER_IDENTIFER}' is not yet available for backup (failed precondition)

Environment

• Tanzu Mission Control (TMC)
• Tanzu Kubernetes Grid (TKG) v2.5.X

Cause

There are network connectivity issues between the Velero pod running in the target Kubernetes cluster and the target location (Minio or AWS S3) could cause the issue.

Resolution

Check the velero pod log to decide the next action.

kubectl -n velero logs deployment/velero

 

Case 1 - SAN field is not configured on Target Location

This only happens when configuring a Custom root/CA certificate (TMC --> Administration --> Target locations).

time="2025-03-19T05:32:55Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/2/0, BackupStorageLocation \"{TARGET_LOCATION_NAME}\" is unavailable: rpc error: code = Unknown desc = operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get \"https://{TARGET_LOCATION_URL}:9000/{TARGET_BUCKET}?delimiter=%2F&list-type=2&prefix=######\": tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:178"

How to resolve: Recreate TARGET_LOCATION server cert including SAN field.

 

Case 2 - Network connection failure from Velero pod to Target Location

time="2025-03-19T05:32:35Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/2/0, BackupStorageLocation \"{TARGET_LOCATION_NAME}\" is unavailable: rpc error: code = Unknown desc = operation error S3: ListObjectsV2, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: , HostID: , deserialization failed, failed to copy error response body, read tcp ###.###.##.###:51674->{TARGET_LOCATION_IPADDRESS}:9000: read: connection reset by peer)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:178"

How to resolve: Check and resolve network connection issues between the Kubernetes node to TARGET_LOCATION, like firewall and network routing issues. 

ssh capv@${K8S_NODE}
curl -v telnet://${TARGET_LOCATION_IPADDRESS}:9000