TMC Self-Managed Backup Location Update Fails with resource not found error

Products

• VMware Tanzu Mission Control - SM

Issue/Introduction

In VMware Tanzu Mission Control Self-Managed (TMC-SM), attempts to update a Backup Location configuration (such as adding or removing cluster groups) fails with the below error.

Update fails with: API Error: Failed to update backup location: unable to update backup location object (resource not found)

From dataprotection-server logs the following error can be observed

{"component":"data-protection","grpc.method":"Update","grpc.service":"vmware.tanzu.manage.v1alpha1.dataprotection.provider.backuplocation.BackupLocationResourceService","level":"error","msg":"unable to update backup location object. Inner Error: rpc error: code = NotFound desc = Model to update does not exist: ERROR #22023 payload string too long"}

Cause

The issue is caused by a hardcoded limitation in PostgreSQL, where the pg_notify() function enforces a maximum payload limit of 8000 bytes.

The dataprotection database contains a trigger called backuplocationnotifyevent that executes the function bl_notify_event() after any update to the backuplocations table. This function serializes the row data into JSON to notify other services of the change.

In environments with many assigned cluster groups, the assigned_groups JSONB column grows significantly. Even with existing mitigations (such as stripping the ca_cert field), the total serialized JSON payload exceeds 8000 bytes. This causes the trigger—and the entire update transaction—to fail and roll back. The API incorrectly interprets this rollback as a "resource not found" error.

Resolution

Since the  notify message size 8000-bytes is the hard limits of PostgresSQL, there isn't any parameter that can change it.

As a workaround, there are two different options 

1. Un-assign some of clustergroups that do not need to enable data protection from the target location.

2. If option#1 is not possible, the customer can create another target location with either same or different configuration, for the new clustergroups can be assigned to this one.  

Note that if you want to do the option 2 and do the same configuration then do not assign the same clusters/clustergroups to both of them.