Namespace stuck in Terminating state due to ArchivedPostgresCluster in VCF 9.x
Article ID: 447639
Updated On:
Products
Issue/Introduction
In VCF 9.x Supervisor environment with VMware Data Services Manager , a namespace remains stuck in a Terminating or Removing state.
Standard kubectl delete namespace commands do not complete, and standard resource checks may appear empty.
Environment
VMware Data Services Manager 9.x
VCF 9.x Supervisor
Cause
The namespace cleanup is blocked because the varchivedpostgrescluster.kb.io admission webhook prevents the direct manual deletion of archived clusters objects.
Because these are Custom Resources (CRs), they often do not appear in a standard kubectl get all command, leaving the namespace controller unable to finish its task while the resources persist.
Querying the api-resources revealed the archivedpostgrescluster
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>
The customer had not removed the archivedpostgrescluster - a custom resource that functions as a stateful snapshot of the decommissioned PostgreSQL database instance.
Resolution
To resolve this, you must identify all remaining objects, including custom resources, and then trigger the automated cleanup of the ArchivedPostgresCluster by updating its expiration metadata.
In the Supervisor context:
- Identify remaining objects: Run the following commands to find both standard and custom resources still residing in the stuck namespace:
# Check standard resources
kubectl get all -n <namespace>
# Check all namespaced resources including Custom Resource Definitions (CRDs)
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>
2. Identify the specific blocking resource, in this case, object types archivedpostgrescluster.databases.dataservices.vmware.com and associated dataserviceversions
3. Patch the resource for cleanup: Set the expiresAt value to a past date (e.g., January 1st, 2025). This signals the internal controller to bypass the webhook protection and purge the resource:
kubectl patch archivedpostgrescluster <name> -n <namespace> --type='json' -p='[{"op": "replace", "path": "/spec/expiresAt", "value": "2025-01-01T00:00:00Z"}]'
4. Remove all remaining objects, in this case, dependent dataserviceversions: Ensure any remaining dataserviceversions identified in Step 1 are also cleared
kubectl delete dataserviceversions --all -n <namespace>
5. Verify namespace deletion: After the controller reconciles the expired cluster (usually within 1-2 minutes), the namespace should proceed to full deletion.
Additional Information
Feedback
