Tanzu Mission Control Self-Managed Install Stuck secret "s3-creds" not found

Products

• VMware Tanzu Mission Control

Issue/Introduction

An installation of Tanzu Mission Control Self-Managed (TMC-SM) is stuck and not progressing.

 

While connected to the cluster's context, the following pods are stuck in CreateContainerConfigError state in the tmc-local namespace:

kubectl get pods -n tmc-local | grep -v Run

• audit-service-consumer
• audit-service-server
• dataprotection-server
• inspection-server

 

When describing the above pods, the following errors are present:

kubectl describe pod -n tmc-local <pod name>

secret "inspection-s3-creds" not found
secret "dp-s3-creds" not found
secret "audit-s3-creds" not found

 

The above noted kubernetes secrets are not present in the intended cluster for installing TMC-SM:

kubectl get secrets -n tmc-local | grep s3

 

Describing the s3-access-operator and tanzu mission control packageInstall (PKGI) shows the below error message:

kubectl describe pkgi -n tmc-local s3-access-operator

kubectl describe pkgi -n tmc-local tanzu-mission-control

usefulErrorMessage: |-
      kapp: Error: Ownership errors:
      - Resource 'customresourcedefinition/s3accesspolicies.infra.tmc.eng.vmware.com (apiextensions.k8s.io/v1) cluster' is already associated with a different label 'kapp.k14s.io/app=<id>'

Environment

Tanzu Mission Control Self-Managed

Cause

A known kapp ownership conflict issue on the customResourceDefinition "s3accesspolicies" used by TMC-SM.

Specifically, kapp-controller needs to assign a new kapp.k14s.io/app label to this CRD, but it will not overwrite the existing label.

CRD was leftover from a previous install or installation attempt of TMC-SM in this cluster.
The CRD will be leftover if a previous tmc-sm uninstall failed or there were manual, unsupported deletions of TMC kubernetes objects in the cluster.

Resolution

The system is actively trying to apply the new kapp.k14s.io/app label to the affected CRD.

As such, the only action needed is to remove the old kapp app label from this CRD and confirm on the status after reconcilation.

1. Connect into the cluster context where tmc-local namespace and this CRD are present.
   
   
2. Confirm on the status of the CRD:
   

   kubectl get crd s3accesspolicies.infra.tmc.eng.vmware.com

    

3. Take a backup of the CRD as per best practice:
   

   kubectl get crd s3accesspolicies.infra.tmc.eng.vmware.com -o yaml > s3accesspolicycrd-backup.yaml

   
   
   
4. Remove the old kapp app label from the CRD:
   
   1. You can use the below command to remove the label:
      

      kubectl label crd s3accesspolicies.infra.tmc.eng.vmware.com kapp.k14s.io/app-

       

   2. Or you can edit the CRD directly and remove the label:
      

      kubectl edit crd s3accesspolicies.infra.tmc.eng.vmware.com

       

5. On next reconciliation, the new kapp app label will be applied to the CRD.
   Reconciliation is performed periodically every 10 minutes.
   
   
   
6. Once reconciliation completes, check that the s3-access-operator pod is present and Running:
   

   kubectl get pods -n tmc-local s3-access-operator

    

7. Confirm that the missing kubernetes secrets are created:
   

   kubectl get secrets -n tmc-local | grep s3

   If the secrets are not created despite the s3-access-operator pod in Running state, you may need to restart the s3-access-operator pod or troubleshoot other TMC pod failures.

   
   
   
8. Check that the previously failing pods are now Running:
   

   kubectl get pods -n tmc-local

    

9. Confirm that the s3-access-operator and tanzu-mission-control PKGI no longer show the kapp ownership error message:
   

   kubectl describe pkgi -n tmc-local s3-access-operator
   kubectl describe pkgi -n tmc-local tanzu-mission-control

   Any further error messages found in the above describes will need to be looked into to complete the TMC-SM install.