[9.1.1]Error: "services.vcfa.broadcom.com/VcfaServiceAccount prelude/<uuid>: Unhealthy - expected 0 or 1 resource, got 2" after a VCF Automation upgrade.

Products

• VCF Automation

Issue/Introduction

• After upgrading VCF Automation, the Migration Service (VCD Migrator) reports an unhealthy status.
  
  
• Symptoms (all of the following are typically observed together):
  
  • The Migration Service tile/resource shows an unhealthy state in the VCF Automation provider UI.
  • The 'service-rde-v2' controller logs report a health error similar to:
    • "msg":"Service health issues detected","controller":"service-rde-v2", ...
    • "errors":["services.vcfa.broadcom.com/VcfaServiceAccount prelude/<uuid>: Unhealthy - expected 0 or 1 resource, got 2"]
      
      
• Two Kubernetes Secrets exist in the 'prelude' namespace carrying the same selector label 'services.vcfa.broadcom.com/name=vcd-migrator-serviceaccount' — one named 'vcd-migrator-serviceaccount' and another named with a UUID (e.g. `68535ed9-####-####-####-###########`).
• If the Migration Service attempts to authenticate using the token in the wrong (UUID-named) secret, authentication against Tenant Manager fails.

Environment

VCF Automation 9.1.1

Cause

If the Migrator was uninstalled and re-installed on 9.1 prior to the upgrade, a second secret for the Migrator will have been created which then leads to a conflict when a search returns more than 1 result.

This issue occurs after the following sequence:

• VCF Automation (e.g. 9.1) is installed — the VCD Migrator service is installed out of the box.
• The VCD Migrator service is uninstalled.
• The VCD Migrator service is installed again.
• VCF Automation is upgraded (e.g. to 9.1.1).

Resolution

1. SSH to a VCF Automation Node
   • https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/fleet-management/configuring-management-components/enable-ssh-access-to-the-vcf-services-runtime-cluster.html
2. Copy the attached script to the Node
3. Ensure the script is executable
   • chmod +x <script name>
4. Execute the script
   • <path to script>/remediate-vcd-migrator-serviceaccount-dup.sh            # prompts before deleting