Supervisor upgrade fails from 9.0 to 9.1 with error message: "failed to get agency info for migration: EAM agency Agency:####### has no VM"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Supervisor upgrade from VCF 9.0 to VCF 9.1 does not complete

 

vSphere UI is showing Supervisor in configuring state with error:

GUI Error

Initialized vSphere resources
Deployed Control Plane VMs
 • A general system error occurred. Error message: failed to get agency info for migration: EAM agency Agency:############## has no VM.
Configured Control Plane VMs
Configured Core Supervisor Services

 

The new deployed Supervisor control plane is not joining the cluster

 

root@####  [ ~ ]# k get nodes
NAME                    STATUS   ROLES                  AGE    VERSION
##############          Ready    control-plane,master   118d   v1.32.9+vmware.2-fips
##############          Ready    control-plane,master   118d   v1.32.9+vmware.2-fips
##############          Ready    control-plane,master   134d   v1.32.9+vmware.2-fips
esx1                    Ready    agent                  134d   v1.32.5-sph-20e691d
esx2                    Ready    agent                  134d   v1.32.5-sph-20e691d
root@####[ ~ ]#

 

Existing Supervisor Control planes have different sizes:

example:

psql -d VCDB -U postgres

root@####### [ ~ ]# psql -d VCDB -U postgres
psql (15.15, server 14.19 (VMware Postgres 14.19.0-25043111 release))
Type "help" for help.

VCDB=# SELECT data_key, split_part(data_value, '|', 2) AS name, split_part(data_value, '|', 9) AS cluster, split_part(data_value, '|', 26) AS size FROM VPX_EXT_DATA WHERE ext_id = 'com.vmware.vim.eam' AND data_key LIKE 'Agency:%';
                  data_key                   |            name             |              cluster              | size
---------------------------------------------+-----------------------------+-----------------------------------+-------
 Agency:#################################### | vmware-vsc-apiserver-###### | ClusterComputeResource:domain-c10 | medium
 Agency:#################################### | vmware-vsc-apiserver-###### | ClusterComputeResource:domain-c10 | medium
 Agency:#################################### | vmware-vsc-apiserver-###### | ClusterComputeResource:domain-c10 | small
(3 rows)

 

Environment

VCF 9.1

Cause

Broadcom engineering is aware of this problem currently working on a fix

Resolution

Open a case with Broadcom support