Management cluster upgrade fails and vCenters show "Unknown" or "EventMesh" disconnected

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• Attempts to upgrade a MC (e.g., from version 1.28.11 to 1.29.15) fail

• Connected vCenter endpoints and the Management vCenter appear as Unknown or Disconnected in the TCA UI.

• Attempting to re-authenticate or update credentials for the disconnected endpoints times out continuously.

• Paired TCA Control Plane (TCA-CP) instances display an eventMeshState of disconnected.

• In tca-mgr app and api logs, you observe errors similar to:

  TCA-M app log:
  
  java.lang.RuntimeException: Unable to grab certificate from TCA-CP
      at com.vmware.tca.pairing.service.sync.TcaCpCertificateGrabber.getCertificatePayload(TcaCpCertificateGrabber.java:94)
      at com.vmware.tca.pairing.service.sync.MigrationManager.importTlsCertificate(MigrationManager.java:186)
  ...
  INFO c.v.t.p.s.sync.MigrationManager- Skipping migration of TCA-CP (...) due to certificate import is not yet completed!
  ...
  ERROR c.v.v.h.s.r.c.JobRemotingRestClient- JobRemoting error, remote: ..., response: Unable to execute POST request to https://<TCA-CP-FQDN>/hybridity/api/queueJobFromRemote... permanently.
  
  TCA-M api log
  
  ERROR c.v.t.v.w.UpgradeVCInfraIdFromTcaCp- Failed to update infra ID for VCenter! VimOfVc[id='...', infraId='...', tcaCpRef='TcaCpRef{..., url='https://<TCA-CP-FQDN>'}', status='unknown']
  com.vmware.tca.jwt.client.TcaCpClient$ExecutionException: Unable to execute POST request to https://<TCA-CP-FQDN>/tca/vim/api/v1/validateconnection/vc permanently.

Environment

TCA 3.4.1

TCP 5.1

Cause

This issue occurs when a TCA environment is operating in a mixed-version state (e.g., TCA Manager upgraded to 3.4.1 while one or more paired TCA-CP nodes remain on an older version):

1. Post-Upgrade Recon Loop Failure: In TCA 3.4.1, TCA Manager (TCA-M) runs a post-upgrade task (UpgradeVCInfraIdFromTcaCp) to update the vCenter Infrastructure ID (infraId) by querying each paired TCA-CP. If TCA-M encounters an un-upgraded TCA-CP, the API call fails and aborts the entire loop, skipping database updates for all subsequent vCenters (even those attached to upgraded TCA-CPs). This leaves all affected vCenter VIM records in an Unknown state.
   
   
2. Thread Saturation & Outbox Backlog: Continuous connection retries to unreachable, un-upgraded TCA-CP endpoints saturate the RemotingService thread pool on TCA-M. Additionally, background resync jobs (SyncVimsToCp) generate high-volume traffic in the TCA-CP outbox queue, starving out EventMesh registration payloads (such as SASL credential exchange) and keeping the eventMeshState trapped in a disconnected state.

Resolution

Upgrade all paired TCA-CPs together, in one maintenance window.

Mixed-version topologies across paired TCA-CP instances are not supported in TCA 3.4.x. All paired control planes must be updated to match the TCA Manager version.