Potential Fail on Join/Leave Operation when attempting to add a VCF Instance to Heterogeneous Federation

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Join / Leave operations for Multi-site federation may fail in a heterogeneous setup.
• You see error similar to: 'Failed to Join Federation. Errors: Internal server Error. Failed to join.'
• In the kafka logs located at   /opt/vmware/vcf/pantheon/data/confluent.XXX/kafka/logs/server.log, you see message similar to

ERROR Exception while processing request from 10.x.x.x (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: UPDATE_METADATA, apiVersion: 6, connectionId: 10.x.x.x, listenerName: ListenerName(SSL), principal: User:CN=xxx Caused by: java.lang.IllegalArgumentException: Invalid version for API key UPDATE_METADATA: 6

Cause

This issue occurs as the third party open-source software versions are not backward compatible.

Resolution

Ensure that the VCF instances designated as Controller are of the same vcf version.

Version can be located on the UI here:
Workload Domains (on the left Nav) -> Management Domain -> Update/Patches -> Current Version -> SDDC Manager

Workaround:
Ensure that all the Controllers in the Federation are of the compatible version as listed below: 

• VCF 3.8.x.x to VCF 4.0.1.x (Excluding VCF 3.10.2.0)
• VCF 4.1.x.x & above (+VCF 3.10.2.0)