HCX - Kafka Error "RecordTooLargeException" due to special characters in VM Notes

Products

• VMware HCX

Issue/Introduction

This document is created as a reference for the HCX system and migration services recovery due to a Kafka resource error

During an HCX migration of a VM, the migration may become stuck “Migrating” and not progress further as seen in the HCX Migration UI. 

When connecting to the source HCX manager via SSH and checking /common/logs/admin/app.log the following Kafka RecordTooLargeException is seen:

2022-03-25 16:36:23.580 UTC [RemotingService_SvcThread-431, Ent: HybridityAdmin, , TxId: e34274a5-c7b0-4d17-903b-231e34bee853] WARN  c.v.v.h.m.k.KafkaProducerDelegate- Publish failed and will retry 
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 2150439 bytes when serialized which is larger than 2097152, which is the value of the max.request.size configuration.

 

 

Environment

VMware HCX

Cause

Notes added to the VM being migrated that contain special characters cause the Kafka error. These special characters can be due to a foreign language being used or other factors:

vCenter - <select VM> - Summary - Notes

 

Resolution

Long term handling of special Notes characters through HCX will be tracked as an enhancement request.

Workaround:

Please remove the notes from the VM at the source site prior to migrating the VM to the target site:

vCenter - <select VM> - Summary - Edit Notes

At the target site the notes can then be re-added to the VM after the migration completes successfully

 

Additional Information

Please also be aware of kb HCX - Kafka Error "RecordTooLargeException" due to EULA in VM