During VIO v2t support bundle of UA running MC fails with insufficient disk space while host migration is still going on

Products

• VMware NSX

Issue/Introduction

Symptoms:
In VIO v2t migration support bundle of UA running MC fails with insufficient disk space while host migration is ongoing

Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

The manager appliance becomes inoperable when the disk is completely used up. This is happening because the cm.log grows exponentially leading to the issue where the system becomes unresponsive and migration fails without the option to collect a support bundle.

This happens in an environment with a lot of hosts and security groups.

File seen to reach to 16 G
root@hostname:/var/log/migration-coordinator/v2t# ls -lh | grep cm.log
-rw-r----- 1 umc umc 16G Nov 23 18:52 cm.log

root@hostname:/var/log/proton# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/nsx-var+log 27G 24G 2.4G 91% /var/log --> Close to disk being out of space

Support bundle collection is failing because of this here are the related logs

2021-11-23 18:34:44,158 2617 napi.root.administration.support_bundles.common_node ERROR Error compressing bundle archive, rc: 1, err:
gzip: /image/nsx_support_b_ke_lzs/nsx_manager_########-####-####-####-########c715_20211123_174928.tar.gz: No space left on device

2021-11-23 18:34:44,158 2617 root ERROR Insufficient disk space to create support bundle archive

2021-11-23 18:34:44,159 2617 root ERROR Out of disk space exception while creating support bundle archive

2021-11-23 18:34:44,411 2617 root INFO Successfully unlocked /tmp/.logrotate.lock

Resolution

Fixed in 4.1.0.0.0.20973353

Workaround:
Manually copy the entire v2t folder to an external location and temporarily mv cm.log to some other location that is not part of support bundle collection location

Additional Information

Impact/Risks:
Support bundle collection will fail, this will make debugging difficult in certain situations