During VIO v2t support bundle of UA running MC fails with insufficient disk space while host migration is still going on
Article ID: 339391
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
In VIO v2t migration support bundle of UA running MC fails with insufficient disk space while host migration is ongoing
In VIO v2t migration support bundle of UA running MC fails with insufficient disk space while host migration is ongoing
Environment
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
Cause
The manager appliance becomes inoperable when the disk is completely used up. This is happening because the cm.log grows exponentially leading to the issue where the system becomes unresponsive and migration fails without the option to collect a support bundle.
This happens in an environment with a lot of hosts and security groups.
File seen to reach to 16 G
root@mgr-c6610fa9:/var/log/migration-coordinator/v2t# ls -lh | grep cm.log
-rw-r----- 1 umc umc 16G Nov 23 18:52 cm.log
root@mgr-c6610fa9:/var/log/proton# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/nsx-var+log 27G 24G 2.4G 91% /var/log --> Close to disk being out of space
Support bundle collection is failing because of this here are the related logs
2021-11-23 18:34:44,158 2617 napi.root.administration.support_bundles.common_node ERROR Error compressing bundle archive, rc: 1, err:
gzip: /image/nsx_support_b_ke_lzs/nsx_manager_cb6f0442-9bfd-1048-fda3-b9ba975bc715_20211123_174928.tar.gz: No space left on device
2021-11-23 18:34:44,158 2617 root ERROR Insufficient disk space to create support bundle archive
2021-11-23 18:34:44,159 2617 root ERROR Out of disk space exception while creating support bundle archive
2021-11-23 18:34:44,411 2617 root INFO Successfully unlocked /tmp/.logrotate.lock
This happens in an environment with a lot of hosts and security groups.
File seen to reach to 16 G
root@mgr-c6610fa9:/var/log/migration-coordinator/v2t# ls -lh | grep cm.log
-rw-r----- 1 umc umc 16G Nov 23 18:52 cm.log
root@mgr-c6610fa9:/var/log/proton# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/nsx-var+log 27G 24G 2.4G 91% /var/log --> Close to disk being out of space
Support bundle collection is failing because of this here are the related logs
2021-11-23 18:34:44,158 2617 napi.root.administration.support_bundles.common_node ERROR Error compressing bundle archive, rc: 1, err:
gzip: /image/nsx_support_b_ke_lzs/nsx_manager_cb6f0442-9bfd-1048-fda3-b9ba975bc715_20211123_174928.tar.gz: No space left on device
2021-11-23 18:34:44,158 2617 root ERROR Insufficient disk space to create support bundle archive
2021-11-23 18:34:44,159 2617 root ERROR Out of disk space exception while creating support bundle archive
2021-11-23 18:34:44,411 2617 root INFO Successfully unlocked /tmp/.logrotate.lock
Resolution
Fixed in 4.1.0.0.0.20973353
Workaround:
Manually copy the entire v2t folder to an external location and temporarily mv cm.log to some other location that is not part of support bundle collection location
Workaround:
Manually copy the entire v2t folder to an external location and temporarily mv cm.log to some other location that is not part of support bundle collection location
Additional Information
Impact/Risks:
Support bundle collection will fail, this will make debugging difficult in certain situations
Support bundle collection will fail, this will make debugging difficult in certain situations
Feedback
Yes
No
Powered by
