EDGE Upgrade Evaluation Tool execution check Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade. FAILURE "NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS. Failed migrations: Data migration timedout
MP Upgrade Evaluation Tool execution check Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade. FAILURE "NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS. Failed migrations: Data migration timedout
e.g.
<nsx-manager>:~# grep "appendCheckpoint" /var/log/corfu/corfu-compactor-audit.log | grep "entries(" | awk '{print $10, $11}' | sort | uniq | sort -rnt'(' -k2
ab55d508-d0cd-37d0-afdf-8edfe622da1a, entries(12653305), >> table: TagBulkOperation
dabf8af4-9eb6-3374-9a18-d273ed7132e9, entries(1276041), >> table: GenericPolicyRealizedResource
8d949aa6-302b-3098-8b48-b52c6626f302, entries(1241),
9c5a0d6a-94cc-3fc3-ade9-cfc2d7c64151, entries(122),
23d4bc74-b217-33fd-a60e-0d69cfc96fe4, entries(120),
bb9b2002-345c-3139-b36a-eb686ca9ecc9, entries(12),
(end of snip)
/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t GenericPolicyRealizedResource > /tmp/gprr.txt
grep stringId /tmp/gprr.txt | awk '{print $2}' | cut -d "/" -f 1-6 | sort | uniq -c | sort -nr
1265335 "/infra/realized-state/enforcement-points/default/tags
3813 "/infra/realized-state/enforcement-points/default/services
3119 "/infra/realized-state/enforcement-points/default/groups
1929 "/infra/realized-state/enforcement-points/default/logical-ports
287 "/infra/realized-state/enforcement-points/default/lb-rules
(end of snip)
2024-06-11T11:21:57.994Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for dabf8af4-9eb6-3374-9a18-d273ed7132e9, entries(1207967), cpSize(744950900) bytes at snapshot Token(epoch=518, sequence=3746995903) in 3816415 ms
..
2024-06-11T11:55:12.002Z ERROR main Migration 3453 - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"] Migration failed
java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.vmware.nsx.management.migration.ufo.UFOMigration.migrate(UFOMigration.java:200) ~[logical-migration.jar:?]
at com.vmware.nsx.management.migration.impl.LogicalMigration.executeMigrations(LogicalMigration.java:78) ~[logical-migration.jar:?]
at com.vmware.nsx.management.migration.impl.Migration.migrate(Migration.java:46) ~[logical-migration.jar:?]
at com.vmware.nsx.management.migration.impl.LogicalMigration.main(LogicalMigration.java:47) ~[logical-migration.jar:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_332]
at com.vmware.nsx.management.migration.ufo.UFOMigration.migrate(UFOMigration.java:194) ~[logical-migration.jar:?]
... 3 more.
This is caused by extremely high number of objects in NSX-T deployment (e.g. segment ports, tags), usually created by user / automation tool, which will exhaust Corfu, causing compactor to fail.
Currently, there is no resolution to this issue.
Workaround:
Should you experience this problem, please contact VMware Global Support team for assistance, with a reference to this KB article.