HCX Manager Migration UI Goes Read-Only: Postgres Wraparound Error "database is not accepting commands to avoid wraparound data loss"
Article ID: 408671
Updated On:
Products
Issue/Introduction
The HCX Manager /common directory utilization becomes high which causes the HCX Manager Migration UI to go into READ ONLY mode.
Common symptoms include:
1) The HCX mobility Migrations UI will start presenting all the active migrations in READ ONLY view mode.
2) High "/common" directory usage (Use%) is seen via the disk free (df) CLI command.
HCX_Manager# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 5.9G 64K 5.9G 1% /dev/shm
tmpfs 2.4G 8.8M 2.4G 1% /run
/dev/sda3 7.7G 4.4G 3.0G 60% /
/dev/sda2 10M 2.0M 8.1M 20% /boot/efi
/dev/sda4 7.7G 4.1G 3.3G 56% /slot2
/dev/mapper/vg01-lv01 44G 40G 2.2G 95% /common
tmpfs 1.2G 0 1.2G 0% /run/user/10003) The following log messages are seen on the HCX Manager:
/common/logs/admin/app.log
ERROR: database is not accepting commands to avoid wraparound data loss in database "hybridity"/common/logs/admin/web.log
ERROR c.v.v.h.a.postgresdb.PostgresAdapter- Error running a postgres query as function serviceCountersForDashboard_countMigrations_byPair with {"context_enterprisename":"HybridityAdmin","context_roles":["System Administrator"],"context_organization":["DEFAULT"]} : ERROR: database is not accepting commands to avoid wraparound data loss in database "hybridity"/common/logs/postgres/postgres<xxx>.log
2025-08-10 04:16:51.556 GMT [30577] CONTEXT: while cleaning up index "Subscriptions_subscriptionSearchIndex" of relation "public.Subscriptions" automatic vacuum of table "hybridity.public.Subscriptions"
2025-08-10 04:16:55.319 GMT [30581] ERROR: could not access file "$libdir/pg_trgm": No such file or directoryEnvironment
VMware HCX 4.11.0
VMware HCX 4.11.1
Cause
This is a regression found in HCX Manager 4.11 software related to the Postgres database auto vaccuum functionality. As a result the Postgres database is not accepting commands to avoid wraparound data loss in database "hybridity" and it locks all the write operation and goes into read only mode.
Resolution
This issue is resolved in VMware HCX 4.11.2, available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.
If you have not encountered this issue but are concerned about the possibility and are currently unable to upgrade, please follow the 'Proactive Remediation Steps' outlined below.
Proactive Remediation Steps (if the issue has not been seen yet):
1) Download the appropriate Postgres rpm file based on the software currently running on the HCX manager from the following link:
For 4.11.0: https://packages.vmware.com/photon/5.0/photon_5.0_x86_64/x86_64/postgresql13-contrib-13.16-1.ph5.x86_64.rpm
For 4.11.1: https://packages.vmware.com/photon/5.0/photon_5.0_x86_64/x86_64/postgresql13-contrib-13.21-1.ph5.x86_64.rpm
NOTE: The Postgres version is different between 4.11.0 and 4.11.1
2) Upload the file onto the HCX Manager under /common/tmp and switch to root user.
3) Run the following commands on the HCX Manager under /common/tmp.
Example for 4.11.0:
root@hcxmg [ /home/admin ]# cd /common/tmp/
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.16-1.ph5.x86_64
postgresql13-client-13.16-1.ph5.x86_64
postgresql13-13.16-1.ph5.x86_64
postgresql13-server-13.16-1.ph5.x86_64
root@hcxmg [ /common/tmp ]# rpm -i postgresql13-contrib-13.16-1.ph5.x86_64.rpm
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.16-1.ph5.x86_64
postgresql13-client-13.16-1.ph5.x86_64
postgresql13-13.16-1.ph5.x86_64
postgresql13-server-13.16-1.ph5.x86_64
postgresql13-contrib-13.16-1.ph5.x86_64 Example for 4.11.1:
root@hcxmg [ /common/tmp ]# rpm -i postgresql13-contrib-13.21-1.ph5.x86_64.rpm
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.21-1.ph5.x86_64
postgresql13-client-13.21-1.ph5.x86_64
postgresql13-server-13.21-1.ph5.x86_64
postgresql13-13.21-1.ph5.x86_64
postgresql13-contrib-13.21-1.ph5.x86_64Then continue from point 5 below .
Remediation Steps (if the issue is present):
If you have encountered this issue , please follow the below steps.
1) If the errors described in this KB are seen in the logs then use the following postgres database query to check if “Subscriptions” has a very low “remaining” count:
SELECT
c.relnamespace::regnamespace as schema_name,
c.relname as table_name,
greatest(age(c.relfrozenxid), age(t.relfrozenxid)) as age,
2^31 - 1000000 - greatest(age(c.relfrozenxid), age(t.relfrozenxid)) as remaining
FROM pg_class c
LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm')
AND c.relnamespace = 'public'::regnamespace
ORDER BY 4;Example:
root@hcxmg [ /home/admin ]# psql -U postgres hybridity
psql (13.16)
Type "help" for help.
hybridity=# SELECT
hybridity-# c.relnamespace::regnamespace as schema_name,
hybridity-# c.relname as table_name,
hybridity-# greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age,
WHERE c.relkind IN ('r', 'm')
AND c.relnamespace = 'public'::regnamespace
hybridity-# 2^31-1000000-greatest(age(c.relfrozenxid),
hybridity(# age(t.relfrozenxid)) as remaining
hybridity-# FROM pg_class c LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
hybridity-# WHERE c.relkind IN ('r', 'm')
hybridity-# AND c.relnamespace = 'public'::regnamespace
hybridity-# ORDER BY 4;
schema_name | table_name | age | remaining
-------------+------------------------------------------------+------------+------------
public | Subscriptions | 2146483647 | 1 <<<<<
public | VirtualWire | 196471880 | 1950011768
public | DatastoreDetails | 196471877 | 1950011771
public | EdgeConfiguration | 196471873 | 1950011775
public | Licenses | 196471868 | 1950011780
<snip>2) After confirming step 1, download the appropriate Postgres rpm file based on the software currently running on the HCX manager from the following link:
For 4.11.0: https://packages.vmware.com/photon/5.0/photon_5.0_x86_64/x86_64/postgresql13-contrib-13.16-1.ph5.x86_64.rpm
For 4.11.1: https://packages.vmware.com/photon/5.0/photon_5.0_x86_64/x86_64/postgresql13-contrib-13.21-1.ph5.x86_64.rpm
NOTE: The Postgres version is different between 4.11.0 and 4.11.1
3) Upload the file onto the HCX Manager under /common/tmp and switch to root user.
4) Run the following commands on the HCX Manager under /common/tmp.
Example for 4.11.0:
root@hcxmg [ /home/admin ]# cd /common/tmp/
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.16-1.ph5.x86_64
postgresql13-client-13.16-1.ph5.x86_64
postgresql13-13.16-1.ph5.x86_64
postgresql13-server-13.16-1.ph5.x86_64
root@hcxmg [ /common/tmp ]# rpm -i postgresql13-contrib-13.16-1.ph5.x86_64.rpm
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.16-1.ph5.x86_64
postgresql13-client-13.16-1.ph5.x86_64
postgresql13-13.16-1.ph5.x86_64
postgresql13-server-13.16-1.ph5.x86_64
postgresql13-contrib-13.16-1.ph5.x86_64 Example for 4.11.1:
root@hcxmg [ /common/tmp ]# rpm -i postgresql13-contrib-13.21-1.ph5.x86_64.rpm
root@hcxmg [ /common/tmp ]# rpm -q -a | grep post
postgresql13-libs-13.21-1.ph5.x86_64
postgresql13-client-13.21-1.ph5.x86_64
postgresql13-server-13.21-1.ph5.x86_64
postgresql13-13.21-1.ph5.x86_64
postgresql13-contrib-13.21-1.ph5.x86_645) Stop the following HCX Manager services.
root@hcxmg [ /common/tmp ]# systemctl stop appliance-management
root@hcxmg [ /common/tmp ]# systemctl stop plan-engine
root@hcxmg [ /common/tmp ]# systemctl stop web-engine
root@hcxmg [ /common/tmp ]# systemctl stop app-engine
root@hcxmg [ /common/tmp ]# systemctl stop kafka
root@hcxmg [ /common/tmp ]# systemctl stop zookeeper6) Use the following postgres database command to freeze the vacuum functionality for "Subscriptions".
root@hcxmg [ /common/tmp ]# psql -U postgres hybridity
psql (13.16)
Type "help" for help.
hybridity=# VACUUM FREEZE public."Subscriptions";7) Use the following postgres database query to check that “Subscriptions” is no longer listed.
SELECT
c.relnamespace::regnamespace as schema_name,
c.relname as table_name,
greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age,
2^31-1000000-greatest(age(c.relfrozenxid),
age(t.relfrozenxid)) as remaining
FROM pg_class c LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm')
AND c.relnamespace = 'public'::regnamespace
ORDER BY 4;Example:
hybridity=#
SELECT
c.relnamespace::regnamespace as schema_name,
c.relname as table_name,
greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age,
2^31-1000000-greatest(age(c.relfrozenxid),
age(t.relfrozenxid)) as remaining
FROM pg_class c LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm')
AND c.relnamespace = 'public'::regnamespace
ORDER BY 4;
schema_name | table_name | age | remaining
-------------+------------------------------------------------+-----------+------------
public | VirtualWire | 196471880 | 1950011768
public | DatastoreDetails | 196471877 | 1950011771
public | EdgeConfiguration | 196471873 | 1950011775
public | Licenses | 196471868 | 1950011780
public | ApplianceConfig | 196471864 | 1950011784
<snip>8) Restart the following HCX Manager services.
root@hcxmg [ /common/tmp ]# systemctl start zookeeper
root@hcxmg [ /common/tmp ]# systemctl start kafka
root@hcxmg [ /common/tmp ]# systemctl start app-engine
root@hcxmg [ /common/tmp ]# systemctl start web-engine
root@hcxmg [ /common/tmp ]# systemctl start plan-engine
root@hcxmg [ /common/tmp ]# systemctl start appliance-management9) Check that all of the service statuses are "active (running)".
root@hcxmg [ /common/tmp ]# systemctl status zookeeper
root@hcxmg [ /common/tmp ]# systemctl status kafka
root@hcxmg [ /common/tmp ]# systemctl status app-engine
root@hcxmg [ /common/tmp ]# systemctl status web-engine
root@hcxmg [ /common/tmp ]# systemctl status plan-engine
root@hcxmg [ /common/tmp ]# systemctl status appliance-managementExample:
root@hcxmg [ /common/tmp ]# systemctl status plan-engine
plan-engine.service - Migration Planner Engine
Loaded: loaded (/etc/systemd/system/plan-engine.service; enabled; preset: enabled)
Active: active (running) since Wed 2025-08-27 20:12:39 UTC; 2min 36s ago
Process: 1562 ExecStartPre=/etc/systemd/service-dependency-check.sh postgresdb database-upgrade (code=exited, status=0/SUCCESS)
Process: 1568 ExecStart=/etc/systemd/plan-engine-start (code=exited, status=0/SUCCESS)
Main PID: 1570 (java)
Tasks: 37
Memory: 330.2M
CPU: 19.584s
CGroup: /system.slice/plan-engine.service
1570 java -Xmx256m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/common/dumps -Dfile.encoding=UTF-8 -jar /opt/vmware/plan-engine/plan_engine.jar
Aug 27 20:12:39 HCX_Mgr.com systemd[1]: Starting Migration Planner Engine...
Aug 27 20:12:39 HCX_Mgr.com service-dependency-check.sh[1564]: localhost:5432 - accepting connections
Aug 27 20:12:39 HCX_Mgr.com service-dependency-check.sh[1562]: postgresdb is running.
Aug 27 20:12:39 HCX_Mgr.com service-dependency-check.sh[1562]: database-upgrade is running.
Aug 27 20:12:39 HCX_Mgr.com systemd[1]: Started Migration Planner Engine.10) Monitor the disk free (df) CLI command to ensure that /common directory usage (Use%) is reduced
root@hcxmg [ /common/tmp ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 5.9G 64K 5.9G 1% /dev/shm
tmpfs 2.4G 8.8M 2.4G 1% /run
/dev/sda3 7.7G 4.3G 3.0G 59% /
/dev/sda2 10M 2.0M 8.1M 20% /boot/efi
/dev/sda4 7.7G 4.1G 3.3G 56% /slot2
/dev/mapper/vg01-lv01 44G 19G 23G 46% /common
tmpfs 1.2G 0 1.2G 0% /run/user/100011) Confirm that operations on the HCX Manager Migration UI are no longer in READONLY view mode.
Additional Information
There is no Impact to existing Network extension datapath traffic.
Impact will be seen but is not limited to the following components that utilize Postgres Database transactions:
- Existing or new migration workflows.
- Service mesh deployment / appliance lifecycle workflows.
- HCX Manager upgrade.
- Site Pairing changes.
Feedback
