VCHA Passive node fails to initialize PostgreSQL replication due to error "PostgreSQL replication is not in progress. For an initial deployment, PostgreSQL standby creation may take time depending on the database size.

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter server HA  became 'Degraded' state since the reconfiguration of the VCHA after the upgrade of vCenter server. 
• Passive node fails to initialize PostgreSQL replication  and the vSphere Client UI displays the following alert under Configure > vCenter HA:
  
  "PostgreSQL replication is not in progress. For an initial deployment, PostgreSQL standby creation may take time depending on the database size. If this issue persists, verify that PostgreSQL is running on the passive node and that the passive node is reachable on the vCenter HA network."
  
  

• Management and VCHA network connectivity across Active, Passive, and Witness nodes are fully functional.
• On passive node, the repl_passive_setup.log has the following events indicating that the vmware-vpostgres has crashed while starting the service. 
  
  /var/log/vmware/vcha/repl_passive_setup.log
  
   YYYY-MM-DDTHH:MM:##Z INFO repl_passive_setup Done running command
   YYYY-MM-DDTHH:MM:##Z  INFO repl_passive_setup rc = [2], stdout = [], stderr = [Start service request failed. Error: Service crashed while starting]
   YYYY-MM-DDTHH:MM:##Z  ERROR repl_passive_setup Failed full_sync: attempt: 42
   YYYY-MM-DDTHH:MM:##Z  ERROR repl_passive_setup ['Traceback (most recent call last):\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_passive.py", line 283, in full_sync\n    start_and_verify_streaming(pg_util, args, early_repl_check=False)\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_passive.py", line 162, in start_and_verify_streaming\n    pg_util.start_replicating(args.active_ip, args.port, args.repl_username,\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 402, in start_replicating\n    raise e\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 393, in start_replicating\n    self.vpostgres_svc_ctl(\'start\')\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 249, in vpostgres_svc_ctl\n    self.vmon_svc_ctl(\'vmware-vpostgres\', action)\n', '  File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 259, in vmon_svc_ctl\n    raise InvokeCommandException(\'Failed to %s %s\' % (action, service_name))\n', 'cis.exceptions.InvokeCommandException: {\n    "detail": [\n        {\n            "id": "install.ciscommon.command.errinvoke",\n            "translatable": "An error occurred while invoking external command : \'%(0)s\'",\n            "args": [\n                "Failed to --start vmware-vpostgres"\n            ],\n            "localized": "An error occurred while invoking external command : \'Failed to --start vmware-vpostgres\'"\n        }\n    ],\n    "componentKey": null,\n    "problemId": null,\n    "resolution": null\n}\n']
  YYYY-MM-DDTHH:MM:##Z  INFO repl_passive_setup running command ['/usr/lib/vmware-vmon/vmon-cli', '--stop', 'vmware-vpostgres']
  YYYY-MM-DDTHH:MM:##Z  INFO repl_passive_setup Done running command
  YYYY-MM-DDTHH:MM:##Z  INFO repl_passive_setup rc = [0], stdout = [Completed Stop service request.
  ], stderr = []
  YYYY-MM-DDTHH:MM:##Z  INFO repl_passive_setup vmware-vpostgres stopped successfully.
  
• /var/log/vmware/vpostgres/serverlog.stderr on the Passive node identifies the exact parameter mismatch causing the daemon to abort startup:
  
     YYYY-MM-DDTHH:MM:##Z  ... LOG: entering standby mode
     YYYY-MM-DDTHH:MM:##Z  ... FATAL: recovery aborted because of insufficient parameter settings
     YYYY-MM-DDTHH:MM:##Z  ... DETAIL: max_connections = 619 is a lower setting than on the primary server, where its value was 620.
     YYYY-MM-DDTHH:MM:##Z  ... HINT: You can restart the server after making the necessary configuration changes.
     YYYY-MM-DDTHH:MM:##Z  ... LOG: startup process (PID 52986) exited with exit code 1
     YYYY-MM-DDTHH:MM:##Z  ... LOG: aborting startup due to startup process failure

Environment

VMware vCenter server 8.0 

Cause

• This issue caused due to the discrepancy in reported memory when comparing both active and passive node typically caused by Virtual Machine Overhead or PCI MMIO hole reservations that can vary slightly depending on the specific ESXi host's hardware features (e.g., BIOS settings, specific CPU instructions, or reserved memory for hardware-assisted virtualization) exposed to the guest.
• In vCenter 8.0, the dynamic database sizing logic (managed by cloudvm-ram-size) calculates the max_connections parameter using a formula based on the total memory visible to the kernel. PostgreSQL strictly forbids a standby node from having a lower max_connections limit than the primary, which triggers the crash.

Resolution

1. Compare the virtual memory allocation on both the active and passive node by executing the command: 
   free -m 
2. If there is a discrepancy in total memory value, increasing the RAM of the Active node can often push both nodes into a higher, stable bracket for the max_connections calculation, providing enough "buffer" so that the allocated memory difference no longer results in different connection limits.

   Example output: 

Active node:

# free -m

               total        used        free      shared  buff/cache   available

Mem:           24065       10468         267         599       14336       13596

Passive node:

# free -m

               total        used        free      shared  buff/cache   available

Mem:           24017         936        9812           1       13677       23081

 

 

In the above example, Photon OS  free -m output shows a discrepancy of 48MB (24065 MB vs. 24017 MB).
Because the passive node sees ~48MB less, the auto-calculation rounds down to 619, while the active node rounds to 620. PostgreSQL strictly forbids a standby node from having a lower max_connections limit than the primary, which triggers the crash.