vmware-vpxd service on VCSA fails to start with the error "contains unexpected zero page at block" on vpxd.log

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Connecting to the vCenter Server Appliance using the vSphere Web Client fails.
  
  
• You see the error similar to:
  
  503 Service Unavailable
  Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f975808ee30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
   
• Running the service-control --status --all command displays the vmware-vpxd service in a stopped status. 
• In the /var/log/vmware/vpxd/vpxd.log file, you see entries similar to;
  
  vpxd.log 
  
  

  YYYY-DD-MMTHH:MM:SS.###+##:## error vpxd[77336] [Originator@6876 sub=Default opID=28db8a4a-71] [Vdb::IsRecoverableErrorCode] Unable to recover from XX002:1

  YYYY-DD-MMTHH:MM:SS.###+##:##  error vpxd[77336] [Originator@6876 sub=Default opID=28db8a4a-71] [VdbStatement] SQLError was thrown: "ODBC error: (XX002) - ERROR: index "pk_vpx_event_31" contains unexpected zero page at block 24;

  --> Error while executing the query" is returned when executing SQL statement " SELECT EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID,        VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE,        COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID,        NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME, CHANGE_TAG_ID FROM VPXV_EVENT_ALL  WHERE (EVENT_TYPE IN (?,?)) AND EVENT_ID<? ORDER BY EVENT_ID DESC LIMIT 15; "

  
  
  YYYY-DD-MMTHH:MM:SS.###+##:##  error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Execute result code: -1
YYYY-DD-MMTHH:MM:SS.###+##:##  error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES (?, ?, ?, ?)
YYYY-DD-MMTHH:MM:SS.###+##:##  error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Execution elapsed time: 2 ms
YYYY-DD-MMTHH:MM:SS.###+##:##  error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Statement diagnostic data from driver is XX002:0:7:ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3;
--> Error while executing the query
   
• In the /var/log/vmware/vpostgres/postgres.log file, you will see entries similar to;
  
  postgres.log
  
  YYYY-DD-MMTHH:MM:SS.###<Time_zone> 591c7546.84c2 1192886 VCDB vc ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3
YYYY-DD-MMTHH:MM:SS.###<Time_zone> 591c7546.84c2 1192886 VCDB vc HINT: Please REINDEX it.
YYYY-DD-MMTHH:MM:SS.###<Time_zone> 591c7546.84c2 1192886 VCDB vc STATEMENT: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES ($1, $2, $3, $4)
YYYY-DD-MMTHH:MM:SS.###<Time_zone> 591c7546.84c1 1192934 VCDB vc LOG: could not receive data from client: Connection reset by peer

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter server 7.x
VMware vCenter server 8.x

Cause

This issue occurs due to an issue with a table within the vCenter Server Appliance postgres database.

Resolution

To resolve this issue, re-index the affected table.

Important: Prior to running these steps, VMware recommends to take a snapshot of the VCSA. For more information, see Best practices for using VMware snapshots in the vSphere environment.

• Take SSH to vCenter server and login as root user. 
  
  
• Run this command to enable the shell:
  shell.set --enabled true
  
  
• Access the postgres database by running this command:
  /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
  
  
• Re-index the affected table by running these commands:
  reindex index <table_name>;

Note: Select the table to be re-indexed from the errors noted from vpxd log and postgres log. 

Example: 
reindex index pk_fle_snap_disk_unit;
reindex index vpx_vm_fle_snap_disk_unit_m1;
reindex index pk_vpx_event_31;

• Quit vPostgres shell with this command:
  \q
  
  
• Start the vmware-vpxd service by running this command:
  service-control --start vmware-vpxd


• Start remaining services by executing this command: 
  service-control --start --all