vmware-vpxd service on VCSA fails with the error "contains unexpected zero page at block" on vpxd.log

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Connecting to the vCenter Server Appliance using the vSphere Web Client fails.
• You see the error similar to:
  
  503 Service Unavailable
  Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f975808ee30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
   
• Running the service-control --status command displays the vmware-vpxd service as stopped.
• A complaint exists on vpx_vm_fle_snap_disk_unit_m1 table.
• In the /var/log/vmware/vpxd/vpxd.log file, you see entries similar to;
  
  XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Execute result code: -1
  XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES (?, ?, ?, ?)
  XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Execution elapsed time: 2 ms
  XXXX-XX-XXTXX:XX:XX.XXXZ error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-x-xxxxxxx-xx-xxxxxxxx] [VdbStatement] Statement diagnostic data from driver is XX002:0:7:ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3;
  --> Error while executing the query
   
• In the /var/log/vmware/vpostgres/postgres.log file, you will see entries similar to;
  
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc HINT: Please REINDEX it.
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc STATEMENT: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES ($1, $2, $3, $4)
  2017-05-17 16:07:44.377 UTC 591c7546.84c1 1192934 VCDB vc LOG: could not receive data from client: Connection reset by peer

Environment

VMware vSphere ESXi 6.5
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x
VMware Update Manager 6.5

Cause

This issue occurs due to an issue with a table within the vCenter Server Appliance postgres database.

Resolution

To resolve this issue, re-index the affected table.

Important: Prior to running these steps, VMware recommends to take a snapshot of the VCSA. For more information, see Best practices for using VMware snapshots in the vSphere environment.

1. Log in as root through an SSH or console session on the vCenter Server Appliance.
2. Run this command to enable the shell:
   
   shell.set --enabled true
    
3. Access the postgres database by running this command:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
    
4. Re-index the affected table by running these commands:
   
   Note: Select the table to be re-indexed from the postgres log error.
   
   reindex index pk_fle_snap_disk_unit;
   
   reindex index vpx_vm_fle_snap_disk_unit_m1;
5. Quit vPostgres shell with this command: \q
6. Start the vmware-vpxd service by running this command:
   
   service-control --start vmware-vpxd