vmware-vpxd service on VCSA fails with the error "contains unexpected zero page at block" on vpxd.log

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Connecting to the vCenter Server Appliance using the vSphere Web Client fails.
• You see the error similar to:
  
  503 Service Unavailable
  Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f975808ee30] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
   
• Running the service-control --status command displays the vmware-vpxd service as stopped.
• A complaint exists on vpx_vm_fle_snap_disk_unit_m1 table.
• In the vpxd.log file, you see entries similar to;
  
  2017-05-17T16:28:26.192Z error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-4-2a039441-01-621c8039] [VdbStatement] Execute result code: -1
  2017-05-17T16:28:26.192Z error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-4-2a039441-01-621c8039] [VdbStatement] SQL execution failed: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES (?, ?, ?, ?)
  2017-05-17T16:28:26.192Z error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-4-2a039441-01-621c8039] [VdbStatement] Execution elapsed time: 2 ms
  2017-05-17T16:28:26.192Z error vpxd[7FBF0A4A3700] [Originator@6876 sub=Default opID=lro-4-2a039441-01-621c8039] [VdbStatement] Statement diagnostic data from driver is XX002:0:7:ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3;
  --> Error while executing the query
   
• In the postgres.log file, you will see entries similar to;
  
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc ERROR: index "pk_fle_snap_disk_unit" contains unexpected zero page at block 3
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc HINT: Please REINDEX it.
  2017-05-17 16:07:42.794 UTC 591c7546.84c2 1192886 VCDB vc STATEMENT: INSERT INTO VPX_VM_FLE_SNAP_DISK_UNIT (SNAPSHOT_DISK_ID, ARRAY_ID, VM_ID, UPDATE_KEY) VALUES ($1, $2, $3, $4)
  2017-05-17 16:07:44.377 UTC 591c7546.84c1 1192934 VCDB vc LOG: could not receive data from client: Connection reset by peer
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.5
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x
VMware Update Manager 6.5

Cause

This issue occurs due to an issue with a table within the vCenter Server Appliance postgres database.

Resolution

To resolve this issue, re-index the affected table.

Important: Prior to running these steps, VMware recommends to take a snapshot of the VCSA. For more information, see Best practices for using snapshots in the vSphere environment (1025279).

1. Log in as root through an SSH or console session on the vCenter Server Appliance.
2. Run this command to enable the shell:
   
   shell.set --enabled true
    
3. Access the postgres database by running this command:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
    
4. Re-index the affected table by running these commands:
   
   Note: Select the table to be re-indexed from the postgres log error.
   
   reindex index pk_fle_snap_disk_unit;
   
   reindex index vpx_vm_fle_snap_disk_unit_m1;

Quit vPostgres shell with this command: \q

1. Start the vmware-vpxd service by running this command:
   
   service-control --start vmware-vpxd