Upgrade of vSphere Replication appliance fail to initiate aborting instantly without throwing any error

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Upgrade of VR appliance from 8.5 to 8.7 fail to start.
  
  
• Post uploading the ISO image to the VR virtual machine, from vSphere Replication Appliance Management, under 'Update' tab > 8.7.0 image is selected.
  
  
• Clicking on 'install' triggers the upgrade but then aborts within 15-20 seconds.
  
  
• Update vSphere Replication window (as seen below) pops up and disappears:
  
  

Environment

vSphere Replication 8.x

Cause

• Log file /opt/vmware/var/log/preupdatelog reports not enough space in VRMS DB, although partition /var/lib/vrmsdb is just 1% used (as checked running command 'df -h' in VR appliance shell.).
  
  
• /opt/vmware/var/log/preupdatelog : 

psql.bin: fe_sendauth: no password supplied
Number of found VCTA tunnels: 0
No DRxx replications found. Continue with upgrade...
+ grep -q vmware-vcta
+ rpm -qa
+ '[' -r /opt/vmware/hms/conf/embedded_db.cfg ']'
+ source /opt/vmware/hms/conf/embedded_db.cfg
++ EMB_DB_INSTALL_DIR=/opt/vmware/vpostgres/current
++ EMB_DB_INSTANCE=vrmsdb
++ EMB_DB_USER=vrmsdb
++ EMB_DB_PASSWORD=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
++ EMB_DB_STORAGE=/var/lib/vrmsdb
++ EMB_DB_MAX_CONNECTIONS=150
+ echo /opt/vmware/vpostgres/current
/opt/vmware/vpostgres/current
++ /opt/vmware/hms/bin/embedded-db.sh --status
+ '[' ok == ok ']'
+ export PGPASSWORD=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ PGPASSWORD=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ /opt/vmware/vpostgres/current/bin/pg_dump -U vrmsdb -Ft vrmsdb
+ /usr/bin/bzip2 --best
pg_dump: [tar archiver] could not write to output file: No space left on device
+ '[' 1 '!=' 0 ']'
+ echo 'pg_dump failure detected'
pg_dump failure detected
+ systemctl stop hms-vpostgres
+ export -f background
+ exit 0
+ nohup bash -c background

Resolution

1. Take snapshot of the vSphere Replication Appliance.
   
   
2. Take SSH to VR appliance and log into the VRMS database using following commands:
   # /opt/vmware/vpostgres/current/bin/psql -U vrmsdb
   
   Note : if being prompted with password, cat the file /opt/vmware/hms/conf/embedded_db.cfg and use the password against the entry 'EMB_DB_PASSWORD'.
   
   
3. Run following commands to check number of events logged in table incomingeventlogentity and outgoinglogentity:
   

   > select count(*) from outgoingeventlogentity
> select count(*) from incomingeventlogentity

4. Table incomingeventlogentity had total of 7 million entries filling up the VRMS database.
   
   
5. Stop hms service using command:
   # systemctl stop hms
   
   
6. Run following command to cleanup up the table incomingeventlogentity and outgoingeventlogentity tables:
   
   # epochtime=`date +%s`000 && conditiontime=$((epochtime-3600000)) && echo "DELETE from incomingeventlogentity WHERE timestamp < $conditiontime;
DELETE from outgoingeventlogentity WHERE timestamp < $conditiontime; " | /opt/vmware/vpostgres/current/bin/psql -U vrmsdb
   
   
7. Start hms service using command:
   # systemctl start hms
   
   
8. Run database vacuum on VRMS DB to free up and reclaim space.
   
   
9. Re-run the upgrade of VR from 8.5 to 8.7.