Upgrading the vCenter Server Appliance to vSphere 6.0 with embedded Postgres fails on starting VMware vCenter Server
search cancel

Upgrading the vCenter Server Appliance to vSphere 6.0 with embedded Postgres fails on starting VMware vCenter Server

book

Article ID: 315324

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
When attempting to upgrade your vCenter Server Appliance to vSphere 6.0, you experience these symptoms:
  • The upgrade process through the HTML5 Deployment Wizard for the vCenter Server Appliance fails.
  • You see this message for multiple hours:

    Starting VMware vCenter Server...
     
  • The upgrade process through the OVF Tool fails.
  • You see this message for multiple hours:

    vpxd_firstboot.py
     
  • Reviewing the VIMTOP from the vCenter Server Appliance 6.0 that you are migrating to, the vCenter Server process is running near or at 100% CPU allocation.
     
  • Reviewing the TOP from the vCenter Server Appliance 6.0 that you are migrating to, the VPXD process is running near or at 100% CPU allocation.
     
  • In the \var\log\vmware\vpostgres\postgresql-##.log file, you see entries similar to:
<YYYY-MM-DD>T<time> UTC 54fb95c7.64e8 0 VCDB postgres STATEMENT: select count(*) from VPX_STAT_hist1_1;</time>
<YYYY-MM-DD>T<time></time> UTC 54fb58ff.36f5 0 VCDB vc LOG: duration: 17195501.642 ms statement: DO
$$
DECLARE r RECORD;
DECLARE sql_stmt VARCHAR;
BEGIN
FOR r IN SELECT table_name
FROM information_schema.tables
WHERE UPPER(table_name) LIKE 'VPX_HIST_STAT%'
AND table_type = 'BASE TABLE'
LOOP
sql_stmt = 'INSERT INTO ' || r.table_name || ' SELECT INSERTED.COUNTER_ID,HS.TIME_ID,';
sql_stmt = sql_stmt || 'SUM(STAT_VAL) ';
sql_stmt = sql_stmt || ' FROM ' || r.table_name || ' HS';
sql_stmt = sql_stmt || ' INNER JOIN VPX_STAT_COUNTER SC ON HS.COUNTER_ID = SC.COUNTER_ID ';
sql_stmt = sql_stmt || ' INNER JOIN VPX_DEVICE D ON SC.DEVICE_ID=D.DEVICE_ID ';
sql_stmt = sql_stmt || ' INNER JOIN VPX_OBJECT_HIERARCHY H ON SC.ENTITY_ID = H.STAT_ENTITY_ID ';
sql_stmt = sql_stmt || ' INNER JOIN (SELECT SC.ENTITY_ID, SC.COUNTER_ID, ';
sql_stmt = sql_stmt || ' CASE WHEN SD49.NAME=''swapused'' AND SD49.GROUP_NAME=''mem'' AND SD49.ROLLUP_TYPE=0 ';
sql_stmt = sql_stmt || ' THEN (SELECT ID FROM VPX_STAT_DEF WHERE NAME=''swapped'' AND GROUP_NAME=''mem'' AND ROLLUP_TYPE=0) ';
sql_stmt = sql_stmt || ' ELSE SC.STAT_ID END MODSTAT_ID, NC.AGGR_TO, NC.AGGR_TYPE ';
sql_stmt = sql_stmt || ' FROM VPX_STAT_ID T ';
sql_stmt = sql_stmt || ' INNER JOIN VPX_STAT_COUNTER SC ON SC.ENTITY_ID=T.ID ';
sql_stmt = sql_stmt || ' INNER JOIN VPX_NEW_COUNTERS NC ON NC.COUNTER_ID = SC.STAT_ID ';
sql_stmt = sql_stmt || ' INNER JOIN VPX_STAT_DEF SD49 ON SD49.ID = SC.STAT_ID) INSERTED ';
sql_stmt = sql_stmt || ' ON H.PARENT_STAT_ENTITY_ID=INSERTED.ENTITY_ID AND INSERTED.MODSTAT_ID=SC.STAT_ID ';
sql_stmt = sql_stmt || ' AND (INSERTED.AGGR_TYPE=H.ENTITY_TYPE OR H.ENTITY_TYPE=''R'' ) ';
sql_stmt = sql_stmt || ' AND (H.ENTITY_TYPE=INSERTED.AGGR_TO OR (INSERTED.AGGR_TO=''C'' AND H.ENTITY_TYPE IN (''V'',''H''))) ';
sql_stmt = sql_stmt || ' WHERE D.DEVICE_NAME='''' OR D.DEVICE_NAME='' ''';
sql_stmt = sql_stmt || ' GROUP BY HS.TIME_ID,SC.STAT_ID,INSERTED.COUNTER_ID; ';
EXECUTE sql_stmt;
END LOOP;
EXCEPTION WHEN OTHERS THEN
NULL;
END$$;
 
  • In the \var\log\vmware\vpxd\dbuhelper-#.log file, you see entries similar to:
<YYYY-MM-DD>T<time></time> Section for VMware VirtualCenter, pid=5226, version=6.0.0, build=2559267, option=Release
<YYYY-MM-DD>T<time></time> verbose dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Dumping early logs:
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Logging uses fast path: true
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Initialized channel manager
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Current working directory: /usr/lib/vmware-vpx/dbupgrade
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E55F700] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F49467F5700] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F49467B4700] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=ThreadPool] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F4946773700] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Syscommand enabled: true
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] ReaperManager Initialized
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Log path: /var/log/vmware/vpxd
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Initializing SSL
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] Vmacore::InitSSL: handshakeTimeoutUs = 20000000
<YYYY-MM-DD>T<time></time> info dbuhelper[7F49460E9700] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=vpxdVdb] Registry Item DB 5 value is ''
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=vpxdVdb] [VpxdVdb] Converted munged password in registry
<YYYY-MM-DD>T<time></time> warning dbuhelper[7F494E5687A0] [Originator@6876 sub=vpxCommon] [VpxConfig] Invalid specified value property: vpxd.memChecker.memStopInMB
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 1000 transactions for 1 GB RAM dedicated to vpxd.
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=Default] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=dbuHelper] [DbuHelper] Upgrading the VM clobs....
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=dbuHelper] [DbuHelper] Done upgrading the VM clobs.
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=dbuHelper] [DbuHelper] Upgrading the datastore clobs.
<YYYY-MM-DD>T<time></time> info dbuhelper[7F494E5687A0] [Originator@6876 sub=dbuHelper] [DbuHelper] Done upgrading the Datstore clobs.


Environment

VMware vCenter Server Appliance 6.0.x

Cause

During the Import and Firstboot phase of the vCenter Server Appliance, pre-calculation scripts are run to populate the Performance Charts. With larger environments, pre-calculations can take multiple hours to complete depending on the size of the historical performance tables (hist_stat###).

Resolution

To resolve this issue, truncate the historical performance information or trim the historical performance information in the vCenter Server Appliance 5.5 before upgrading to vSphere 6.0.
 
To selectively trim historical information stored on the vCenter Server Appliance 5.5 before upgrade, see Selective deletion of tasks, events, and historical performance data in vSphere 5.x and 6.x (2110031).
 

Process to truncate the historical performance information:

To truncate all historical performance information stored on the vCenter Server Appliance 5.5 before upgrading:
  1. Snapshot the vCenter Server Appliance 5.5.
  2. Open an SSH session and connect to the vCenter Server Appliance 5.5 using root credentials.
  3. Run this command to truncate all historical performance data:

    /opt/vmware/vpostgres/current/bin/psql -U postgres -t -d VCDB -c "select 'truncate table ' || table_name || ' cascade;' from information_schema.tables where table_name like 'vpx_hist_stat%' and table_type = 'BASE TABLE' order by 1" | /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

    After completion of the preceding steps, upgrade to vSphere 6.0.


Additional Information


Selective deletion of tasks, events, and historical performance data in vSphere 5.x and 6.x
将 vCenter Server Appliance 升级到具有嵌入式 Postgres 的 vSphere 6.0 失败：正在启动 VMware vCenter Server
La actualización de vCenter Server Appliance a vSphere 6.0 con Postgres integrada falla al iniciar VMware vCenter Server
Falha na atualização do vCenter Server Appliance para o vSphere 6.0 com Postgres integrado na inicialização do VMware vCenter Server

Powered by Wolken Software